Golang Bufio (A Complete Guide)
Introduction
Bufio is a package in the standard library of Go that provides buffered I/O. It implements a buffered reader and writer that both implement the io.Reader and io.Writer interfaces.
What is Buffered I/O?
Buffered I/O is a technique that allows a program to read or write data in chunks rather than one byte at a time. This is useful because it allows the program to read or write data more efficiently. It also allows the program to read or write data more predictably.
In Go, this is done by using the bufio package. This package provides buffered readers and writers.
How to Use Bufio
The file we’ll be using in this guide is:
Creating a Buffered Reader
To create a buffered reader, you can use the bufio.NewReader function. This function takes an io.Reader as an argument. This means that you can pass in any type that implements the io.Reader interface. This includes os.File, strings.Reader, and bytes.Buffer.
package main
import (
"bufio"
"fmt"
"os"
)
func main() {
file, err := os.Open("file.txt")
if err != nil {
fmt.Println(err)
return
}
defer file.Close()
reader := bufio.NewReader(file)
fmt.Println(reader)
}
In the above example, we are creating a buffered reader from a file. We are then printing the buffered reader to the console. The buffered reader is a pointer to a bufio.Reader struct.
Reading from a Buffered Reader
To read from a buffered reader, you can use the bufio.Reader.Read function. This function takes a byte slice as an argument. This byte slice is where the data will be read into.
package main
import (
"bufio"
"fmt"
"os"
)
func main() {
file, err := os.Open("file.txt")
if err != nil {
fmt.Println(err)
return
}
reader := bufio.NewReader(file)
data := make([]byte, 100)
_, err = reader.Read(data)
if err != nil {
fmt.Println(err)
return
}
fmt.Println(string(data))
}
In the above example, we are creating a buffered reader from a file. We are then reading 100 bytes from the file into a byte slice. We are then converting the byte slice to a string and printing it to the console.
The output of the above program is:
Ex ad mollit laborum non esse nostrud excepteur. Mollit fugiat nisi magna est deserunt excepteur par
Creating a Buffered Writer
To create a buffered writer, you can use the bufio.NewWriter function. This function takes an io.Writer as an argument. This means that you can pass in any type that implements the io.Writer interface. This includes os.File, strings.Builder, and bytes.Buffer.
package main
import (
"bufio"
"fmt"
"os"
)
func main() {
file, err := os.Create("file.txt")
if err != nil{
fmt.Println(err)
return
}
defer file.Close()
writer := bufio.NewWriter(file)
fmt.Println(writer)
}
In the above example, we are creating a buffered writer from a file. We are then printing the buffered writer to the console. The buffered writer is a pointer to a bufio.Writer struct.
Writing to a Buffered Writer
To write to a buffered writer, you can use the bufio.Writer.Write function. This function takes a byte slice as an argument. This byte slice is the data that will be written to the writer.
bufio.Writer.Flush is used to write the data to the writer. This function must be called before the program exits or the data will not be written to the writer.
package main
import (
"bufio"
"fmt"
"os"
)
func main() {
file, err := os.Create("file2.txt")
if err != nil {
fmt.Println(err)
return
}
writer := bufio.NewWriter(file)
_, err = writer.Write([]byte("Hello World!"))
if err != nil {
fmt.Println(err)
return
}
err = writer.Flush()
if err != nil {
fmt.Println(err)
return
}
}
This will create a file called file.txt and write Hello World! to it. The bufio.Writer.Write function will not write the data to the file until the bufio.Writer.Flush function is called.
Change the Buffer Size
The default buffer size for a buffered writer is 4096 bytes. This means that the data will be written to the writer in chunks of 4096 bytes. If you want to change the buffer size, you can use the bufio.NewWriterSize function.
package main
import (
"bufio"
"fmt"
"os"
)
func main() {
file, err := os.Create("file2.txt")
if err != nil {
fmt.Println(err)
return
}
writer := bufio.NewWriterSize(file, 100)
_, err = writer.Write([]byte("Hello World!"))
if err != nil {
fmt.Println(err)
return
}
err = writer.Flush()
if err != nil {
fmt.Println(err)
return
}
}
In the above program, we changed the buffer size to 100 bytes. This means that the data will be written to the writer in chunks of 100 bytes. This will make the writer slower but it will also use less memory.
Bufio vs. I/O
The main difference between buffered I/O and normal I/O is that buffered I/O reads or writes data in chunks rather than one byte at a time. While on the other side normal I/O reads or writes data one byte at a time. This might not seem like a big difference but it can make a big difference in performance.
In a case where you are reading or writing a lot of data, buffered I/O can be much faster than normal I/O. To see this, we can compare the performance of buffered I/O and normal I/O using benchmarks.
package main
import (
"fmt"
"bufio"
"io"
"os"
)
func funcToWithIO() {
file, err := os.Open("file.txt")
if err != nil {
fmt.Println(err)
return
}
defer file.Close()
data := make([]byte, 100)
for {
_, err := file.Read(data)
if err == io.EOF {
break
}
if err != nil {
fmt.Println(err)
return
}
}
}
func funcToWithBufio() {
file, err := os.Open("file.txt")
if err != nil {
fmt.Println(err)
return
}
defer file.Close()
reader := bufio.NewReader(file)
data := make([]byte, 100)
for {
_, err := reader.Read(data)
if err == io.EOF {
break
}
if err != nil {
fmt.Println(err)
return
}
}
}
func createFile() {
file, err := os.Create("file.txt")
if err != nil {
fmt.Println(err)
return
}
defer file.Close()
for i := 0; i < 1000000; i++ {
file.Write([]byte("Hello World!"))
}
}
func main() {
createFile()
funcToWithIO()
funcToWithBufio()
}
package main
import "testing"
func BenchmarkFuncToWithIO(b *testing.B) {
for i := 0; i < b.N; i++ {
funcToWithIO()
}
}
func BenchmarkFuncToWithBufio(b *testing.B) {
for i := 0; i < b.N; i++ {
funcToWithBufio()
}
}
In the above example, we are creating a file called file.txt and writing Hello World! 1,000,000 times to it. We are then reading the file using normal I/O and buffered I/O. We are then benchmarking the two functions to see which one is faster.
The output of the above program is:
goos: linux
goarch: amd64
pkg: Rbufio
cpu: Intel(R) Core(TM) i7-10750H CPU @ 2.60GHz
BenchmarkFuncToWithIO-4 2 901096096 ns/op
BenchmarkFuncToWithBufio-4 33 39213737 ns/op
PASS
ok Rbufio 4.194s
As you can see, the buffered I/O function is much faster with a runtime of 39.213737ms per iteration. The normal I/O function has a runtime of 901.096096ms per iteration. This is a huge difference in performance.
Learn more about benchmarks in the Golang Benchmarking article.
Other Bufio Functions
There are many other functions in the bufio package that can be used to read and write data. Here are some of the most commonly used functions:
bufio.Reader.ReadString
This function reads data until a specific delimiter is found. It returns a string containing the data up to and including the delimiter.
package main
import (
"bufio"
"fmt"
"os"
)
func main() {
file, err := os.Open("file.txt")
if err != nil {
fmt.Println(err)
return
}
defer file.Close()
reader := bufio.NewReader(file)
data, err := reader.ReadString('\n')
if err != nil {
fmt.Println(err)
return
}
fmt.Println(data)
}
In this case, the bufio.Reader.ReadString function will read data from the file until it finds a new line character. It will then return the data up to and including the newline character.
bufio.Reader.ReadBytes
This function is similar to the bufio.Reader.ReadString function. The only difference is that it returns a byte slice instead of a string.
package main
import (
"bufio"
"fmt"
"os"
)
func main() {
file, err := os.Open("file.txt")
if err != nil {
fmt.Println(err)
return
}
defer file.Close()
reader := bufio.NewReader(file)
data, err := reader.ReadBytes('\n')
if err != nil {
fmt.Println(err)
return
}
fmt.Println(data)
}
bufio.Reader.ReadSlice
This function is similar to the bufio.Reader.ReadString function. The only difference is that it returns a byte slice instead of a string. The difference between bufio.Reader.ReadSlice and bufio.Reader.ReadBytes is that bufio.Reader.ReadSlice will return an error if the delimiter is not found.
package main
import (
"bufio"
"fmt"
"os"
)
func main() {
file, err := os.Open("file.txt")
if err != nil {
fmt.Println(err)
return
}
defer file.Close()
reader := bufio.NewReader(file)
data, err := reader.ReadSlice('\n')
if err != nil {
fmt.Println(err)
return
}
fmt.Println(data)
}
bufio.Reader.ReadLine
package main
import (
"bufio"
"fmt"
"os"
)
func main() {
file, err := os.Open("file.txt")
if err != nil {
fmt.Println(err)
return
}
defer file.Close()
reader := bufio.NewReader(file)
data, isPrefix, err := reader.ReadLine()
if err != nil {
fmt.Println(err)
return
}
fmt.Println(data, isPrefix)
}
bufio.Reader.ReadRune
The bufio.Reader.ReadRune function reads a single UTF-8 encoded Unicode character and returns the Unicode code point. It returns an error if the character is not a valid Unicode character.
package main
import (
"bufio"
"fmt"
"os"
)
func main() {
file, err := os.Open("file.txt")
if err != nil {
fmt.Println(err)
return
}
defer file.Close()
reader := bufio.NewReader(file)
data, _, err := reader.ReadRune()
if err != nil {
fmt.Println(err)
return
}
fmt.Println(data)
}
bufio.Reader.Read
The bufio.Reader.Read function reads data into a byte slice. It returns the number of bytes read and an error if any.
package main
import (
"bufio"
"fmt"
"os"
)
func main() {
file, err := os.Open("file.txt")
if err != nil {
fmt.Println(err)
return
}
defer file.Close()
reader := bufio.NewReader(file)
data := make([]byte, 5)
n, err := reader.Read(data)
if err != nil {
fmt.Println(err)
return
}
fmt.Println(n, data)
}
bufio.Reader.Peek
The bufio.Reader.Peek function returns the next n bytes without advancing the reader. It returns an error if there are not enough bytes available.
package main
import (
"bufio"
"fmt"
"os"
)
func main() {
file, err := os.Open("file.txt")
if err != nil {
fmt.Println(err)
return
}
defer file.Close()
reader := bufio.NewReader(file)
data, err := reader.Peek(5)
if err != nil {
fmt.Println(err)
return
}
fmt.Println(data)
}
bufio.Reader.Discard
The bufio.Reader.Discard function discards the next n bytes without returning them. It returns an error if there are not enough bytes available.
package main
import (
"bufio"
"fmt"
"os"
)
func main() {
file, err := os.Open("file.txt")
if err != nil {
fmt.Println(err)
return
}
defer file.Close()
reader := bufio.NewReader(file)
_, err = reader.Discard(5)
if err != nil {
fmt.Println(err)
return
}
}
bufio.Reader.Buffered
The bufio.Reader.Buffered function returns the number of bytes that can be read from the current buffer without blocking.
package main
import (
"bufio"
"fmt"
"os"
)
func main() {
file, err := os.Open("file.txt")
if err != nil {
fmt.Println(err)
return
}
defer file.Close()
reader := bufio.NewReader(file)
fmt.Println(reader.Buffered())
}
Other Bufio Types
bufio.Writer
The bufio.Writer type implements a buffered writer. It wraps an io.Writer and provides buffering and some help for textual I/O. It is not safe for concurrent use by multiple goroutines.
It has the following methods:
bufio.NewWriter- creates a new buffered writerbufio.Writer.Flush- flushes any buffered data to the underlying writerbufio.Writer.Available- returns the number of bytes that can be written to the buffer without blockingbufio.Writer.Buffered- returns the number of bytes that are currently bufferedbufio.Writer.Reset- resets the buffer to be emptybufio.Writer.Write- writes a byte slice to the bufferbufio.Writer.WriteString- writes a string to the bufferbufio.Writer.WriteByte- writes a single byte to the bufferbufio.Writer.WriteRune- writes a single UTF-8 encoded Unicode character to the buffer
bufio.Scanner
The bufio.Scanner type implements a simple scanner for reading data. It wraps an io.Reader and provides a simple interface for reading data, line by line. It is not safe for concurrent use by multiple goroutines.
It has the following methods:
bufio.NewScanner- creates a new scannerbufio.Scanner.Scan- advances the scanner to the next token, which will then be available through theTextmethod. It returns false when the scan stops, either by reaching the end of the input or by an error.bufio.Scanner.Text- returns the most recent token generated by a call toScanas a newly allocated string holding its bytes.bufio.Scanner.Bytes- returns the most recent token generated by a call toScanas a slice of bytes that will be overwritten by the next call toScan.bufio.Scanner.Err- returns the first non-nilerror that was encountered by theScanner.
Conclusion
In this article, we learned about the bufio package and its types. We also learned about the different methods of the bufio.Reader type. We also learned about the other types of the bufio package.
