How to read and write with Golang bufio
In Golang,
Writing with bufio
With bufio, we can use the bufio.Writer method to accumulate data into a buffer before writing to IO. In the example below, we have demonstrated three likely situations that you may encounter:
- The buffer is full
- The buffer has space after a write
- A write larger than buffer capacity is made
1. The buffer is full
As soon as the buffer is full, the write operation takes place.
2. The buffer has space after write
If the buffer still has space after the last write, it will not attempt to complete that write until specifically urged to do so by the Flush() method.
3. A write larger than buffer capacity is made
If a write is larger than buffer capacity, the buffer is skipped because there is no need to buffer.
package mainimport ("fmt""bufio")// Writer type used to initialize buffer writertype Writer intfunc (*Writer) Write(p []byte) (n int, err error) {fmt.Printf("Writing: %s\n",p)return len(p), nil}func main() {// declare a buffered writer// with buffer size 4w := new(Writer)bw := bufio.NewWriterSize(w, 4)// Case 1: Writing to buffer until fullbw.Write([]byte{'1'})bw.Write([]byte{'2'})bw.Write([]byte{'3'})bw.Write([]byte{'4'}) // write - buffer is full// Case 2: Buffer has spacebw.Write([]byte{'5'})err := bw.Flush() // forcefully write remainingif err != nil {panic(err)}// Case 3: (too) large write for buffer// Will skip buffer and write directlybw.Write([]byte("12345"))}
Miscellaneous write features
Re-use
We can re-use the same bufio.NewWriterSize for different writers using the reset() method:
writerOne := new(Writer)
bw := bufio.NewWriterSize(writerOne,2)
writerTwo := new(Writer)
bw.Reset(writerTwo)
Check available space
We can check available space with the Available() method.
Reading with bufio
bufio allows us to read in batches with bufio.Readerio.Reader under the hood
PeekReadSliceReadLineReadByteScanner
1. Peek
The Peek method lets us see the first bytes (referred to as peek value) in the buffer without consuming them. The method operates in the following way.
- If the peek value is less than buffer capacity, the characters equal to the peek value are returned.
- If the peek value is greater than buffer capacity, bufio.ErrBufferFull is returned.
- If the peek value includes EOF and is less than buffer capacity, EOF is returned.
2. ReadSlice
ReadSlice has a signature of:
func (b *Reader) ReadSlice(delim byte) (line []byte, err error)
It returns a slice of the string including the delimiter. For example, if the input is 1, 2, 3 and we use commas as delimiters, the output will be:
1, 2, 3
If the delimiter cannot be found, and EOF has been reached, then io.EOF is returned. If the delimiter is not reached and readSlice has exceeded buffer capacity, then io.ErrBufferFull is returned.
3. ReadLine
ReadLine is defined as:
ReadLine() (line []byte, isPrefix bool, err error)
ReadLine uses ReadSlice under the hood. However, it removes new-line characters (\n or \r\n) from the returned slice.
Note that its signature is different because it returns the
isPrefixflag as well. This flag returns true when the delimiter has not been found and the internal buffer is full.
Readline does not handle lines longer than the internal buffer. We can call it multiple times to finish reading.
4. ReadByte
ReadByte has a signature of:
func (b *Reader) ReadBytes(delim byte) ([]byte, error)
Similar to ReadSlice, ReadBytes returns slices before and including the delimiter. In fact, ReadByte works over ReadSlice, which acts as the underlying low-level function. However, ReadByte can call multiple instances of ReadSlice to accumulate return data; therefore, circumventing buffer size limitations. Additionally, since ReadByte returns a new slice of byte, it is safer to use because consequent read operations will not overwrite the data.
5. Scanner
Scanner breaks a stream of data by splitting it into tokens. Scanning stops at EOF, at first IO error, or if a token is too large to fit into the buffer. If more control over error handling is required, use bufio.Reader.
Scanner has a signature of:
func NewScanner(r io.Reader) *Scanner
This is the split function used for dividing the text into token defaults to ScanLines; however, you change it if need be.
package mainimport ("bufio""fmt""strconv""strings")const singleLine string = "I'd love to have some coffee right about now"const multiLine string = "Reading is my...\r\n favourite"func main() {fmt.Println("Lenght of singleLine input is " + strconv.Itoa(len(singleLine)))str := strings.NewReader(singleLine)br := bufio.NewReaderSize(str, 25)fmt.Println("\n---Peek---")// Peek - Case 1: Simple peek implementationb, err := br.Peek(3)if err != nil {fmt.Println(err)}fmt.Printf("%q\n",b) // output: "I'd"// Peek - Case 2: Peek larger than buffer sizeb, err = br.Peek(30)if err != nil {fmt.Println(err) // output: "bufio: buffer full"}// Peek - Case 3: Buffer size larger than stringbr_large := bufio.NewReaderSize(str,50)b, err = br_large.Peek(50)if err != nil {fmt.Println(err) // output: EOF}// ReadSlicefmt.Println("\n---ReadSlice---")str = strings.NewReader(multiLine)r := bufio.NewReader(str)for {token, err := r.ReadSlice('.')if len(token) > 0 {fmt.Printf("Token (ReadSlice): %q\n", token)}if err != nil {break}}// ReadLinefmt.Println("\n---ReadLine---")str = strings.NewReader(multiLine)r = bufio.NewReader(str)for {token, _ , err := r.ReadLine()if len(token) > 0 {fmt.Printf("Token (ReadLine): %q\n", token)}if err != nil {break}}// ReadBytesfmt.Println("\n---ReadBytes---")str = strings.NewReader(multiLine)r.Reset(str)for {token, err := r.ReadBytes('\n')fmt.Printf("Token (ReadBytes): %q\n", token)if err != nil {break}}// Scannerfmt.Println("\n---Scanner---")str = strings.NewReader(multiLine)scanner := bufio.NewScanner(str)for scanner.Scan() {fmt.Printf("Token (Scanner): %q\n", scanner.Text())}}
Read more about
bufioin the. official documentation https://pkg.go.dev/bufio
Free Resources
