当某些行足够长而导致“ bufio.Scanner：令牌太长”错误时，如何在Go中逐行读取文本文件？

Question

I have a text file where each line represents a JSON object. I am processing this file in Go with a simple for loop like this:

scanner := bufio.NewScanner(file)
for scanner.Scan() {
   jsonBytes = scanner.Bytes()
   var jsonObject interface{}
   err := json.Unmarshal(jsonBytes, &jsonObject)

   // do stuff with "jsonObject"...

}
if err := scanner.Err(); err != nil {
   log.Fatal(err)
}

When this code reaches a line with a particularly large JSON string (~67kb), I get the error message, "bufio.Scanner: token too long".

Is there an easy way to increase the max line size readable by NewScanner? Or is there another approach you can take altogether, when needing to read lines that are too large for NewScanner but are known to not be of unsafe size generally?

Answer 1

From the package docs:

Programs that need more control over error handling or large tokens, or must run sequential scans on a reader, should use bufio.Reader instead.

It looks like the preferred solution is bufio.Reader.ReadLine.