在Golang中读取非常大的文件

Question

I'm currently trying to read in a file with 200+ columns and 1000+ rows. I use the following code:

var result []string

file, err := os.Open("t8.txt")
if (err != nil) {
  fmt.Println(err)
}
defer file.Close()
scan := bufio.NewScanner(file)
for scan.Scan() {
  result = append(result, scan.Text())

}


fmt.Println(scan.Err()) //token too long

However when I print out the results, all I get is the first line because it says the token is too long. When I try it on smaller files, it works fine. Is there a way in Golang that I could scan in large files?

Answer 1

As already pointed out by @Dave C in the comments you are running into MaxScanTokenSize = 64 * 1024

To get around that limitation, use bufio.Reader which has a ReadString(delim byte) method which seems appropriate for your case.

From the Scanner go doc (specifically the last sentence):

Scanner provides a convenient interface for reading data such as a file of newline-delimited lines of text. Successive calls to the Scan method will step through the 'tokens' of a file, skipping the bytes between the tokens. The specification of a token is defined by a split function of type SplitFunc; the default split function breaks the input into lines with line termination stripped. Split functions are defined in this package for scanning a file into lines, bytes, UTF-8-encoded runes, and space-delimited words. The client may instead provide a custom split function.

Scanning stops unrecoverably at EOF, the first I/O error, or a token too large to fit in the buffer. When a scan stops, the reader may have advanced arbitrarily far past the last token. Programs that need more control over error handling or large tokens, or must run sequential scans on a reader, should use bufio.Reader instead.