bufio.ScanLines与转义的新行

Question

I am trying to adapt bufio.ScanLines so it is aware of escaped new lines \ .

Input:

line1 \
continues on line2

Expected output:

["line1 continues on line2"]

Right now the output of bufio.ScanLines (see example code below) is:

["line1 \\", "continues on line2"]

Example code:

s := bufio.NewScanner(f)
s.Split(bufio.ScanLines)

for s.Scan() {
    fmt.Println(s.Text())
}

What would be the best approach here? Looking for an implementation that still passes the tests in https://golang.org/src/bufio/scan_test.go.

Answer 1

A few obvious approaches come to mind.

First, take a look at the source for bufio.ScanLines, it's not large and you could easily implement your own bufio.SplitFunc from scratch starting from a copy of that modified to do what you want.

Second, you could write a bufio.SplitFunc that called bufio.ScanLines in a loop, combining tokens as long as it returns ones that end in your escape character, and then returning the combined token.

Given the short size and simplicity of the first approach that's what I'd probably do. The second approach would likely end up just as long, be less efficient, and probably require state since you'd need to store the combined-token-so-far when returning (0, nil, nil) to ask for more input.

Another solution would be to implement a Transformer (from the golang.org/x/text/transform package) that strips relevant escaped characters from the input (e.g. removes "\\ ") and use transform.NewReader to make a filtered reader that you'd then do use however you want (e.g. passing to a bufio.Scanner with the regular ScanLines).

In any case, you could copy the appropriate tests from scan_test.go as well as adding your own for the escaped newline behaviour. Beware of bufio.MaxScanTokenSize as well.