Golang：在处理CSV时，是否重新格式化单行？

Question

My golang CSV processing routine copies almost exactly from the Package CSV example:

func processCSV(path string){

    file:= utils.OpenFile(path)
    reader:= csv.NewReader(file)
    reader.LazyQuotes = true

    cs:= []*Collision{} //defined elsewhere

    for {

        line, err := reader.Read()

        //Kill processing if we're at EOF
        if err == io.EOF {
            break
        }

        c := get(line) //defined elsewhere
        cs= append(cs, c)
    }

    //Do other stuff...
}

The code works great until it encounters a malformed (?) line of CSV, which generally looks something like this:

item1,item2,"item3,"has odd quoting"","item4",item5

The csvReader.LazyQuotes = true option doesn't seem to offer enough tolerance to read this line as I need it.

My question is this: can I ask the csv reader for the original line so that I can "massage" it to pull out what I need? The files I'm working with are moderately large (~150mb) and I'm not sure I want to re-do them, especially as only a few lines per file have such problems.

Thanks for any tips!

Answer 1

As far as I can tell encoding/csv doesn't provide any such functionality, so you can either look for some 3rd party csv package that does that, or you can implement a solution yourself.

If you want to go the DIY route I can offer you a tip, whether it's a good tip that you should implement is up to you.

You could implement an io.Reader that wraps your file and tracks the last line read, then every time you encouter an error because of malformed csv you can use your reader to reread that line, massage it, add it to the results, and have the loop continue as if nothing happened.

Here's an example of how your processCSV would change:

func processCSV(path string){

    file := utils.OpenFile(path)
    myreader := NewMyReader(file)
    reader := csv.NewReader(myreader)
    reader.LazyQuotes = true

    cs:= []*Collision{} //defined elsewhere

    for {

        line, err := reader.Read()

        //Kill processing if we're at EOF
        if err == io.EOF {
            break
        }

        // malformed csv
        if err != nil {
            // Just reread the last line and on the next iteration of
            // the loop myreader.Read should continue returning bytes 
            // that come after this malformed line to the csv.Reader.
            l, err := myreader.CurrentLine()
            if err != nil {
                panic(err)
            }

            // massage the malformed csv line
            line = fixcsv(l) 
        }

        c := get(line) //defined elsewhere
        cs= append(cs, c)
    }

    //Do other stuff...
}