如何在GO lang中将大型csv文件拆分为多个文件？

Question

I am a novice Go lang programmer,trying to learn Go lang features.I wanted to split a large csv file into multiple files in GO lang, each file containing the header.How do i do this? I have searched everywhere but couldnt get the right solution.Any help in this regard will be greatly appreciated. Also please suggest me a good book for reference.

Thanking You

Answer 1

Depending on your shell fu this problem might be better suited for common shell utilities but you specifically mentioned go.

Let's think through the problem.

How big is this csv file? Are we talking 100 lines or is it 5G ?

If it's smallish I typically use this:

http://golang.org/pkg/io/ioutil/#ReadFile

However, this package also exists:

http://golang.org/pkg/encoding/csv/

Regardless - let's return to the abstraction of the problem. You have a header (which is the first line) and then the rest of the document.

So what we probably want to do (if ignoring csv for the moment) is to read in our file.

Then we want to split the file body by all the newlines in it.

You can use this to do so:

http://golang.org/pkg/strings/#Split

You didn't mention but do you know how many files you want to split by or would you rather split by the line count or byte count? What's the actual limitation here?

Generally it's not going to be file count but if we pretend it is we simply want to divide our line count by our expected file count to give lines/file.

Now we can take slices of the appropriate size and write the file back out via:

http://golang.org/pkg/io/ioutil/#WriteFile

A trick I use sometime to help think me threw these things is to write down our mission statement.

"I want to split a large csv file into multiple files in go"

Then I start breaking that up into pieces but take the divide/conquer approach - don't try to solve the entire problem in one go - just break it up to where you can think about it.

Also - make gratiutious use of pseudo-code until you can comfortably write the real code itself. Sometimes it helps to just write a short comment inline with how you think the code should flow and then get it down to the smallest portion that you can code and work from there.

By the way - many of the golang.org packages have example links where you can literally run in your browser the example code and cut/paste that to your own local environment.

Also, I know I'll catch some haters with this - but as for books - imo - you are going to learn a lot faster just by trying to get things working rather than reading. Action trumps passivity always. Don't be afraid to fail.