如何减少[]字符串的内存使用量？

Question

Go is pretty new to me and i have some troubles understanding the memory usage :

I want to load a file similar to csv into an array of rows, each row being a struct composed of a key on 22 char and an array of values (string).
My code look like this : https://play.golang.org/p/hJ4SHjVXaG

Problem is that for a file of 450M it uses around 2G1 of memory.
Does anyone have a solution to reduce that memory use ?

Update using SirDarius solution : https://play.golang.org/p/DBmOFOkZdx still use around 1G9

Answer 1

How many lines and fields are there in the file?

It is plausible that what you are describing is using the minimum amount of memory.

Looking at the code I think it will use 450MB of memory for the underlying string data.

It will then slice that up into strings. These consist of a pointer and a length which take 16 bytes on a 64 bit platform.

So 1.5GB/16 = 93Million.

So if there are >50 Million fields in your file then the memory use seems reasonable.

There are other overheads like number of rows etc so this isn't an exact calculation.

EDIT

Given
5 millions row, 10 column each

That is 50 million string headers of 16 bytes which will take 800MB. Plus the data itself 450MB, plus 5 * 8 * 5 million Rows = 200MB makes 1.45GB

So I don't think even with perfect memory allocation, you'll be able to reduce the usage below 1.5GB.