Use case:
- Read the content from a table in Postgres
- Process the row Write to a file
- Write the content to a file
So I have used postgres/sql to read the rows from the table and buffered writer to write the processed row to file. I have also taken care to make sure to flush the buffer at the end of each row so that I can keep control of the memory. But still I can see the memory going as high as 1.5 GB while processing close to 200000 rows.
The code snippet is as below:
r, e := c.Query("select * from tab1")
if e != nil {
fmt.Println("Error while query", e.Error())
}
c.Close()
//file is the filehandler for the output file
fileWriter := bufio.NewWriter(file)
for r.Next() {
var res *StatsData = &StatsData{}
e = sqlstruct.Scan(res, r)
if e != nil {
fmt.Println("Failed to scan", e.Error())
continue
}
dt := p.CalculateData(*res)
fileWriter.Write([]byte(dt))
fileWriter.Flush()
res = nil
}
r.Close()
file.Close()
I expected that since I am flushing the fileWriter at the end of each loop, the buffer will be flushed to the underlying writer which will write the data to the file. Hence the memory usage will be almost constant throughout the operation.
Where am I wrong in my understanding? What can I do more to make sure that the memory can be kept under control?