doumao1917 2016-10-03 14:40
浏览 252

在Golang中将数据写入文件时内存变高

Use case:

  1. Read the content from a table in Postgres
  2. Process the row Write to a file
  3. Write the content to a file

So I have used postgres/sql to read the rows from the table and buffered writer to write the processed row to file. I have also taken care to make sure to flush the buffer at the end of each row so that I can keep control of the memory. But still I can see the memory going as high as 1.5 GB while processing close to 200000 rows.

The code snippet is as below:

    r, e := c.Query("select * from tab1")
    if e != nil {
            fmt.Println("Error while query", e.Error())
    }
    c.Close()
    //file is the filehandler for the output file
    fileWriter := bufio.NewWriter(file)
    for r.Next() {
            var res *StatsData = &StatsData{}
            e = sqlstruct.Scan(res, r)
            if e != nil {
                    fmt.Println("Failed to scan", e.Error())
                    continue
            }
            dt := p.CalculateData(*res)
            fileWriter.Write([]byte(dt))
            fileWriter.Flush()
            res = nil
    }
    r.Close()
    file.Close()

I expected that since I am flushing the fileWriter at the end of each loop, the buffer will be flushed to the underlying writer which will write the data to the file. Hence the memory usage will be almost constant throughout the operation.

Where am I wrong in my understanding? What can I do more to make sure that the memory can be kept under control?

  • 写回答

0条回答 默认 最新

    报告相同问题?

    悬赏问题

    • ¥15 python的qt5界面
    • ¥15 无线电能传输系统MATLAB仿真问题
    • ¥50 如何用脚本实现输入法的热键设置
    • ¥20 我想使用一些网络协议或者部分协议也行,主要想实现类似于traceroute的一定步长内的路由拓扑功能
    • ¥30 深度学习,前后端连接
    • ¥15 孟德尔随机化结果不一致
    • ¥15 apm2.8飞控罗盘bad health,加速度计校准失败
    • ¥15 求解O-S方程的特征值问题给出边界层布拉休斯平行流的中性曲线
    • ¥15 谁有desed数据集呀
    • ¥20 手写数字识别运行c仿真时,程序报错错误代码sim211-100