duanpin9531 2018-08-22 18:53
浏览 50
已采纳

Go中最快的数据加载方式

I need to regularly load over 300'000 rows x 78 columns of data into my Go program.

Currently I use (import github.com/360EntSecGroup-Skylar/excelize):

xlsx, err := excelize.OpenFile("/media/test snaps.xlsm")
if err != nil {
    fmt.Println(err)
    return
}

//read all rows into df
df := xlsx.GetRows("data")

It takes about 4 minutes on a decent PC using Samsung 960 EVO Series - M.2 Internal SSD.

Is there a faster way to load this data? Currently it takes me more time to read the data than processing it. I'm also opened to other file formats.

  • 写回答

1条回答 默认 最新

  • doupi7619 2018-08-22 20:11
    关注

    As suggested in the comments, instead of using the XLS format, use a custom, fast data format for reading and writing your table.

    In the most basic case, just write the number of columns and rows to a binary file, then write all the data in one go. This will be very fast, I have created a little example here which just writes 300.000 by 40 float32s to a file and reads them back. On my machine this takes about 400ms and 250ms (notice that the file is hot in cache after writing it, may take longer on initial read).

    package main
    
    import (
        "encoding/binary"
        "os"
    
        "github.com/gonutz/tic"
    )
    
    func main() {
        const (
            rowCount = 300000
            colCount = 40
        )
        values := make([]float32, rowCount*colCount)
        func() {
            defer tic.Toc()("write")
            f, _ := os.Create("file")
            defer f.Close()
            binary.Write(f, binary.LittleEndian, int64(rowCount))
            binary.Write(f, binary.LittleEndian, int64(colCount))
            check(binary.Write(f, binary.LittleEndian, values))
        }()
        func() {
            defer tic.Toc()("read")
            f, _ := os.Open("file")
            defer f.Close()
            var rows, cols int64
            binary.Read(f, binary.LittleEndian, &rows)
            binary.Read(f, binary.LittleEndian, &cols)
            vals := make([]float32, rows*cols)
            check(binary.Read(f, binary.LittleEndian, vals))
        }()
    }
    
    func check(err error) {
        if err != nil {
            panic(err)
        }
    }
    
    本回答被题主选为最佳回答 , 对您是否有帮助呢?
    评论

报告相同问题?

悬赏问题

  • ¥15 求jacquard数据集
  • ¥15 w10部分软件不能联网
  • ¥15 关于安装hbase的问题(操作系统-windows)
  • ¥15 cadence617版本,如何做一个参数可调的反相器
  • ¥15 novnc连接pve虚拟机报错安全协议不支持262
  • ¥15 设备精度0.03给多少公差能达到CPK1.33
  • ¥15 qt+ffmpeg报错non-existing PPS 0 referenced
  • ¥15 FOC simulink
  • ¥50 MacOS 使用虚拟机安装k8s
  • ¥20 玩游戏gpu和cpu利用率特别低,玩游戏卡顿