doufei9805 2018-08-30 18:23
浏览 127

如何使用io.Pipe()通过ReedSolomon库通过擦除编码对大文件进行分块

Using https://github.com/klauspost/reedsolomon, erasure coding a large file into smaller chunks, I want to use io.Pipe() to create a stream of pipes which connects each other. For example: one pipe which helps in Chunking the file and other which calls func Upload for uploading the file

func (rd *ReedSolomon) Chunking() {
    fname := rd.File

    // Create encoding matrix.
    enc, err := reedsolomon.NewStream(rd.data, rd.Par)
    checkErr(err)

    fmt.Println("Opening", fname)
    f, err := os.Open(fname)
    checkErr(err)

    instat, err := f.Stat()
    checkErr(err)

    shards := rd.Data + rd.Par
    out := make([]*os.File, shards)

    // Create the resulting files.
    dir, file := filepath.Split(fname)
    if fi.OutDir != "" {
        dir = fi.OutDir
    }

    for i := range out {
        outfn := fmt.Sprintf("%s.%d", file, i)
        out[i], err = os.Create(filepath.Join(dir, outfn))
        checkErr(err)
    }

    // Split into files.
    data := make([]io.Writer, rd.Data)
    for i := range data {
        data[i] = out[i]
    }

    // Do the split
    err = enc.Split(f, data, instat.Size())
    checkErr(err)

    // Close and re-open the files.
    input := make([]io.Reader, rd.Data)
    target_url := "http://localhost:8080"

    for i := range data {
        out[i].Close()
        f, err := os.Open(out[i].Name())
        checkErr(err)
        input[i] = f
        defer f.Close()
    }

    // Create parity output writers
    parity := make([]io.Writer, rd.Par)
    for i := range parity {
        parity[i] = out[rd.Data+i]
        defer out[rd.Data+i].Close()
    }

    err = enc.Encode(input, parity)
    checkErr(err)

    for _, sd := range out {
        postFile(sd.Name(), target_url)
    }

}

I am pretty new to the use of the io.Pipe()

  • 写回答

0条回答 默认 最新

    报告相同问题?

    悬赏问题

    • ¥15 安卓adb backup备份应用数据失败
    • ¥15 eclipse运行项目时遇到的问题
    • ¥15 关于#c##的问题:最近需要用CAT工具Trados进行一些开发
    • ¥15 南大pa1 小游戏没有界面,并且报了如下错误,尝试过换显卡驱动,但是好像不行
    • ¥15 没有证书,nginx怎么反向代理到只能接受https的公网网站
    • ¥50 成都蓉城足球俱乐部小程序抢票
    • ¥15 yolov7训练自己的数据集
    • ¥15 esp8266与51单片机连接问题(标签-单片机|关键词-串口)(相关搜索:51单片机|单片机|测试代码)
    • ¥15 电力市场出清matlab yalmip kkt 双层优化问题
    • ¥30 ros小车路径规划实现不了,如何解决?(操作系统-ubuntu)