doutongfu9484
2014-11-28 21:34
浏览 260
已采纳

Golang从管道中读取大量数据

I'm trying to read an archive that's being tarred, streaming, to stdin, but I'm somehow reading far more data in the pipe than tar is sending.

I run my command like this:

tar -cf - somefolder | ./my-go-binary

The source code is like this:

package main

import (
    "bufio"
    "io"
    "log"
    "os"
)

// Read from standard input
func main() {
    reader := bufio.NewReader(os.Stdin)
    // Read all data from stdin, processing subsequent reads as chunks.
    parts := 0
    for {
        parts++
        data := make([]byte, 4<<20) // Read 4MB at a time
        _, err := reader.Read(data)
        if err == io.EOF {
            break
        } else if err != nil {
            log.Fatalf("Problems reading from input: %s", err)
        }
    }
    log.Printf("Total parts processed: %d
", parts)
}

For a 100MB tarred folder, I'm getting 1468 chunks of 4MB (that's 6.15GB)! Further, it doesn't seem to matter how large the data []byte array is: if I set the chunk size to 40MB, I still get ~1400 chunks of 40MB data, which makes no sense at all.

Is there something I need to do to read data from os.Stdin properly with Go?

  • 写回答
  • 好问题 提建议
  • 追加酬金
  • 关注问题
  • 邀请回答

2条回答 默认 最新

相关推荐 更多相似问题