如何将exec函数的标准输出传递给另一个函数的读取器？

Question

I'm trying to stream the stdout from mongodump into s3. I have gotten the syntax for the S3 arbitrary-length stream right, but I don't understand how to couple the two functions. I don't want to process the entire mongodump command before I start uploading to S3. This is what I have so far:

dumpCmd := exec.Command("mongodump", "--host", "<host>", "--port", "<port>", "--archive")
dumpCmd.Stdout = os.Stdout


uploader := s3manager.NewUploader(session.New(&aws.Config{Region: aws.String("us-east-1")}))
result, err := uploader.Upload(&s3manager.UploadInput{
    Body:   dumpCmd.Stdout,
    Bucket: aws.String("myBucket"),
    Key:    aws.String("myKey"),
})
if err != nil {
    log.Fatalln("Failed to upload", err)
}

log.Println("Successfully uploaded to", result.Location)

unfortunately dumpCmd.Stdout is a writer, not a reader, and I don't know how to pipe the output of the writer into the reader.

After using a pipe I'm now getting a new error, which may belong in a new question:

dumpCmd := exec.Command("mongodump", "--host", "<host>", "--port", "<port>", "--archive")
body, err := dumpCmd.StdoutPipe()
if err != nil {
    // handle error
}

if err := dumpCmd.Start(); err != nil {
    // handle error
}

uploader := s3manager.NewUploader(session.New(&aws.Config{Region: aws.String("us-east-1")}))
result, err := uploader.Upload(&s3manager.UploadInput{
    Body:   body,
    Bucket: aws.String("myBucket"),
    Key:    aws.String("myKey"),
})
if err != nil {
    log.Fatalln("Failed to upload", err)
}

if err := dumpCmd.Wait(); err != nil {
    // handle error
}

log.Println("Successfully uploaded to", result.Location)

error:

2016/03/10 12:39:18 Failed to upload MultipartUpload: upload multipart failed
    upload id: QOWW4jBHH4PKjs1Tloc8dlCTtFN94vDHIJIWJChsrjxLZggScZbRUhM4FU9V.xOnIg9uYnBWqOA1x1xqStfA1p8vdAOHNyUp4gOO5b1gbuXvUitQyLdfFhKg9MnyxsV1
caused by: RequestError: send request failed
caused by: Put https://myBucket/myKey?partNumber=1&uploadId=QOWW4jBHH4PKjs1Tloc8dlCTtFN94vDHIJIWJChsrjxLZggScZbRUhM4FU9V.xOnIg9uYnBWqOA1x1xqStfA1p8vdAOHNyUp4gOO5b1gbuXvUitQyLdfFhKg9MnyxsV1: read |0: illegal seek
exit status 1

Answer 1

Use a pipe to connect the output of the command to the input of the upload.

The tricky issue here is that the uploader seeks on the body, but the seek is not supported by the pipe. To circumvent this, a wrapper is created around the pipe to hide the Seek method from the uploader.

dumpCmd := exec.Command("mongodump", "--host", "<host>", "--port", "<port>", "--archive")
body, err := dumpCmd.StdoutPipe()
if err != nil {
    // handle error
}

if err := dumpCmd.Start(); err != nil {
    // handle error
}

// Wrap the pipe to hide the seek methods from the uploader
bodyWrap := struct {
    io.Reader
}{body}

uploader := s3manager.NewUploader(session.New(&aws.Config{Region: aws.String("us-east-1")}))
result, err := uploader.Upload(&s3manager.UploadInput{
    Body:   bodyWrap,
    Bucket: aws.String("net-openwhere-mongodb-snapshots-dev"),
    Key:    aws.String("myKey"),
})
if err != nil {
    log.Fatalln("Failed to upload", err)
}

if err := dumpCmd.Wait(); err != nil {
   // handle error
}

log.Println("Successfully uploaded to", result.Location)