通过gzip将命令输出到Reader

Question

I'm trying to execute a shell command and compress it's output. The problem is that I then need to interface with an API that expects a Reader.

For that I tried with the following (simplified code):

package main

import (
    "encoding/hex"
    "testing"
    "fmt"
    "io"
    "io/ioutil"
    "os/exec"
    "compress/gzip"
)

func TestPipe(t *testing.T) {
    cmd := exec.Command("echo", "hello_from_echo")

    reader, writer := io.Pipe()
    gzW := gzip.NewWriter(writer)

    cmd.Stdout = gzW 
    cmd.Start()

    go func() {
        fmt.Println("Waiting")
        cmd.Wait()
        fmt.Println("wait done")
        // writer.Close()
        // gzW.Close()
    }()

    msg, _ := ioutil.ReadAll( reader )
    fmt.Println( hex.EncodeToString( msg ) )
}

The problem is that ReadAll hangs forever. If I close gzW nothing really changes. However, if I close the writer variable, now the program finishes without hanging, but the output is:

$ go test -run Pipe
Waiting
wait done
1f8b080000096e8800ff
PASS

However, no matter what I echo the output is the same. If I try it from the command line like this: echo "hello_from_echo" | gzip | hexdump the output is totally different, so there's something wrong with that approach.

Any clue what could be the problem?

Thanks in advance

Answer 1

You're closing the gzip writer and pipe writer in the wrong order. You need to close the gzip.Writer to flush any buffers and write the gzip footer, then you can close the PipeWriter to unblock the ReadAll. Also adding the WaitGroup ensures that you're not blocked on any of the close calls.

cmd := exec.Command("echo", "hello_from_echo and more")

pr, pw := io.Pipe()
gzW := gzip.NewWriter(pw)

cmd.Stdout = gzW
cmd.Start()

var wg sync.WaitGroup
wg.Add(1)
go func() {
    defer wg.Done()
    err := cmd.Wait()
    if err != nil {
        log.Println(err)
    }
    gzW.Close()
    pw.Close()
}()

buf, err := ioutil.ReadAll(pr)
if err != nil {
    t.Fatal(err)
}
wg.Wait()
fmt.Println(hex.EncodeToString(buf))