当在Docker中将PEPI输出时从OS.Stdin读取时被阻止

Question

I'm trying to pipe the output(logs) of a program to a Go program which aggregates/compress the output and uploads to S3. The command to run the program is "/program1 | /logShipper". The logShipper is written in Go and it's simply read from os.Stdin and write to a local file. The local file will be processed by another goroutine and upload to S3 periodically. There are some existing docker log drivers but we are running the container on a fully managed provider and the log processing charge is pretty expensive, so we want to bypass the existing solution and just upload to S3.

The main logic of the logShipper is simply read from the os.Stdin and write to some file. It's work correctly when running on the local machine but when running in docker the goroutine blocked at reader.ReadString(' ') and never return.

go func() {
    reader := bufio.NewReader(os.Stdin)
    mu.Lock()
    output = openOrCreateOutputFile(&uploadQueue, workPath)
    mu.Unlock()
    for {
    text, _ := reader.ReadString('
')
    now := time.Now().Format("2006-01-02T15:04:05.000000000Z")
    mu.Lock()
    output.file.Write([]byte(fmt.Sprintf("%s %s", now, text)))
    mu.Unlock()
     }
}()

I did some research online but not find why it's not working. One possibility I'm thinking is might docker redirect the stdout to somewhere so the PIPE not working the same way as it's running on a Linux box? (As looks like it can't read anything from program1) Any help or suggestion why it not working is welcome. Thanks.

Edit: After doing more research I realized it's a bad practice to handle the logs in this way. I should more rely on the docker's log driver to handle the log aggregate and shipping. However, I'm still interested to find out why it's not read anything from the PIPE source program.

Answer 1

I'm not sure about the way the Docker handles output, but I suggest that you extract the file descriptor with os.Stdin.Fd() and then resort to using golang.org/x/sys/unix package as follows:

// Long way, for short one jump
// down straight to it.
//
// retrieve the file descriptor
// cast it to int, because Fd method
// returns uintptr
fd := int(os.Stdin.Fd())

// extract file descriptor flags
// it's safe to drop the error, since if it's there
// and it's not nil, you won't be able to read from
// Stdin anyway, unless it's a notice
// to try again, which mostly should not be 
// the case
flags, _ := unix.FcntlInt(fd, unix.F_GETFL, 0)

// check if the nonblocking reading in enabled
nb := flags & unix.O_NONBLOCK != 0

// if this is the case, just enable it with
// unix.SetNonblock which is also a
// -- SHORT WAY HERE --
err = unix.SetNonblock(fd, true)

The difference between the long and a short way is that the long way will definitely tell you, if the problem is in the nonblocking state absence or not.

If this is not the case. Then I have no other ideas personally.