douan0729
2016-01-29 05:50
浏览 93
已采纳

无法将Shell命令的输出写入Go中的文件

I have written the following function for executing the snt2cooc command (one of the preprocessing steps for running GIZA++. For our purposes I think we can consider the snt2cooc script to be a black box):

func SNTToCOOC(srcVocab, tgtVocab, sntPath, outpath string) error {
    // open the out file for writing
    outfile, err := os.Create(outpath)
    if err != nil {
        return err
    }
    defer outfile.Close()

    cmdStr := "snt2cooc"
    args := []string{srcVocab, tgtVocab, sntPath}
    cmd := exec.Command(cmdStr, args...)
    cmd.Stdout = outfile
    if err = cmd.Run(); err != nil {
        return err
    }
    cmd.Wait()
    return err
}

When running, the function executes without an error, but the output file is empty. This same code works for other similar commands, but not for this specific snt2cooc command, and I noticed that when I run this command in the shell directly:

snt2cooc file1.vcb file2.vcb file3.snt

I get the following output:

END.
0 2
0 3
0 4
0 5
0 6

(truncated for brevity)

And if I send the output of the command to a file directly from the shell:

snt2cooc file1.vcb file2.vcb file3.snt > out.txt

The contents of out.txt are as expected:

0 2
0 3
0 4
0 5
0 6

Notice how in the first case, the line END. is output to stdout first, and only then is the real output of the command sent to stdout. I therefore think there is a race condition going on, where the Go code finishes executing before the command's final output is written to file. This despite calling cmd.Wait(). I'm not too sure what exactly the snt2cooc command is doing internally. Could someone provide a hint on how to solve this?

Edit 1:

It seems like the following code, with the sleep of 500ms included, consistently writes output to the file for the snt2cooc command:

cmdStr := "snt2cooc"
args := []string{srcVocab, tgtVocab, sntPath}
cmd := exec.Command(cmdStr, args...)
stdout, err := cmd.StdoutPipe()
time.Sleep(500 * time.Millisecond)
if err != nil {
    return err
}
err = cmd.Start()
if err != nil {
    return err
}

out := bufio.NewScanner(stdout)
for out.Scan() {
    outfile.Write(out.Bytes())
    outfile.WriteString("
")
}
if err := out.Err(); err != nil {
    return err
}

This proves to me that there is some race condition going on, with the Go program exiting before all output is written to file. I added a bounty to this question, with the hope that someone can 1) explain why this is happening and 2) provide a non-hacky way (i.e. 500ms sleep) to fix it.

  • 写回答
  • 关注问题
  • 收藏
  • 邀请回答

2条回答 默认 最新

  • duan117890 2016-02-02 00:05
    已采纳

    First, clean up your code.

    cmd.Stderr = os.DevNull, so you ignore stderr. Stdout and Stderr specify the process's standard output and error. If either is nil, Run connects the corresponding file descriptor to the null device (os.DevNull).

    cmd.Wait() returns error, you ignore it. func (c *Cmd) Wait() error.

    Wait waits for the command to exit. It must have been started by Start. You use Run, not Start.

    What output do you get when you run this code?

    failure.go:

    package main
    
    import (
        "fmt"
        "os"
        "os/exec"
    )
    
    func main() {
        err := SNTToCOOC("file1.vcb", "file2.vcb", "file3.snt", "out.txt")
        if err != nil {
            fmt.Println(err)
        }
    }
    
    func SNTToCOOC(srcVocab, tgtVocab, sntPath, outpath string) error {
        outfile, err := os.Create(outpath)
        if err != nil {
            return err
        }
        defer outfile.Close()
        cmdStr := "snt2cooc"
        args := []string{srcVocab, tgtVocab, sntPath}
        cmd := exec.Command(cmdStr, args...)
        cmd.Stdout = outfile
        cmd.Stderr = os.Stderr
        err = cmd.Run()
        if err != nil {
            return err
        }
        return err
    }
    

    Run:

    $ rm -f out.txt && go run failure.go && cat out.txt
    

    Also, what output do you get when you run this code with cmd.Stdout = os.Stdout substituting for cmd.Stdout = outfile.

    打赏 评论
  • doufeixuan8882 2016-01-29 06:39

    The problem isn't with the innard of SNTtoCooc but how you write to the file itself using cmd.Stdout:

    func anyWrite(args []string, outpath string) error {
            outfile, err := os.Create(outpath)
            if err != nil {
                    return err
            }
    
            defer outfile.Close()
    
            // I use simple "echo" here
            cmd := exec.Command("echo", args...)
            stdout, err := cmd.Output()
            if err != nil {
                    return err
            }
    
            // Use this instead of cmd.Stdout seems to solve the problem
            outfile.Write(stdout)
    
            return nil
    }
    
    func main() {
            args := []string{"Line 1", "Line 2", "Line 3"}
            if err := anyWrite(args, "./outfile.txt"); err != nil {
                    panic(err) 
            }
    } 
    

    As per the comments in os/exec

    Stdout and Stderr specify the process's standard output and error. If either is nil, Run connects the corresponding file descriptor to the null device (os.DevNull).

    打赏 评论

相关推荐 更多相似问题