并发使用时，BufferedWriter不起作用

Question

I need to write large amount of data to text file from a number of goroutines (say 30) concurrently. What I do is this:

workers.Add(core.Concurrency)
    for i := 0; i < core.Concurrency; i++ {
        go func() {
            defer workers.Done()
            writer := bufio.NewWriter(f)
            defer writer.Flush()
            a.Worker(workChan, writer)
        }()
    }

But this doesn't seems to work for some cases. Here f is the *os.File object. This doesn't writes to the file in some cases at all, and in some cases it writes some data but doesn't do future writes. The behaviour is very inconsistent and there are no errors too.

Any ideas why this might be happening?

Answer 1

The problem is that you have multiple goroutines trying to write to a file at once, without any synchronization. This will lead to an unpredicatable output order, and possibly lost writes. Using buffered I/O on top of that just helps to obscure the behavior.

The best solution is to kick off a single goroutine that writes to your output (with or without buffered I/O, depending on your needs), and have all of your workers send the data to be written to the writing goroutine over a channel.