为什么在Go中首选长度为512，即[512] byte的缓冲区字节数组进行读写？

Question

I was studying Go from "Network programming with Go" by Jan Newmarch, and I noticed that almost all his examples involve [512]byte as a buffer for writing and reading to a connection.

I tried to search online but failed to get an answer. I suspect it might have something to do with i/o, but am not sure what is the exact reason behind this design.

Could anyone elaborate a bit on the choice of the buffer?

Some sample codes from the book:

func handleConn(c net.Conn){ 
    defer c.Close()

    var buf [512]byte 

    for{ 
        n, err := c.Read(buf[0:])

        if err != nil{ return }

        _, err2 := c.Write(buf[0:]) 

        if err2 != nil{ 
          return 
         }
    }
 }

Answer 1

Not a direct answer but some background in addition to what other folks stated in comments.

The Go types which wrap files and sockets are relatively thin in the sense any Read() and Write() call done on them results in a syscall being performed (with the sockets, it's more tricky as they use async I/O via the system-provided poller such as epoll, kqueue, IOCP etc). This means that reading from a file or network by chunks of 1 byte is woefully ineffective.

To consider another extreme, it's possible to allocate, say, a 100MiB buffer and attempt to pass it to Read(). While the kernel's syscall will happily accept a destination of that size, it should be noted that contemporary OSes have internal buffers on network sockets of around 64KiB¹ in size so your Read() call under most circumstances will return having read just that much data or less. This means you will be wasting most of your buffer space.

Now comes another set of considerations: what pattern of reading data from a socket your application has?

Say, when you're streaming data from a socket to an open file, you don't really care about buffering (you'd like it would be someone else's decision to pick the "right" size). For this case, just use io.Copy() (which currently (Go 1.6) uses an internal buffer of 32KiB).

Conversely, if you're parsing some application-level protocol utilizing TCP as its transport, you often need to read the data in chunks of arbitrary fixed size. For this case, the best pattern is wrapping the socket in a bufio.Reader — to combat the "small reads" problem outlined above — and then use io.ReadFull() to read data into your local arrays/slices of the size you need (if possible, do reuse your arrays and slices to lower the pressure on the garbage collector).

Another case is text-based "linewise" protocols such as SMTP or HTTP. In these protocols, the maximum line length is typically fixed, and it makes sense to use buffers of the maximum size of the protocol's line to deal with them. (But anyway, to deal with such protocols, it's best to use the net/textproto standard package.)

As to your question per se, my stab at it is that 512 is just a beautiful number which has no special meaning. When you write a book like this you have to pick some value anyway.

As you can see from my descriptions of real-work patterns of reading from the network, most of the time you simply have no business dealing with buffering — let the standard tools do this for you. You should only resort to thinking about tuning that stuff when you're facing a real problem with the defaults provided by the standard packages.

TL;DR

• The book you're reading merely explains you the basic concepts, so it has to use some number.
• Real-world code seems to use other numbers when it needs to buffer (usually higher)…
• …but you should not concern you with these numbers until absolutely necessary: use the ready-made tools where possible.

---

¹ Of course, I can't say for all operating systems, and they have different knobs to tweak this stuff, and "contemporary" might start to mean different things in over a year or less, you know… still I consider my estimation as being quite close to the truth.