我可以在发布前获取http.NewRequest的大小吗？

Question

We have an API that accepts JSON. We encourage people to gzip the payload before they POST it because we impose a limit on size. I wanted to see the actual difference in size between raw JSON and gzip'd but I can't seem to reliably and accurately get the size of the http request after it's built.

For the non-compressed version, MyJSON []byte is the raw data before I stuff it into the http request like this:

req, err := http.NewRequest("POST", url, bytes.NewBuffer(MyJSON))

For the gzip'd version I compress the data into a buffer and then add it to the http request like this:

req, err := http.NewRequest("POST", url, &buffer)

Is it possible to get the size of the http request? And if not, I should be able to get the size of the raw MyJSON []byte with len(), but I can't seem to get the size of the bytes.Buffer for the compressed version.

Answer 1

In a word: No.

This is because the http.Request object's body is an io.Reader, which could have any size, and the only way to know the size of an io.Reader is to read it all, and count. And in practice, that io.Reader is not even consumed until after the data is already in flight.

What this means for your application is that you have two options:

1. You can either read the entire body first, and buffer it (in memory or on disk), count its size, then send it.
2. You can find a way to calculate (or estimate) the size after the request is sent over the wire.

The latter will be more efficient, but means you can't proactively act on the data before sending the request. If you choose the second approach, you can write a custom io.Reader that counts the bytes read, and pass that to your NewRequest() call.

---

Also note: The fact that you have your JSON in a byte slice ([]byte) is a bit of a code smell. This may be appropriate in some cases, but most of the time, it's more efficient (in terms of memory, and time) to stream your JSON directly to your HTTP request. Example:

var someBigHairyObject = /* Perhaps a struct or map */
r, w := io.Pipe()
go func() {
    err := json.NewEncoder(w).Encode(someBigHairyObject)
    w.CloseWithError(err)
}()
req, _ := http.NewRequest("POST", "http://example.com/", r)

In this way, the JSON marshaling (and you can include gzip here, too) is done "directly" to the network, without an intermediate buffer, which uses memory, and postpones your request.