It does not look like you could read the body much faster, specially if you already tried just doing a
ioutil.ReadAll(r.Body) instead of decoding it.
And since from what you observe most requests are indeed fast, probably your problem is not that
Here are some things to try:
- Play with different "max_workers" settings
If the amount of pending requests is too large, even though goroutines are cheap they do take some memory so your server might slow down anyway, specially if garbage collection plays a role there, which takes us to the next bullet.
- Try to profile GC, or else play with different settings
Here you have the runtime documentation including the
GOGC variable you can try tuning, plus some flags you can try using to profile the GC pauses (see
gctrace flag there for example)
These blogs might also help, they detail some issues they were seeing with GC at high processing volume and how they worked around those improving GC performance:
- Scale horizontally
You don't describe your whole setup, but probably you have a group of servers load-balancing? If volume is high one possibility is that the amount of servers you have can't handle the load, in that case you can try adding more servers.