结合使用AWS SQS和Golang的最有效方法

Question

When using the AWS SQS (Simple Queue Service) you pay for each request you make to the service (push, pull, ...). There is a maximum of 256kb for each message you can send to a queue.

To save money I'd like to buffer messages sent to my Go application before I send them out to SQS until I have enough data to efficiently use the 256kb limit.

Since my Go application is a webserver, my current idea is to use a string mutex and append messages as long as I would exceed the 256kb limit and then issue the SQS push event. To save even more space I could gzip every single message before appending it to the string mutex.

I wonder if there is some kind of gzip stream that I could use for this. My assumption is that gzipping all concatenated messages together will result in smaller size then gzipping every message before appending it to the string mutex. One way would be to gzip the string mutex after every append to validate its size. But that might be very slow.

Is there a better way? Or is there a total better approach involving channels? I'm still new to Go I have to admit.

Answer 1

I'd take the following approach

• Use a channel to accept incoming "internal" messages to a go routine
• In that go routine keep the messages in a "raw" format, so 10 messages is 10 raw uncompressed items
• Each time a new raw item arrives, compress all the raw messages into one. If the size with the new message > 256k then compress messages EXCEPT the last one and push to SQS

This is computationally expensive. Each individual message causes a full compression of all pending messages. However it is efficient for SQS use