weixin_39915820
2020-12-27 05:48 阅读 0

S2 compression

According to recent post of on Reddit:

S2 is meant to be the choice for absolute speed. While it is only available in Go other languages can simply use Snappy for compression and implementing the decoder changes should be quite trivial, since the changes are very small.

For the rest I would probably go for zstd - especially if I can find the time to implement even stronger compression for the Go version.

S2, Go data compression at GBs per second

Maybe it could be implemented in taskq?

该提问来源于开源项目:vmihailenco/taskq

  • 点赞
  • 写回答
  • 关注问题
  • 收藏
  • 复制链接分享

6条回答 默认 最新

  • weixin_39838829 weixin_39838829 2020-12-27 05:48

    It should be a fine match since it can pretty much always compress at IO speeds.

    点赞 评论 复制链接分享
  • weixin_39633090 weixin_39633090 2020-12-27 05:48

    I don't mind supporting another compression method - it is only a matter of time required to implement it and I don't have much of it...

    But I wonder why do you ask - is zstd too slow or is it a cross compilation problem?

    点赞 评论 复制链接分享
  • weixin_39915820 weixin_39915820 2020-12-27 05:48

    In project I work on S2 shows slightly better IO performance, so I already use it as an option (s2, zstd, etc, thanks to the excellent libraries of ).

    I will try to implement PR if you don't mind.

    点赞 评论 复制链接分享
  • weixin_39633090 weixin_39633090 2020-12-27 05:48

    s2 is welcome for sure. But please note that I've just pushed v3 (it is working but API not stable). So if you plan to make changes - please make sure you have the latest version.

    点赞 评论 复制链接分享
  • weixin_39915820 weixin_39915820 2020-12-27 05:48

    Thank you! I will start with v3.

    点赞 评论 复制链接分享
  • weixin_39633090 weixin_39633090 2020-12-27 05:48

    This is closed by https://github.com/vmihailenco/taskq/commit/2b763bb443ff8773bb8502a18c14f5ae08487f82.

    s2 now is the default since it is 2-3x faster especially for small payloads up to 256kb (which is the SQS limit). Comperssion is only 5%-10% worse than zstd.

    点赞 评论 复制链接分享

相关推荐