2020-12-26 23:54 阅读 1

testing compression

I'm trying to test compression but i don't see any compression.

$ kopia repository create filesystem --path /ocean/kz
$ kopia policy set --compression=gzip-best-compression /ocean/kz
$ kopia policy show /ocean/kz
Policy for wilk:/ocean/kz:
  Compressor: "gzip-best-compression" (defined for this target)
  Compress files regardless of extensions.
  Compress files of all sizes.

$ kopia snapshot ~/projets/flibuste

The repository has exactly the same size without setting compression (580M for 617M of source, 259M with targz). Did I missed some configuration ?

I also tried setting compression with --global

edit: I build kopia with today clone


  • 点赞
  • 写回答
  • 关注问题
  • 收藏
  • 复制链接分享

6条回答 默认 最新

  • weixin_39946364 weixin_39946364 2020-12-26 23:54

    you need to set policy on ~/projects/... and not on the repository.

    What kinds of files are you compressing? Many files these days are not really compressible at all. Try a directory full of source code, such as Kopia itself.

    To verify the compression do kopia snapshot ls to find the directory entry (starts with k<hash>) then do kopia ls -l k<hash> - if you see any entries with Z in them - those are compressed.

    点赞 评论 复制链接分享
  • weixin_39757040 weixin_39757040 2020-12-26 23:54

    It's ok with setting policy on the source. I've tried with kopia source and it's ok also.

    But i found that with a sql dump as text, very compressible (153M possibly gziped to 10M) there is an issue.

    $ ls -lh /tmp/data
    total 153M
    -rw-r--r-- 1 wilk wilk 153M janv.  8 16:30 dump.sql

    Without compression

    $  kopia snapshot /tmp/data
    uploaded snapshot 034c... (root k8b...) in 4.407741852s
    $ du -shc /ocean/kk
    153M    /ocean/kk

    With compression

    uploaded snapshot 70... (root k9b6...) in 1m9.976920549s
    $ du -shc /ocean/kk
    311M    /ocean/kk

    I cleared cache and create new repository each time

    edit: also mem usage is going very high (800M)

    点赞 评论 复制链接分享
  • weixin_39946364 weixin_39946364 2020-12-26 23:54

    Weird, I'll investigate this - I downloaded a 163MB SQL dump from https://launchpad.net/test-db/+download and I can also see this.

    点赞 评论 复制链接分享
  • weixin_39757040 weixin_39757040 2020-12-26 23:54

    I can confirm that it fix the issue \o/. You've done it very rapidly. My repository (617M) is going to 262M which is less than with borg 305M. But slower (kopia 34s, borg 16s). On second snapshot kopia is a lot faster than borg...

    I will test it now with my home on a daily basis to compare with real data.

    Thanks !

    点赞 评论 复制链接分享
  • weixin_39946364 weixin_39946364 2020-12-26 23:54

    btw you can see which compression algo is the fastest with:

    kopia benchmark compression --data-file ~/employees_db/load_salaries.dump --repeat 1
    点赞 评论 复制链接分享
  • weixin_39757040 weixin_39757040 2020-12-26 23:54

    Changing the algo, it's a lot faster with s2. Still a little bit slower than borg (lz4) but, for me, i don't think it need to be investigated more than that.

    点赞 评论 复制链接分享