weixin_39990819
weixin_39990819
2021-01-12 04:07

git clone premature end of pack file

Version of s3fs being used: V1.84 Kernel information : 4.15.0-1023-aws GNU/Linux Distribution: image /etc/fstab: s3fs#csgo-ranking-prediction /home/ubuntu/s3 fuse _netdev,exec,allow_other,passwd_file=/etc/passwd-s3fs,cipher_suites=AESGCM,kernel_cache,max_background=1000,max_stat_cache_size=100000,multipart_size=52,parallel_count=30,multireq_max=30,umask=0002,uid=1000,gid=1000,dbglevel=warn 0 0 system logs: log.txt

error details: image

该提问来源于开源项目:s3fs-fuse/s3fs-fuse

  • 点赞
  • 写回答
  • 关注问题
  • 收藏
  • 复制链接分享
  • 邀请回答

10条回答

  • weixin_39952502 weixin_39952502 3月前

    Problem persists with 1.85 but with slightly different symptoms:

    
    $ git clone https://github.com/s3fs-fuse/s3fs-fuse.git
    Cloning into 's3fs-fuse'...
    remote: Enumerating objects: 55, done.
    remote: Counting objects: 100% (55/55), done.
    remote: Compressing objects: 100% (32/32), done.
    remote: Total 4898 (delta 22), reused 41 (delta 22), pack-reused 4843
    Receiving objects: 100% (4898/4898), 2.74 MiB | 26.00 KiB/s, done.
    warning: die() called many times. Recursion error or racy threaded death!
    fatal: premature end of pack file, 48 bytes missing
    fatal: premature end of pack file, 48 bytes missing
    fatal: index-pack failed
    
    点赞 评论 复制链接分享
  • weixin_39952502 weixin_39952502 3月前

    I made some progress on diagnosing this issue via mirrorfs. s3fs unexpectedly truncates a temporary file:

    
    $ ls -l blobstore/s3proxy-bucket/java-httpbin/.git/objects/pack/tmp_pack_Zzimzc
    -rw-rw----. 1 gaul gaul 0 May  2 18:50 blobstore/s3proxy-bucket/java-httpbin/.git/objects/pack/tmp_pack_Zzimzc
    
    点赞 评论 复制链接分享
  • weixin_39952502 weixin_39952502 3月前

    I found that s3fs truncates unflushed open files when opening a second fd to them. The following example returns 4 with ext4 and 0 with s3fs:

    
    (echo bar ; wc -c < foo >&2) > foo
    
    点赞 评论 复制链接分享
  • weixin_39688636 weixin_39688636 3月前

    I was able to reproduce this problem.

    If I try on the local disk, it will look like

    
    $ echo test > foo
    $ (echo bar ; wc -c < foo >&2) > foo
      4
    $ cat foo
      bar
    

    The file system mounted with s3fs is as follows.

    
    $ echo test > foo
    $ (echo bar ; wc -c < foo >&2) > foo
      0
    $ cat foo
    

    I think that the cause is the problem of time lag until file upload, and tested it with the following command.

    
    (echo bar ; sleep 10 ; wc -c < foo >&2) > foo
    

    As a result, both got the same result.

    
    $ echo test > foo
    $ (echo bar ; sleep 10 ; wc -c < foo >&2) > foo
      4
    $ cat foo
      bar
    

    From this result, it seems that the following command is executed before the file output(upload) is completed. The time to upload the object to S3 and complete will take much more time to complete writing to the local disk. I think this is the reason.

    点赞 评论 复制链接分享
  • weixin_39952502 weixin_39952502 3月前

    I successfully cloned the s3fs-fuse repository into an s3fs mount. Fixed by c596441f5872c5309908739de66397182229123c.

    点赞 评论 复制链接分享
  • weixin_39963853 weixin_39963853 3月前

    Hi, I have the same issue. Have you succeeded in solving it?

    点赞 评论 复制链接分享
  • weixin_39990819 weixin_39990819 3月前

    I have switched to goofys and decided to handle all kind of git operations on my local

    点赞 评论 复制链接分享
  • weixin_39688636 weixin_39688636 3月前

    I can not reproduce this problem. I looked for similar problems, and they seemed to be occurring when repository capacity is large, git command is old, etc. We would like to confirm the following for this Issue solution. Is this problem also occurring in other repositories? Is the git command occurred even at the latest?

    点赞 评论 复制链接分享
  • weixin_39947961 weixin_39947961 3月前

    Hi, It still happened, please reference below.

    Error message when clone to s3 mounted directory:

    
    Cloning into '/public/reference/jupyter-tutorial'...
    remote: Enumerating objects: 142, done.
    remote: Total 142 (delta 0), reused 0 (delta 0), pack-reused 142
    Receiving objects: 100% (142/142), 223.22 KiB | 38.00 KiB/s, done.
    fatal: premature end of pack file, 63 bytes missing
    fatal: index-pack failed
    
    [root-10-94-89-236 ]# cat /etc/fstab
    #
    UUID=e9c3e7c2-e039-4b2b-9991-955a8c7cd8c0     /           xfs    defaults,noatime  1   1
    /dev/xvdb  swap  swap defaults 0 0
    /dev/xvdc /dat1  ext4  defaults 0 2
    s3fs#jupyterhub-dev /s3f1 fuse _netdev,allow_other,iam_role=auto,use_cache=/tmp 0 0
    
    [root-10-94-89-236 ]# s3fs --version
    Amazon Simple Storage Service File System V1.84(commit:unknown) with OpenSSL
    Copyright (C) 2010 Randy Rizun <rrizun.com>
    License GPL2: GNU GPL version 2 <http:>
    This is free software: you are free to change and redistribute it.
    There is NO WARRANTY, to the extent permitted by law.
    
    [root-10-94-89-236 ]# git --version
    git version 2.14.5
    
    [root-10-94-89-236 ]# cat /etc/os-release
    NAME="Amazon Linux"
    VERSION="2"
    ID="amzn"
    ID_LIKE="centos rhel fedora"
    VERSION_ID="2"
    PRETTY_NAME="Amazon Linux 2"
    ANSI_COLOR="0;33"
    CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
    HOME_URL="https://amazonlinux.com/"
    
    [root-10-94-89-236 ]# cat /etc/system-release
    Amazon Linux release 2 (Karoo)
    </http:></rrizun.com>
    点赞 评论 复制链接分享
  • weixin_39952502 weixin_39952502 3月前

    I tested this again against master including 5f5da4b2cbf2753024ff58b4b3a75b686879a7c5 but still experience these symptoms:

    
    $ git clone https://github.com/s3fs-fuse/s3fs-fuse.git
    Cloning into 's3fs-fuse'...
    remote: Enumerating objects: 36, done.
    remote: Counting objects: 100% (36/36), done.
    remote: Compressing objects: 100% (26/26), done.
    remote: Total 4534 (delta 16), reused 20 (delta 10), pack-reused 4498
    Receiving objects: 100% (4534/4534), 2.57 MiB | 4.00 KiB/s, done.
    error: inflate: data stream error (unknown compression method)
    error: inflate: data stream error (unknown compression method)
    fatal: serious inflate inconsistency
    fatal: index-pack failed
    
    点赞 评论 复制链接分享