weixin_39984952
weixin_39984952
2020-12-09 09:41

Attempting to pull single revision from Mozilla Try repo seems to fetch entire try repo with 0.5.0

I'm trying to get searchfox to support indexing try builds on https://bugzilla.mozilla.org/show_bug.cgi?id=1463888. Searchfox starts from gecko-dev so this is currently not the best idea, but under 0.5.0b3 this is turning out worse than I was expecting, so filing an issue here in case there's a simple setting I missed or in the case this helps fix a bug. Or if anyone has good advice on the best path forward for my use-case! (This is for review purposes where the blame/history info is particularly useful, so I wanted to try to have things operate largely as they normally do.)

I issued the command git cinnabar fetch hg::https://hg.mozilla.org/try 2d7b20644045a119994ade27be6524bab9798fd7 and got:


Reading 24310 graft candidates
Reading 14 changesets
Reading and importing 14 manifests
Reading and importing 156 files
..."WARNING Cannot graft" a lot...
Importing 14 changesets
Reading 1053843 changesets

And that's still ongoing.

This is after a series of operations starting from searchfox's gecko-dev.tar that looks like the following, with me giving up via control-C in many cases in the hope I could help things along.

  • git cinnabar fetch hg::https://hg.mozilla.org/try 2d7b20644045a119994ade27be6524bab9798fd7 (note that this is without any setup beyond symlinking cinnabar onto the path)

Getting clone bundle from https://hg.cdn.mozilla.net/mozilla-central/9055d9d89a4bca5cf48dda789299559aefca4e54.gzip-v2.hg
Reading 419308 changesets
Reading and importing 253357 manifests^C
  • tried the import.py script from https://github.com/glandium/git-cinnabar/wiki/Mozilla:-Using-a-git-clone-of-gecko%E2%80%90dev-to-push-to-mercurial and ran into https://github.com/glandium/git-cinnabar/issues/137 but didn't try downgrading because I didn't find the issue and assumed the page had bit-rotted.
  • git config cinnabar.graft true
  • git cinnabar fetch hg::https://hg.mozilla.org/try 2d7b20644045a119994ade27be6524bab9798fd7

Reading 668992 graft candidates
Getting clone bundle from https://hg.cdn.mozilla.net/mozilla-central/9055d9d89a4bca5cf48dda789299559aefca4e54.gzip-v2.hg
Reading 419308 changesets
Reading and importing 77947 manifests^C
  • git remote add try hg::https://hg.mozilla.org/try
  • git config remote.try.skipDefaultUpdate true
  • git config fetch.prune true
  • git cinnabar fetch try 2d7b20644045a119994ade27be6524bab9798fd7

Reading 668992 graft candidates
Getting clone bundle from https://hg.cdn.mozilla.net/mozilla-central/9055d9d89a4bca5cf48dda789299559aefca4e54.gzip-v2.hg
Reading 419308 changesets
Reading and importing 9274 manifests^C
  • git remote add mozilla hg::https://hg.mozilla.org/mozilla-unified -t bookmarks/central
  • git remote update

Fetching origin
From https://github.com/mozilla/gecko-dev
 x [deleted]         (none)     -> origin/b2g48_v2_6
Auto packing the repository in background for optimum performance.
See "git help gc" for manual housekeeping.
Fetching mozilla
Reading 668992 graft candidates
Getting clone bundle from https://hg.cdn.mozilla.net/mozilla-unified/66f6b8d84e5d35d0de8d07d3e9de44a9cce913f2.gzip-v2.hg
Reading 467485 changesets
Reading and importing 466567 manifests
Reading and importing 2720931 files
20563.639 WARNING [bundle2] ignoring bundle2 part: hgtagsfnodes
... WARNING Cannot graft*
Importing 467485 changesets
Reading 484 changesets
Reading and importing 482 manifests
... WARNING Cannot graft*
Importing 484 changesets
From hg::https://hg.mozilla.org/mozilla-unified
 * [new branch]      bookmarks/central -> mozilla/bookmarks/central
  • git cinnabar fetch try 2d7b20644045a119994ade27be6524bab9798fd7

Reading 24310 graft candidates
Reading 14 changesets
Reading and importing 14 manifests
Reading and importing 156 files
... WARNING Cannot graft*
Importing 14 changesets
Reading 1053779 changesets
Reading and importing 248858 manifests^C
  • git cinnabar fetch hg::https://hg.mozilla.org/try 2d7b20644045a119994ade27be6524bab9798fd7 (I was paranoid that the explicit remote might be doing something), this is still ongoing.

Reading 24310 graft candidates
Reading 14 changesets
Reading and importing 14 manifests
Reading and importing 156 files
... WARNING Cannot graft*
Importing 14 changesets
Reading 1053843 changesets

该提问来源于开源项目:glandium/git-cinnabar

  • 点赞
  • 写回答
  • 关注问题
  • 收藏
  • 复制链接分享
  • 邀请回答

8条回答

  • weixin_39566387 weixin_39566387 5月前

    Note that there are race conditions pulling non-graft mercurial things into a gecko-dev graft repository. If your grafted gecko-dev data is older than some changeset from mozilla-central/inbound/autoland that happens to be in the history of the try push you're fetching, you'll end up unable to graft new things from gecko-dev. (That would stop being a problem if gecko-dev was using git-cinnabar going forward)

    Until #77 allows to pull data without storing metadata, what you can do is rollback the cinnabar metadata to the state before pulling from try with git cinnabar rollback refs/cinnabar/metadata^6, assuming the metadata is only updated once. If you want to be extra sure, take the sha1 for refs/cinnabar/metadata before pulling from try, and run git cinnabar rollback <sha1> after you're done.

    点赞 评论 复制链接分享
  • weixin_39992312 weixin_39992312 5月前

    Could searchfox switch to a non-grafted git-cinnabar clone similar to https://github.com/mozilla/gecko? Pulling try heads would still be expensive until this issue is resolved, but perhaps they'd be cheap enough after the initial pull?

    点赞 评论 复制链接分享
  • weixin_39566387 weixin_39566387 5月前

    pulling try heads is not expensive. What makes it seemingly expensive is that the grafting process was essentially making the pull a no-op, which triggered the fallthrough to pull even more stuff. With grafting disabled during the pull, there is no difference between using a non-grafted clone of m-c or a grafted gecko-dev.

    点赞 评论 复制链接分享
  • weixin_39566387 weixin_39566387 5月前

    FTR, the problem is not, as I assumed, with post-clonebundle, but with grafting not doing the right thing when it couldn't graft anything that was pulled.

    点赞 评论 复制链接分享
  • weixin_39566387 weixin_39566387 5月前

    With fd7c938, things now properly fail early as originally intended:

    
    Reading 14 changesets
    Reading and importing 14 manifests
    Reading and importing 156 files
    WARNING Cannot graft 5719046d7694e0c5c50557d096f7430062e9fc0c, not importing.
    WARNING Cannot graft b8801d88ce07afdd400b704f02633f181fd7cfd2, not importing.
    WARNING Cannot graft 53e9f273db3ecc8448789909494e107870a42d1b, not importing.
    WARNING Cannot graft 1e2d7803321393133cf30b2074715203431e4d00, not importing.
    WARNING Cannot graft c7feaa6eed73b2c41d800e2ce444bd671f6bcc06, not importing.
    WARNING Cannot graft d97ce279b48571e405343b77172e3f53195fb645, not importing.
    WARNING Cannot graft b50fe9c22fcdae4e871a85a363c312167a2794d2, not importing.
    WARNING Cannot graft 4102170e1464a2e91a79c0119b6d86533794b45d, not importing.
    WARNING Cannot graft 29108f91e39b6f3423df65b591d3c102cfd237bc, not importing.
    WARNING Cannot graft 7f339f5c6b3dee02446084d1c568fc3d99004f5f, not importing.
    WARNING Cannot graft 1148bb5838d9a77a071d3558b2bb3af31c858764, not importing.
    WARNING Cannot graft 5650fe0d4f426f8e579a49805b34afdace9864ee, not importing.
    WARNING Cannot graft a046f41981a1b1507046e11d0a581a07044a947a, not importing.
    WARNING Cannot graft 2d7b20644045a119994ade27be6524bab9798fd7, not importing.
    Importing 14 changesets
    fatal: Couldn't find remote ref hg/revs/2d7b20644045a119994ade27be6524bab9798fd7
    
    点赞 评论 复制链接分享
  • weixin_39984952 weixin_39984952 5月前

    Thanks for the fix!

    Yes, I had read the thread at https://groups.google.com/d/msg/mozilla.dev.platform/kqMX4H6Iw5M/ehtXT_MeAgAJ and the docs you created/updated at https://wiki.mozilla.org/GitHub/Gecko_Repositories and it seemed like https://github.com/mozilla/gecko is the best clean option. I just wanted to see what happened with the naive gecko-dev approach. Given that https://github.com/mozilla/gecko has the CVS info in there too and git cinnabar git2hg can do the revision mapping, it seems like everything searchfox needs is there. It's just a question of legwork. I've filed https://bugzilla.mozilla.org/show_bug.cgi?id=1464297 to do this, although :kats seems better informed about all of this than I, so he may have an alternate proposal.

    点赞 评论 复制链接分享
  • weixin_39566387 weixin_39566387 5月前

    So, there's a legitimate problem with the post-clonebundle fall through, which is what causes a full clone of try, but even with that fixed, you wouldn't end up with what you want because of your setup. All you'll get is tons of "Cannot graft" warnings, and the try pushes wouldn't be pulled.

    When grafting is enabled, only changesets that have a corresponding existing commit are imported, and obviously, gecko-dev doesn't have commits corresponding to try heads.

    So you need to disable grafting when fetching try changesets.

    I'll keep this issue open for the post-clonebundle fall through issue, though, because it should only pull the changesets you requested, not all heads.

    点赞 评论 复制链接分享
  • weixin_39984952 weixin_39984952 5月前

    Thanks very much for the guidance! As you might surmise, I was just cargo-culting things and did not understand what graft was doing, but your succinct explanation has helped greatly. Thanks for the great tool and the fast response, both are greatly appreciated.

    点赞 评论 复制链接分享

相关推荐