weixin_39937635
weixin_39937635
2021-01-12 05:27

Bundle install fails first time without cache, succeeds second time

While deploying our application to new instances recently we've seen:


    buildkite-puma: + su - deploy -c 'cd /app && bundle install --deployment --without=development:test'
    buildkite-puma: Fetching gem metadata from https://rubygems.org/
    buildkite-puma: Fetching version metadata from https://rubygems.org/
    buildkite-puma: Fetching version metadata from https://rubygems.org/
    buildkite-puma: Fetching dependency metadata from https://rubygems.org/
    buildkite-puma: Could not find intercom-3.5.1 in any of the sources
==> buildkite-puma: Terminating the source AWS instance...

I've been able to replicate this with the following on a generic ubuntu AWS instance (rubygems and bundler freshly updated):

 ruby
$ bundle env
Environment

    Bundler   1.12.5
    Rubygems  2.6.4
    Ruby      2.2.4p230 (2015-12-16 revision 53155) [x86_64-linux-gnu]
    Git       2.5.0

Bundler settings

    frozen
      Set for your local app (/home/deploy/test/.bundle/config): "1"
    path
      Set for your local app (/home/deploy/test/.bundle/config): "vendor/bundle"
    disable_shared_gems
      Set for your local app (/home/deploy/test/.bundle/config): "true"
    orig_path
      Set via BUNDLE_ORIG_PATH: "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games"

Gemfile

    source "https://rubygems.org"

    gem "intercom"

Gemfile.lock

    GEM
      remote: https://rubygems.org/
      specs:
        intercom (3.5.1)
          json (~> 1.8)
        json (1.8.3)

    PLATFORMS
      ruby

    DEPENDENCIES
      intercom

    BUNDLED WITH
       1.12.4

If I bundle install this (using deployment for convenience, the problem happens either way):


$ bundle install --deployment
Fetching gem metadata from https://rubygems.org/
Fetching version metadata from https://rubygems.org/
Could not find intercom-3.5.1 in any of the sources

If I bundle install subsequently:


$ bundle install --deployment
Fetching gem metadata from https://rubygems.org/
Fetching version metadata from https://rubygems.org/
Installing json 1.8.3 with native extensions
Using bundler 1.12.5
Installing intercom 3.5.1
Bundle complete! 1 Gemfile dependency, 3 gems now installed.
Use `bundle show [gemname]` to see where a bundled gem is installed.

But if I remove ~/.bundle and the installed gems, then it fails again:


$ rm -rf ~/.bundle vendor/bundle
$ bundle install --deployment
Fetching gem metadata from https://rubygems.org/
Fetching version metadata from https://rubygems.org/
Could not find intercom-3.5.1 in any of the sources

Here's a broken bundle install with verbose mode on:


$ bundle install --deployment --verbose
HTTP GET https://index.rubygems.org/versions
HTTP 200 OK
Fetching gem metadata from https://rubygems.org/
Looking up gems ["json", "intercom"]
HTTP GET https://index.rubygems.org/info/json
HTTP GET https://index.rubygems.org/info/intercom
HTTP 200 OK
HTTP 200 OK
Looking up gems ["rest-client"]
HTTP GET https://index.rubygems.org/info/rest-client
HTTP 200 OK
Looking up gems ["mime-types", "rdoc", "netrc", "ffi", "http-cookie"]
HTTP GET https://index.rubygems.org/info/mime-types
HTTP GET https://index.rubygems.org/info/rdoc
HTTP GET https://index.rubygems.org/info/netrc
HTTP GET https://index.rubygems.org/info/ffi
HTTP GET https://index.rubygems.org/info/http-cookie
HTTP 200 OK
HTTP 200 OK
HTTP 200 OK
HTTP 200 OK
HTTP 200 OK
Looking up gems ["domain_name", "sqlite3", "hoe", "minitest", "archive-tar-minitar", "nokogiri", "rcov", "mime-types-data", "bones", "rake"]
HTTP GET https://index.rubygems.org/info/domain_name
HTTP GET https://index.rubygems.org/info/sqlite3
HTTP GET https://index.rubygems.org/info/minitest
HTTP GET https://index.rubygems.org/info/archive-tar-minitar
HTTP GET https://index.rubygems.org/info/nokogiri
HTTP GET https://index.rubygems.org/info/rcov
HTTP GET https://index.rubygems.org/info/mime-types-data
HTTP GET https://index.rubygems.org/info/bones
HTTP GET https://index.rubygems.org/info/rake
HTTP GET https://index.rubygems.org/info/hoe
HTTP 200 OK
HTTP 200 OK
HTTP 200 OK
HTTP 200 OK
HTTP 200 OK
HTTP 200 OK
HTTP 200 OK
HTTP 200 OK
HTTP 200 OK
HTTP 200 OK
Looking up gems ["racc", "rake-compiler", "rexical", "tenderlove-frex", "weakling", "mini_portile", "mini_portile2", "pkg-config", "unf", "little-plugger", "loquacious", "bones-extras", "bones-git", "rspec", "rubyforge", "RubyInline", "gemcutter"]
HTTP GET https://index.rubygems.org/info/racc
HTTP GET https://index.rubygems.org/info/rake-compiler
HTTP GET https://index.rubygems.org/info/rexical
HTTP GET https://index.rubygems.org/info/tenderlove-frex
HTTP GET https://index.rubygems.org/info/weakling
HTTP GET https://index.rubygems.org/info/mini_portile
HTTP GET https://index.rubygems.org/info/mini_portile2
HTTP GET https://index.rubygems.org/info/unf
HTTP GET https://index.rubygems.org/info/little-plugger
HTTP GET https://index.rubygems.org/info/loquacious
HTTP GET https://index.rubygems.org/info/bones-extras
HTTP GET https://index.rubygems.org/info/bones-git
HTTP GET https://index.rubygems.org/info/rspec
HTTP GET https://index.rubygems.org/info/rubyforge
HTTP GET https://index.rubygems.org/info/RubyInline
HTTP GET https://index.rubygems.org/info/gemcutter
HTTP GET https://index.rubygems.org/info/pkg-config
HTTP 200 OK
HTTP 200 OK
HTTP 200 OK
HTTP 200 OK
HTTP 200 OK
HTTP 200 OK
HTTP 200 OK
HTTP 200 OK
HTTP 200 OK
HTTP 200 OK
HTTP 200 OK
HTTP 200 OK
HTTP 200 OK
HTTP 200 OK
HTTP 200 OK
HTTP 200 OK
HTTP 200 OK
Looking up gems ["git", "bones-rcov", "bones-rspec", "bones-rubyforge", "bones-zentest", "json_pure", "net-scp", "ZenTest", "unf_ext", "cucumber", "diff-lcs", "spicycode-rcov", "syntax", "rspec-core", "rspec-expectations", "rspec-mocks"]
HTTP GET https://index.rubygems.org/info/git
HTTP GET https://index.rubygems.org/info/bones-rcov
HTTP GET https://index.rubygems.org/info/bones-rspec
HTTP GET https://index.rubygems.org/info/bones-rubyforge
HTTP GET https://index.rubygems.org/info/json_pure
HTTP GET https://index.rubygems.org/info/net-scp
HTTP GET https://index.rubygems.org/info/ZenTest
HTTP GET https://index.rubygems.org/info/unf_ext
HTTP GET https://index.rubygems.org/info/cucumber
HTTP GET https://index.rubygems.org/info/diff-lcs
HTTP GET https://index.rubygems.org/info/spicycode-rcov
HTTP GET https://index.rubygems.org/info/syntax
HTTP GET https://index.rubygems.org/info/rspec-core
HTTP GET https://index.rubygems.org/info/rspec-expectations
HTTP GET https://index.rubygems.org/info/rspec-mocks
HTTP GET https://index.rubygems.org/info/bones-zentest
HTTP 200 OK
HTTP 200 OK
HTTP 200 OK
HTTP 200 OK
HTTP 200 OK
HTTP 200 OK
HTTP 200 OK
HTTP 200 OK
HTTP 200 OK
HTTP 200 OK
HTTP 200 OK
HTTP 200 OK
HTTP 200 OK
HTTP 200 OK
HTTP 200 OK
HTTP 200 OK
Looking up gems ["spruz", "rspec-support", "builder", "polyglot", "term-ansicolor", "treetop", "gherkin", "multi_json", "multi_test", "cucumber-core", "gherkin3", "cucumber-wire", "event-bus", "net-ssh"]
HTTP GET https://index.rubygems.org/info/spruz
HTTP GET https://index.rubygems.org/info/rspec-support
HTTP GET https://index.rubygems.org/info/builder
HTTP GET https://index.rubygems.org/info/polyglot
HTTP GET https://index.rubygems.org/info/term-ansicolor
HTTP GET https://index.rubygems.org/info/treetop
HTTP GET https://index.rubygems.org/info/gherkin
HTTP GET https://index.rubygems.org/info/multi_json
HTTP GET https://index.rubygems.org/info/multi_test
HTTP GET https://index.rubygems.org/info/gherkin3
HTTP GET https://index.rubygems.org/info/cucumber-wire
HTTP GET https://index.rubygems.org/info/event-bus
HTTP GET https://index.rubygems.org/info/net-ssh
HTTP GET https://index.rubygems.org/info/cucumber-core
HTTP 200 OK
HTTP 200 OK
HTTP 200 OK
HTTP 200 OK
HTTP 200 OK
HTTP 200 OK
HTTP 200 OK
HTTP 200 OK
HTTP 200 OK
HTTP 200 OK
HTTP 200 OK
HTTP 200 OK
HTTP 200 OK
HTTP 200 OK
Looking up gems ["tins", "trollop", "facets", "echoe", "needle", "jruby-pageant", "bcrypt_pbkdf", "rbnacl", "rbnacl-libsodium"]
HTTP GET https://index.rubygems.org/info/tins
HTTP GET https://index.rubygems.org/info/trollop
HTTP GET https://index.rubygems.org/info/facets
HTTP GET https://index.rubygems.org/info/echoe
HTTP GET https://index.rubygems.org/info/needle
HTTP GET https://index.rubygems.org/info/jruby-pageant
HTTP GET https://index.rubygems.org/info/bcrypt_pbkdf
HTTP GET https://index.rubygems.org/info/rbnacl
HTTP GET https://index.rubygems.org/info/rbnacl-libsodium
HTTP 200 OK
HTTP 200 OK
HTTP 200 OK
HTTP 200 OK
HTTP 200 OK
HTTP 200 OK
HTTP 200 OK
HTTP 200 OK
HTTP 200 OK
Looking up gems ["curses", "highline", "allison"]
HTTP GET https://index.rubygems.org/info/curses
HTTP GET https://index.rubygems.org/info/highline
HTTP GET https://index.rubygems.org/info/allison
HTTP 200 OK
HTTP 200 OK
HTTP 200 OK
Looking up gems ["termios"]
HTTP GET https://index.rubygems.org/info/termios
HTTP 200 OK
Fetching version metadata from https://rubygems.org/
Looking up gems ["rest-client", "json", "mime-types", "rdoc", "netrc", "ffi", "http-cookie", "domain_name", "sqlite3", "hoe", "minitest", "archive-tar-minitar", "nokogiri", "rcov", "mime-types-data", "bones", "rake", "racc", "rake-compiler", "rexical", "tenderlove-frex", "weakling", "mini_portile", "mini_portile2", "pkg-config", "unf", "little-plugger", "loquacious", "bones-extras", "bones-git", "rspec", "rubyforge", "RubyInline", "gemcutter", "git", "bones-rcov", "bones-rspec", "bones-rubyforge", "bones-zentest", "json_pure", "net-scp", "ZenTest", "unf_ext", "cucumber", "diff-lcs", "spicycode-rcov", "syntax", "rspec-core", "rspec-expectations", "rspec-mocks", "spruz", "rspec-support", "builder", "polyglot", "term-ansicolor", "treetop", "gherkin", "multi_json", "multi_test", "cucumber-core", "gherkin3", "cucumber-wire", "event-bus", "net-ssh", "tins", "trollop", "facets", "echoe", "needle", "jruby-pageant", "bcrypt_pbkdf", "rbnacl", "rbnacl-libsodium", "curses", "highline", "allison", "termios"]
Could not find intercom-3.5.1 in any of the sources

Here's a zip of ~/.bundle when it's done: http://d.pr/f/16XKV

Interestingly, in the cache directory the info for intercom is missing 3.5.1, but if I manually curl it is present:


$ grep 3.5.1 .bundle/cache/compact_index/rubygems.org.443.29b0360b937aa4d161703e6160654e47/info/intercom
$ curl https://index.rubygems.org/info/intercom | grep 3.5.1
3.5.1 json:~> 1.8|checksum:62e94f23327c9e9f323243b2f2d42d845e8403e42c151bb54a6abfb162c2e87b,ruby:>= 2.1.0,rubygems:>= 0

With a second bundle run verbosely:


$ bundle install --deployment --verbose
HTTP GET https://index.rubygems.org/versions
HTTP 304 Not Modified
Fetching gem metadata from https://rubygems.org/
Looking up gems ["json", "intercom"]
HTTP GET https://index.rubygems.org/info/intercom
HTTP 206 Partial Content
Looking up gems ["rest-client"]
Looking up gems ["mime-types", "rdoc", "netrc", "ffi", "http-cookie"]
HTTP GET https://index.rubygems.org/info/mime-types
HTTP 304 Not Modified
Looking up gems ["bones", "rake", "hoe", "minitest", "domain_name", "sqlite3", "archive-tar-minitar", "nokogiri", "rcov", "mime-types-data"]
Looking up gems ["unf", "racc", "rake-compiler", "rexical", "tenderlove-frex", "weakling", "mini_portile", "mini_portile2", "pkg-config", "little-plugger", "loquacious", "bones-extras", "bones-git", "rspec", "rubyforge", "RubyInline", "gemcutter"]
HTTP GET https://index.rubygems.org/info/rake-compiler
HTTP GET https://index.rubygems.org/info/rspec
HTTP 304 Not Modified
HTTP 304 Not Modified
Looking up gems ["unf_ext", "git", "bones-rcov", "bones-rspec", "bones-rubyforge", "bones-zentest", "ZenTest", "json_pure", "net-scp", "cucumber", "diff-lcs", "spicycode-rcov", "syntax", "rspec-core", "rspec-expectations", "rspec-mocks"]
HTTP GET https://index.rubygems.org/info/rspec-mocks
HTTP GET https://index.rubygems.org/info/rspec-core
HTTP GET https://index.rubygems.org/info/rspec-expectations
HTTP GET https://index.rubygems.org/info/git
HTTP 304 Not Modified
HTTP 304 Not Modified
HTTP 304 Not Modified
HTTP 304 Not Modified
Looking up gems ["builder", "polyglot", "term-ansicolor", "treetop", "gherkin", "multi_json", "multi_test", "cucumber-core", "gherkin3", "cucumber-wire", "event-bus", "net-ssh", "spruz", "rspec-support"]
HTTP GET https://index.rubygems.org/info/treetop
HTTP 304 Not Modified
Looking up gems ["tins", "trollop", "echoe", "needle", "jruby-pageant", "bcrypt_pbkdf", "rbnacl", "rbnacl-libsodium", "facets"]
HTTP GET https://index.rubygems.org/info/rbnacl
HTTP GET https://index.rubygems.org/info/rbnacl-libsodium
HTTP 304 Not Modified
HTTP 304 Not Modified
Looking up gems ["curses", "highline", "allison"]
Looking up gems ["termios"]
Fetching version metadata from https://rubygems.org/
Looking up gems ["rest-client", "json", "mime-types", "rdoc", "netrc", "ffi", "http-cookie", "bones", "rake", "hoe", "minitest", "domain_name", "sqlite3", "archive-tar-minitar", "nokogiri", "rcov", "mime-types-data", "unf", "racc", "rake-compiler", "rexical", "tenderlove-frex", "weakling", "mini_portile", "mini_portile2", "pkg-config", "little-plugger", "loquacious", "bones-extras", "bones-git", "rspec", "rubyforge", "RubyInline", "gemcutter", "unf_ext", "git", "bones-rcov", "bones-rspec", "bones-rubyforge", "bones-zentest", "ZenTest", "json_pure", "net-scp", "cucumber", "diff-lcs", "spicycode-rcov", "syntax", "rspec-core", "rspec-expectations", "rspec-mocks", "builder", "polyglot", "term-ansicolor", "treetop", "gherkin", "multi_json", "multi_test", "cucumber-core", "gherkin3", "cucumber-wire", "event-bus", "net-ssh", "spruz", "rspec-support", "tins", "trollop", "echoe", "needle", "jruby-pageant", "bcrypt_pbkdf", "rbnacl", "rbnacl-libsodium", "facets", "curses", "highline", "allison", "termios"]
Installing json 1.8.3 with native extensions
Building native extensions.  This could take a while...
0:  json (1.8.3) from /home/deploy/test/vendor/bundle/ruby/2.2.0/specifications/json-1.8.3.gemspec
Using bundler 1.12.5
0:  bundler (1.12.5) from /var/lib/gems/2.2.0/gems/bundler-1.12.5/lib/bundler/source
Installing intercom 3.5.1
0:  intercom (3.5.1) from /home/deploy/test/vendor/bundle/ruby/2.2.0/specifications/intercom-3.5.1.gemspec
Bundle complete! 1 Gemfile dependency, 3 gems now installed.
Bundled gems are installed into ./vendor/bundle.

Now the version is present in the cache as expected:


$ grep 3.5.1 .bundle/cache/compact_index/rubygems.org.443.29b0360b937aa4d161703e6160654e47/info/intercom
3.5.1 json:~> 1.8|checksum:62e94f23327c9e9f323243b2f2d42d845e8403e42c151bb54a6abfb162c2e87b,ruby:>= 2.1.0,rubygems:>= 0

Are the caches fetched in different ways first and subsequent times? Is it possible that the CDN has a stale version cached for the exact request bundler makes the first time versus what looks like a GET with Range in subsequent runs?

该提问来源于开源项目:rubygems/bundler

  • 点赞
  • 写回答
  • 关注问题
  • 收藏
  • 复制链接分享
  • 邀请回答

5条回答

  • weixin_39937635 weixin_39937635 4月前

    Side-note: this is the second time that I have personally had problems with the new compact index in production deployment scenarios. 😢

    点赞 评论 复制链接分享
  • weixin_39937635 weixin_39937635 4月前

    tl;dr: could you bust the cache on the intercom index please?

    点赞 评论 复制链接分享
  • weixin_39980184 weixin_39980184 4月前

    I'm having the same problem with openid_connect-0.11.2.

    I haven't investigated it, I merely see Could not find openid_connect-0.11.2 in any of the sources in the logs on initial install, it works on 2nd attempt.

    If feedback is not enough I can try to provide more details too.

    点赞 评论 复制链接分享
  • weixin_39856208 weixin_39856208 4月前

    I believe this has been fixed on the server side, so I'm going to close this ticket. Let us know if you see more issues like this, though!

    点赞 评论 复制链接分享
  • weixin_39937635 weixin_39937635 4月前

    Thanks so much for your hard work !

    点赞 评论 复制链接分享

相关推荐