"Retrying dependency api due to error" issue

[tmharber]

Hi all. I'm currently trying to do a bundle install for a particular project, but keep getting the error:

Retrying dependency api due to error (2/4): Bundler::HTTPError Network error while fetching
https://bundler.rubygems.org/api/v1/dependencies?gems=faraday%2Cactivesupport%2Crack%2Coogway_profile%2Cslim%2Cbskyb-global-clients%2Cmysky_logging%2Cslogger%2Cjson_pure%2Crubyzip%2Crest-client%2Cterminal-table%2Cinteract%2Caddressable%2Cuuidtools%2Crb-readline%2Ccaldecott-client%2Cescape_utils%2Coj%2Cjson%2Csinatra%2Csinatra-contrib%2Csprockets%2Csprockets-helpers%2Cbskyb-rack-middleware%2Ccustomer_overview%2Cether-config-mysky%2Crack-fiber_pool%2Ceventmachine%2Cem-synchrony%2Cem-http-request%2Credis%2Chiredis%2Credis-store%2Credis-rack%2Cnewrelic_rpm%2Cnewrelic-faraday%2Cthin%2Cnimbus%2Cvmc%2Chashie%2Cpage_magic%2Cpoltergeist%2Cchildprocess%2Cwait%2Cmirage%2Cnokogiri%2Ccapybara%2Csinatra-websocket%2Cwaitforit

However, if I try following that URL, it takes a few seconds then downloads a 'dependencies' file, so clearly I'm able to hit it. It's also not a transient network error as I did a bundle install on the particular project (A) and bundle install on another, unrelated project (B) at the same time - project A's bundle install failed but B's bundle install worked fine and didn't get these errors.

One thing that works is if I use the suffix --full-index - in that case, it works perfectly, although it would be nice to know why it's failing without. Is it timing out when finding dependencies because there are too many gems possibly?

I've tried running it on a different PC (two dev boxes and a deployment box) and get the same issue. Any ideas for what could be causing this? Using ruby-2.1.1 and bundler 1.11.2.

[segiddins]

Can you share the output of bundle env so we can try and reproduce the issue?

[tmharber]

Warning: this Gemfile contains multiple primary sources. Using `source` more than once without a block is a security risk, and may result in installing unexpected gems. To resolve this warning, use a block to indicate which gems should come from the secondary source. To upgrade this warning to an error, run `bundle config disable_multisource true`.
Your Gemfile lists the gem httparty (>= 0) more than once.
You should probably keep only one of them.
While it's not a problem now, it could cause errors if you change the version of just one of them later.
Environment

    Bundler   1.11.2
    Rubygems  2.2.2
    Ruby      2.1.1p76 (2014-02-24 revision 45161) [x86_64-darwin12.0]
    GEM_HOME  /Users/tha28/.rvm/gems/ruby-2.1.1@my-photos
    GEM_PATH  /Users/tha28/.rvm/gems/ruby-2.1.1@my-photos:/Users/tha28/.rvm/gems/ruby-2.1.1@global
    RVM       1.26.11 (latest)
    Git       2.5.4 (Apple Git-61)

Bundler settings

    gem.coc
      Set for the current user (/Users/tha28/.bundle/config): "false"
    gem.mit
      Set for the current user (/Users/tha28/.bundle/config): "false"
    gem.test
      Set for the current user (/Users/tha28/.bundle/config): "false"

Gemfile

    source 'https://rubygems.org'
    source '---'

    gem 'sinatra'
    gem 'sass', require: 'sass/plugin/rack'
    gem 'ether', '~> 1.1.1', require: 'mysky/platform'
    gem 'sprockets-sass'
    gem 'pry'
    gem 'blood-pressure', '0.0.1'
    gem 'zip', '~> 2.0.2'
    gem 'httparty'

    group :development do
      gem 'rake'
      gem 'nimbus-deployer'
      gem 'parallel_tests'
      gem 'foreman'
    end

    group :test do
      gem 'rspec', '2.99.0'
      gem 'bskyb-sinatra-platform', '~> 8.0', require: %w(ext/string mysky/global/tasks)
      gem 'cucumber'
      gem 'selenium-webdriver'
      gem 'poltergeist', '~> 1.6.0'
      gem 'jasmine'
      gem 'jshintrb'
      gem 'execjs'
      gem 'therubyracer'
      gem 'webmockets'
      gem 'httparty'
    end

Gemfile.lock

    GEM
      remote: http://rubygems.org/
      remote: ---/
      specs:
        activesupport (4.2.3)
          i18n (~> 0.7)
          json (~> 1.7, >= 1.7.7)
          minitest (~> 5.1)
          thread_safe (~> 0.3, >= 0.3.4)
          tzinfo (~> 1.1)
        addressable (2.3.8)
        backports (3.6.7)
        blood-pressure (0.0.1)
        bskyb-global-clients (0.0.4)
          activesupport
          faraday
        bskyb-rack-middleware (1.1.0)
          bskyb-global-clients
          oogway_profile
          rack
          slim
        bskyb-sinatra-platform (8.0.2)
          activesupport (~> 4.0)
          capybara (~> 2.2)
          childprocess (~> 0.5)
          hashie (~> 2.0)
          json (~> 1.8)
          mirage (~> 3.0)
          nokogiri (~> 1.6)
          poltergeist (~> 1.5)
          wait (~> 0.5)
        builder (3.2.2)
        capybara (2.5.0)
          mime-types (>= 1.16)
          nokogiri (>= 1.3.3)
          rack (>= 1.0.0)
          rack-test (>= 0.5.4)
          xpath (~> 2.0)
        childprocess (0.5.9)
          ffi (~> 1.0, >= 1.0.11)
        cliver (0.3.2)
        coderay (1.1.0)
        cookiejar (0.3.2)
        cucumber (1.3.17)
          builder (>= 2.1.2)
          diff-lcs (>= 1.1.3)
          gherkin (~> 2.12)
          multi_json (>= 1.7.5, < 2.0)
          multi_test (>= 0.1.1)
        customer_overview (0.0.24)
          faraday (= 0.8.9)
          mysky_logging
        daemons (1.2.3)
        diff-lcs (1.2.5)
        em-http-request (1.1.2)
          addressable (>= 2.3.4)
          cookiejar
          em-socksify (>= 0.3)
          eventmachine (>= 1.0.3)
          http_parser.rb (>= 0.6.0)
        em-socksify (0.3.1)
          eventmachine (>= 1.0.0.beta.4)
        em-synchrony (1.0.4)
          eventmachine (>= 1.0.0.beta.1)
        em-websocket (0.3.8)
          addressable (>= 2.1.1)
          eventmachine (>= 0.12.9)
        ether (1.1.1)
          activesupport
          bskyb-rack-middleware (~> 1.1)
          customer_overview (~> 0.0.23)
          em-http-request (~> 1.0)
          em-synchrony (~> 1.0)
          ether-config-mysky (~> 1.1)
          eventmachine (>= 1.0.7)
          faraday (= 0.8.9)
          hiredis (~> 0.4.5)
          newrelic-faraday (~> 0.6)
          newrelic_rpm (~> 3.7)
          rack-fiber_pool (~> 0.9)
          redis (~> 3.0.1)
          redis-rack
          redis-store
          sinatra (~> 1.4)
          sinatra-contrib (~> 1.4)
          slim (~> 2.0)
          sprockets (~> 2.11)
          sprockets-helpers (~> 1.1)
        ether-config-mysky (1.1.0)
        eventmachine (1.0.8)
        execjs (2.5.2)
        faraday (0.8.9)
          multipart-post (~> 1.2.0)
        ffi (1.9.10)
        foreman (0.78.0)
          thor (~> 0.19.1)
        gherkin (2.12.2)
          multi_json (~> 1.3)
        haml (4.0.7)
          tilt
        hashie (2.1.2)
        hike (1.2.3)
        hiredis (0.4.5)
        http_parser.rb (0.6.0)
        httparty (0.13.7)
          json (~> 1.8)
          multi_xml (>= 0.5.2)
        i18n (0.7.0)
        interact (0.2)
        jasmine (2.3.1)
          jasmine-core (~> 2.3)
          phantomjs
          rack (>= 1.2.1)
          rake
        jasmine-core (2.3.4)
        jshintrb (0.3.0)
          execjs
          multi_json (>= 1.3)
          rake
        json (1.8.3)
        json_pure (1.5.5)
          spruz (~> 0.2.8)
        libv8 (3.16.14.7)
        method_source (0.8.2)
        mime-types (1.25.1)
        mini_portile (0.6.2)
        minitest (5.7.0)
        mirage (3.0.10)
          childprocess
          haml
          hashie
          httparty
          ptools
          sinatra
          thor
          waitforit
        multi_json (1.11.2)
        multi_test (0.1.1)
        multi_xml (0.5.5)
        multipart-post (1.2.0)
        mysky_logging (0.2.0)
          faraday
          slogger
        newrelic-faraday (0.7.0)
          faraday (>= 0.6, < 0.10)
          newrelic_rpm (~> 3.0)
        newrelic_rpm (3.14.0.305)
        nimbus-deployer (0.1.4)
          activesupport (~> 4.0)
          vmc (= 0.3.13)
        nokogiri (1.6.6.2)
          mini_portile (~> 0.6.0)
        oogway_profile (1.4.0)
          faraday
          json
          mysky_logging (>= 0.0.6)
        parallel (1.3.3)
        parallel_tests (1.0.7)
          parallel
        phantomjs (1.9.8.0)
        poltergeist (1.6.0)
          capybara (~> 2.1)
          cliver (~> 0.3.1)
          multi_json (~> 1.0)
          websocket-driver (>= 0.2.0)
        pry (0.10.1)
          coderay (~> 1.1.0)
          method_source (~> 0.8.1)
          slop (~> 3.4)
        ptools (1.3.3)
        rack (1.6.4)
        rack-fiber_pool (0.9.3)
        rack-protection (1.5.3)
          rack
        rack-test (0.6.3)
          rack (>= 1.0)
        rake (10.4.2)
        rdoc (4.1.2)
          json (~> 1.4)
        redis (3.0.7)
        redis-rack (1.5.0)
          rack (~> 1.5)
          redis-store (~> 1.1.0)
        redis-store (1.1.7)
          redis (>= 2.2)
        ref (1.0.5)
        rest-client (1.6.8)
          mime-types (~> 1.16)
          rdoc (>= 2.4.2)
        rspec (2.99.0)
          rspec-core (~> 2.99.0)
          rspec-expectations (~> 2.99.0)
          rspec-mocks (~> 2.99.0)
        rspec-core (2.99.2)
        rspec-expectations (2.99.2)
          diff-lcs (>= 1.1.3, < 2.0)
        rspec-mocks (2.99.2)
        rubyzip (1.1.7)
        rubyzip2 (2.0.2)
        sass (3.4.6)
        selenium-webdriver (2.46.2)
          childprocess (~> 0.5)
          multi_json (~> 1.0)
          rubyzip (~> 1.0)
          websocket (~> 1.0)
        sinatra (1.4.6)
          rack (~> 1.4)
          rack-protection (~> 1.4)
          tilt (>= 1.3, < 3)
        sinatra-contrib (1.4.6)
          backports (>= 2.0)
          multi_json
          rack-protection
          rack-test
          sinatra (~> 1.4.0)
          tilt (>= 1.3, < 3)
        sinatra-websocket (0.3.0)
          em-websocket (~> 0.3.6)
          eventmachine
          thin (>= 1.3.1)
        slim (2.1.0)
          temple (~> 0.6.9)
          tilt (>= 1.3.3, < 2.1)
        slogger (0.0.11)
        slop (3.6.0)
        sprockets (2.12.4)
          hike (~> 1.2)
          multi_json (~> 1.0)
          rack (~> 1.0)
          tilt (~> 1.1, != 1.3.0)
        sprockets-helpers (1.2.1)
          sprockets (>= 2.2)
        sprockets-sass (1.2.0)
          sprockets (~> 2.0)
          tilt (~> 1.1)
        spruz (0.2.13)
        temple (0.6.10)
        terminal-table (1.4.5)
        therubyracer (0.12.1)
          libv8 (~> 3.16.14.0)
          ref
        thin (1.6.3)
          daemons (~> 1.0, >= 1.0.9)
          eventmachine (~> 1.0)
          rack (~> 1.0)
        thor (0.19.1)
        thread_safe (0.3.5)
        tilt (1.4.1)
        tzinfo (1.2.2)
          thread_safe (~> 0.1)
        vmc (0.3.13)
          interact (~> 0.2.0)
          json_pure (~> 1.5.1)
          rest-client (>= 1.6.1, < 1.7.0)
          rubyzip2 (~> 2.0.1)
          terminal-table (~> 1.4.2)
        wait (0.5.2)
        waitforit (0.0.1)
        webmockets (0.0.7)
          childprocess
          httparty
          json
          ptools
          sinatra
          sinatra-websocket (= 0.3.0)
          slim
          thor
          waitforit
        websocket (1.2.2)
        websocket-driver (0.6.3)
          websocket-extensions (>= 0.1.0)
        websocket-extensions (0.1.2)
        xpath (2.0.0)
          nokogiri (~> 1.3)
        zip (2.0.2)

    PLATFORMS
      ruby

    DEPENDENCIES
      blood-pressure (= 0.0.1)
      bskyb-sinatra-platform (~> 8.0)
      cucumber
      ether (~> 1.1.1)
      execjs
      foreman
      httparty
      jasmine
      jshintrb
      nimbus-deployer
      parallel_tests
      poltergeist (~> 1.6.0)
      pry
      rake
      rspec (= 2.99.0)
      sass
      selenium-webdriver
      sinatra
      sprockets-sass
      therubyracer
      webmockets
      zip (~> 2.0.2)

    BUNDLED WITH
       1.11.2

[RochesterinNYC]

Out of curiosity, what does this line do in Gemfiles?

source '---'

[tmharber]

haha sorry, it's an internal gem server - just cut out the url

[tmharber]

Any updates? Still getting this for some projects continually, but some seem to be exempt from this.

[indirect]

@segiddins is just me, or is our Postgres instance getting slower and slower? :/ it looks like we need to raise the statement timeout again

[jamesRaybould]

Don't suppose anyone has had a chance to look at this? It is causing a few issues now when we try and run bundler package and the --full-index workaround won't work on that package

[indirect]

@jamesRaybould I think you can run bundle install --full-index --freeze, then bundle cache, then bundle install --no-freeze. Maybe. :P

[jamesRaybould]

Thanks for getting back to me, unfortunately running:

bundle install --full-index --frozen
bundle cache
bundle install --no-frozen

Still doesn't work and still errors out on the cache stage:

Updating files in vendor/cache
 * rake-10.3.2.gem
 * i18n-0.7.0.gem
Fetching gem metadata from http://gem-server.internal.server/....
Fetching gem metadata from https://rubygems.org/.Retrying dependency api due to error (2/4): Bundler::HTTPError Network error while fetching https://bundler.rubygems.org/api/v1/...

[tmharber]

Again, seems to be an issue only when we're using the vmc gem. We've raised the issue with VMC here, but they haven't responded yet. The issue is still happening though.

[TheNotary]

I hit this error just now, it has something to do with my gem server geminabox. Disabling one of its configuration lines fixed it, though is cumbersome due to having to manually push things. maybe we've miconfigured geminabox somehow?

Note the freshly commented out line in my config.ru file:

require "rubygems"
require "geminabox"

Geminabox.data = "/media/john/lijfdk/storage/geminabox_store/data"

# Geminabox.rubygems_proxy = true

run Geminabox::Server

I wonder if manually pushing vmc to your internal gem server will resolve this (or lead to an error on the next gem down that isn't already cached on your private gem server).

[RochesterinNYC]

Are there any action items here? It seems VMC is no longer maintained.

Additionally, is it reproducible/still an issue with the latest version of bundler? (v1.12.4)

[TheNotary]

I would chalk this up to a problem with geminabox, close the ticket and see if @tmharber comes back with fresh information.