Bosh compilation of curator package should not need internet access

[fbonelle]

When deploying version 23.0.0, we get an error while bosh is compiling curator package.

It appears that to build curator job, some download is required. But in case of corporate network, it should not be necessary to download on internet packages.

Trace:
Error 450001: Action Failed get_task: Task 08485bbc-4059-4961-6194-f39f2701b30e result: Compiling package curator: Running packaging script: Command exited with 1; Stdout: Installing collected packages: urllib3
  Running setup.py install for urllib3
Successfully installed urllib3
Processing ./curator/click-3.3.tar.gz
Installing collected packages: click
  Running setup.py install for click
Successfully installed click
Processing ./curator/elasticsearch-py.tar.gz
Collecting urllib3<2.0,>=1.8 (from elasticsearch==1.6.0)
  Retrying (Retry(total=4, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', gaierror(-2, 'Name or service not known'))': /simple/urllib3/
  Retrying (Retry(total=3, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', gaierror(-2, 'Name or service not known'))': /simple/urllib3/
  Retrying (Retry(total=2, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', gaierror(-2, 'Name or service not known'))': /simple/urllib3/
  Retrying (Retry(total=1, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', gaierror(-2, 'Name or service not known'))': /simple/urllib3/
  Retrying (Retry(total=0, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', gaierror(-2, 'Name or service not known'))': /simple/urllib3/
  Retrying (Retry(total=4, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', gaierror(-2, 'Name or service not known'))': /simple/urllib3/
  Retrying (Retry(total=3, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', gaierror(-2, 'Name or service not known'))': /simple/urllib3/
  Retrying (Retry(total=2, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', gaierror(-2, 'Name or service not known'))': /simple/urllib3/
  Retrying (Retry(total=1, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', gaierror(-2, 'Name or service not known'))': /simple/urllib3/
  Retrying (Retry(total=0, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', gaierror(-2, 'Name or service not known'))': /simple/urllib3/
  Could not find any downloads that satisfy the requirement urllib3<2.0,>=1.8 (from elasticsearch==1.6.0)
  No distributions at all found for urllib3<2.0,>=1.8 (from elasticsearch==1.6.0)
, Stderr: + set -u
+ export PATH=/var/vcap/packages/python3/bin:/var/vcap/bosh/bin:/usr/local/bin:/usr/local/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/X11R6/bin
+ PATH=/var/vcap/packages/python3/bin:/var/vcap/bosh/bin:/usr/local/bin:/usr/local/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/X11R6/bin
+ pip3 install --install-option=--prefix=/var/vcap/packages/curator curator/urllib3-1.8.3.tar.gz
+ pip3 install --install-option=--prefix=/var/vcap/packages/curator curator/click-3.3.tar.gz
+ pip3 install --install-option=--prefix=/var/vcap/packages/curator curator/elasticsearch-py.tar.gz

[dpb587]

Hrm, agreed; it should not require internet access.

[mrdavidlaing]

I've received similar bug reports from other users.

@bonzofenix; do you have any idea why this is happening. I know you guys deploy to an environment with no 'net access; so I'm surprised you haven't bumped into this.

I'm also confused as to why the error is happening - the install runs:

pip3 install --install-option=--prefix=/var/vcap/packages/curator curator/urllib3-1.8.3.tar.gz
--> Successfully installed urllib3

followed by

Processing ./curator/elasticsearch-py.tar.gz
Collecting urllib3<2.0,>=1.8 (from elasticsearch==1.6.0)

Surely urllib3-1.8.3 is >= 1.8 and < 2.0 ?

[bonzofenix]

@mrdavidlaing sorry for the late response. the version is correct:

@fbonelle which stemcell are you using? We will try to reproduce it.

@bonzofenix && @grayisgreat

[fbonelle]

@bonzofenix Now, we are on stemcell 3074, 3093 & 3098

[bonzofenix]

on which infrastructure?

On Oct 12, 2015, at 13:34, François Bonelle [email protected] wrote:

@bonzofenix https://github.com/bonzofenix Now, we are on stemcell 3074, 3093 & 3098

—
Reply to this email directly or view it on GitHub #177 (comment).

[fbonelle]

@bonzofenix It's on VCloud

[mrdavidlaing]

I was able to reproduce this on AWS:

+-----------------------------------------+---------------+---------+--------------------+
| Name                                    | OS            | Version | CID                |
+-----------------------------------------+---------------+---------+--------------------+
| bosh-aws-xen-hvm-ubuntu-trusty-go_agent | ubuntu-trusty | 3094*   | ami-03370074 light |
+-----------------------------------------+---------------+---------+--------------------+

when I disabled internet access for the curator compilation vm

[mrdavidlaing]

@fbonelle, @bonzofenix - I think 4ddbe43 addresses this problem - by explicitly telling pip not to use the pypi, and which folder to search for dependencies in.

Please could you give this a test in your environments.

PS - you might want to cherry-pick this change onto https://github.com/logsearch/logsearch-boshrelease/tree/v23.0.0, since there are a fair amount of unrelated changes on develop