CI: dependencies are conflicting with poetry 1.0 vendored dependencies

[dmfigol]

It seems that in poetry 1.0, its dependencies are vendored. Also, we are not using virtualenv in the Dockerfile.

nornir/Dockerfile

Lines 19 to 24 in fda8644

	RUN poetry install --no-interaction
	COPY . .
	# Install the project as a package
	RUN poetry install --no-interaction

This leads to some conflicts and an extra dependency update (?) process when building a container:

Step 8/11 : RUN poetry install --no-interaction
 ---> Running in b626817c6265
Skipping virtualenv creation, as specified in config file.
Installing dependencies from lock file
Warning: The lock file is not up to date with the latest changes in pyproject.toml. You may be getting outdated dependencies. Run update to update them.


Package operations: 95 installs, 19 updates, 0 removals

  - Installing decorator (4.4.0)
  - Installing ipython-genutils (0.2.0)
  - Updating six (1.13.0 /root/.poetry/lib/poetry/_vendor/py3.6 -> 1.12.0)
  - Updating attrs (19.3.0 /root/.poetry/lib/poetry/_vendor/py3.6 -> 19.1.0)
  - Installing parso (0.5.1)
  - Updating ptyprocess (0.6.0 /root/.poetry/lib/poetry/_vendor/py3.6 -> 0.6.0)
  - Updating pycparser (2.19 /root/.poetry/lib/poetry/_vendor/py3.6 -> 2.19)
  - Updating pyrsistent (0.14.11 /root/.poetry/lib/poetry/_vendor/py3.6 -> 0.15.4)
  - Installing traitlets (4.3.2)
  - Installing wcwidth (0.1.7)
  - Installing asn1crypto (0.24.0)
  - Installing backcall (0.1.0)
  - Updating cffi (1.13.2 /root/.poetry/lib/poetry/_vendor/py3.6 -> 1.12.3)
  - Installing colorama (0.4.1)
  - Installing jedi (0.15.1)
  - Updating jsonschema (3.2.0 /root/.poetry/lib/poetry/_vendor/py3.6 -> 3.0.2)
  - Installing jupyter-core (4.5.0)
  - Installing markupsafe (1.1.1)
  - Updating pexpect (4.7.0 /root/.poetry/lib/poetry/_vendor/py3.6 -> 4.7.0)
  - Installing pickleshare (0.7.5)
  - Installing prompt-toolkit (2.0.9)
  - Installing pygments (2.4.2)
  - Installing python-dateutil (2.8.0)
  - Installing pyzmq (18.1.0)
  - Installing tornado (6.0.3)
  - Updating webencodings (0.5.1 /root/.poetry/lib/poetry/_vendor/py3.6 -> 0.5.1)
  - Installing bcrypt (3.1.7)
  - Installing bleach (3.1.0)
  - Updating cryptography (2.8 /root/.poetry/lib/poetry/_vendor/py3.6 -> 2.7)
  - Installing defusedxml (0.6.0)
  - Installing entrypoints (0.3)
  - Installing ipython (7.8.0)
  - Installing jinja2 (2.10.1)
  - Installing jupyter-client (5.3.3)
  - Installing mistune (0.8.4)
  - Updating more-itertools (8.0.2 /root/.poetry/lib/poetry/_vendor/py3.6 -> 7.2.0)
  - Installing nbformat (4.4.0)
  - Installing pandocfilters (1.4.2)
  - Installing pynacl (1.3.0)
  - Installing testpath (0.4.2)
  - Installing future (0.17.1)
  - Installing ipykernel (5.1.2)
  - Installing nbconvert (5.6.0)
  - Installing paramiko (2.6.0)
  - Installing prometheus-client (0.7.1)
  - Installing send2trash (1.5.0)
  - Installing terminado (0.8.2)
  - Updating zipp (0.6.0 /root/.poetry/lib/poetry/_vendor/py3.6 -> 0.6.0)
  - Updating certifi (2019.11.28 /root/.poetry/lib/poetry/_vendor/py3.6 -> 2019.9.11)
  - Updating chardet (3.0.4 /root/.poetry/lib/poetry/_vendor/py3.6 -> 3.0.4)
  - Updating idna (2.8 /root/.poetry/lib/poetry/_vendor/py3.6 -> 2.8)
  - Updating importlib-metadata (1.1.3 /root/.poetry/lib/poetry/_vendor/py3.6 -> 0.23)
  - Installing lxml (4.4.1)
  - Installing notebook (6.0.1)
  - Updating pyparsing (2.4.6 /root/.poetry/lib/poetry/_vendor/py3.6 -> 2.4.2)
  - Installing pyserial (3.4)
  - Installing pytz (2019.2)
  - Installing scp (0.13.2)
  - Installing textfsm (1.1.0)
  - Updating urllib3 (1.25.7 /root/.poetry/lib/poetry/_vendor/py3.6 -> 1.25.5)
  - Installing alabaster (0.7.12)
  - Installing atomicwrites (1.3.0)
  - Installing babel (2.7.0)
  - Installing docutils (0.15.2)
  - Installing imagesize (1.1.0)
  - Installing ncclient (0.6.6)
  - Installing netaddr (0.7.19)
  - Installing netmiko (2.4.2)
  - Installing packaging (19.2)
  - Installing pluggy (0.13.0)
  - Installing py (1.8.0)
  - Installing pyyaml (5.1.2)
  - Updating requests (2.22.0 /root/.poetry/lib/poetry/_vendor/py3.6 -> 2.22.0)
  - Installing snowballstemmer (1.9.1)
  - Installing sphinxcontrib-websupport (1.1.2)
  - Installing typing (3.7.4.1)
  - Installing widgetsnbextension (3.5.1)
  - Installing appdirs (1.4.3)
  - Installing click (7.0)
  - Installing coverage (4.5.4)
  - Installing dataclasses (0.6)
  - Installing ipywidgets (7.5.1)
  - Installing junos-eznc (2.2.1)
  - Installing jupyter-console (6.0.0)
  - Installing mccabe (0.6.1)
  - Installing mypy-extensions (0.4.1)
  - Installing nxapi-plumbing (0.5.2)
  - Installing pockets (0.7.2)
  - Installing pycodestyle (2.5.0)
  - Installing pydocstyle (4.0.1)
  - Installing pyeapi (0.8.2)
  - Installing pyflakes (2.1.1)
  - Installing pyiosxr (0.53)
  - Installing pytest (5.1.2)
  - Installing qtconsole (4.5.5)
  - Installing sphinx (1.8.5)
  - Installing toml (0.10.0)
  - Installing typed-ast (1.4.0)
  - Installing typing-extensions (3.7.4)
  - Installing black (19.3b0)
  - Installing flake8-import-order (0.18.1)
  - Installing jupyter (1.0.0)
  - Installing mypy (0.740)
  - Installing napalm (2.4.0)
  - Installing nbsphinx (0.4.2)
  - Installing nbval (0.9.2)
  - Installing pydantic (1.1)
  - Installing pylama (7.7.1)
  - Installing pytest-cov (2.7.1)
  - Installing requests-mock (1.7.0)
  - Installing ruamel.yaml (0.15.100)
  - Installing sphinx-issues (1.2.0)
  - Installing sphinx-rtd-theme (0.4.3)
  - Installing sphinxcontrib-napoleon (0.7)
Removing intermediate container b626817c6265
 ---> bef8d86d5e6b
Step 9/11 : COPY . .
 ---> 78d495161129
Step 10/11 : RUN poetry install --no-interaction
 ---> Running in 13f1f9d93236
Skipping virtualenv creation, as specified in config file.
Installing dependencies from lock file
Warning: The lock file is not up to date with the latest changes in pyproject.toml. You may be getting outdated dependencies. Run update to update them.


Package operations: 0 installs, 19 updates, 0 removals

  - Updating six (1.13.0 /root/.poetry/lib/poetry/_vendor/py3.6 -> 1.12.0)
  - Updating attrs (19.3.0 /root/.poetry/lib/poetry/_vendor/py3.6 -> 19.1.0)
  - Updating ptyprocess (0.6.0 /root/.poetry/lib/poetry/_vendor/py3.6 -> 0.6.0)
  - Updating pycparser (2.19 /root/.poetry/lib/poetry/_vendor/py3.6 -> 2.19)
  - Updating pyrsistent (0.14.11 /root/.poetry/lib/poetry/_vendor/py3.6 -> 0.15.4)
  - Updating cffi (1.13.2 /root/.poetry/lib/poetry/_vendor/py3.6 -> 1.12.3)
  - Updating jsonschema (3.2.0 /root/.poetry/lib/poetry/_vendor/py3.6 -> 3.0.2)
  - Updating pexpect (4.7.0 /root/.poetry/lib/poetry/_vendor/py3.6 -> 4.7.0)
  - Updating webencodings (0.5.1 /root/.poetry/lib/poetry/_vendor/py3.6 -> 0.5.1)
  - Updating cryptography (2.8 /root/.poetry/lib/poetry/_vendor/py3.6 -> 2.7)
  - Updating more-itertools (8.0.2 /root/.poetry/lib/poetry/_vendor/py3.6 -> 7.2.0)
  - Updating zipp (0.6.0 /root/.poetry/lib/poetry/_vendor/py3.6 -> 0.6.0)
  - Updating certifi (2019.11.28 /root/.poetry/lib/poetry/_vendor/py3.6 -> 2019.9.11)
  - Updating chardet (3.0.4 /root/.poetry/lib/poetry/_vendor/py3.6 -> 3.0.4)
  - Updating idna (2.8 /root/.poetry/lib/poetry/_vendor/py3.6 -> 2.8)
  - Updating importlib-metadata (1.1.3 /root/.poetry/lib/poetry/_vendor/py3.6 -> 0.23)
  - Updating pyparsing (2.4.6 /root/.poetry/lib/poetry/_vendor/py3.6 -> 2.4.2)
  - Updating urllib3 (1.25.7 /root/.poetry/lib/poetry/_vendor/py3.6 -> 1.25.5)
  - Updating requests (2.22.0 /root/.poetry/lib/poetry/_vendor/py3.6 -> 2.22.0)
  - Installing nornir (2.3.0)

The easiest solution for this is to use virtual environment in the docker container

[dbarrosop]

Not sure I understand the issue here. To the best of my knowledge, poetry's dependencies has always been vendored and, in any case, if they are vendored that shouldn't require a venv as the whole point of vendoring is to avoid conflict with the system.

The reason why there are two installs have to do with the fact that poetry is extremely slow installing dependencies so we try to cache a layer with the dependencies already installed.

[dmfigol]

I am not sure why poetry vendoring comes into play here, but it is seen in the logs that it is happening when it shouldn't.
I understand why we have two poetry install. The second install is expected to look like this:

Step 10/11 : RUN poetry install --no-interaction
 ---> Running in 13f1f9d93236
Skipping virtualenv creation, as specified in config file.
Installing dependencies from lock file

Package operations: 0 installs, 1 updates, 0 removals
  - Installing nornir (2.3.0)

but it isn't

[dbarrosop]

IMHO a container shouldn't require a venv. I think we should wait for python-poetry/poetry#1879 because I tried several combinations of dockerfiles and none yielded the expected result, maybe someone will document soon what's the best approach so we can re from it.

If there is no better solution proposed in a timely manner we can go with this but unless this is a major issue I'd prefer to hold on for a while.

[dmfigol]

@dbarrosop I have been using virtual environments in docker containers with Python apps almost exclusively for the last year and my experience has been great. Though I had different reasoning. I take advantage of multi-stage builds and with virtual environments, I can copy the whole virtual environment between stages.
I am ok if you prefer to wait on proposed changes in the PR.
I've also opened python-poetry/poetry#1883