Pip does not reinstall with latest setuptools

[ianw]

• Pip version: 8.1.2
• Python version: 2.7
• Operating System:

Description:

DevStack (OpenStack) has a "libs from git" option where we can choose to install the latest client libraries from a git checkout, rather than using the dependencies from PyPi. This is intended to let us do CI testing with the latest versions of these clients if desired.

Thus what ends up happening is that the clients are installed by pip from pypi during the standard run, and then later we re-install them from a git checkout.

So in the below, I setup a new virtualenv, "pip install python-keystoneclient", then go to a checkout of python-keystone client and "pip install -e" it ... at this point you can see pip thinks that it is installing "3.2.1.dev19" ... but when it is finished, "pip freeze" still reports the old client from PyPi installed.

$ . ./new-setuptools/bin/activate
(new-setuptools)$ pip list
pip (8.1.2)
setuptools (25.0.0)
wheel (0.29.0)
(new-setuptools)$ pip install python-keystoneclient
Collecting python-keystoneclient
  Using cached python_keystoneclient-3.2.0-py2.py3-none-any.whl
... (blah) ...

Installing collected packages: requests, rfc3986, pbr,  ...

(new-setuptools)$ pip freeze
...
pyparsing==2.1.5
python-keystoneclient==3.2.0

(new-setuptools)$ cd ~/programs/python-keystoneclient/

(new-setuptools)$ pip install --upgrade -e .
Obtaining file:///home/iwienand/programs/python-keystoneclient
Requirement already up-to-date: pbr>=1.6 in /tmp/new-setuptools/lib/python2.7/site-packages (from python-keystoneclient==3.2.1.dev19)
Requirement already up-to-date: pyparsing>=2.0.1 in /tmp/new-setuptools/lib/python2.7/site-packages (from oslo.utils>=3.15.0->python-keystoneclient==3.2.1.dev19)
Requirement already up-to-date: monotonic>=0.6 in /tmp/new-setuptools/lib/python2.7/site-packages (from oslo.utils>=3.15.0->python-keystoneclient==3.2.1.dev19)
...
Installing collected packages: python-keystoneclient
  Running setup.py develop for python-keystoneclient
Successfully installed python-keystoneclient-3.2.0

(new-setuptools)$ pip freeze
...
python-keystoneclient==3.2.0
...

I'll avoid it in here for brevity, but you can see in http://paste.openstack.org/show/541509/ where I do the same thing but downgrade setuptools to <25 before, and we get the library installed from the checkout after the "pip -e" install

The suspicious setuptools change is pypa/setuptools@e211f45 but this is talking about easy-install.pth which I'm not sure is related? Investigations continuing...

[ianw]

Let me just add a new comment, after investigation. Maybe this is obvious to those who follow this closely, but for others ...

The issue is pypa/setuptools@e211f45

When you install with "pip install -e" (i.e. "python setup.py develop") a /usr/lib/python2.7/site-packages/easy-install.pth file is generated. Before setuptools 25.0 this would do some magic and insert the new directory first in sys.path [1]. i.e., a package installed with "pip install -e" would be picked up before any package installed via regular pip, or system-packages (i.e. the "rewite" method).

After [1], the directory is simply listed in easy-install.pth (i.e. the "raw" method) which means that it just gets inserted at the back of sys.path, and the installed package overwrites it.

In the original pull request [2] it says

Since nobody should be using easy_install (or even python setup.py install, as I understand it), I think the only entries in easy-install.pth should be the result of pip install -e. It seems that the only difference in behavior between pip install and pip install -e should be that they are 'symlinked' (via .pth), and not a wildly different import priority.

Pip doesn't really say anthing about "-e" other than it's "setuptools develop mode". I don't think it explicitly states this in the documentation for development mode [3] but it seems very odd that if I'm deploying a later version from my checkout tree, with --upgrade even, that I will still by default end up importing the old packaged version after this command.

I feel like the usual workflow is something like "pip install -e abc (drags in xyz) ... hack abc ... oops this is a bug in xyz ... pull the xyz git tree ... "pip install -e ." ... hack bug in xyz ... rerun abc tests"

Now you have to take the rather unintuitive step of removing "xyz" before installing it. And if you don't, nothing will warn you or alert you that, actually, you're still using the packaged version of "xyz". You will probably figure this out if you're directly hacking on xyz as in this scenario and your changes don't apply...

Silence is frustrating for interactive use, but very bad for CI. For example, luckily we check in devstack that we are actually using the installs from git we think we are, because we've had problems in this area before -- that flagged this issue. If we didn't, then all CI would still be passing and everyone thinks they're testing their clients from the git tree, but really they're just checking against the latest release because it was installed first! It's not until maybe months later when you cut your new release you realise the "pip install -e" wasn't doing anything all along.

POLS would suggest that pip should probably uninstall the previous package, or otherwise take on the standard upgrade action that removes the old version before doing the "setup develop" so that only the development mode install is available.

[1] https://bitbucket.org/minrk/setuptools/commits/ff7ed172b054c0e598a3aaa77c6b9f1b68a7f158?at=default
[2] https://bitbucket.org/pypa/setuptools/pull-requests/135/remove-import-sys-from-easy-installpth/diff
[3] http://setuptools.readthedocs.io/en/latest/setuptools.html#development-mode

[jaraco]

Thanks @ianw for the detailed research on this issue. Note that there is a short-term workaround to set SETUPTOOLS_SYS_PATH_TECHNIQUE=rewrite and it will behave the way it always has... but the intention is to deprecate that behavior, once issues like this one have a clear solution. Setuptools will also consider reverting this new behavior and simply using rewrite in all scenarios, but only if that's what the PyPA decides is best.

[dstufft]

Presumably pip could uninstall what is already there when doing an editable install too.

[piotr-dobrogost]

Compare with issue #1548 and pull #1552. Also issue #1662.

[xavfernandez]

Should be fixed by #3898