Failed install with outdated pipenv version

[nikolaik]

I think this build fails because of a bug fixed in a more recent pipenv version (latest is at 2020.6.2 at the moment)

See attached build log below:

-----> Found python-3.7.3, removing
cp: cannot stat '/tmp/build_d0defce75280ad585c4ef561790992d0/requirements.txt': No such file or directory
-----> Installing python-3.8.3
-----> Installing pip
-----> Installing dependencies with Pipenv 2018.5.18…
       Installing dependencies from Pipfile.lock (ddf461)…
       An error occurred while installing pycryptodome==3.9.*! Will try again.
       Installing initially–failed dependencies…
       Collecting pycryptodome==3.9.* 
         Using cached https://files.pythonhosted.org/packages/26/29/73158aec5b839234920c934ca6b81af260eaedd1f93ce341e05fa32c89b4/pycryptodome-3.9.8-cp38-cp38-manylinux1_x86_64.whl
       
       THESE PACKAGES DO NOT MATCH THE HASHES FROM Pipfile.lock!. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
           pycryptodome==3.9.* from https://files.pythonhosted.org/packages/26/29/73158aec5b839234920c934ca6b81af260eaedd1f93ce341e05fa32c89b4/pycryptodome-3.9.8-cp38-cp38-manylinux1_x86_64.whl#sha256=dd302b6ae3965afeb5ef1b0d92486f986c0e65183cd7835973f0b593800590e6 (from -r /tmp/pipenv-f08ruld6-requirements/pipenv-d2z_bfac-requirement.txt (line 1)):
               Expected sha256 07024fc364869eae8d6ac0d316e089956e6aeffe42dbdcf44fe1320d96becf7f
               Expected     or 09b6d6bcc01a4eb1a2b4deeff5aa602a108ec5aed8ac75ae554f97d1d7f0a5ad
               Expected     or 0e10f352ccbbcb5bb2dc4ecaf106564e65702a717d72ab260f9ac4c19753cfc2
               Expected     or 1f4752186298caf2e9ff5354f2e694d607ca7342aa313a62005235d46e28cf04
               Expected     or 2fbc472e0b567318fe2052281d5a8c0ae70099b446679815f655e9fbc18c3a65
               Expected     or 3ec3dc2f80f71fd0c955ce48b81bfaf8914c6f63a41a738f28885a1c4892968a
               Expected     or 426c188c83c10df71f053e04b4003b1437bae5cb37606440e498b00f160d71d0
               Expected     or 626c0a1d4d83ec6303f970a17158114f75c3ba1736f7f2983f7b40a265861bd8
               Expected     or 767ad0fb5d23efc36a4d5c2fc608ac603f3de028909bcf59abc943e0d0bc5a36
               Expected     or 7ac729d9091ed5478af2b4a4f44f5335a98febbc008af619e4569a59fe503e40
               Expected     or 83295a3fb5cf50c48631eb5b440cb5e9832d8c14d81d1d45f4497b67a9987de8
               Expected     or 8be56bde3312e022d9d1d6afa124556460ad5c844c2fc63642f6af723c098d35
               Expected     or 8f06556a8f7ea7b1e42eff39726bb0dca1c251205debae64e6eebea3cd7b438a
               Expected     or 9230fcb5d948c3fb40049bace4d33c5d254f8232c2c0bba05d2570aea3ba4520
               Expected     or 9378c309aec1f8cd8bad361ed0816a440151b97a2a3f6ffdaba1d1a1fb76873a
               Expected     or 9977086e0f93adb326379897437373871b80501e1d176fec63c7f46fb300c862
               Expected     or 9a94fca11fdc161460bd8659c15b6adef45c1b20da86402256eaf3addfaab324
               Expected     or 9c739b7795ccf2ef1fdad8d44e539a39ad300ee6786e804ea7f0c6a786eb5343
               Expected     or b1e332587b3b195542e77681389c296e1837ca01240399d88803a075447d3557
               Expected     or c109a26a21f21f695d369ff9b87f5d43e0d6c768d8384e10bc74142bed2e092e
               Expected     or c818dc1f3eace93ee50c2b6b5c2becf7c418fa5dd1ba6fc0ef7db279ea21d5e4
               Expected     or cff31f5a8977534f255f729d5d2467526f2b10563a30bbdade92223e0bf264bd
               Expected     or d4f94368ce2d65873a87ad867eb3bf63f4ba81eb97a9ee66d38c2b71ce5a7439
               Expected     or d61b012baa8c2b659e9890011358455c0019a4108536b811602d2f638c40802a
               Expected     or d6e1bc5c94873bec742afe2dfadce0d20445b18e75c47afc0c115b19e5dd38dd
               Expected     or ea83bcd9d6c03248ebd46e71ac313858e0afd5aa2fa81478c0e653242f3eb476
               Expected     or ed5761b37615a1f222c5345bbf45272ae2cf8c7dff88a4f53a1e9f977cbb6d95
               Expected     or f011cd0062e54658b7086a76f8cf0f4222812acc66e219e196ea2d0a8849d0ed
               Expected     or f1add21b6d179179b3c177c33d18a2186a09cc0d3af41ff5ed3f377360b869f2
               Expected     or f655addaaaa9974108d4808f4150652589cada96074c87115c52e575bfcd87d5
                    Got        dd302b6ae3965afeb5ef1b0d92486f986c0e65183cd7835973f0b593800590e6
       
       You are using pip version 9.0.2, however version 20.1.1 is available.
       You should consider upgrading via the 'pip install --upgrade pip' command.

[ipmb]

This was the response last time upgrading pipenv came up: #786 (comment)

[jh0ker]

Hey @nikolaik, have you found a solution to this? I am having a similar problem.

[nikolaik]

(...) have you found a solution to this? I am having a similar problem.

Ended up downgrading pipenv to the same version used by the heroku python buildpack and relocking deps, ie. something like:

pip install pipenv==2018.05.18
rm Pipfile.lock # possibly
pipenv lock

You might be able to skip downgrading pipenv (which can reintroduce other buggy behaviour, like with editable git deps or similar)

[jh0ker]

Thanks for your answer! It's not really an ideal solution, i was hoping i could somehow force installing an up-to-date version of pipenv, but this at least works.

[edmorley]

Hi! On my list is to update the buildpack's pipenv to a newer version. I know in the past this was deferred due to it being a breaking change (due to changes in the lockfile format or similar), and then cancelled due to the future of pipenv looking more bleak.

Since then activity in the pipenv project has picked up, and I've taken over as Python owner, so I'm happy to re-evaluate.

Before I can perform the upgrade I'd need to:

• work out in what way the pipenv upgrade is a breaking change for people with an existing lockfile (also does this affect all lockfiles, or only some of them?)
• figure out how the buildpack can detect old vs new style lock files to avoid it being a breaking change (eg by installing old pipenv for old style lockfiles), or else at least output sensible warning/error messages
• decide rollout/documentation/changelog/... strategy

I don't use pipenv personally, so would need to do some experimentation to answer the above (sadly not much was documented in previous issues in this repo) - but if anyone is able to help give more context it would definitely speed things up / mean I can justify bumping this up the priority queue :-)

[jh0ker]

@edmorley I'm glad this is something you are considering, it would indeed be great to be able to use one of the recent releases of pipenv.

To your checklist: Interestingly enough, there is a field in the Pipfile.lock called pipfile-spec which did not seem to change after downgrading. I would think that means it that at least a lockfile created with 2018.05.18 would be upwards compatible to 2020.8.13 (what i was using before).

I might do some investigating on my own if i can find the time. If i find anything interesting, should i post it here?

[edmorley]

@jh0ker Yeah if you or anyone else finds out more, here is a great place to put it :-)

[edmorley]

Hi! Has anyone been able to find answers to the above, which might speed up this work? :-)

[ipmb]

I just put together a little script to test this and I'm not seeing any differences:

$ diff pipenv/2018.11.14/Pipfile pipenv/2018.05.18/Pipfile && echo "no difference"
2d1
< name = "pypi"
5,6c4
<
< [dev-packages]
---
> name = "pypi"
10a9,10
> [dev-packages]
>
$ diff pipenv/2018.11.14/Pipfile.lock pipenv/2018.05.18/Pipfile.lock && echo "no difference"
no difference

I think a graceful way to handle this would be with pyproject.toml similar to how package.json is used in the Node builds. Worst case, a specific heroku section could be added, but you may be able to reuse the standard build-system section https://www.python.org/dev/peps/pep-0517/

[edmorley]

Hi! This week's cryptography 3.4 release made this much more urgent, since cryptography installs fail outright when using older pip, such as that pinned when using pipenv in this buildpack. (And whilst only <5% of Heroku Python builds use pipenv, cryptography is pretty widely used as a sub-dependency.)

I've just released a new buildpack version that includes #1169 and so latest pipenv + newer pip when using pipenv.

Sorry it took so long to get this issue closed. Non-Python projects have consumed most of my time the last 4 months (eg Heroku-20 beta/GA/default stack, Cedar-14 EOL + sunset, Heroku-16 deprecation process, amongst others), but I'm now finally able to return to the Python buildpack and give it the attention it deserves :-)

The recent test suite refactor/adding new testcases (eg #1146) reduced the risk of pipenv upgrades considerably, so hopefully moving forwards we can track recent pipenv versions more closely (with the caveat that we never want to be the test subjects for a brand new release, so won't ever update immediately - particularly given the varied stability of pipenv over the years).