--generate-hashes with remote URLs

[blueyed]

When using a requirement like -e git+https://github.com/lock8/django-rest-framework.git@d9ee7d68178a6b50b55caacdb50a531b2cc0eaf6#egg=djangorestframework --hash=sha256:8a25e5ea1727e83bb55c3459b1116161ebf67314696672227a053265564c6af9 pip-compile with --generate-hashes generates:

-e git+https://github.com/lock8/django-rest-framework.git@d9ee7d68178a6b50b55caacdb50a531b2cc0eaf6#egg=djangorestframework

I.e. it removes the --hash that is already there.

pip install complains:

Hashes are required in --require-hashes mode, but they are missing from some requirements. Here is a list of those requirements along with the hashes their downloaded archives actually had. Add lines like these to your requirements files to prevent tampering. (If you did not enable --require-hashes manually, note that it turns on automatically when any package has a hash.)
    https://github.com/lock8/django-rest-framework/archive/d9ee7d68178a6b50b55caacdb50a531b2cc0eaf6.tar.gz#egg=djangorestframework --hash=sha256:8a25e5ea1727e83bb55c3459b1116161ebf67314696672227a053265564c6af9

Note that pip-tools requires the -e for remote URLs in general, and that I turn it into https://github.com/lock8/django-rest-framework/archive/d9ee7d68178a6b50b55caacdb50a531b2cc0eaf6.tar.gz#egg=djangorestframework manually after pip-compiles runs over it.

As a quick fix I suggest keeping the --hash from the .in file?!

[vphilippon]

Thanks for the report @blueyed

That sounds like a good idea, it's a user given parameter, we should keep it in the output.

[atugushev]

I've tried to install the editable package:

$ cat requirements.txt
-e git+https://github.com/lock8/django-rest-framework.git@d9ee7d68178a6b50b55caacdb50a531b2cc0eaf6#egg=djangorestframework --hash=sha256:8a25e5ea1727e83bb55c3459b1116161ebf67314696672227a053265564c6af9

and got this error:

$ pip install -r requirements.txt --require-hashes
Obtaining djangorestframework from git+https://github.com/lock8/django-rest-framework.git@d9ee7d68178a6b50b55caacdb50a531b2cc0eaf6#egg=djangorestframework (from -r reqs.txt (line 1))
The editable requirement djangorestframework from git+https://github.com/lock8/django-rest-framework.git@d9ee7d68178a6b50b55caacdb50a531b2cc0eaf6#egg=djangorestframework (from -r reqs.txt (line 1)) cannot be installed when requiring hashes, because there is no single file to hash.

@blueyed it seems there is no way to pip install editable packages with hash compare. See the issue in pip.

[blueyed]

@atugushev
Oh, I forgot that I am turning git+ URLs into https already (note the second snippet, which does not use git+ anymore).
So I think the issue still applies, except for that you have to use non-git URLs in the first place.

[atugushev]

@blueyed
ok, i've tried this one:

$ cat requirements.in
https://github.com/lock8/django-rest-framework/archive/d9ee7d68178a6b50b55caacdb50a531b2cc0eaf6.tar.gz#egg=djangorestframework

and got this error:

$ pip-compile requirements.in
piptools.exceptions.UnsupportedConstraint: pip-compile does not support URLs as packages, unless they are editable. Perhaps add -e option? (constraint was: djangorestframework from https://github.com/lock8/django-rest-framework/archive/d9ee7d68178a6b50b55caacdb50a531b2cc0eaf6.tar.gz#egg=djangorestframework (from -r requirements.in (line 1))

Just wandering how did you compile non-editable URL package?

[blueyed]

@atugushev
It needs to be editable, i.e. add -e in front.

[atugushev]

@blueyed

For -e https://... i got:

pip._internal.exceptions.InstallationError: https://github.com/lock8/django-rest-framework/archive/d9ee7d68178a6b50b55caacdb50a531b2cc0eaf6.tar.gz#egg=djangorestframework should either be a path to a local project or a VCS url beginning with svn+, git+, hg+, or bzr+

[blueyed]

@atugushev
Oh, sorry.. editable is not necessary.. this should work (as a result):

https://github.com/lock8/django-rest-framework/archive/d9ee7d68178a6b50b55caacdb50a531b2cc0eaf6.tar.gz#egg=djangorestframework --hash=sha256:8a25e5ea1727e83bb55c3459b1116161ebf67314696672227a053265564c6af9

I am using this to transform/compile requirements:

$(PIP_REQUIREMENTS_ALL):: $(PIP_REQUIREMENTS_DIR)/%.txt: $(PIP_REQUIREMENTS_DIR)/%.in
	@pip-compile --no-header --generate-hashes $(PIP_COMPILE_ARGS) --output-file "$@.tmp" "$<" >"$@.out" || { \
	  ret=$$?; echo "pip-compile failed:" >&2; cat "$@.out" >&2; \
	  $(RM) "$@.tmp" "$@.out"; \
	  exit $$ret; }
	@sed -n '1,10 s/# Depends on/-r/; s/\.in/.txt/p' "$<" > "$@"
	@# Keep and transform '-e git+' as-is (includes the hash).
	@sed -n -e '/-e git+/ {s~^-e git+\(http.*\)@\([^#]\+\)\(#.*\)\?~\1/archive/\2.tar.gz\3~; s~\.git/archive~/archive~; p}' "$<" >> "$@"
	@# Remove any editables (not supported with hashes).
	@sed -e '/^-e /d' "$@.tmp" >> "$@"
	@$(RM) "$@.tmp" "$@.out"

The relevant part is sed -n -e '/-e git+/ {s~^-e git+\(http.*\)@\([^#]\+\)\(#.*\)\?~\1/archive/\2.tar.gz\3~; s~\.git/archive~/archive~; p}' to transform -e git+ (for pip-compile to work) into https://… in the result.

[jcushman]

I think the underlying goal of using --generate-hashes with remote URLs is working after #807. Running on master:

$ echo 'https://github.com/lock8/django-rest-framework/archive/d9ee7d68178a6b50b55caacdb50a531b2cc0eaf6.tar.gz#egg=djangorestframework' > requirements.in
$ pip-compile --generate-hashes
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --generate-hashes
#
https://github.com/lock8/django-rest-framework/archive/d9ee7d68178a6b50b55caacdb50a531b2cc0eaf6.tar.gz#egg=djangorestframework \
    --hash=sha256:8a25e5ea1727e83bb55c3459b1116161ebf67314696672227a053265564c6af9

Would it still be useful to preserve --hash in requirements.in for other reasons? Not sure if this should be closed or updated.

[blueyed]

@jcushman
This looks fine, thanks!