[PoC] PEP 751 pip lock command

[sbidoul]

This is a rough PoC for a pip lock command that writes PEP 751 compliant toml to stdout.

Use with:

• pip --quiet lock -o - . to lock the local project to stdout.
• pip lock -e . to lock the local project in editable mode to pylock.toml

I wrote this mainly as a way to better understand the PEP.

A few comments:

• This command works with the same logic as pip install --dry-run --ignore-installed, with mostly the same options.
• I removed the platform/abi/implementation/python-version options due to their limitations (Various package-index filtering flags do not affect the environment markers #11664). We could keep them with the same caveat as for the other commands...
• ...but it is unclear to me if pip in its current state could output something useful for environments, requires-python, packages.marker.
• This command begs for a --only-deps option (Add --only-deps (and --only-build-deps) option(s) #11440)
• The tables are currently not inlined which makes the output somewhat hard to grok.

[potiuk]

Nice. I would love to help with making it happen (of course it depends on PEP 751 to be approved). We are currently doing our own custom and pretty "poor" implementation of locks with cosntraints in Airlfow and having a "standard" way of reproducible installs would be a great thing.

Not sure what is the status now of the PEP and whether it has a chance to be approved soon, but that looks really great that we are trying to standardise it now.

[notatallshaw]

I appreciate this is draft, but some small early feedback.

Firstly, I am advocating for an "output file" option. Writing pip's output to a file in other commands has several footguns, e.g knowing the correct flags to turn off non-relevant output, determining if pip hit errors or warnings when you do, and writing non-UTF-8 files when you direct standard out from powershell on Windows.

pip-compile (and uv pip compile) solve this by having an output file option which is considered best practice to use. That said, I don't know the history of why other commands don't offer this, perhaps it's non-trivial.

I removed the platform/abi/implementation/python-version options due to their limitations

I agree, I think pip should start as limited as possible around known difficult issues, and expand from there. This will give the chance for the lock API to evolve, if able.

[pfmoore]

Cool. I did some early PoC work on the PEP, but things have evolved drastically since then, and I never got back to it. But one of my goals was always for the pip installation report format to be easily convertible into a lockfile. Another thing I'd like to see pip implement (I may well get round to this myself, in due course, it's not a request for someone else to pick it up!) is pip freeze --format=lockfile. I'd hope that would be relatively easy to implement.

I'd very definitely aim pip at solely producing "environment reproduction" lockfiles, and not cross-platform ones. To that end, I think that disallowing the --platform etc., arguments¹ is the right call.

I don't have a strong opinion on the question of an --output-file option, but I think that defaulting to writing to stdout is a reasonable place to start. With an output file option, we'd have to consider validating the provided filename to match the standard-required form pylock[.xxx].toml, which IMO is a distraction from the main exercise of ensuring that we produce a correct lockfile.

Not sure what is the status now of the PEP and whether it has a chance to be approved soon

The PEP is in its final stages before being submitted for pronouncement. At this point, it's extremely unlikely that any major changes will occur. Speaking as the PEP-delegate, I can say that I've been heavily involved in the discussions on the PEP, and I'm confident that it has a good chance of being accepted².

Footnotes

1. Or, as you say, putting them under a "use at your own risk" warning. ↩

2. I'm not giving anything away here - I've always been supportive of the idea of a lockfile standard, but the problem has been getting community consensus on what it should look like. At this point, we're the closest we've ever been to consensus. ↩

[sbidoul]

pip freeze --format=lockfile

I think a pre-requisite for that is PEP 710

[notatallshaw]

I don't have a strong opinion on the question of an --output-file option, but I think that defaulting to writing to stdout is a reasonable place to start. With an output file option, we'd have to consider validating the provided filename to match the standard-required form pylock[.xxx].toml, which IMO is a distraction from the main exercise of ensuring that we produce a correct lockfile.

I'm fine with this being in a follow up PR if there are non-trivial questions to answer.

Though looking now at the PEP I do think a pedantic reading of it would be that the data should only ever be written as a file and not outputted as stdout as that would technically violate those same file name specifications 😉. (If not made clear by emoji I think that's terrible and that stdout should absolutely be an option at least).

[pfmoore]

🙂 I wasn't trying to be pedantic, just noting the likelihood that someone would ask the question about validating the filename. Getting the encoding correct by using the --output-file option is a much more important point, though.

[sbidoul]

I choose not to emit version for direct URL packages, as the version could by dynamic. This is to avoid problems with projects using setuptools-scm, where the version possibly changes with each commit and committing the lock file would change the version.

Also, I choose to emit a relative path for directory packages, so the common use case (namely pip lock .) produces something that is portable.