Package source priority (HPC cluster use case)

[pradyunsg]

Originally posted by @mboisson in #8606 (comment)

Here are what we can and can not do, in our environment (HPC clusters).

1. We can't run a server. This is for HPC clusters which are not under our control. The only thing we distribute is a filesystem. We can't long-standing processes and we don't control the network.
2. We can't run binary (manylinux or other) wheels from PyPI for two primary reasons reasons: missing libraries or libraries installed in non-standard locations, non-optimized CPU instructions
3. We can't host all of PyPI, that's just too much, and pure-python wheels work just fine.
4. We can't change our requirement files/install differently, because we aren't the ones doing the installation. We support end users (researchers), we don't do their works, they install whatever package they need.

What we can do, and have been doing:

1. Provide users a directory with wheels that we compile from source
2. Define a PIP_CONFIG_FILE in the user's environments to make sure their pip install ... commands use some settings. We have been using this to point their pip to our wheel directory, which used to be preferred over pypi. We are now also using that to put a constraint on pip < 21, since that's now broken for us.

[pradyunsg]

There was additional discussion in #8606, but I'm not moving that over for my own sanity. :)

[mboisson]

Other relevant information in this comment:
#8606 (comment)

We are not trying to provide a "secure" version and exclude installing other versions from PyPI. We are trying to provide an "optimized" and "working" version which PyPI fails to do in some instances.

Our users install everything from PyPI anyway. We are not trying to make it more secure than PyPI, we are trying to make it more optimal, and to repair broken binary packages provided on PyPI.

In that use case, shadowing PyPI in some circumstances is precisely what we want and need to do.

Yes, I acknowledge that PyPI is subject to supply chain attack. But that's not what we are trying to fix.

[pradyunsg]

IIUC, your use case is "we have wheels that we want to use preferentially over what's on PyPI for the same versions of the package".

Since you're building these wheels yourself, can you add local version identifiers to those built wheels? Specifically those would look like 1.0.0+awesomelyfast. :)

[mboisson]

I am not familiar with those, but carry on ?

Would these involve just renaming the wheel files ? Or do we need to inject some metadata inside of them ? We have over 6000 wheel files at the moment, so rebuilding them is a large endeavour.

Assuming we can, how do these impact dependency and version resolution in pip ?

[pradyunsg]

In that use case, shadowing PyPI in some circumstances is precisely what we want and need to do.

I guess a way to better explain why your use case broke would be -- substitute "PyPI" for "target company's package index" and hopefully you see that it models the intent of a supply chain attack.

Would these involve just renaming the wheel files ? Or do we need to inject some metadata inside of them ?

Unless I'm missing something, you'll need to do 2 things -- rename the wheels and update the version in the METADATA file inside the wheel (that's package-name.dist-info/METADATA) [1]

Assuming we can, how do these impact dependency and version resolution in pip ?

pip will preferentially use local versions of packages. If there's 1.0.0 and 1.0.0+awesomelyfast, pip will use 1.0.0+awesomelyfast.

If (somehow) that doesn't happen, that's a bug in pip that we'd need to fix. :)

---

To be clear, it's not that anyone thinks that your use case isn't important / worth catering to somehow, but it was extremely unclear what you're trying to do and the story came together in pieces (which can all be put together now, hopefully). That said, it is very likely that you'll need to change something to be compatible with the new way that things work -- based on ComputeCanada/software-stack#80 being filed, I'm gonna be optimistic and guess that you'd be open to changes on this front to keep things working without needing to fork things and add maintenance workload for yourself. :)

---

[1] The file's format is documented at https://packaging.python.org/specifications/core-metadata/ and how-to-parse example at https://github.com/pradyunsg/installer/blob/35ce9141733f1d3fcedfd5e19ea5e34d732fe822/src/installer/utils.py#L69.

[pradyunsg]

Another option is wheel build tags -- which might be an even better answer. If a wheel has a build tag, that wheel is preferred over a wheel that doesn't.

That should only require renaming the wheel files. You can read more about the filename here: https://packaging.python.org/specifications/binary-distribution-format/#file-name-convention

[pradyunsg]

From #8606 (comment):

So... from the stackoverflow comment

Where's this SO answer/comment?

[mboisson]

Would these involve just renaming the wheel files ? Or do we need to inject some metadata inside of them ?

Unless I'm missing something, you'll need to do 2 things -- rename the wheels and update the version in the METADATA file inside the wheel (that's package-name.dist-info/METADATA) [1]

I am hoping we don't need to hack the wheels manually (i.e. unzip them and sed the METADATA file) ?
We basically build our wheels with python setup.py bdist_wheel (after some hacking/patching/changing environment variable to use other dependencies, etc.). I guess there is an option to add local version identifier ?

Assuming we can, how do these impact dependency and version resolution in pip ?

pip will preferentially use local versions of packages. If there's 1.0.0 and 1.0.0+awesomelyfast, pip will use 1.0.0+awesomelyfast.

Even with pip install <something>==1.0.0 ? or with a requirement file that specify an exact version ?

[mboisson]

From #8606 (comment):

So... from the stackoverflow comment

Where's this SO answer/comment?

https://stackoverflow.com/a/67442488

[mboisson]

Another option is wheel build tags -- which might be an even better answer. If a wheel has a build tag, that wheel is preferred over a wheel that doesn't.

That should only require renaming the wheel files. You can read more about the filename here: https://packaging.python.org/specifications/binary-distribution-format/#file-name-convention

Thanks. I'll need to do some testing of how it actually behaves in dependency and version resolution with the latest pip. The local version is more tempting because we could tag them with a meaningful word (computecanada in our case), rather than just a number.

It would be useful to have a description of how dependency resolution is expected to behave, because unfortunately, history has shown that we can run test and have something that works, just to have the way it works changed in the next version of pip.

In particular, if I have, locally:

package-1.0.0+local-cp38-none-linux_x86_64.whl
package-1.0.0-cp38-none-linux_x86_64.whl
package-1.0.0-1-cp38-none-linux_x86_64.whl
package-1.0.0-2-cp38-none-linux_x86_64.whl

and PyPI has

package-1.0.0-cp38-none-linux_x86_64.whl
package-1.0.0-1-cp38-none-linux_x86_64.whl

and I run
pip install package==1.0.0 --find-links=/my/local/wheelhouse/directory
what is the expected order of priority ?

[pradyunsg]

The local version is more tempting because we could tag them with a meaningful word (computecanada in our case), rather than just a number.

Build tags can be 1computecanada or something like that. It's a digit + (optional) string.

Wheels on PyPI can't have build tags, IIRC. Ignoring that, it's gonna be the local version, followed by decreasing build tags followed by remote 1.0.0 followed by local 1.0.0 I think.

None the less, I think the local version will be prefered over build tags. The order of preference is specified at: https://github.com/pypa/pip/blob/main/src/pip/_internal/index/package_finder.py#L530

[uranusjr]

Wheels on PyPI can't have build tags, IIRC.

I believe build tags are possible on PyPI, although practically nobody uses them.