chore: support git dependencies with pip

[saikonen]

Support for git repo dependencies.

• Add a proper platform restriction for remote deployment that compares tags from built wheels to the target platform. Error out in case of mismatch
• fix caching for remote execution platforms

[saikonen]

this seems to be running into issues with some repositories where a specific setuptools version is requested. the pip download part is failing due to not finding any matching version in the environment

f.ex. with git+https://github.com/pytest-dev/pytest.git it runs into

ERROR: Could not find a version that satisfies the requirement setuptools>=45.0 (from versions: none)
          ERROR: No matching distribution found for setuptools>=45.0

[saikonen]

fixed original issue by applying --no-build-isolation for the pip download command, but there still exists an edge case where in some cases pre-release versions seem to build an incorrect version 0.0.0 when downloaded with --no-index --no-build-isolation, but save correctly without either of the arguments.

As an example:

pip download --dest . --no-deps git+https://github.com/pytest-dev/pytest.git

saves as pytest-8.0.0.dev273+g54623f0f.zip where as

pip download --no-index --no-build-isolation --dest . --no-deps git+https://github.com/pytest-dev/pytest.git

saves as pytest-0.0.0.zip

[saikonen]

for context: setuptools-scm is a required dependency for correctly building VCS sources with pip, otherwise the default behavior with setuptools is to use a placeholder 0.0.0 version instead of erroring out due to missing tools

[romain-intel]

So this builds when installing correct? In other words, it doesn't build during env resolution but during env hydration? Is that correct? If so, as I mentioned previously, I think this potentially breaks reproducibility. I may have missed the part though where it builds when resolving.

[saikonen]

So this builds when installing correct? In other words, it doesn't build during env resolution but during env hydration? Is that correct? If so, as I mentioned previously, I think this potentially breaks reproducibility. I may have missed the part though where it builds when resolving.

This shouldn't break reproducibility as for non-wheel sources, specifically VCS, as a wheel will try to be built before reaching create? I still need to add the platform mismatch check though.

The intended flow is as follows for pypi env resolution:

• solve pip dry-run with report
• download pip download to wheels, only for solved locations ending in .whl / build_wheels for solved locations not ending in .whl
• after download/build_wheels we have a metadata mapping of url->path_to_whl which is used in
• create pip install --no-compile from the local wheels

[romain-intel]

I don't think this is quite correct. You probably want a set intersection and if empty, then error out.

[saikonen]

Great catch! Fixed this and added some unit tests for validation. Coverage is not extensive but should at least handle obvious cross-platform deployment breaks (e.g. osx-arm64 -> linux-64)

[saikonen]

superseded by #1681