all_requirements fails for multiple python versions

[thomasbaekkegaard]

🐞 bug report

Affected Rule

The all_requirements constant in the generated requirements.bzl file in the PyPI hub repo.

Is this a regression?

Not really.

Description

When running with multiple python version for a single hub, the lock files will generally not contain the exact same dependencies. This means that when using all_requirements from the generated @python_deps//:requirements.bzl file, it will reference some dependencies which does not exist for all the python versions. In practice this means an error is raised by @rules_python//python/config_settings:is_not_matching_current_config.

Instead, all_requirements should only return the requirements for the python version matching the current toolchain, OR alternatively only the requirements common to all python versions.

🔬 Minimal Reproduction

Steps:

• Create requirements.in with:

mypy==1.15.0

• Generate lock files requirements_3_10.txt and requirements_3_11.txt for Python 3.10 and Python 3.11, respectively.

• Add to MODULE.Bazel:

python = use_extension("@rules_python//python/extensions:python.bzl", "python")
python.toolchain(
    is_default = True,
    python_version = "3.11",
)

pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip")
pip.parse(
    hub_name = "python_deps",
    python_version = "3.11",
    requirements_linux = ":requirements_lock_3_11.txt",
)

pip.parse(
    hub_name = "python_deps",
    python_version = "3.10",
    requirements_linux = ":requirements_lock_3_10.txt",
)
use_repo(pip, "python_deps")

• Now try to use all_requirements - e.g. for running checks on all licenses:

load("@python_deps//:requirements.bzl", "all_requirements")

license_test(
    name = "all_python_license_check",
    deps = all_requirements,
)

🔥 Exception or Error

Running the described example on the default toolchain (python 3.11) results in the error:

ERROR: {path_to_bazel_cache}/external/rules_python++pip+python_deps/tomli/BUILD.bazel:5:12: configurable attribute "actual" in @@rules_python++pip+python_deps//tomli:_no_matching_repository doesn't match this configuration: No matching wheel for current configuration's Python version.

The current build configuration's Python version doesn't match any of the Python
wheels available for this distribution. This distribution supports the following Python
configuration settings:
    //_config:is_cp310

To determine the current configuration's Python version, run:
    `bazel config <config id>` (shown further below)

For the current configuration value see the debug message above that is
printing the current flag values. If you can't see the message, then re-run the
build to make it a failure instead by running the build with:
    --@@rules_python+//python/config_settings:current_config=fail

However, the command above will hide the `bazel config <config id>` message.


This instance of @@rules_python++pip+python_deps//tomli:_no_matching_repository has configuration identifier ... . To inspect its configuration, run: bazel config ... .

For more help, see https://bazel.build/docs/configurable-attributes#faq-select-choose-condition.

🌍 Your Environment

Operating System:

  
Linux Ubuntu
  

Output of bazel version:

  
8.2.1
  

Rules_python version:

  
1.4.1
  

[aignas]

A PR to drop packages that are not common to all versions would be welcome.

EDIT: just a few pointers what would need to change.

1. Currently this line just combines the list of exposed packages by doing a set union, instead we should do an intersection, something like exposed_packages = _intersection(exposed_packages or out.exposed_packages, out.exposed_packages)

def _intersection(a, b):
    result = {}
    for item in a:
        if item not in b:
            continue
        
        result[item] = None
    return result

2. Add a test in //tests/pypi/extension/extension_tests.bzl.