compile_pip_requirements update and all_whl_requirements seem like a circular dependency

[binarystardust]

🐞 bug report

Affected Rule

The issue is caused by the rule:

compile_pip_requirements
all_whl_requirements

Is this a regression?

Have not tested previous versions.

Description

A clear and concise description of the problem...

When trying to switch the index url I noticed that requirements.txt was getting populated with a package that didnt exist in the requirements.txt file when running the compile_pip_requirements.update rule. This should have failed from the beginning as the registry for the index_url was bogus. Then with further poking around I ran it again and this time it failed. There are 2 concerns:

1. How is the requirements.txt being filled in with a package when overriding the index url to one that doesnt contain the package?
2. The rule all_whl_requirements seems to be loading before the update command causing it to pass with the empty requirements.txt and then fail the second time when running the same command since the package doesnt actually exist in the new index registry.

🔬 Minimal Reproduction

MODULE.bazel:

bazel_dep(name = "rules_python", version = "1.3.0")

python = use_extension("@rules_python//python/extensions:python.bzl", "python")
python.toolchain(
    configure_coverage_tool = True,
    ignore_root_user_error = True,
    python_version = "3.11",
)
use_repo(python, "python_3_11")

pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip")
pip.parse(
    experimental_index_url = "https://us-python.pkg.dev/null/simple",
    hub_name = "pypi",
    python_version = "3.11",
    requirements_lock = "//:requirements.txt",
)
use_repo(pip, "pypi")

requirements.in:

itsdangerous

requirements.txt is empty to start.

BUILD.bazel:

load("@pypi//:requirements.bzl", "all_whl_requirements")
load("@rules_python//python:pip.bzl", "compile_pip_requirements")

compile_pip_requirements(
    name = "requirements",
    src = "requirements.in",
    extra_args = [
        "--allow-unsafe",
    ],
    requirements_txt = "requirements.txt",
)

To produce error:

1. Execute the command: bazel run //:requirements.update
2. Execute the command again: bazel run //:requirements.update

🔥 Exception or Error

$ bazel run //:requirements.update
INFO: Analyzed target //:requirements.update (131 packages loaded, 5471 targets configured).
INFO: Found 1 target...
Target //:requirements.update up-to-date:
  bazel-bin/requirements.update
INFO: Elapsed time: 6.987s, Critical Path: 0.59s
INFO: 5 processes: 5 internal.
INFO: Build completed successfully, 5 total actions
INFO: Running command line: bazel-bin/requirements.update '--src=_main/requirements.in' _main/requirements.txt //:requirements.update '--resolver=backtracking' --allow-unsafe --generate-hashes --allow-unsafe
Updating requirements.txt
WARNING: --strip-extras is becoming the default in version 8.0.0. To silence this warning, either use --strip-extras to opt into the new default or use --no-strip-extras to retain the existing behavior.
#
# This file is autogenerated by pip-compile with Python 3.11
# by the following command:
#
#    bazel run //:requirements.update
#
itsdangerous==2.2.0 \
    --hash=sha256:c6242fc49e35958c8b15141343aa660db5fc54d4f13a1db01a3f5891b98700ef \
    --hash=sha256:e0050c0b7da1eea53ffaf149c0cfbb5c6e2e2b69c4bef22c81fa6eb73e5f6173
    # via -r requirements.in
$ bazel run //:requirements.update
WARNING: Download from https://us-python.pkg.dev/null/simple/itsdangerous/ failed: class java.io.FileNotFoundException GET returned 404 Not Found
ERROR: ~/.cache/bazel/_bazel_user/f7d37797d74a062b8f39bc84d0ebc222/external/rules_python~/python/private/pypi/simpleapi_download.bzl:130:14: Traceback (most recent call last):
        File "~/.cache/bazel/_bazel_user/f7d37797d74a062b8f39bc84d0ebc222/external/rules_python~/python/private/pypi/extension.bzl", line 615, column 25, in _pip_impl
                mods = parse_modules(module_ctx)
        File "~/.cache/bazel/_bazel_user/f7d37797d74a062b8f39bc84d0ebc222/external/rules_python~/python/private/pypi/extension.bzl", line 490, column 36, in parse_modules
                out = _create_whl_repos(
        File "~/.cache/bazel/_bazel_user/f7d37797d74a062b8f39bc84d0ebc222/external/rules_python~/python/private/pypi/extension.bzl", line 155, column 50, in _create_whl_repos
                requirements_by_platform = parse_requirements(
        File "~/.cache/bazel/_bazel_user/f7d37797d74a062b8f39bc84d0ebc222/external/rules_python~/python/private/pypi/parse_requirements.bzl", line 180, column 36, in parse_requirements
                index_urls = get_index_urls(
        File "~/.cache/bazel/_bazel_user/f7d37797d74a062b8f39bc84d0ebc222/external/rules_python~/python/private/pypi/extension.bzl", line 470, column 79, in lambda
                get_index_urls = lambda ctx, distributions: simpleapi_download(
        File "~/.cache/bazel/_bazel_user/f7d37797d74a062b8f39bc84d0ebc222/external/rules_python~/python/private/pypi/simpleapi_download.bzl", line 130, column 14, in simpleapi_download
                _fail("Failed to download metadata for {} for from urls: {}".format(
Error in fail: Failed to download metadata for ["itsdangerous"] for from urls: ["https://us-python.pkg.dev/null/simple"]
ERROR: error evaluating module extension pip in @@rules_python~//python/extensions:pip.bzl
INFO: Elapsed time: 0.521s
INFO: 0 processes.
ERROR: Build did NOT complete successfully
ERROR: Build failed. Not running target
FAILED: 
    Fetching module extension pip in @@rules_python~//python/extensions:pip.bzl; starting

🌍 Your Environment

Operating System:

  
Linux
  

Output of bazel version:

  
$ bazel version
Bazelisk version: development
Build label: 7.4.0
Build target: @@//src/main/java/com/google/devtools/build/lib/bazel:BazelServer
Build time: Tue Oct 22 17:24:25 2024 (1729617865)
Build timestamp: 1729617865
Build timestamp as int: 1729617865
  

Rules_python version:

  
1.3.0
  

Anything else relevant?

[binarystardust]

After experimenting some more I realized that compile_pip_requirements and pip.parse in the root level are not necessarily using the same registry settings. I had to update the rule to add it into the extra args section as seen here:

compile_pip_requirements(
    name = "requirements",
    src = "requirements.in",
    extra_args = [
        "--allow-unsafe",
        "--index-url=https://us-python.pkg.dev/null/simple",
    ],
    requirements_txt = "requirements.txt",
)

I think ultimately the issue is that during the Bazel loading stage it tries to load the requirements.txt file but this gets generated with the compile_pip_requirements rule and when that is being ran I believe the code should ignore errors with the requirements.txt when loading. If not there is some sort of cyclical dependency between these 2 things.

I also think a nice to have would be to have compile_pip_requirements use the default values loaded with pip.parse rather than making separate arguments in compile_pip_requirements. Otherwise putting a note in the documentation mentioning that the the index should likely be in both spots for better clarity.

[aignas]

The file that is loaded by pip.parse is not generated, it is the copy from your source tree. Having the compile_pip_requirements and pip.parse be more integrated/documented would be a great improvement, but so far nobody is working on that yet, patches/PRs are welcome!