pip doesn't resolve ENV vars in the requirements.txt file of the dependencies of a given dependency (recursive in depth)

[javier-wf]

Description

Let's say we have a project called A in a Github repository from an organization, that has a requirements.txt file with the following dependencies:

B @ git+https://${PAT_TOKEN}@github.com/MY_ORGANIZATION/B@v0.1.0

Then we have the project B also in a Github repository from the same organization, with his own requirements.txt file. For example:

C @ git+https://${PAT_TOKEN}@github.com/MY_ORGANIZATION/C@v0.1.0

As said, all projects belong to the same Github organization (also C), so we can create a Personal Access Token (PAT) that can be used for authenticating in all projects. We export the PAT_TOKEN and set it as an environment variable in a terminal.

If we do pip install -r requirements.txt for project B everything works fine. The C project is installed as a dependency of B and the ${PAT_TOKEN} is correctly read from the environment variables, resolving the url, doing the checkout, etc.

However, if we do the same for project A it fails. Initially, it correctly resolves the B dependency, injecting the ${PAT_TOKEN} from env vars, but now B requires to install C, and in that case the ${PAT_TOKEN} is not replaced by the env variable, but passed literally to the checkout, then failing.

Expected behavior

The value of ${PAT_TOKEN} environment variable must be passed recursively to all dependencies and dependencies of dependencies.

pip version

24.3.1

Python version

3.11

OS

linux

How to Reproduce

1. Create a private repository A in GitHub inside an organization O, and commit some random code that imports package B.
2. Create a private repository B in GitHub inside an organization O, and commit some random code that imports package C.
3. Create a private repository C in GitHub inside an organization O, and commit some random code.
4. Create a requirements.txt file for project Aand add the following line:
   B @ git+https://${PAT_TOKEN}@github.com/O/B@v0.1.0
5. Create a requirements.txt file for project Band add the following line:
   C @ git+https://${PAT_TOKEN}@github.com/O/C@v0.1.0
6. In your account settings, create a personal access token that can be used for all the respositories within the organization.
7. Run pip install -r requirements.txt for A project.

Output

...
Collecting B@ git+https://****@github.com/O/B (from -r requirements.txt (line 1))
  Cloning https://****@github.com/O/B (to revision v0.1.0) to /tmp/pip-install-l1b_8yth/B_afff0d9438154e0bbd86cc84bd9a6408
  Running command git clone --filter=blob:none --quiet 'https://****@github.com/O/B' /tmp/pip-install-l1b_8yth/B_afff0d9438154e0bbd86cc84bd9a6408
  Resolved https://****@github.com/O/B to commit 04060492b1907ca817366f20be6a87a32680bf04
Installing build dependencies: started
...
Collecting C@ git+https://****@github.com/O/C@v0.1.0 (from B@ git+https://***@github.com/O/B@v0.1.0->-r requirements.txt (line 1))
  Cloning https://****@github.com/O/C (to revision v0.1.0) to /tmp/pip-install-l1b_8yth/C_66e1729f35a1430fa58d589084f2aacd
  Running command git clone --filter=blob:none --quiet 'https://****@github.com/O/C' /tmp/pip-install-l1b_8yth/C_66e1729f35a1430fa58d589084f2aacd
  fatal: could not read Password for 'https://${PAT_TOKEN}@github.com': No such device or address
  error: subprocess-exited-with-error
  
  × git clone --filter=blob:none --quiet 'https://****@github.com/O/C' /tmp/pip-install-l1b_8yth/C_66e1729f35a1430fa58d589084f2aacd did not run successfully.
  │ exit code: 128
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[notatallshaw]

As you can see from the error:

note: This error originates from a subprocess, and is likely not a problem with pip

For pip install -r requirements.txt will read those requirements and substitute any environmental variables.

But when you pass a source tree, like a file path or a git link, pip invokes the defined build backend (setuptools, hatchling, poetry.core, flit, etc.) to build the package, it then receives that packages dependency metadata from the build backend, pip does not try to read any requirements.txt from that project, that's up to how the build backend is set up.

You've not provided a reproducible example of how you've set these projects up, but if you have a setup.py that reads and passes the dependencies from a requirements.txt then you need to check setuptools and see if it supports environmental substitution, I'm not very familiar myself sorry.

[javier-wf]

Ok. I thought that the behavior was:

1. I run pip install -r requirements.txt for project A
2. pip finds project B as a dependency due to the presence of B @ git+https://${PAT_TOKEN}@github.com/O/B@v0.1.0 in the requirements.txt
3. pip, therefore, checkouts project B
4. Now, pip finds the requirements.txt file in project B and therefore runs pip install -r requirements.txt for project B
5. pip finds project C as a dependency due to the presence of C @ git+https://${PAT_TOKEN}@github.com/O/C@v0.1.0 in the requirements.txt file of project B
6. Then here it fails because the environment variables are not well substituted in this last call

I am using setuptools to setup my projects, and setutools-scm for dynamic versioning and also for dynamic dependencies so as to I don't have to hardcode the dependencies of each project in the pyproject.toml but just reading them from the requirements.txt file. Here there is an example (truncated and simplified):

[build-system]
requires = ["setuptools>=68", "setuptools-scm>=8"]
build-backend = "setuptools.build_meta"

[tool.setuptools_scm]

[tool.setuptools.dynamic]
dependencies = { file = ["requirements.txt"] }

[project]
name = "B"
dynamic = ["version", "dependencies"]

description = "B"
readme = "README.md"
requires-python = ">=3.11"

Maybe the problem comes from the dynamic dependencies injection of setuptools. Any idea?

Thanks.

[pfmoore]

Maybe the problem comes from the dynamic dependencies injection of setuptools.

Correct. The specification

[tool.setuptools.dynamic]
dependencies = { file = ["requirements.txt"] }

is interpreted by setuptools. The requirements file format is not standardised, so how setuptools interprets that line is up to setuptools, and it appears that they don't expand environment variables when evaluating the file = construct.

You would have to ask the setuptools project who to achieve what you're trying to do - pip isn't involved in building projects apart from invoking the build backend (setuptools in this case).

[glechic]

I used the following workaround.

with open('requirements.txt') as reqs:
    for line in reqs:
        line = line.strip()
        for key, value in os.environ.items():
            line = line.replace(f"${{{key}}}", value)
        
        install_requires.append(line)

setup(
    ...,
    install_requires=install_requires,
)

[AndreyOrb]

@javier-wf , did you open/find a relevant issue in setuptools project?
This issue is probably the same: pypa/setuptools#3584

[notatallshaw]

Thanks @glechic for providing a workaround for users of setuptools (presumably done in a setup.py).

Closing this issue as pip can't do anything here.

[AndreyOrb]

I am using the pyproject.toml and the proposed workaround is not relevant.
Interesting. Both issues were closed with pip vs setuptools blaming each other.

But the issue is still there no matter which project must address it.

@notatallshaw , @abravalheri Could we proceed with this thread until the issue is resolved?

[abravalheri]

You would have to ask the setuptools project who to achieve what you're trying to do - pip isn't involved in building projects apart from invoking the build backend (setuptools in this case).

Thanks Paul.

We were aware about some potential confusion when implementing [tool.setuptools.dynamic] dependencies, and to be honest it only got in because of "popular demand".

For anyone using this [tool.setuptools.dynamic] dependencies, please note that the setuptools never committed to accept a requirements.txt format.

What we support is a restricted syntax that merely resembles requirements.txt (and is a subset of it), so that you can carefully engineer a text file to be used as an input to pip and setuptools at the same time.

You can have commented lines starting with #, and the non commented lines are required to comply with the PEP 508 specification. Non pip-specific syntax or flags is accepted.

Please refer to the https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html#dynamic-metadata and https://setuptools.pypa.io/en/latest/userguide/declarative_config.html#opt-6:

Supporting file for dependencies is meant for a convenience for packaging applications with possibly strictly versioned dependencies.
...
Also note that the file format for specifying dependencies resembles a requirements.txt file, however please keep in mind that all non-comment lines must conform with PEP 508 (pip specific syntaxes, e.g. -c/-r/-e and other flags, are not supported).

Accepting pip-compatible and fully-featured requirements.txt is not in the scope of setuptools.