pip doesn't resolve ENV vars in the requirements.txt file of the dependencies of a given dependency (recursive in depth)

[javier-wf]

Description

Let's say we have a project called A in a Github repository from an organization, that has a requirements.txt file with the following dependencies:

B @ git+https://${PAT_TOKEN}@github.com/MY_ORGANIZATION/[email protected]

Then we have the project B also in a Github repository from the same organization, with his own requirements.txt file. For example:

C @ git+https://${PAT_TOKEN}@github.com/MY_ORGANIZATION/[email protected]

As said, all projects belong to the same Github organization (also C), so we can create a Personal Access Token (PAT) that can be used for authenticating in all projects. We export the PAT_TOKEN and set it as an environment variable in a terminal.

If we do pip install -r requirements.txt for project B everything works fine. The C project is installed as a dependency of B and the ${PAT_TOKEN} is correctly read from the environment variables, resolving the url, doing the checkout, etc.

However, if we do the same for project A it fails. Initially, it correctly resolves the B dependency, injecting the ${PAT_TOKEN} from env vars, but now B requires to install C, and in that case the ${PAT_TOKEN} is not replaced by the env variable, but passed literally to the checkout, then failing.

Expected behavior

The value of ${PAT_TOKEN} environment variable must be passed recursively to all dependencies and dependencies of dependencies.

pip version

24.3.1

Python version

3.11

OS

linux

How to Reproduce

1. Create a private repository A in GitHub inside an organization O, and commit some random code that imports package B.
2. Create a private repository B in GitHub inside an organization O, and commit some random code that imports package C.
3. Create a private repository C in GitHub inside an organization O, and commit some random code.
4. Create a requirements.txt file for project Aand add the following line:
   B @ git+https://${PAT_TOKEN}@github.com/O/[email protected]
5. Create a requirements.txt file for project Band add the following line:
   C @ git+https://${PAT_TOKEN}@github.com/O/[email protected]
6. In your account settings, create a personal access token that can be used for all the respositories within the organization.
7. Run pip install -r requirements.txt for A project.

Output

...
Collecting B@ git+https://****@github.com/O/B (from -r requirements.txt (line 1))
  Cloning https://****@github.com/O/B (to revision v0.1.0) to /tmp/pip-install-l1b_8yth/B_afff0d9438154e0bbd86cc84bd9a6408
  Running command git clone --filter=blob:none --quiet 'https://****@github.com/O/B' /tmp/pip-install-l1b_8yth/B_afff0d9438154e0bbd86cc84bd9a6408
  Resolved https://****@github.com/O/B to commit 04060492b1907ca817366f20be6a87a32680bf04
Installing build dependencies: started
...
Collecting C@ git+https://****@github.com/O/[email protected] (from B@ git+https://***@github.com/O/[email protected]>-r requirements.txt (line 1))
  Cloning https://****@github.com/O/C (to revision v0.1.0) to /tmp/pip-install-l1b_8yth/C_66e1729f35a1430fa58d589084f2aacd
  Running command git clone --filter=blob:none --quiet 'https://****@github.com/O/C' /tmp/pip-install-l1b_8yth/C_66e1729f35a1430fa58d589084f2aacd
  fatal: could not read Password for 'https://${PAT_TOKEN}@github.com': No such device or address
  error: subprocess-exited-with-error
  
  × git clone --filter=blob:none --quiet 'https://****@github.com/O/C' /tmp/pip-install-l1b_8yth/C_66e1729f35a1430fa58d589084f2aacd did not run successfully.
  │ exit code: 128
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[notatallshaw]

As you can see from the error:

note: This error originates from a subprocess, and is likely not a problem with pip

For pip install -r requirements.txt will read those requirements and substitute any environmental variables.

But when you pass a source tree, like a file path or a git link, pip invokes the defined build backend (setuptools, hatchling, poetry.core, flit, etc.) to build the package, it then receives that packages dependency metadata from the build backend, pip does not try to read any requirements.txt from that project, that's up to how the build backend is set up.

You've not provided a reproducible example of how you've set these projects up, but if you have a setup.py that reads and passes the dependencies from a requirements.txt then you need to check setuptools and see if it supports environmental substitution, I'm not very familiar myself sorry.

[javier-wf]

Ok. I thought that the behavior was:

1. I run pip install -r requirements.txt for project A
2. pip finds project B as a dependency due to the presence of B @ git+https://${PAT_TOKEN}@github.com/O/[email protected] in the requirements.txt
3. pip, therefore, checkouts project B
4. Now, pip finds the requirements.txt file in project B and therefore runs pip install -r requirements.txt for project B
5. pip finds project C as a dependency due to the presence of C @ git+https://${PAT_TOKEN}@github.com/O/[email protected] in the requirements.txt file of project B
6. Then here it fails because the environment variables are not well substituted in this last call

I am using setuptools to setup my projects, and setutools-scm for dynamic versioning and also for dynamic dependencies so as to I don't have to hardcode the dependencies of each project in the pyproject.toml but just reading them from the requirements.txt file. Here there is an example (truncated and simplified):

[build-system]
requires = ["setuptools>=68", "setuptools-scm>=8"]
build-backend = "setuptools.build_meta"

[tool.setuptools_scm]

[tool.setuptools.dynamic]
dependencies = { file = ["requirements.txt"] }

[project]
name = "B"
dynamic = ["version", "dependencies"]

description = "B"
readme = "README.md"
requires-python = ">=3.11"

Maybe the problem comes from the dynamic dependencies injection of setuptools. Any idea?

Thanks.

[pfmoore]

Maybe the problem comes from the dynamic dependencies injection of setuptools.

Correct. The specification

[tool.setuptools.dynamic]
dependencies = { file = ["requirements.txt"] }

is interpreted by setuptools. The requirements file format is not standardised, so how setuptools interprets that line is up to setuptools, and it appears that they don't expand environment variables when evaluating the file = construct.

You would have to ask the setuptools project who to achieve what you're trying to do - pip isn't involved in building projects apart from invoking the build backend (setuptools in this case).