Eventually implement support for the PEP 735 "dependency groups"

[webknjaz]

What's the problem this feature will solve?

It's just a new source of deps and an extension to the first round of the pyproject.toml runtime deps resolution.

Describe the solution you'd like

$sbj. If somebody's interested in making a PoC, it's probably fine to start drafting something already. Need to wait for the PEP acceptance to actually merge it in, though.

Alternative Solutions

... not doing this?

Additional context

Refs:

• https://peps.python.org/pep-0735/
• https://discuss.python.org/t/pep-735-dependency-groups-in-pyproject-toml/39233/237
• POC for pip: pypa/pip@main...sirosen:pip:pep-735-support

[chrysle]

What are you planning to have for dependency group support? An option to collect only such dependencies, or alongside with normal and build deps? And the possibility to collect only specified dependency groups?

[webknjaz]

Same as #1960 attempted, just in the proper place. Semantically, dependency groups don't imply including runtime dependencies so they should definitely be collected and output separately. However, we shouldn't actively prevent combining them with other deps (runtime, or runtime+extra, or other dependency groups; though the latter seems to be covered by the PEP itself).

[chrysle]

However, we shouldn't actively prevent combining them with other deps

IIUC that would mean actually creating an option like --include-dependency-groups/--all-dependency-groups, next to --only-dependency-groups?

[webknjaz]

Aren't the past two the same?

[chrysle]

Aren't the past two the same?

Yes, this was unintended. Is the rest of my assumption valid?

[webknjaz]

I think so. It of course could be iterated on during a PoC review..

[webknjaz]

UPD: said PEP is accepted as of two weeks ago. So we should implement support for it.

[davidism]

Is there an idea for how this will work? I'd expect to be able to move each .in template into a separate dependency group instead. Then pip-compile pyproject.toml --group tests would read the tests group from pyproject.toml and output to tests.txt (or to -o {name}). Does that sound reasonable?

Currently I have different environments as separate templates: requirements/test.in, requirements/typing.in, etc. Now I'd have tests and typing dependency groups instead but otherwise get the same behavior.

[webknjaz]

Is there an idea for how this will work? I'd expect to be able to move each .in template into a separate dependency group instead.

Yeah, that's kinda the use-case this PEP solves.

Then pip-compile pyproject.toml --group tests would read the tests group from pyproject.toml and output to tests.txt (or to -o {name}). Does that sound reasonable?

This needs to be defined, I think. While working on support for the distribution package build dependencies (#1681 / https://pip-tools.readthedocs.io/en/stable/#maximizing-reproducibility), we realized that there would be cases where one would want to extract those dependencies in isolation and cases of getting combined deps (--build-deps-for=wheel / --all-build-deps / --only-build-deps ).

There's a precedent that doing pip-compile pyproject.toml requirements.in works and sources the deps from several places. That can also be combined with --extras, IIRC.

So to proceed meaningfully, we need to define the semantics and UX for all of these combinations. Do we need --[no-]runtime or some other ways to distinguish what the users would like to encode in their constraint file generation requests? Additionally, I would like to keep #2124 in mind as well, while answering these questions.

Currently I have different environments as separate templates: requirements/test.in, requirements/typing.in, etc. Now I'd have tests and typing dependency groups instead but otherwise get the same behavior.

Simlarly, I have even more envs defined, mostly one per tox target with multiple constraint files (aka DIY lock files) for each, since combinations of different platforms/runtimes can count as separate environments (fun stuff: https://github.com/ansible/awx-plugins/tree/c7fc0a1/dependencies/lock-files#readme).

[sirosen]

Now that the pip implementation seems on track to merge and be part of pip 25.1 later this year, I'd like to start thinking about contributing a parallel feature here in pip-tools specifically for pip-compile.¹
I'd like to discuss the exact desired UX here, and hopefully I can work on a PR

The UX in pip is going to be pip install --group test as a shorthand for pip install --group "./pyproject.toml:test".
You can specify --group multiple times and potentially use multiple distinct pyproject.toml files. e.g., It allows the following:

pip install \
    --group ./subproject1/pyproject.toml:typing \
    --group ./subproject2/pyproject.toml:typing

I'd like pip-compile to imitate this UX exactly, so that users have a consistent experience between tools.

What I'd implement would therefore be a --group option like pip's, and the same deduplication/normalization pass over data using dependency_groups. We could either depend on dependency_groups directly or require pip >= 25.1 (once available) and leverage the vendored copy from there. IMO either is fine, since pip-tools already reaches into pip internals.

I don't have any exciting releases planned for dependency-groups. Probably once pip supports --group I'll do a 2.0 release to remove the pip wrapper command. It depends only on packaging, nothing else.

Does this overall plan sound good? Does pip-compile want a different UX?

Footnotes

1. I'm assuming that the implementation will merge and release as-is. It's been approved but not yet merged, and nobody has raised objections to it in several weeks since the last changes. ↩

[merwok]

What is the meaning of $sbj. ?

[webknjaz]

@merwok sbj == subject. As in a reference to what's written in the issue title.

[webknjaz]

@sirosen I think we have a tradition of double colons. I.e. ::. So it only seems natural to use that as a separator.

[sirosen]

Okay. I don't like that it will drift from pip syntax but if it's consistent within pip-tools that might be best.

Just to lay out how it would look, here are the single vs double colon options, plus a third idea:

Single Colon

• pip install --group './foo/pyproject.toml:bar'
• pip-compile --group './foo/pyproject.toml:bar'

Double Colon

• pip install --group './foo/pyproject.toml:bar'
• pip-compile --group './foo/pyproject.toml::bar'

1-2 Colons

i.e. Make :: and : both work as aliases/alternatives.

• pip install --group './foo/pyproject.toml:bar'
• pip-compile --group './foo/pyproject.toml:bar'
• pip-compile --group './foo/pyproject.toml::bar'

---

I'll work on an PR as soon as I'm able, using the double-colon syntax. If the 1-2 colon option looks nice, I'd be happy to pursue that as well.