Implement a --group option for installing from [dependency-groups] found in pyproject.toml files

[sirosen]

Update: This PR has undergone a major revision. See this comment for the revised proposal.
The initial comment here is preserved, as it contains useful context and is necessary to understand the following discussion.

---

This changeset implements --group as an option for loading dependency groups, leveraging the dependency-groups library¹.
resolves #12963

Example usage

pip install --group GROUP1 --group GROUP2
pip download --group GROUP1

pyproject.toml loading and working dir

Per the discussion in #12963 , this is implemented with support only for loading data out of the pyproject.toml file found in the working directory.

Although in theory there might be a desire to have any of the following interfaces in the future...

pip install --group GROUP1 --pyproject-file /scratch/foo/pyproject.toml
PIP_PYPROJECT_PATH=/scratch/foo/pyproject.toml pip install --group GROUP1
pip install --project-dir /scratch/foo --group GROUP1

...there is not a clear "winner" amongst these and they are only usable with --group.
Until or unless there is a more strongly demonstrated need to specify the path in some way, no attempt is made here to support it.²

--group as the Option Name

--group is known to match the option provided by uv for installation from groups.
There are some lines of argument in favor (e.g., the initial post which mentions this), and some arguments against (e.g., my note that identical option names could incorrectly imply identical behavior).
Ultimately, I chose to propose --group because of a UV-independent argument that it is short and declarative.

I am open to arguments from pip maintainers in favor of other names -- but I think --group is a good name and that "potential confusion with uv" is a bad reason not to choose it if it's the best name for the feature.
Any alternative name should have some reason that it would also be good for users. e.g., --dependency-group GROUP1 could work because it's very explicit.

--group as a repeated option vs comma-delimited --groups

I originally proposed an interface along the lines of --groups foo,bar,baz in #12963.
I'll just make a quick note that as soon as I started implementing this, I could not see any particularly good reason to add specialized parsing.
Option parsing can collect these for us trivially, and it removes the potential for ambiguity in cases like --groups foo,bar --groups foo,baz.

Wrapped dependency-groups lib

Use of dependency-groups passes through a wrapper module which conforms the errors from that library to InstallationError.
Details of the implementation do (intentionally) leak out, most notably the error strings.

This can be revised if there's a desire for pip to control the error messages more tightly.

Tests

It was not clear to me exactly how much effort I should invest in unit and functional tests.
I therefore generally tried to follow the guide of "test the subset of features which map to existing tests of requirements.txt files".

New unit tests and functional tests explore some particular cases, but they are intentionally a little bit limited.
If desirable, we can retest most of the behaviors provided by dependency_groups.resolve under the unit tests.

Commits & Review

Here's a summary of the (at time of writing) commit series:

• Add 'dependency-groups==1.3.0' to vendored libs
• Implement Dependency Group option: --group
• Add unit tests for dependency group loading
• Add initial functional tests for dependency-groups
• Add a news fragment for --group support

Notably, the first commit contains the whole vendoring step with no usage. So it should be possible to diff the HEAD of the PR against that commit to see the "real work" here.

Footnotes

1. As a potentially important aside, I intend to open a thread to move dependency-groups into github.com/pypa/ for better continuity of ownership and maintenance. ↩

2. Some users will likely be disappointed with this decision. But the nice thing about not deciding anything today is that it can easily be added if a strong argument is made. "No is temporary; yes is forever." ↩

[zanieb]

Exciting to see this up!

Per the discussion in #12963 , this is implemented with support only for loading data out of the pyproject.toml file found in the working directory.

To clarify for those that have not read the discussion, I feel like we did not reach consensus that automatically discovering a pyproject.toml in the working directory was the right solution. I'll repeat that I think this is a surprising and significant change for pip.

I'll disclaim that I am one of the people opposed, but so was @pfmoore — I feel like this merits further discussion.

that "potential confusion with uv" is a bad reason not to choose it if it's the best name for the feature.

As a minor note, we have not added dependency group support to the uv pip interface — we're waiting for a name choice here. I still think --group is a good option, but we are very likely to support whatever is chosen here.

[sirosen]

I feel like we did not reach consensus that automatically discovering a pyproject.toml in the working directory was the right solution.

I agree with that assessment. I'm proposing it as an initial option, but am 100% ready to adjust course if there's an alternative with pip maintainer support.

I'll repeat that I think this is a surprising and significant change for pip.

Maybe I'm inferring too much, but does this suggest that there's an alternative UX you would find less surprising?

I've thought up alternatives, but none of them seem clearly better to me.

---

And 👍 to the note about keeping uv pip / pip in sync.

[pfmoore]

I'm OK with --group, if I'd written the code I would probably have gone with the more explicit --dependency-group, but there's very little in it (--group is shorter to type, for what it's worth...)

It did occur to me when this PR was submitted that we hadn't reached agreement on auto-discovering pyproject.toml, and I'm still a little uncomfortable about it. One concern I have is that we will potentially get people asking us to locate "the project's pyproject.toml when the working directory is not at the root of the project, and that's when we really do need to accept that we're introducing the concept of "the current project" to pip.

I don't know the right answer here. It feels to me like the "traditional" role of pip, as a standalone installer, is being eroded by the ecosystem drift towards the "everything is a project" model. I don't like that, but maybe at some point I have to accept the inevitable and stop blocking things based on an outdated view of what pip is for. I would definitely like to know what the other @pypa/pip-committers think about this, though. There's also a wider discussion about how pip fits into the modern packaging ecosystem that I think the maintainers need to have, but that hasn't happened yet either.

I don't want to block things on the basis of some grand philosophical debate, so if someone says "let's just do it for now and worry about the bigger picture later", I'm OK with that.

[henryiii]

What about adding a --project-dir option that defaults to .? Then at least it can be made explicit? ${project-dir}/pyproject.toml would then be the path to the pyproject.toml.

FWIW, I also like --group, it's a lot shorter than --dependency-group; short enough I don't think there would be a strong desire for a short option for it. Pip mostly installs dependencies already.

Original post:

I think users would like for this to work without having to add some new --project option to tell pip where the project it - it's specified in -e. already:

pip install -e. --group test

However, then I think most people would expect this to work:

pip install . --group test

which is a lot more dubious; another clearer example would be

pip install ./some/thing ./other/thing --group test

Which isn't clear to me at all which of the two directories would be used - or both. Or neither and just the current directory.

If there was a --project-dir option, this would be solved, though then the command line is really long and repetitive:

pip install -e. --project-dir . --group test

But that's fully explicit. I think either defaulting the project-dir to . or allowing -e to also set the default directory would help shorten the command line usage (both would be too confusing, IMO) - at the expense of potential confusion, but having a clear and consistent rule would help. But maybe it might make sense to start with the "explicit" version, then propose either of those simplifications?

FWIW, I'm pretty happy if this gets in in any state, and I like shorter CLIs if possible. :)

[sirosen]

What about adding a --project-dir option that defaults to .? Then at least it can be made explicit? ${project-dir}/pyproject.toml would then be the path to the pyproject.toml.

This was one option I considered, as well as, similar, --pyproject-file="./pyproject.toml" as a default behavior.
I'll circle back on this below, but I much prefer an option for the filename to one for the dir.

It did occur to me when this PR was submitted that we hadn't reached agreement on auto-discovering pyproject.toml, and I'm still a little uncomfortable about it.

I read some of your comments there as weakly supporting the idea of ./pyproject.toml as the only behavior, but that may have been my mistake.

One concern I have is that we will potentially get people asking us to locate "the project's pyproject.toml when the working directory is not at the root of the project, and that's when we really do need to accept that we're introducing the concept of "the current project" to pip.

I do not like the idea that pip would expand to do any "discovery" process. Even a simple process, like crawling up parent dirs to find a pyproject.toml, would represent a pretty dramatic change in scope.

Is this perhaps an argument in favor of accepting the pyproject file (or dir?) as an option? Any user who asked "why doesn't pip find my pyproject.toml?" would get a pretty easy answer of "because you didn't pass --pyproject-file=... and it's not in the working dir".

---

I'm much more comfortable with the idea of adding an option for the pyproject.toml file than one for the project dir.

A file option keeps pip uniformly file-oriented. You can have -c, -r, and now --pyproject-file, all of which may drive behaviors of your commands. This adds the notion that pip may have behaviors (this being the first) which are driven by reading a pyproject.toml file, but it doesn't seem to be a huge paradigm shift to me.

A dir option adds the notion that there is a "project directory", and I'm much less clear on what that means. It would only drive pyproject.toml today, but it implies that pip has some non-file-driven behaviors which this controls.

[zanieb]

Maybe I'm inferring too much, but does this suggest that there's an alternative UX you would find less surprising?

I made two concrete recommendations in the issue (both of which are similar to @henryiii's suggestions here):

1. Require --project <dir> to explicitly define the project path. Groups can only be read from the project.
2. Read groups for for any source tree in the installation.

For discussion purposes, let's list a couple more proposals:

3. Discover the pyproject.toml in the working directory
4. Discover the pyproject.toml in any parent directory

I don't want to just repeat the discussion from there — I think there's a fair bit of context on the upsides and downsides to each of those in the thread already. Unfortunately, I don't think there's an obviously superior option here. I'll try to summarize some thoughts briefly:

(1) is very explicit which is good for teaching but can be repetitive and verbose
(2) is the most intuitive for existing users but hides a lot of complexity which will cause some confusion
(3) is a departure from existing pip behavior (not project or working directory aware)
(4) is what uv does, but it seems like there is consensus this is out of scope for pip right now

Since @sirosen replied while I was authoring this..

I much prefer an option for the filename to one for the dir.

The pyproject.toml filename is standardized. It don't think it make sense to ask users to type it. It'd need to be included in every invocation of the option which makes it feel redundant. If we allowed alternative filenames, I'd feel differently. It does have the benefit of clarifying the expected value for the option and might help with ambiguity I raised previously like pip install --editable . --project . --group test. I worry there are still confusing cases though, like pip install --editable . --pyproject ./pyproject.toml.

[potiuk]

Fascinating discussion. I am also ok for accepting any option, but I think the best way will be to use --group (and I like short version of it) for all directories that are specified explicitly (either via --editable or not).

Just to make a discussion more concrete - (and provide context) we are eyeing into refactoring airflow monorepo for Airflow 3 (it's partially done already but we are also waiting for this one to land), and we will have 90+ sub-projects eventually in Airflow monorepo (yep, I know it's crazy).

We are now recommending uv for contributors now (due to workspace feature), but also very strong on making sure that pip workflows works for our contributors (even if they are a bit more complex for the contributors to run). We have > 3000 contributors, and I would hate to lose them if - for whatever reason - they are not able to use auv (for example because admins in their corporate entity have some strange rules on what tools could be run). I've already heard stories about "having to configure company proxy for uv` to make it works.

So - for multiple reasons we make sure in our docs and workflows that both uv and pip are supported.

In our case we currently have (this wil change and improve but here it is):

pyproject.toml -> main airflow project
task_sdk
    pyproject.toml   ->  new task_sdk project for Airlfow 3
providers
    pyproject.toml -> 90+ providers - we will split it to 90+ subprojects (90+ pyproject.toml files) 😱  

eventually it will be:

airflow
     pyproject.toml -> main airflow project
task_sdk
    pyproject.toml   ->  new task_sdk project for Airlfow 3
providers
         amazon
               pyproject.toml
         google
               pyproject.toml

What I really would like with groups is:

• pip install -e ./airflow --group devel -> only install "airflow" with devel group of deps for airflow
• pip install -e ./airflow -e ./task_sdk --group devel -> install both airlfow and tas_sdk - both with devel group of deps if defined in both
• pip install -e ./airflow -e ./providers/google --group devel -> install both airlfow and google provider both with devel groups of deps if defined in both (and only from airflow if devel is not defined in google)

Generally i'd be for not having --project flag, but choosing pyproject.toml files coming from the "folders" or "--editable" projects specified explicitly. I somehow find it pretty confusing to specify them independently.

In uv with workspace defined, the --group could work on the whole workspace instead:

• uv sync --devel woudl automatically install all development deps, additionally we could specify the default group in uv workspace that should be used for uv sync by default (details to be worked out).

[pfmoore]

I read some of your comments there as weakly supporting the idea of ./pyproject.toml as the only behavior, but that may have been my mistake.

Weakly in the sense of "yeah, I guess we might have to do that". It has problems, as you mention, though, so I'm definitely not enthusiastic about it 🙁

Is this perhaps an argument in favor of accepting the pyproject file (or dir?) as an option? Any user who asked "why doesn't pip find my pyproject.toml?" would get a pretty easy answer of "because you didn't pass --pyproject-file=... and it's not in the working dir".

Yes. And stronger, it's an argument for requiring the --pyproject-file option, so we don't even need the "in the working dir" qualifier. The problem with defaulting to the working directory is that people can argue that "the default isn't helpful". Whereas requiring that the user specifies the file every time avoids this by not having a default (helpful or otherwise). If nothing else, I'd argue that we should start with no default, and if experience shows that there's a clear consensus on a default value, we can add it later. Users could experiment with defaults by setting PIP_PYPROJECT_FILE=./pyproject.toml in their environment.

I'm much more comfortable with the idea of adding an option for the pyproject.toml file than one for the project dir.

+1. Your arguments make sense to me. (But I'm not convinced about defaulting the pyproject.toml file).

This adds the notion that pip may have behaviors (this being the first) which are driven by reading a pyproject.toml file, but it doesn't seem to be a huge paradigm shift to me.

There are two aspects I don't like:

1. It's the first time pip has assumed any sort of file structure. At the moment, we don't default the argument to --editable as ., or the argument of -r as requirements.txt, and we don't assume pip is being run in a source tree. But now we're going to start assuming that --group implies we're running in a source tree. But only when --group is specified. And it doesn't actually need to be a source tree, it could just be a file called pyproject.toml with a dependency groups section and nothing else. It doesn't even need to be called pyproject.toml...
2. It suggests to users that other behaviours should assume you're working in a source tree. Which will add a maintenance cost of having to explain repeatedly that --group is a special case. Your comment "this being the first" hints at this.

[pfmoore]

Read groups for for any source tree in the installation.

I'm a strong -1 on this. There are so many edge cases that don't have intuitive (or in some cases even plausible) interpretations that I don't even think this is the "user friendly" option. It's deceptively straightforward in simple cases, but will end up causing nasty bugs as soon as people do something unusual.

I suggest that we simply drop this as a possibility, as it feels like people are only thinking about the "obvious" cases, and I have no appetite for coming up with multiple problematic cases just so that people can suggest workaround after workaround. (For example, if a requirements file includes -e some/global/project what happens if the user specifies --group dev and dev is specified in that project as well as the user's project? What if the user doesn't want the dev group from the global project? What if they do?)

[sirosen]

In general, I am only interested in implementing behaviors which are easy to reason about and don't contain avoidable ambiguities. Even within that confined space, we are debating how to create a UX in which most naive users won't be easily confused.

I worry there are still confusing cases though, like pip install --editable . --pyproject ./pyproject.toml.

I think this is a good point of concern. A user could do something like...

pip install -e ./foo --pyproject ./bar/pyproject.toml

and expect... ?
A user who does this intentionally and expects a special result clearly has some incorrect mental model for what's happening.

I'd like to have a solution in which the above usage, or its analogue, emits a warning or error, instructing the user that they're misapplying the options.

And stronger, it's an argument for requiring the --pyproject-file option, so we don't even need the "in the working dir" qualifier. ... Users could experiment with defaults by setting PIP_PYPROJECT_FILE=./pyproject.toml in their environment.

I find this convincing in principle, but I worry that it makes for a very verbose CLI experience for interactive usage.
Is it reasonable to consider adding a short opt to make it easier? e.g. -p / --pyproject.file FILENAME? That allows

pip install -p ./pyproject.toml --group lint

If we go down this path, I have a question:
Are you suggesting that the default be to read a PIP_PYPROJECT_FILE env var if one is set? I would be ok with that. It seems consistent with the env-config loader behavior which exists (though I haven't investigated much).

And an initial expectation about requirements and behavior

• --pyproject-file as an would be required if you use --group
• --pyproject-file without group would be an error, indicating that it must be used as --group

The check for --group could be expanded in the future, if other options use --pyproject-file.

I much prefer an option for the filename to one for the dir.

The pyproject.toml filename is standardized. It don't think it make sense to ask users to type it. It'd need to be included in every invocation of the option which makes it feel redundant. If we allowed alternative filenames, I'd feel differently.

This is still on my mind. I agree that it seems redundant, but it also is the most in-keeping with pip not having a built-in notion of projects, workflows, etc.

Does it make a difference that the proposed behavior would allow someone to pass something other than pyproject.toml? For example, you could use, under this proposed UX, the following:¹

pip install --group test --pyproject-file ./tox.toml

Maybe that's a point against this, but I want to note it and understand if it elicits a strong positive or negative response from anyone. The behavior as implemented today doesn't have to worry about the filename. I was thinking --pyproject-file would accept an arbitrary filename, and not validate that the basename is pyproject.toml.
If --pyproject-file is expected to check the filename, I want to clarify that now.

Footnotes

1. And it's probably a bad idea! But it's kind of interesting. ↩

[zanieb]

In general, I am only interested in implementing behaviors which are easy to reason about and don't contain avoidable ambiguities

I think this rules out (2) — I'm happy not to discuss that option further as suggested by @pfmoore but it's worth reiterating (as you have) that users will expect this and there should be warnings that guide them to the correct behavior.

Does it make a difference that the proposed behavior would allow someone to pass something other than pyproject.toml?

The idea that this would be allowed is what is most concerning to me about including the filename, especially since it's such a clearly defined standard (a file named pyproject.toml in the root directory of the project).

As a minor note regarding -p, this short flag is already pretty overloaded. In uv it's short for --python (selection of an interpreter) and we also have --package(selection of a workspace member package) and --project (selection of a project directory). I agree a short-flag would be nice to have here. -p is taken in uv though, we would not be able to support it.

As I was looking at the CLI to write the above, I realized we recently added --project to target a project directory in uv (astral-sh/uv#7603). That wasn't why I was pushing for it, but that does mean that it would be the best choice for compatibility — uv pip already supports it today.

[pfmoore]

I find this convincing in principle, but I worry that it makes for a very verbose CLI experience for interactive usage.

I know. That's the biggest drawback here. But I'd rather that we're verbose rather than confusing...

Is it reasonable to consider adding a short opt to make it easier?

I'm against that - while this is important functionality in the wider ecosystem, it's not well aligned to pip's core feature set, and I don't think it deserves one of the limited single-character option names.

Are you suggesting that the default be to read a PIP_PYPROJECT_FILE env var if one is set?

No, pip has a general feature that all command line options can be specified in a config file, or via an environment variable. So simply by having the command line option we automatically get the ability to set a default in those ways.

Does it make a difference that the proposed behavior would allow someone to pass something other than pyproject.toml?

I'm not comfortable about it. But to be fair, the implementation currently allows an invalid pyproject.toml (no [project] section, no [build-system] section). So this is just another aspect of that. If it matters, you could validate the argument to require it to have a filename of pyproject.toml.

[notatallshaw]

There's a lot of design discussion going on in this PR that to me seems to boil down to:

1. Should pip have a concept of a project?
2. And if so what should it assume the project structure looks like?

If the answer to 1 for this PR is "no", I'd like to point out that it doesn't stop pip adding a concept of a project in the future. If --pyproject-file is added now, it doesn't stop there being a future PR that makes pip "project aware"

And if the answer to 1 for this PR is "yes", then I'd like people to consider that the answer for 2 could affect a lot more than just this feature, for example if at some point pip reads it's own configuration out of pyproject.toml (i.e. #13003). So I would caution to consider this design quite carefully, looking at what other tools have done here and what challenges they've faced. I would be in favor of something as minimal and unopinionated about user workflows as possible, but I don't have a strong sense for what that is.

[sirosen]

1. Should pip have a concept of a project?
2. And if so what should it assume the project structure looks like?

If the answer to 1 for this PR is "no", I'd like to point out that it doesn't stop pip adding a concept of a project in the future. If --pyproject-file is added now, it doesn't stop there being a future PR that makes pip "project aware"

I'm inclined to answer (1) as "no", at least within the scope of this PR.
My intent was to propose a the most narrow version of this change that I could.

If a narrow change is not possible, I'd rather step back and make a more complete proposal which starts from the notion that pip will be aware of a "current project", to see if that gets traction. But, at least right now, I still believe that a simple and narrow version of this is possible.

Are you suggesting that the default be to read a PIP_PYPROJECT_FILE env var if one is set?

No, pip has a general feature that all command line options can be specified in a config file, or via an environment variable. So simply by having the command line option we automatically get the ability to set a default in those ways.

Ah, that explains why I didn't see the kind of env var logic I expected! I noticed the config loading PIP_* but didn't follow what it meant.

Upcoming update

For now, I plan to update the PR to implement --pyproject-file as required when --group is used, and with errors if either is used alone. I'll update the PR title as well, once I make the change.

I'll also validate that the filename provided is pyproject.toml. I see no strong reason not to validate this.

Idea: --pyproject-path

At the cost of some mildly more complex behavior, it would be possible to accept a path arg which can be the path to a pyproject.toml file or a directory. Thus allowing:

# equivalent
pip install --group foo --pyproject-path .
pip install --group foo --pyproject-path ./pyproject.toml

# equivalent
pip download --group bar --pyproject-path ./baz
pip download --group bar --pyproject-path ./baz/pyproject.toml

with helptext to the tune of

--pyproject-path   The path to a pyproject.toml file or a directory containing one.
                   Used to resolve `--group` options to `[dependency-groups]`.

Is this a good idea, worth pursuing? It adds some slightly more elaborate behavior for a mildly better user-experience. Because it's still described as "the file or dir containing the file", it doesn't give up the "file and path oriented" direction.

I'll update the PR without it for now, but am happy to switch to it if it seems like a good solution.

[pfmoore]

Is this a good idea, worth pursuing?

That works for me. The --python option uses similar logic.

[potiuk]

Is this a good idea, worth pursuing?

That works for me. The --python option uses similar logic.

I like it too. It also works nice as a building block of our pip guideline part for airflow contributors.

[notatallshaw]

I'll also validate that the filename provided is pyproject.toml. I see no strong reason not to validate this.

I still think pip should accept any filename for the reasons I outlined in #12963 (comment).

However given the path is explicit there is the obvious workaround of creating a directory and sticking a custom pyproject.toml there.

And nothing stops removing this validation in the future if opinion on this changes.

[potiuk]

I still think pip should accept any filename for the reasons I outlined in #12963 (comment).

Side comment @notatallshaw - this problem in "messy monorepo" could be solved by converting the "custom named" toml files into putting pyproject.toml file for each team in a separate sub-folder of that directory where you currently have custom named files (directories named same way as currrent files). I think that is way better approach - especially that IDEs and such already have "if pyproject.toml" use the right schema etc.

[henryiii]

I personally don't like the "any filename" support, as the PEP specifically is for pyproject.toml, and I can provide a bit of context from an identical feature in cibuildwheel: there, it looks for [tool.cibuildwheel] in pyproject.toml, but you can also pass a path to a config file. Unlike almost all (usually later) implementations of this, like hatch, pdm, tox, ruff, and uv, the "other" file still keeps the tool.cibuildwheel header, so it's simpler to parse and simpler to teach. However, by not using a fixed file name (like cibuildwheel.toml), there's no way for validation tools like those using SchemaStore to know that your file is actually a cibuildwheel configuration file. If I were to get a second chance, I'd require the file to be cibuildwheel.toml. And because most tools have not kept the same structure for their configuration, tool.<tool>.dependency-groups would be a top-level [dependency-groups] in their configuration, so I don't think you can just drop this into tox.toml or wherever. IMO, this was proposed specifically as a pyproject.toml feature, and multiple groups are allowed instead of multiple files.

Is this a good idea, worth pursuing?

I don't mind this, though. While it does allow someone to put a specific file in, it is optimized for the "correct" case following the PEPs.

Also, I second that subdirectories with pyproject.toml's would be better, IDEs and validators would like that at least.

[notatallshaw]

this problem in "messy monorepo" could be solved by converting the "custom named" toml files into putting pyproject.toml file for each team in a separate sub-folder of that directory where you currently have custom named files (directories named same way as currrent files).

Then we would have a clean monorepo 😛.

Sure, that's the goal, but technical nuances and resource capacity have so far got in the way. For fresh projects we do this and can use opinionated tools like poetry, but for the older projects we stick to tools that don't force specific workflows or project structure, and pip has always been excellent in this regard.

Anyway, I think I made my point in the linked post, and as long as there is an explicit path there is a workaround. I didn't mean to belabor it here.

[sirosen]

And nothing stops removing this validation in the future if opinion on this changes.

This, to me, is an important reason to include the requirements that the filename is pyproject.toml.

If pip supports only pyproject.toml today and is later loosened to allow use of other filenames, we won't expect this to break workflows. But if we start the other way around -- allowing any filename -- then adding the validation at a later date probably would break workflows.

So the better short-term choice, from a compatibility perspective, is to validate the name.

[zanieb]

Why include the -path suffix? That looks like it breaks from the naming scheme pip uses for flags that accept paths.

[sirosen]

It can be --pyproject, if that's more consistent. I don't have any desire to break with convention -- I'm genuinely looking for the least controversial path here! 😁

[pfmoore]

Personally I think --pyproject is too similar to --project. People will mistype it, and it pushes back to the whole idea of "pip should have a concept of a current project". I don't have anything new to say on the latter topic, so I'll just make this comment and leave it at that for now.

[zanieb]

The groups have to be read from a Python project so I don't think there's a way to implement --group support without pip having some concept of a current project (unless we do something like (2) — which has been ruled out).

[notatallshaw]

The groups have to be read from a Python project so I don't think there's a way to implement --group support without pip having some concept of a current project (unless we do something like (2) — which has been ruled out).

As I understand the building consensus, they have to be read from a pyproject.toml, but not a project¹ .

e.g. pip install -e ./my_project --pyproject-file /some/where/else/pyproject.toml --groups my-group. Which, as I understand it, would install the editable package in "my_project" and all it's dependencies, as well as the requirements defined by the group "my-group" in /some/where/else/pyproject.toml. Similiar to pip install -e ./my_project -r /some/where/else/requirements.txt.

Footnotes

1. For some definition of "project" that implies some structure ↩

[zanieb]

I don't quite see the difference. To me (and from uv's perspective), a Python project is simply a directory with a pyproject.toml in it — any additional structure is imposed by build tooling. The pyproject.toml is, by definition, the metadata for a Python project. Saying --pyproject-file /some/where/else/pyproject.toml still implies that there is a Python project we are inspecting.

I see the allure of requiring the path to the pyproject.toml file specifically: I think it might help with user confusion by reducing conflation with -e /project/path. Requiring the file path instead of a directory leads to additional confusion about what you should pass, so we're tacking -file or -path on the end of the flag. However, I don't think that's well aligned with the rest of pip, e.g.: it's --requirements not --requirements-file, --target not --target-dir, --editable not --editable-path, etc. I think the only exception to this is --cache-dir (which is actually a little confusing because then there's --no-cache-dir to disable the cache). It might be worth it to break from the usual pattern, it's hard to say. I may be biased because I feel it is excessively verbose to require the file instead of the directory.

Regardless, I feel like pip needs to introduce the concept of project awareness to implement this feature.

[notatallshaw]

I don't quite see the difference. To me (and from uv's perspective), a Python project is simply a directory with a pyproject.toml in it — any additional structure is imposed by build tooling. The pyproject.toml is, by definition, the metadata for a Python project. Saying --pyproject-file /some/where/else/pyproject.toml still implies that there is a Python project we are inspecting.

In my example I think the distinction is that -e ./my_project doesn't imply reading pyproject.toml out of ./my_project, and that there are no requirements for there to exist any other file in /some/where/else/ other than pyproject.toml, even regardless of the contents of pyproject.toml, as only the dependency groups are being read (at least for this feature).

Regardless, I feel like pip needs to introduce the concept of project awareness to implement this feature.

Then this conversation is regressing rather than moving forward, I posed that question directly in #13065 (comment) and it was answered "no" by the PR author in #13065 (comment) and other comments that followed seem to agree with this sentiment.

I also agree with the PR author that if the answer is "yes" then the design should be addressed in a seperate issue, and not discussed in the review of this PR

[pfmoore]

Regardless, I feel like pip needs to introduce the concept of project awareness to implement this feature.

As @notatallshaw pointed out, this would be a step backward. I've said all this before, but I think that introducing a concept of project awareness to pip is a big change, not just to the implementation details but to the underlying design. And at this point, I'm not even sure it's the right thing to do - we have loads of project-based tools, what distinguishes pip is precisely the fact that it doesn't require you to follow that model.

However, I think that being able to install dependency groups is a useful feature for pip, and it's something I absolutely expect users to request. So that leads me to the view that we need to find a way to express that without needing a project-based model. And saying "install this dependency group from this pyproject.toml file" feels like a natural way to do this.

I'm more than comfortable with a minor wart in how we name the option (including -path when we don't need to for other options) if that's what it takes to get this implemented without a big redesign of pip.

[pradyunsg]

I’m broadly in favour of the latest design, although I would like to hear the other maintainers’ views

I am as well. I haven't had the bandwidth to review the implementation though!

[potiuk]

I’m broadly in favour of the latest design, although I would like to hear the other maintainers’ views

I am as well. I haven't had the bandwidth to review the implementation though!

Just FYI - we'd love to get this in for Aiflow. We are heavily modernising our packagins setup in preparation for Airlfow 3 (and we have a very complex setup with eventually ~ 100 projects in a single monorepo that we will manage) - and we have now setup wher we use uv as our "ideal" development tooling (i.e. one with the least friction - with features like Python installation and workspaces) - but being an Apache Software Foundation project, we are vary of not being too dependent on projects that are not under some Foundation / Open Source Steward umbrella - so we make sure that our users can do whatever they need to do with other PEP-compliant packaging/building tools. And pip is our "golden standard" - all our docs explain how you can work with our monoroepo with both uv and pip.

Since PEP for that one is approved - we would love to entirely move to have - for example - our dev dependencies to be kept as dependency groups (rather than as we have it as dynamic editable extras), And lack of support for it in pip holds us back from doing it fully (we have now both dependency groups defined for uv and editable extras have to by dynamic to include devel dependencies).

[potiuk]

It's not a huge blocker, of course, more "nice to have" - but if there is anything we can do help with it, let us know please.

[pfmoore]

@sirosen are you going to add that documentation discussed above? Assuming you do, I'm inclined to merge this - it's been around for long enough that if anyone had any major issues, they would have raised them.

@sbidoul do you have any objections to this being in 25.0 if @sirosen gets the docs added? We can wait till 25.1 if the short notice is a concern.

[sbidoul]

+1

[sbidoul]

I'll release 25.0 this weekend.

I'm fine getting this in, but perhaps should we then label it as experimental for one cycle, so we have some latitude to postpone possible bug fixes (if any) to the next release?

[sirosen]

@sirosen are you going to add that documentation discussed above?

Yes, I have every intention of doing so. I had an unfortunate personal/medical matter come up, so I haven't been able to spend any time at a computer for a while.

[pfmoore]

I'll release 25.0 this weekend.

Ah, that's a bit sooner than I realised. I suggest we defer until 25.1 in that case, both to avoid a last minute merge and to take any pressure off @sirosen.

[sirosen]

I've rebased and added an initial pass at a doc (this commit for the diff).

[pfmoore]

LGTM!