add RaisesGroup & Matcher

[jakkdl]

Previous PR's: #11656 (ExpectedExceptionGroup), #11671 (closed to give this a fresh PR w/o stale comments)
fixes: #11538 #12504

This adds RaisesGroup (also exported as raises_group) and Matcher, to allow a robust way of expecting ExceptionGroup and the ability to specify the structure of nested groups, etc etc.

with RaisesGroups(ValueError):
    raise ExceptionGroup("", (ValueError(),))

This is the exact implementation available in trio.testing.RaisesGroup which has been in use since december 2023 and gone through a couple iterations since to settle on a good interface. The most recent addition was python-trio/trio#3145 which added error messages, where the formatting can probably still be fine-tuned.

Pytest currently has excinfo.group_contains for checking exceptiongroups... which is very problematic. I think it should be deprecated, either in the same release as this PR, or very soon after.

This currently does not touch the implementation of pytest.raises, but if we made Matcher a contextmanager we could easily support

with Matcher(ValueError):
    raise ValueError()

this would fix e.g. #12763 with Matcher having the check parameter. The only complication is the legacy form of pytest.raises but that should be solvable with a small helper function.
This can however be done in a followup PR.

This currently does not add an .assert_matches method, which previous iterations of the PR had. The reason to add that would be to improve the error message on failure, though it's possible to get the same result with

assert (m := Matcher(TypeError)).matches(e), m.fail_reason
# versus
Matcher(TypeError).assert_matches(e)

though note that .matches() is already a fairly niche feature that is entirely separate to the context-manager use (not to be confused with the match parameter).

TODO:

• bump the issue # on the newsfragment file
• documentation

[RonnyPfannschmidt]

We ought not to use such a generic name easily

There's a general wishlist plan to enable matchers as a whole that act similar to dirty equals

The matcher api that's being used here conflates multiple aspects of that in a manner that's problematic to partition later

Ideally the whole api around matchers is first evolved as a plugin to avoid early api lock in as it's hard to make good matchers

I'll elaborate further when the flu is over

[jakkdl]

sure, I can rename it... maybe something that's closer to RaisesGroup... RaisesExc/RaisesException? or something like ExpectedExc/ExpectedException

This has been available & in use for >1 year in trio, which has allowed the interface to mature. I don't think pytest needs to repeat that process by putting it in a plugin.

I know you've mentioned matcher objects in #11538 and I'm excited for what that might entail in the future, but I don't think we want to wait for that to add usable exceptiongroup support. If you have concrete ways of improving the RaisesGroup interface to harmonize with it I'm of course open to suggestions though.

[Zac-HD]

I've been heavily involved in the design and review for this feature as we've developed and shipped it in Trio. I'm excited to get it into Pytest (and ideally soonish), but I shouldn't be the only approving core dev for a substantial feature.

[jakkdl]

@Zac-HD please review the documentation :)

It feels a bit silly to add big warning boxes to ExceptionInfo.group_contains and not deprecate it, but it also feels silly to deprecate it one release after adding it.

[Zac-HD]

@Tusenka unfortunately pull requests are the wrong place to ask questions like this, unless you're either the author of the PR or a maintainer of the package. I suggest StackOverflow instead.

[jakkdl]

please bring basic questions like these to StackOverFlow, the python discord, ask an LLM of your choice, or otherwise. This is not relevant to the PR at hand and only wastes the time of maintainers without any benefit to pytest.

[jakkdl]

I think I'm mostly just waiting for a quick pass on docs changes from @Zac-HD (or others), and otherwise this can be merged and I can look at potential followup for pytest.raises and/or deprecating excinfo.group_contains

[nicoddemus]

Thanks a lot @jakkdl for the great work here!

I'm OK with us having pytest.RaisesGroup and it being different than pytest.raises, you have made your points and since nobody else has given their opinions, I guess it is no big deal in the end.

However I think we should reconsider also exposing pytest.raises_group, as having both ways to do the exact same thing might not be a net positive? So I propose just going ahead and having only pytest.RaisesGroup, leaving out the pytest.raises_group alias.

[nicoddemus]

Hi @jakkdl, thanks for sticking with it so far!

Can you just fix the conflict please? Then I will be happy to merge. 👍

[jakkdl]

lel, amusing how big a conflict you can get from a simple be2600e

I'll merge before it happens again! 🎉 🚀 🎉 🚀

[jakkdl]

ugh I just noticed this didn't quite have 100% patch coverage. We should enforce that in config

[nicoddemus]

Thanks @jakkdl.

In the future, please consider squashing the commits before merging, as this PR introduced a lot of commits which don't make sense separately.

See https://github.com/pytest-dev/pytest/blob/main/CONTRIBUTING.rst#mergesquash-guidelines. 👍

[jakkdl]

oh shit, I'm way too used to squashing being the default

do you want me to squash them on master (if I can?), or will you do that?

[RonnyPfannschmidt]

It's already on main and main is protected

[nicoddemus]

Yeah, unfortunately now it has landed on main so we cannot change it, as a Git good practice.

Not the end of the world, but something to take care in the future. Thanks again!

[bluetech]

Let's make match and check kw-only arguments (also in the real __init__ below)

[jakkdl]

Fixed in af1c472 in #13279

[RonnyPfannschmidt]

i just noted that we have a important dissonance - the raises contextmanager is reentreant, the RaisesExc one isnt

[jakkdl]

Are you referring to a failed match now raising AssertionError instead of letting the original exception through unchanged (reverted in #13279 (comment)), or something else?

[jakkdl]

this does fail:

    r = pytest.raises(ValueError)
    with r:
        with r:
            raise ValueError
        raise ValueError

but that was an error previously as well.

For reuse this does work:

    r = pytest.raises(ValueError)
    with r:
        raise ValueError
    with pytest.raises(Failed):
        with r:
            ...
    with r:
        raise ValueError

probably worth adding tests for those

[RonnyPfannschmidt]

I'm referring to the state of the contextmanager instances

[jakkdl]

yes there's state - but afaik there's no relevant state that's not reset upon doing a new __enter__ or matches() call. Do you have an example that demonstrates the change in behavior?

[jakkdl]

@RonnyPfannschmidt ?