[pytest] New rule idea- Prefer multiple pytest.param(..., id=...) over pytest.mark.parametrize(..., ids=...)

[UnknownPlatypus]

Hi !

This is a new rule suggestion for the pytest-style plugin.

It checks for pytest.mark.parametrize calls using the ids keyword argument and replace each set of test values with a pytest.params(.., id=...) parameter set, effectively binding the id with the set of arguments.

When using the ids=[…] syntax, each id is bound to a set of parameters by index.

This is quite error prone and makes test hard to modify (adding a test case in the middle means adding an id at the corresponding index in the ids list). Moreover, if one test case fails, it's not directly obvious which set of parameters is associated with the failing id. This issue gets worse the more test case you have in your parametrize call.

Pytest seems to agree this is not a good pattern:

In test_timedistance_v1, we specified ids as a list of strings which were used as the test IDs. These are succinct, but can be a pain to maintain.

An autofix would look like this:

Don't use:

@pytest.mark.parametrize(
    "a, b, expected",
    [
        (datetime(2001, 12, 12), datetime(2001, 12, 11), timedelta(1)),
        ... # 10 other test cases
        (datetime(2001, 12, 11), datetime(2001, 12, 12), timedelta(-1)),
    ],
    ids=["forward", ..., "backward"],
)
def test(a, b, expected):
    pass

Use instead:

@pytest.mark.parametrize(
    "a, b, expected",
    [
        pytest.param(
            datetime(2001, 12, 12), datetime(2001, 12, 11), timedelta(1), id="forward"
        ),
        ... # 10 other test cases
        pytest.param(
            datetime(2001, 12, 11), datetime(2001, 12, 12), timedelta(-1), id="backward"
        ),
    ],
)
def test(a, b, expected):
    pass

I've already implemented such fixer in python (using pyupgrade/django-upgrade building blocs) and would be very interested in porting this to ruff.

If you have in mind another MR I could take inspiration from (especially the auto-fix part) it would be awesome!
Also I'm not sure which code to use, PT028 could overlap with a potential new pytest-style rule

[MichaReiser]

Thanks @UnknownPlatypus for the nice write up.

For now, we only rarely add new rules that don't exist in the upstream linter (in this case, pytest-style). Do you know if your lint has been proposed and discussed for flake8-pytest

[UnknownPlatypus]

@MichaReiser To my knowledge, it has not been discussed in flake8-pytest and I intentionally didn't open an issue there because this is the kind of rule that would be extremely cumbersome to manually fix (hence the reason I wrote an autofixer for it only).

But I can open an issue there if you want to get some more input from the pytest community that might be more present there ?

[TomerBin]

Thank you @UnknownPlatypus !! This rule sounds amazing