DEV: Evaluate dependency update concept for CI

[stefan6419846]

At the moment, the requirement files used by the CI are only updated partially and if required. The same applies to the pre-commit configuration.

There have been some related issues where I stumbled upon such cases:

• In DEV: Ensure testing against the latest versions #2455, I would like to ensure that our package does not break with the latest package changes as it did for pytest some weeks ago and for Pillow with yesterdays' release.
• In DEV: Remove unused dependencies #2572, I stumbled upon some dependencies which nobody is using anymore and thus could be cleaned.
• In DEV: Review pinned requirements in include files #2573, I had a look at explicitly pinned packages, where the reason sometimes is not documented appropriately. Most of these just require some unpinning and small fixes. But there also is the pre-commit framework which has a separate set of updates to maintain as well, although I do not think that this is actively used indeed(?)

I would appreciate if we would come up with an appropriate process for handling such cases and document it accordingly. Some immediately apparent issue might already be that updating the requirement files is not very easy due to requiring different Python versions on a system, which requires additional local setups and might depend on the OS/distribution.

[MartinThoma]

I stumbled upon some dependencies which nobody is using anymore and thus could be cleaned.

A partial solution for that might be to comment explicitly why a dependency is there within the requirements.in file.

[MartinThoma]

I do use pre-commit and I think it's pretty valuable. We could also run pre-commit in CI. That could even replace the way we run ruff / mypy.

To me pre-commit is valuable as it's a well-known way to apply linting rules before CI. To me as a contributor, it's nicer to fail earlier (and even nicer if auto-fixes are applied).

At the same time, if somebody does not know pre-commit it's not necessary. CI will ensure the coding standards are fulfilled.

[MartinThoma]

requiring different Python versions

I was thinking about dropping the next Python version at the end of 2024: #2418

I'm not sure if that makes things simpler, though.

[stefan6419846]

A partial solution for that might be to comment explicitly why a dependency is there within the requirements.in file.

This is correct (or at least inside the history to allow blaming with an additional issue to track the progress of resolving this). An ideal solution to this would still be to have no pins at all and ensure that each update applies cleanly similar to https://github.com/stefan6419846/pypdf_recent (although I am ignoring linting issues there for now due to some larger changes enforced by ruff).

I do use pre-commit and I think it's pretty valuable. We could also run pre-commit in CI. That could even replace the way we run ruff / mypy.

The standalone calls in CI are completely fine in my opinion as this separates the different tools in a clean manner through separate steps. We should still keep in mind that the versions there need some love as well ;)

I was thinking about dropping the next Python version at the end of 2024: #2418

I'm not sure if that makes things simpler, though.

I do not think so unless we are going to drop both Python 3.7 and 3.8 at the end of the year. In October 2024, Python 3.8 will go EOL as well and Python 3.13 is planned to be released: https://devguide.python.org/versions/#supported-versions With the current approach, we are always lagging one version behind, possibly complicating dependency handling in CI due to newer releases not supporting Python 3.7 any more for example (like recent Pillow versions).

[j-t-1]

Although currently in beta, I think it is possible to add 3.13 to tests now.

[pubpub-zz]

Although currently in beta, I think it is possible to add 3.13 to tests now.

At some time also it would be interesting to test python 3.13 with NOGIL

[stefan6419846]

I have just added Python 3.13 to our test matrix. AFAIK the new GIL handling requires a build parameter to be set and only affects threading, which (if ever) would be relevant for the parallel test runner in CI only, but not for regular pypdf usage.