chore: remove support for python 3.9

[felixscherz]

Description

Hi, this is removing support for python 3.9 and updates the python syntax accordingly. Intended to resolve #818.

Development notes

• removed typing.Union in favor of | operator
• removed typing.Callable in favor of collections.abc.Callable
• removed python 3.9 from unit test matrix in CI

Checklist

• Opened this PR as a 'Draft Pull Request' if it is work-in-progress
• Updated the documentation to reflect the code changes
• Added a description of this change in the relevant RELEASE.md file
• Added tests to cover my changes

[deepyaman]

I think can also simplify https://github.com/felixscherz/kedro-plugins/blob/chore/remove-python-3.9/kedro-datasets/pyproject.toml#L230-L231; not sure if there are other such places.

On a separate note (and this is in no way a reflection on this PR, which resolves the issue very well), I've started thinking more about #695, especially

Is it necessary to drop 3.9 support when it's compatible and not causing any issue?

While I was a big proponent of NEP-29, some experience on another project makes me wonder if we should hold off a bit, especially given almost nothing of value is gained through these changes (the typing changes are nice, but don't have any real impact...). I wouldn't be opposed to holding off a major version, unless somebody can point to more bad things we're doing right now to support 3.9.

[felixscherz]

Thanks for having a look, good catch with the environment markers! I'll remove the one you referenced which seems to be the only one that deals with python 3.9.
I would agree with you that removing support is not actually necessary as long as there are no major workarounds required and python 3.9 is still officially supported.

[merelcht]

Thanks for the contribution @felixscherz ! I'm still in favour of adhering to NEP-29, which is what we agreed on recently in backlog grooming as well.

@astrojuanlu I'm guessing you also still want to go ahead with this?

[astrojuanlu]

Agreed to continue adhering to NEP 29 indeed! Thanks @felixscherz for this PR 🙌🏼

Approving to signal my acceptance although naturally the CI will need to be green

[felixscherz]

Thanks for the review! I'm having a look at the CI failures, it seems like they are failing on the main branch as well.

EDIT: Seems like this is related to kedro itself removing support for python 3.8 kedro-org/kedro#4212.
Some of the CI jobs are installing kedro from HEAD: https://github.com/kedro-org/kedro-plugins/actions/runs/11235726053/job/31234293125?pr=870#step:8:3

[kevin1kevin1k]

@felixscherz Hey I am the author of kedro-org/kedro#4212.
I noticed that under .github/workflows you modified kedro-datasets.yml.
According to the error patterns I guess the CI errors will be gone after modifying kedro-{airflow, docker, telemetry}.yml as well.

[felixscherz]

@kevin1kevin1k thank you! I opened an issue to deal with the removal of python 3.8: #872, we need to remove it from github actions but also make updates to the python syntax and update the pyrpoject.toml files, I'm working on those PRs at the moment:)