Drop NEP 29 in favor of official python version support

[tokoko]

This is a continuation of a discussion started here in Zulip. Ibis is currently following NEP 29 python version deprecation policy, which means ibis is dropping python version support sooner than versions reach official EOL. For example, ibis has already dropped support for python 3.9 which reaches EOL a year from now in 2025-10. This is much faster than most of the engines that ibis supports as backends, all of the following: duckdb, polars, datafusion, pyspark and pyarrow have yet to drop python 3.8 which reaches official EOL next month.

This accelerated lifecycle might be a problem for both users and other libraries that plan on depending on ibis for core functionalities. Ibis is a fast-evolving library with somewhat frequent major version upgrades and dependent libraries are in practice forced to mirror similar versioning policy not to fall too far behind. For context, such a library which aims to also follow the official python version statuses and has yet to drop python 3.8 would now be forced to depend on ibis-framework 5.1.0, Next month when 3.8 reaches EOL, it would bump 4 major versions at once to ibis-framework 9.0.0 and still not be on latest and greatest.

[cpcloud]

@tokoko Thanks for opening this issue.

It has been nice as a developer of to not have to deal with a large range of Python versions, but as adoption keeps increasing people are going to start using Ibis in environments that are more constrained in how fast they can upgrade.

I think there are some predictable outcomes here if we followed the official Python release schedule:

• More money spent on CI, because we'd have to upgrade to GH enterprise so that we have enough queuing capacity to run at least one more python version in the build matrix.
• There will more work to maintain things, not clear how much.

And some harder-to-predict outcomes:

• What features do we have if any, that are Python-version dependent (hopefully none)?. I suspect there might be some use of typing that doesn't work in 3.8 and 3.9.
• How much extra work does this support add?

[cpcloud]

Actually, I guess in this case 3.8 EOL is next month, so it's very unlikely we'd add back support for that.

[tokoko]

@cpcloud thanks for the reply.

• More money spent on CI, because we'd have to upgrade to GH enterprise so that we have enough queuing capacity to run at least one more python version in the build matrix.

This is probably true, but I'd just point out that this also depends on an additional choice of what versions we actually want to test in ci. At any given point in time there are 5 supported python versions if you're following official deprecations, while per NEP 29 you have 3. In an ideal world you would probably be testing all of them for each test, but IIUC even now ibis is mostly testing only lowest and highest versions for most tests (3.10 and 3.12 now). One option, admittedly not ideal, would be to keep that policy even with official versions, so starting from next month we would be testing only 3.9 and 3.13. In this case there should be no changes in costs. Alternatively we could add a middle version in the mix as well just to be more sure (3.9, 3.11 and 3.13).

[cpcloud]

I'm not too worried about the monetary cost, but yeah, we could adopt the same strategy for backend testing, and then run the all versions on our non-backend, fast, core test suite which covers the expression API. This is what we are doing now as well, so maybe not too much would change in terms of CI.