fix: numpy 2.0 compatibility

[lgray]

This uses latest numpy in the build system, oldest possible numpy is no longer a correct choice.

Checked on laptop with numpy 1 and this package built against numpy 2.

[lgray]

@tpgillam whenever you get a chance to pull this in and make a new release, it would be appreciated!

[tpgillam]

Thanks for the contribution @lgray!

Looking at the state of the tests, I think this package is overdue a little spring cleaning first. I’ll get to that when I can, then will be great to have numpy 2 compatibility working.

Assorted list of what needs doing:

• Extend test suite to make sure we test with a (minimal numpy, latest 1.x, latest 2.x)
• Extend support up to python 3.12
• Test and build wheels for for macos arm (or universal? Will need to check current recommendations)
• (Optionally) migrate everything over to pyproject.toml

[tpgillam]

Also, as far as I can tell, oldest-supported-numpy is still maintained and recommended.

https://pypi.org/project/oldest-supported-numpy/

What breaks for you with it?

[lgray]

https://github.com/scipy/oldest-supported-numpy

Please have a look at the deprecation notice on their GitHub. It might be on pypi but it is no longer the right way to do things, and does indeed cause problems. There's no forwards compatibility. As given the package won't work when numpy 2 is installed.

I can update your CI as well to use the presently supported range of python 3s. Ah, it's on your list upstairs. Checking against numpy1 should be easy, you do the build, uninstall numpy 2, install numpy 1 and run tests again.

[lgray]

It looks like your oldest supported numpy is going to be 1.26, given that you would like to support python 3.12.

[lgray]

@tpgillam that should be most of the spring cleaning :-)

[lgray]

The only thing I need to add is to test with numpy 2 and numpy 1

[tpgillam]

Just drive-by commenting, don't have time to look in detail at your work right now, sorry!

https://github.com/scipy/oldest-supported-numpy

Thanks for the reference! I'll read this in more detail later.

It looks like your oldest supported numpy is going to be 1.26, given that you would like to support python 3.12.

I feel like it should be possible to support older versions when using a suitable python interpreter. I'm inclined to be as protective as possible towards users that might be on older systems.

[lgray]

and there we go.

[lgray]

The answer that users will get from the python core developers for anything less than python 3.9 is "that version of python is no longer supported". You're pretty well situated to rattle things around and tell people to update their systems.

The oldest numpy version supported in python 3.12 is 1.26, the newest numpy version supported in python 3.9 is 2.0.1. It's rather cut and dried as to what to do if you would like to support python 3.12.

[lgray]

Ah lovely - I can fix up the problems so long as you let the CI run. Thanks!

[Rupt]

Thanks for your diligent contributions @lgray.

I can't comment on the implementation, but your statements about NumPy compatibility surprise me, because they contradict NumPy's support specification https://scientific-python.org/specs/spec-0000/ (forwarded from https://numpy.org/neps/nep-0029-deprecation_policy.html).

I can sadly confirm that you are correct: NumPy versions <1.26 does not support Python 3.12:

PYTHON_BUILD=cpython-3.12.4+20240726-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz
wget https://github.com/indygreg/python-build-standalone/releases/download/20240726/${PYTHON_BUILD}
tar xvf ${PYTHON_BUILD}
python/bin/python --version  # Python 3.12.4
python/bin/pip install numpy==2.0  # ok
python/bin/pip install numpy==1.26  # ok
python/bin/pip install numpy==1.25  # error
python/bin/pip install numpy==1.24  # error

[lgray]

Ah, you're right @Rupt with respect to the numpy support window. The issue comes when you want to support 2 and 3.12 simultaneously. I don't have a machine where I can test numpy 2 vs. much older numpy (arm macos machine right now), but if you really need it we can support back to 1.22 (and I think you can do that compiling with 2).

However, there's not too much of a good reason to do that, IMO, considering current environment management practices.

[tpgillam]

@lgray thanks for your continuing efforts to get the CI working! Wondering whether it would be easier for you to create a PR into your own fork such that you can run the CI yourself for testing (at least I presume that would work)? I'm happy to press 'approve & run' when I see it, but maybe you'd prefer to have faster iteration times.

[lgray]

A bunch of the failures in the wheel CI are from the wheels already existing since you seem to be uploading even in PR! A little dangerous.

I'll make it check for a release tag.

[lgray]

Oh, nevermind, that's just the github artifacts.

[tpgillam]

I'll make it check for a release tag.

We have this check:

mt2/.github/workflows/ci.yml

Line 110 in 646818c

if: github.event_name == 'push' && startsWith(github.event.ref, 'refs/tags/v')

maybe it can be made better.

[lgray]

@tpgillam could you have a look in github and see if you can kill the uploaded artifacts for this PR?

[lgray]

I've changed the CI so it only uploads artifacts when there's a version tag. So I think this is almost there.

[lgray]

ok - nice - it works back to numpy 1.19.3 when built against numpy 2, according to the tests in python 3.9. Have adjusted all the pins.

[lgray]

Indeed, running the CI over on my own fork everything passes now, and we correctly test numpy back to oldest supported in 3.9.

https://github.com/lgray/mt2/actions/runs/10186047164/job/28176969069?pr=1

[tpgillam]

Thanks @lgray . github doesn't list any persisted artifacts for this workflow, so let's see what happens in this CI run.

Could you please bump the version in all appropriate places to 1.2.1 and add entry in HISTORY so that we can then tag once merged?

[lgray]

looks like it's all good to go aside from your github options expecting a job simply called "Build"!

[lgray]

let me make a meta task with that name that just requires all the other jobs to pass, then it'll change as the matrix changes... sorry if I deleted that earlier.

[lgray]

OK - @tpgillam added two things and I think this is done.

1. Build / pass job label represents all jobs passing, you'll need to add that to your repository config as the expected check, instead of "Build"
2. as commits are added to a PR if there are previously running CI jobs on the same PR it will cancel them, better for GHA resources

[tpgillam]

Thanks @lgray , this looks great!

The required checks on github are baffling me. I wish they were configurable more sensibly, but hey-ho. I've tried but so far failed to get it to match Build / pass or Build / pass (pull request).

edit: apparently the magic was just to require pass on the github side.

[lgray]

Cool, thanks a lot for dealing with all the churn. :-)

Whenever you're ready, please merge and make a release!