[BUG] Completely broken on numpy 2.x (wrong vector sent to fitness())

[samimia-swks]

With numpy 2.x, the fitness function is called with the wrong values in the decision vector x.

import pygmo as pg
class sphere_function:
    def fitness(self, x):
        cost = sum(x * x)
        print(f"FITNESS EVAL: x = {x}, cost = {cost}")
        return [cost]

    def get_bounds(self):
        return ([-1] * 3, [1] * 3)
prob = pg.problem(sphere_function())
pop = pg.population(prob, 5)
print(pop)

The example above results in the following. Notice how the population member #0 is [-0.504554, 0.87294, 0.639481] but the fitness function gets [-0.50455375 -0.50455375 -0.50455375].

FITNESS EVAL: x = [-0.50455375 -0.50455375 -0.50455375], cost = 0.7637234652148235
FITNESS EVAL: x = [0.4426457 0.4426457 0.4426457], cost = 0.5878056348538054
FITNESS EVAL: x = [-0.47532954 -0.47532954 -0.47532954], cost = 0.6778145209225833
FITNESS EVAL: x = [-0.29921424 -0.29921424 -0.29921424], cost = 0.2685874763420541
FITNESS EVAL: x = [0.63765334 0.63765334 0.63765334], cost = 1.219805338218631
Problem name: <class '__main__.sphere_function'>
        C++ class name: pybind11::object

        Global dimension:                       3
        Integer dimension:                      0
        Fitness dimension:                      1
        Number of objectives:                   1
        Equality constraints dimension:         0
        Inequality constraints dimension:       0
        Lower bounds: [-1, -1, -1]
        Upper bounds: [1, 1, 1]
        Has batch fitness evaluation: false

        Has gradient: false
        User implemented gradient sparsity: false
        Has hessians: false
        User implemented hessians sparsity: false

        Fitness evaluations: 5

        Thread safety: none

Population size: 5

List of individuals: 
#0:
        ID:                     14351618211829196000
        Decision vector:        [-0.504554, 0.87294, 0.639481]
        Fitness vector:         [0.763723]
#1:
        ID:                     4233680892357765143
        Decision vector:        [0.442646, 0.462569, -0.970743]
        Fitness vector:         [0.587806]
#2:
        ID:                     457977541108690035
        Decision vector:        [-0.47533, 0.97677, -0.805264]
        Fitness vector:         [0.677815]
#3:
        ID:                     3780462174932513958
        Decision vector:        [-0.299214, -0.134968, -0.699614]
        Fitness vector:         [0.268587]
#4:
        ID:                     12888162120884326344
        Decision vector:        [0.637653, 0.840588, -0.0665339]
        Fitness vector:         [1.21981]

Champion decision vector: [-0.299214, -0.134968, -0.699614]
Champion fitness: [0.268587]

On numpy 1.x, we get :

FITNESS EVAL: x = [-0.38570461 -0.2763677  -0.19182114], cost = 0.2619425019598244
FITNESS EVAL: x = [-0.20674582 -0.56690414 -0.68899929], cost = 0.8388441636353365
FITNESS EVAL: x = [-0.57724158  0.56498671  0.34808134], cost = 0.77357844517818
FITNESS EVAL: x = [-0.13960828  0.79237103  0.04465593], cost = 0.6493364798439734
FITNESS EVAL: x = [-0.19441917  0.14853349  0.61230586], cost = 0.4347794724943631

Population size: 5

List of individuals: 
#0:
        ID:                     4942601127234171947
        Decision vector:        [-0.385705, -0.276368, -0.191821]
        Fitness vector:         [0.261943]
#1:
        ID:                     11313699314120481030
        Decision vector:        [-0.206746, -0.566904, -0.688999]
        Fitness vector:         [0.838844]
#2:
        ID:                     8331669631110857304
        Decision vector:        [-0.577242, 0.564987, 0.348081]
        Fitness vector:         [0.773578]
#3:
        ID:                     3796521727361963477
        Decision vector:        [-0.139608, 0.792371, 0.0446559]
        Fitness vector:         [0.649336]
#4:
        ID:                     18100786683321109976
        Decision vector:        [-0.194419, 0.148533, 0.612306]
        Fitness vector:         [0.434779]

• OS: ubuntu
• Installation method: uv
• Version: v2.19.5

[darioizzo]

CanNOT reproduce on osx (M1 arch): py313 numpy 2.2.0 ... this env: all looks fine, code behaves as expected

# Name                    Version                   Build  Channel
ampl-asl                  1.0.0                h286801f_2    conda-forge
appnope                   0.1.4              pyhd8ed1ab_1    conda-forge
asttokens                 3.0.0              pyhd8ed1ab_1    conda-forge
bzip2                     1.0.8                h99b78c6_7    conda-forge
ca-certificates           2024.12.14           hf0a4a13_0    conda-forge
cloudpickle               3.1.0              pyhd8ed1ab_1    conda-forge
colorama                  0.4.6              pyhd8ed1ab_1    conda-forge
comm                      0.2.2              pyhd8ed1ab_1    conda-forge
debugpy                   1.8.11          py313h928ef07_0    conda-forge
decorator                 5.1.1              pyhd8ed1ab_1    conda-forge
exceptiongroup            1.2.2              pyhd8ed1ab_1    conda-forge
executing                 2.1.0              pyhd8ed1ab_1    conda-forge
icu                       75.1                 hfee45f7_0    conda-forge
importlib-metadata        8.5.0              pyha770c72_1    conda-forge
ipopt                     3.14.16             h3e4dc2c_11    conda-forge
ipykernel                 6.29.5             pyh57ce528_0    conda-forge
ipyparallel               9.0.0              pyhd8ed1ab_1    conda-forge
ipython                   8.30.0             pyh707e725_0    conda-forge
jedi                      0.19.2             pyhd8ed1ab_1    conda-forge
jupyter_client            8.6.3              pyhd8ed1ab_1    conda-forge
jupyter_core              5.7.2              pyh31011fe_1    conda-forge
krb5                      1.21.3               h237132a_0    conda-forge
libblas                   3.9.0           25_osxarm64_openblas    conda-forge
libboost                  1.86.0               hc9fb7c5_3    conda-forge
libcblas                  3.9.0           25_osxarm64_openblas    conda-forge
libcxx                    19.1.5               ha82da77_0    conda-forge
libedit                   3.1.20191231         hc8eb9b7_2    conda-forge
libexpat                  2.6.4                h286801f_0    conda-forge
libffi                    3.4.2                h3422bc3_5    conda-forge
libgfortran               5.0.0           13_2_0_hd922786_3    conda-forge
libgfortran5              13.2.0               hf226fd6_3    conda-forge
libhwloc                  2.11.2          default_hbce5d74_1001    conda-forge
libiconv                  1.17                 h0d3ecfb_2    conda-forge
liblapack                 3.9.0           25_osxarm64_openblas    conda-forge
liblzma                   5.6.3                h39f12f2_1    conda-forge
libmpdec                  4.0.0                h99b78c6_0    conda-forge
libopenblas               0.3.28          openmp_hf332438_1    conda-forge
libscotch                 7.0.5                hc2b2845_4    conda-forge
libsodium                 1.0.20               h99b78c6_0    conda-forge
libsqlite                 3.47.2               h3f77e49_0    conda-forge
libxml2                   2.13.5               h178c5d8_1    conda-forge
libzlib                   1.3.1                h8359307_2    conda-forge
llvm-openmp               19.1.5               hdb05f8b_0    conda-forge
matplotlib-inline         0.1.7              pyhd8ed1ab_1    conda-forge
metis                     5.1.0             h15f6cfe_1007    conda-forge
mumps-include             5.7.3                hce30654_5    conda-forge
mumps-seq                 5.7.3                he17653c_5    conda-forge
ncurses                   6.5                  h7bae524_1    conda-forge
nest-asyncio              1.6.0              pyhd8ed1ab_1    conda-forge
networkx                  3.4.2              pyh267e887_2    conda-forge
nlopt                     2.9.0           py313hf67b645_0    conda-forge
numpy                     2.2.0           py313ha4a2180_0    conda-forge
openssl                   3.4.0                h39f12f2_0    conda-forge
packaging                 24.2               pyhd8ed1ab_2    conda-forge
pagmo                     2.19.1               hc1711db_4    conda-forge
parso                     0.8.4              pyhd8ed1ab_1    conda-forge
pexpect                   4.9.0              pyhd8ed1ab_1    conda-forge
pickleshare               0.7.5           pyhd8ed1ab_1004    conda-forge
pip                       24.3.1             pyh145f28c_1    conda-forge
platformdirs              4.3.6              pyhd8ed1ab_1    conda-forge
prompt-toolkit            3.0.48             pyha770c72_1    conda-forge
psutil                    6.1.0           py313h63a2874_0    conda-forge
ptyprocess                0.7.0              pyhd8ed1ab_1    conda-forge
pure_eval                 0.2.3              pyhd8ed1ab_1    conda-forge
pybind11-abi              4                    hd8ed1ab_3    conda-forge
pygments                  2.18.0             pyhd8ed1ab_1    conda-forge
pygmo                     2.19.7          py313h6e67849_3    conda-forge
python                    3.13.1          h4f43103_102_cp313    conda-forge
python-dateutil           2.9.0.post0        pyhff2d567_1    conda-forge
python_abi                3.13                    5_cp313    conda-forge
pyzmq                     26.2.0          py313h0e8b002_3    conda-forge
readline                  8.2                  h92ec313_1    conda-forge
scipy                     1.14.1          py313hc010ede_2    conda-forge
six                       1.17.0             pyhd8ed1ab_0    conda-forge
stack_data                0.6.3              pyhd8ed1ab_1    conda-forge
tbb                       2022.0.0             h0cbf7ec_0    conda-forge
tk                        8.6.13               h5083fa2_1    conda-forge
tornado                   6.4.2           py313h90d716c_0    conda-forge
tqdm                      4.67.1             pyhd8ed1ab_0    conda-forge
traitlets                 5.14.3             pyhd8ed1ab_1    conda-forge
typing_extensions         4.12.2             pyha770c72_1    conda-forge
tzdata                    2024b                hc8b5060_0    conda-forge
wcwidth                   0.2.13             pyhd8ed1ab_1    conda-forge
zeromq                    4.3.5                hc1bb282_7    conda-forge
zipp                      3.21.0             pyhd8ed1ab_1    conda-forge
zstd                      1.5.6                hb46c0d2_0    conda-forge

Here is the output of the code above from @samimia-swks:

FITNESS EVAL: x = [-0.21025549  0.88580692 -0.46590172], cost = 1.0459256824558625
FITNESS EVAL: x = [-0.46672973  0.12356347  0.1057874 ], cost = 0.2442955502367031
FITNESS EVAL: x = [ 0.77906405 -0.61535512 -0.28134835], cost = 1.064759614020898
FITNESS EVAL: x = [-0.7031307  -0.97260902  0.02324783], cost = 1.4409015614762801
FITNESS EVAL: x = [-0.53295126  0.51910963  0.23055949], cost = 0.6066695325846386
Problem name: <class '__main__.sphere_function'>
	C++ class name: pybind11::object

	Global dimension:			3
	Integer dimension:			0
	Fitness dimension:			1
	Number of objectives:			1
	Equality constraints dimension:		0
	Inequality constraints dimension:	0
	Lower bounds: [-1, -1, -1]
	Upper bounds: [1, 1, 1]
	Has batch fitness evaluation: false

	Has gradient: false
	User implemented gradient sparsity: false
	Has hessians: false
	User implemented hessians sparsity: false

	Fitness evaluations: 5

	Thread safety: none

Population size: 5

List of individuals: 
#0:
	ID:			16151862124706316699
	Decision vector:	[-0.210255, 0.885807, -0.465902]
	Fitness vector:		[1.04593]
#1:
	ID:			8871279321885368107
	Decision vector:	[-0.46673, 0.123563, 0.105787]
	Fitness vector:		[0.244296]
#2:
	ID:			14228770191485562944
	Decision vector:	[0.779064, -0.615355, -0.281348]
	Fitness vector:		[1.06476]
#3:
	ID:			10302531725314368677
	Decision vector:	[-0.703131, -0.972609, 0.0232478]
	Fitness vector:		[1.4409]
#4:
	ID:			527065389800724630
	Decision vector:	[-0.532951, 0.51911, 0.230559]
	Fitness vector:		[0.60667]

Champion decision vector: [-0.46673, 0.123563, 0.105787]
Champion fitness: [0.244296]

[juan11iguel]

I was driving myself crazy trying to debug my problem, but it turned out I was facing the same issue. On my complex UDP the decision vector printed from the fitness function was always all zeros even though when I print the population it has values within the box-bounds.

Reproducing the example shown, here, using uv inline dependencies so it's easy to reproduce:

# /// script
# requires-python = ">=3.11"
# dependencies = [
#     "numpy==1.26.4", 
#     "pygmo>=2.19.5",
# ]
# ///

import pygmo as pg

class sphere_function:
    def fitness(self, x):
        cost = sum(x * x)
        print(f"FITNESS EVAL: x = {x}, cost = {cost}")
        return [cost]

    def get_bounds(self):
        return ([-1] * 3, [1] * 3)

prob = pg.problem(sphere_function())
pop = pg.population(prob, 5)
print(pop)

Output numpy==1.26.4:

uv run scripts/test.py
Reading inline script metadata from `scripts/test.py`
Installed 4 packages in 37ms
FITNESS EVAL: x = [-0.02480168 -0.42369717  0.88062573], cost = 0.955636093710774
FITNESS EVAL: x = [-0.80955458  0.66238881 -0.2029845 ], cost = 1.1353402532541443
FITNESS EVAL: x = [-0.59551142  0.69739813  0.48376777], cost = 1.0750292661256955
FITNESS EVAL: x = [-0.27541319  0.6792471   0.70580915], cost = 1.0353956125547077
FITNESS EVAL: x = [ 0.14815883 -0.26794631 -0.23531787], cost = 0.14912076248283743
Problem name: <class '__main__.sphere_function'>
        C++ class name: pybind11::object

        Global dimension:                       3
        Integer dimension:                      0
        Fitness dimension:                      1
        Number of objectives:                   1
        Equality constraints dimension:         0
        Inequality constraints dimension:       0
        Lower bounds: [-1, -1, -1]
        Upper bounds: [1, 1, 1]
        Has batch fitness evaluation: false

        Has gradient: false
        User implemented gradient sparsity: false
        Has hessians: false
        User implemented hessians sparsity: false

        Fitness evaluations: 5

        Thread safety: none

Population size: 5

List of individuals: 
#0:
        ID:                     8529584264747102174
        Decision vector:        [-0.0248017, -0.423697, 0.880626]
        Fitness vector:         [0.955636]
#1:
        ID:                     10728610797989727912
        Decision vector:        [-0.809555, 0.662389, -0.202985]
        Fitness vector:         [1.13534]
#2:
        ID:                     2137646126746706000
        Decision vector:        [-0.595511, 0.697398, 0.483768]
        Fitness vector:         [1.07503]
#3:
        ID:                     14534875256347363244
        Decision vector:        [-0.275413, 0.679247, 0.705809]
        Fitness vector:         [1.0354]
#4:
        ID:                     1992802031618337540
        Decision vector:        [0.148159, -0.267946, -0.235318]
        Fitness vector:         [0.149121]

Champion decision vector: [0.148159, -0.267946, -0.235318]
Champion fitness: [0.149121]

Output with numpy==2.2.1

uv run scripts/test.py
Reading inline script metadata from `scripts/test.py`
FITNESS EVAL: x = [-0.04206312 -0.04206312 -0.04206312], cost = 0.005307918824063201
FITNESS EVAL: x = [-0.14058251 -0.14058251 -0.14058251], cost = 0.05929032912493235
FITNESS EVAL: x = [-0.68639352 -0.68639352 -0.68639352], cost = 1.4134082007342241
FITNESS EVAL: x = [0.54235462 0.54235462 0.54235462], cost = 0.8824455972651195
FITNESS EVAL: x = [-0.67071594 -0.67071594 -0.67071594], cost = 1.3495796064578014
Problem name: <class '__main__.sphere_function'>
        C++ class name: pybind11::object

        Global dimension:                       3
        Integer dimension:                      0
        Fitness dimension:                      1
        Number of objectives:                   1
        Equality constraints dimension:         0
        Inequality constraints dimension:       0
        Lower bounds: [-1, -1, -1]
        Upper bounds: [1, 1, 1]
        Has batch fitness evaluation: false

        Has gradient: false
        User implemented gradient sparsity: false
        Has hessians: false
        User implemented hessians sparsity: false

        Fitness evaluations: 5

        Thread safety: none

Population size: 5

List of individuals: 
#0:
        ID:                     15288477237018542281
        Decision vector:        [-0.0420631, -0.592621, 0.914466]
        Fitness vector:         [0.00530792]
#1:
        ID:                     11870251820978763992
        Decision vector:        [-0.140583, -0.837447, -0.608265]
        Fitness vector:         [0.0592903]
#2:
        ID:                     12577766529020178429
        Decision vector:        [-0.686394, 0.846751, 0.668364]
        Fitness vector:         [1.41341]
#3:
        ID:                     276612175360994195
        Decision vector:        [0.542355, 0.715306, 0.513774]
        Fitness vector:         [0.882446]
#4:
        ID:                     8312149774674074044
        Decision vector:        [-0.670716, -0.886136, -0.674748]
        Fitness vector:         [1.34958]

Champion decision vector: [-0.0420631, -0.592621, 0.914466]
Champion fitness: [0.00530792]

Steps to reproduce:

1. Install uv
2. Run script with uv run test.py
3. Repeat replacing the version of numpy to 2.2.1 to test.

[juan11iguel]

To add on this, I think it's due to how the package is installed, by using conda I also can't reproduce the bug, but did (as my previous comment shows) on the pip version.

When using conda, if the channel is not conda-forge (but pip), the package also behaves erratically.

In my case it's completely broken, I spent some days trying to debug why when I run any algorithm implemented in C++ it just hangs and blocks the thread until the process is killed. After a lot of debugging, with my specific problem, I pinned it down to the get_bounds method, it got to the point where it would work when returning bounds using numpy.zeros or numpy.ones but not when manually typing them (numpy.array([0, 0, ..., 0]), etc). After installing from conda-forge channel everything worked from the gateway.

[bluescarni]

@juan11iguel generally speaking, mixing together conda and pip packages is not supported.

The pip packages at this point are quite old and what is probably happening is that the pybind11 version that was used to produce the wheels does not support numpy 2 correctly.

@juan11iguel @samimia-swks can you confirm that within a conda environment everything is working correctly?

[juan11iguel]

The conda version works as expected