runtime warning from numpy about invalid value in cast

[JonathanGregory]

There's an example field in the attached zero.cdl.txt. The following produces the warning "/usr/share/anaconda3/lib/python3.8/site-packages/numpy/ma/core.py:467: RuntimeWarning: invalid value encountered in cast".

>>> cf.__version__
'3.14.1'
>>> numpy.__version__
'1.24.2'
n=cf.read('zero.nc')[0]
>>> n.dim('T').dump()
/usr/share/anaconda3/lib/python3.8/site-packages/numpy/ma/core.py:467: RuntimeWarning: invalid value encountered in cast
  fill_value = np.array(fill_value, copy=False, dtype=ndtype)
Dimension coordinate: time
    axis = 'T'
    calendar = '360_day'
    standard_name = 'time'
    units = 'days since 2007-1-1'
    Data(294) = [2007-01-01 00:00:00, ..., 2300-01-01 00:00:00] 360_day

[sadielbartholomew]

Hi @JonathanGregory, thanks for reporting this. I'll have a look into it whilst David is on leave.

I see you've included the cf and NumPy versions, but (if you get a chance), it may prove useful to know the whole set of dependency versions (as e.g. it might be something in cfdm), so please could you share the output also of running cf.environment() or cf.environment(paths=False) for the setup in which you see this. Thanks. Regardless I'll get back to you on this thread when I get a chance to investigate and know more.

[JonathanGregory]

Dear @sadielbartholomew

Thanks. I have found that this does not happen on a brand-new installation I have just made. So I expect it must be to do with a not-entirely consistent installation in which I have recently updated cf-python:

>>> cf.environment()
Platform: Linux-5.4.0-139-generic-x86_64-with-glibc2.10 
HDF5 library: 1.12.0 
netcdf library: 4.7.4 
udunits2 library: libudunits2.so.0 
ESMF: not available 
Python: 3.8.5 /usr/share/anaconda3/bin/python
dask: 2023.3.1 /usr/share/anaconda3/lib/python3.8/site-packages/dask/__init__.py
netCDF4: 1.5.6 /usr/share/anaconda3/lib/python3.8/site-packages/netCDF4/__init__.py
psutil: 5.7.2 /usr/share/anaconda3/lib/python3.8/site-packages/psutil/__init__.py
packaging: 20.4 /usr/share/anaconda3/lib/python3.8/site-packages/packaging/__init__.py
numpy: 1.24.2 /usr/share/anaconda3/lib/python3.8/site-packages/numpy/__init__.py
scipy: 1.5.2 /usr/share/anaconda3/lib/python3.8/site-packages/scipy/__init__.py
matplotlib: 3.3.2 /usr/share/anaconda3/lib/python3.8/site-packages/matplotlib/__init__.py
cftime: 1.6.2 /usr/share/anaconda3/lib/python3.8/site-packages/cftime/__init__.py
cfunits: 3.3.5 /usr/share/anaconda3/lib/python3.8/site-packages/cfunits/__init__.py
cfplot: 3.1.18 /usr/share/anaconda3/lib/python3.8/site-packages/cfplot/__init__.py
cfdm: 1.10.0.3 /usr/share/anaconda3/lib/python3.8/site-packages/cfdm/__init__.py
cf: 3.14.1 /usr/share/anaconda3/lib/python3.8/site-packages/cf/__init__.py

Best wishes

Jonathan

[sadielbartholomew]

Thanks for the further info, Jonathan. I'll take a look tomorrow and see if I can recreate the warning in some environment. Regardless I will report back soon.

[sadielbartholomew]

Hi again Jonathan, I haven't a chance this week to look into this after all, but in reviewing something else I did see the same warning arise in multiple places (and very repeatedly in those locations) on one of our test CI jobs, running with Linux OS (latest Ubuntu available) and Python 3.8, namely https://github.com/NCAS-CMS/cf-python/actions/runs/4551782556/jobs/8026279312?pr=627. This is further evidence of an environment which hits the warnings and it will help me to narrow down where in the cf-python or (cfdm) codebase the issue originates (as the tracebacks here obviously only point to the numpy.ma code location and the fault could get hit anywhere in the calls which show the warning).

Anyway, I should be able to find time to resolve this early next week.

[JonathanGregory]

Thanks, Sadie. J

[sadielbartholomew]

Can confirm now what was always the likely case, that the reason these warnings appear is due to a newer version of numpy, namely at numpy: 1.22.2 these warnings did not appear, but after I upgraded locally to numpy: 1.24.2 I see the warnings come through. This is all consistent with the reports above, but as final evidence, by running the test_Query.py as an example, changing nothing other than the upgrade of NumPy version between the runs:

$python test_Query.py 
Run date: 2023-04-03 16:35:21.728435
Platform: Linux-4.15.0-54-generic-x86_64-with-glibc2.10 
HDF5 library: 1.10.6 
netcdf library: 4.8.0 
udunits2 library: /home/sadie/anaconda3/envs/cf-env/lib/libudunits2.so.0 
ESMF: 8.1.1 /home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/ESMF/__init__.py
Python: 3.8.10 /home/sadie/anaconda3/envs/cf-env/bin/python
dask: 2023.1.0 /home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/dask/__init__.py
netCDF4: 1.5.6 /home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/netCDF4/__init__.py
psutil: 5.9.0 /home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/psutil/__init__.py
packaging: 21.3 /home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/packaging/__init__.py
numpy: 1.22.2 /home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/numpy/__init__.py
scipy: 1.8.0 /home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/scipy/__init__.py
matplotlib: 3.4.3 /home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/matplotlib/__init__.py
cftime: 1.6.0 /home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/cftime/__init__.py
cfunits: 3.3.5 /home/sadie/cfunits/cfunits/__init__.py
cfplot: 3.1.18 /home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/cfplot/__init__.py
cfdm: 1.10.0.3 /home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/cfdm/__init__.py
cf: 3.14.1 /home/sadie/cf-python/cf/__init__.py

test_Query (__main__.QueryTest) ... ok
test_Query__and__ (__main__.QueryTest) ... ok
test_Query__or__ (__main__.QueryTest) ... ok
test_Query_as_where_condition (__main__.QueryTest)
Check queries work correctly as conditions in 'where' ... ok
test_Query_contains (__main__.QueryTest) ... ok
test_Query_datetime1 (__main__.QueryTest) ... ok
test_Query_dteq_dtne_dtge_dtgt_dtle_dtlt (__main__.QueryTest) ... ok
test_Query_evaluate (__main__.QueryTest) ... ok
test_Query_iscontains (__main__.QueryTest) ... ok
test_Query_object_units (__main__.QueryTest)
Check units are processed correctly in and from queries. ... ok
test_Query_set_condition_units (__main__.QueryTest) ... ok
test_Query_year_month_day_hour_minute_second (__main__.QueryTest) ... ok

----------------------------------------------------------------------
Ran 12 tests in 0.784s

OK
$ python test_Query.py 
Run date: 2023-04-03 18:26:44.837007
Platform: Linux-4.15.0-54-generic-x86_64-with-glibc2.10 
HDF5 library: 1.14.0 
netcdf library: 4.9.2 
udunits2 library: /home/sadie/anaconda3/envs/cf-env/lib/libudunits2.so.0 
ESMF: not available 
Python: 3.8.16 /home/sadie/anaconda3/envs/cf-env/bin/python
dask: 2023.1.0 /home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/dask/__init__.py
netCDF4: 1.6.3 /home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/netCDF4/__init__.py
psutil: 5.9.4 /home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/psutil/__init__.py
packaging: 23.0 /home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/packaging/__init__.py
numpy: 1.24.2 /home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/numpy/__init__.py
scipy: 1.10.1 /home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/scipy/__init__.py
matplotlib: 3.4.3 /home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/matplotlib/__init__.py
cftime: 1.6.0 /home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/cftime/__init__.py
cfunits: 3.3.5 /home/sadie/cfunits/cfunits/__init__.py
cfplot: 3.1.18 /home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/cfplot/__init__.py
cfdm: 1.10.0.3 /home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/cfdm/__init__.py
cf: 3.14.1 /home/sadie/cf-python/cf/__init__.py

test_Query (__main__.QueryTest) ... ok
test_Query__and__ (__main__.QueryTest) ... ok
test_Query__or__ (__main__.QueryTest) ... ok
test_Query_as_where_condition (__main__.QueryTest)
Check queries work correctly as conditions in 'where' ... ok
test_Query_contains (__main__.QueryTest) ... ok
test_Query_datetime1 (__main__.QueryTest) ... /home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/numpy/ma/core.py:467: RuntimeWarning: invalid value encountered in cast
  fill_value = np.array(fill_value, copy=False, dtype=ndtype)
/home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/numpy/ma/core.py:467: RuntimeWarning: invalid value encountered in cast
  fill_value = np.array(fill_value, copy=False, dtype=ndtype)
/home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/numpy/ma/core.py:467: RuntimeWarning: invalid value encountered in cast
  fill_value = np.array(fill_value, copy=False, dtype=ndtype)
/home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/numpy/ma/core.py:467: RuntimeWarning: invalid value encountered in cast
  fill_value = np.array(fill_value, copy=False, dtype=ndtype)
/home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/numpy/ma/core.py:467: RuntimeWarning: invalid value encountered in cast
  fill_value = np.array(fill_value, copy=False, dtype=ndtype)
/home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/numpy/ma/core.py:467: RuntimeWarning: invalid value encountered in cast
  fill_value = np.array(fill_value, copy=False, dtype=ndtype)
/home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/numpy/ma/core.py:467: RuntimeWarning: invalid value encountered in cast
  fill_value = np.array(fill_value, copy=False, dtype=ndtype)
/home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/numpy/ma/core.py:467: RuntimeWarning: invalid value encountered in cast
  fill_value = np.array(fill_value, copy=False, dtype=ndtype)
/home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/numpy/ma/core.py:467: RuntimeWarning: invalid value encountered in cast
  fill_value = np.array(fill_value, copy=False, dtype=ndtype)
ok
test_Query_dteq_dtne_dtge_dtgt_dtle_dtlt (__main__.QueryTest) ... /home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/numpy/ma/core.py:467: RuntimeWarning: invalid value encountered in cast
  fill_value = np.array(fill_value, copy=False, dtype=ndtype)
/home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/numpy/ma/core.py:467: RuntimeWarning: invalid value encountered in cast
  fill_value = np.array(fill_value, copy=False, dtype=ndtype)
/home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/numpy/ma/core.py:467: RuntimeWarning: invalid value encountered in cast
  fill_value = np.array(fill_value, copy=False, dtype=ndtype)
/home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/numpy/ma/core.py:467: RuntimeWarning: invalid value encountered in cast
  fill_value = np.array(fill_value, copy=False, dtype=ndtype)
/home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/numpy/ma/core.py:467: RuntimeWarning: invalid value encountered in cast
  fill_value = np.array(fill_value, copy=False, dtype=ndtype)
/home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/numpy/ma/core.py:467: RuntimeWarning: invalid value encountered in cast
  fill_value = np.array(fill_value, copy=False, dtype=ndtype)
/home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/numpy/ma/core.py:467: RuntimeWarning: invalid value encountered in cast
  fill_value = np.array(fill_value, 
```bashcopy=False, dtype=ndtype)
/home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/numpy/ma/core.py:467: RuntimeWarning: invalid value encountered in cast
  fill_value = np.array(fill_value, copy=False, dtype=ndtype)
/home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/numpy/ma/core.py:467: RuntimeWarning: invalid value encountered in cast
  fill_value = np.array(fill_value, copy=False, dtype=ndtype)
/home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/numpy/ma/core.py:467: RuntimeWarning: invalid value encountered in cast
  fill_value = np.array(fill_value, copy=False, dtype=ndtype)
/home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/numpy/ma/core.py:467: RuntimeWarning: invalid value encountered in cast
  fill_value = np.array(fill_value, copy=False, dtype=ndtype)
/home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/numpy/ma/core.py:467: RuntimeWarning: invalid value encountered in cast
  fill_value = np.array(fill_value, copy=False, dtype=ndtype)
ok
test_Query_evaluate (__main__.QueryTest) ... ok
test_Query_iscontains (__main__.QueryTest) ... ok
test_Query_object_units (__main__.QueryTest)
Check units are processed correctly in and from queries. ... ok
test_Query_set_condition_units (__main__.QueryTest) ... ok
test_Query_year_month_day_hour_minute_second (__main__.QueryTest) ... ok

----------------------------------------------------------------------
Ran 12 tests in 0.883s

OK

Now I've recreated the issue with the warnings coming through I'll work out how we need to update our cf code and then put in the PR. Then this can be closed.

[sadielbartholomew]

With the updated numpy version, the warning is also appearing with cfdm, notably showing up for test_example_field (only) in the full test suite run there (and I can see that only cfdm.example_field(4) triggers it, one but not the only one of the cfdm.example_fields() with a _FillValue and of -999.9) so this might be emerging from cfdm rather than cf, else we need to adapt something in both codebases, separately, to prevent the warning coming through. I will try to fix the root cause(s) next week.

[sadielbartholomew]

Just as an update on this, I believe the warnings have been addressed somehow (inadvertently I believe) by one or both of the PRs #255 (upstream) and its sibling here NCAS-CMS/cf-python#630 in that they aren't showing in the test suites for branches that use those, which I noticed during my reviewing of each (which was a thorough rather than quick review).

I was due to get to the bottom of where the warnings were emerging from via pdb but I'll postpone that for now since the problem may have went away by itself during the extensive code changes as part of those PRs. We're doing a cf release next week so I'll double check we aren't seeing such warnings emerge from the woodwork before we do that release and close this accordingly when I am sure.

[sadielbartholomew]

Hi all, to finally get back with a definitive report on this (the warnings still emerged after the merging of the PR I mentioned in my comment above).

In short, the warnings are appearing purely due to an issue with cftime rather than us (cf or cfdm, etc.) and it has been fixed for cftime version 1.6.2, so by upgrading that version appropriately (I used pip install cftime==1.6.2), the spammy warnings disappear.

(For @davidhassell especially) I have included a report to demonstrate why this is 'not our issue' to justify my imminent closure of this Issue here. Feel free to take my word for it instead of reading though all that below(!), but the detail should provide clarification.

@JonathanGregory please can you confirm that by upgrading your cftime library version, as above, the issue is resolved as far as you are concerned. That way I know we are all satisfied that this can be closed. In an earlier comment, you said:

I have found that this does not happen on a brand-new installation I have just made.

and indeed I see for that the cftime version is 1.6.2, so that suggests this could be the case. It certainly resolves things on my local machine. A final confirmation would be appreciated, though.

Details

With the following code using a minimal reproducible example I adapted from the test_Query case showing the warning, notably calling warnings.simplefilter("error", RuntimeWarning) to force a traceback which identifies the location in our code that the warning emerges, rather than just the final downstream numpy call which triggers it:

>>> import cf
>>> import warnings
>>> warnings.simplefilter("error", RuntimeWarning)
>>> d = cf.Data([[1,], [2.0,]], "days since 2000-12-29 21:00:00")
>>> d == cf.eq(cf.dt("2001-01-03 21:00:00"))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/sadie/cf-python/cf/query.py", line 260, in __eq__
    return self._evaluate(x, ())
  File "/home/sadie/cf-python/cf/query.py", line 686, in _evaluate
    f"Can't perform regular expression search on a non-string: {x!r}"
  File "/home/sadie/cf-python/cf/data/data.py", line 775, in __repr__
    return super().__repr__().replace("<", "<CF ", 1)
  File "/home/sadie/cfdm/cfdm/data/data.py", line 212, in __repr__
    return f"<{ self.__class__.__name__}{shape}: {self}>"
  File "/home/sadie/cfdm/cfdm/data/data.py", line 253, in __format__
    return super().__format__("")
  File "/home/sadie/cfdm/cfdm/data/data.py", line 526, in __str__
    first, last = type(self)(
  File "/home/sadie/cf-python/cf/data/data.py", line 5138, in datetime_array
    a = dx.compute()
  File "/home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/dask/base.py", line 314, in compute
    (result,) = compute(self, traverse=False, **kwargs)
  File "/home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/dask/base.py", line 599, in compute
    results = schedule(dsk, keys, **kwargs)
  File "/home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/dask/threaded.py", line 89, in get
    results = get_async(
  File "/home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/dask/local.py", line 511, in get_async
    raise_exception(exc, tb)
  File "/home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/dask/local.py", line 319, in reraise
    raise exc
  File "/home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/dask/local.py", line 224, in execute_task
    result = _execute_task(task, data)
  File "/home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/dask/core.py", line 119, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
  File "/home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/dask/optimization.py", line 990, in __call__
    return core.get(self.dsk, self.outkey, dict(zip(self.inkeys, args)))
  File "/home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/dask/core.py", line 149, in get
    result = _execute_task(task, cache)
  File "/home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/dask/core.py", line 119, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
  File "/home/sadie/cf-python/cf/cfdatetime.py", line 456, in rt2dt
    array = cftime.num2date(
  File "src/cftime/_cftime.pyx", line 584, in cftime._cftime.num2date
  File "src/cftime/_cftime.pyx", line 385, in cftime._cftime.cast_to_int
  File "/home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/numpy/ma/core.py", line 2820, in __new__
    _data = np.array(data, dtype=dtype, copy=copy,
  File "/home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/numpy/ma/core.py", line 3054, in __array_finalize__
    self._fill_value = _check_fill_value(self._fill_value, self.dtype)
  File "/home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/numpy/ma/core.py", line 467, in _check_fill_value
    fill_value = np.array(fill_value, copy=False, dtype=ndtype)
RuntimeWarning: invalid value encountered in cast

we can see the perpetrator line from cf is:

https://github.com/NCAS-CMS/cf-python/blob/6078e08345a33263b3dca98c2d5ae66144d16b92/cf/cfdatetime.py#L456-L458

and there's no issue I found with our usage of num2date, so it was likely on the cftime side.

When I checked their codebase and issue tracker, I saw num2date was updated various times for the two versions after the 1.6.0 I was using when I saw the warnings appear. So that helps to explain why updating seems to fix things.

[JonathanGregory]

Dear @sadielbartholomew

Sorry for slow reply. I have just tested it with my original example file and I still get the error :-(

>> cf.environment()
Platform: Linux-5.15.0-60-generic-x86_64-with-glibc2.35 
HDF5 library: 1.12.2 
netcdf library: 4.9.3-development 
udunits2 library: libudunits2.so.0 
esmpy/ESMF: not available 
Python: 3.10.6 /usr/local/bin/python
dask: 2023.7.0 /usr/local/lib/python3.10/dist-packages/dask/__init__.py
netCDF4: 1.6.4 /usr/local/lib/python3.10/dist-packages/netCDF4/__init__.py
psutil: 5.9.0 /usr/lib/python3/dist-packages/psutil/__init__.py
packaging: 21.3 /usr/lib/python3/dist-packages/packaging/__init__.py
numpy: 1.25.1 /usr/local/lib/python3.10/dist-packages/numpy/__init__.py
scipy: 1.11.1 /usr/local/lib/python3.10/dist-packages/scipy/__init__.py
matplotlib: not available 
cftime: 1.6.2 /usr/local/lib/python3.10/dist-packages/cftime/__init__.py
cfunits: 3.3.6 /usr/local/lib/python3.10/dist-packages/cfunits/__init__.py
cfplot: not available 
cfdm: 1.10.1.0 /usr/local/lib/python3.10/dist-packages/cfdm/__init__.py
cf: 3.15.1 /usr/local/lib/python3.10/dist-packages/cf/__init__.py
>>> n=cf.read('../temp.nc')[0]
>>> n.dim('T').dump()
/usr/local/lib/python3.10/dist-packages/numpy/ma/core.py:467: RuntimeWarning: invalid value encountered in cast
  fill_value = np.array(fill_value, copy=False, dtype=ndtype)
Dimension coordinate: time
    axis = 'T'
    calendar = '360_day'
    standard_name = 'time'
    units = 'days since 2007-1-1'
    Data(294) = [2007-01-01 00:00:00, ..., 2300-01-01 00:00:00] 360_day

As you see, I have cftime 1.6.2, so it isn't the culprit for me, apparently.

Jonathan

[sadielbartholomew]

I have just tested it with my original example file and I still get the error :-(

Ah, that's a shame since I thought I'd pinpointed the culprit 🥲. Sorry about that. No worries, I look over the traceback I reached in my previous comment and see if it leads me to the true culprit. I'll report back when I find out something more useful.

[sadielbartholomew]

Sorry @JonathanGregory, it is well overdue that we sort this, I am realising as I review #292.

From some quick investigation after re-reading my report of previous deductions in this thread, I am fairly certain it relates to Unidata/cftime#328. I will investigate and try to make the fix in cfdm after I submit my review to that PR.