Casting connectivity data to masked arrays?

[mo-gill]

You can currently create a non-masked connectivity array. Members of the ANTs team have had a discussion with @stephenworsley whether it might be beneficial to implement behaviour of automatically setting connectivity arrays to maskedarrays during creation of connectivities.

There are two reasons we would appreciate this being implemented:

1. To prevent issues such as this occuring downstream in future.
2. The current behaviour of iris is to load all netcdf data as a masked array irrespective of whether there is missing data in the netcdf file. In its current state, we've found it a bit surprising that you can create a non-masked connectivity array.

If there's a decision not to implement this, it would be much appreciated if the documentation could be made more explicit in regard to the connectivity behaviour and masked arrays.

[trexfeathers]

The behaviour of Connectivity in this regard is shared with all other dimensional objects - Cube, DimCoord, etcetera - it is common code between them all.

The current behaviour of iris is to load all netcdf data as a masked array irrespective of whether there is missing data in the netcdf file. In its current state, we've found it a bit surprising that you can create a non-masked connectivity array.

Do you find it equally surprising that it is possible to create a non-masked Cube?

I personally think it important that it is still possible to have non-masked arrays on these objects because unfortunately masked arrays often have slower performance ¹; this seems to be because they are considered somewhat niche by NumPy and Dask developers and so there is less optimisation effort when it comes to masks.

To prevent issues such as this occuring downstream in future.

Given the performance costs, I would say that the best solution is to simply be vigilant for mistakes like this. This can work: elsewhere in Iris it is a given that a dimensional object may or may not be masked, and we write/review accordingly.

@mo-gill perhaps there is a case to be made that Connectivity should be an exception from the behaviour shared with other Iris dimensional objects?

Footnotes

1. This was the driver behind Masked array performance #3470 and Decorator to speed up operations on all-False masked arrays #5178 although that was ultimately unsuccessful: https://github.com/SciTools/iris/pull/5178#issuecomment-1497857818 ↩

[hdyson]

I'll jump in on @mo-gill 's behalf here - we've been working on this together.

Do you find it equally surprising that it is possible to create a non-masked Cube?

No, and I think there is a fundamental difference: cubes with meshes can only be loaded from netCDF. Non-mesh cubes can be loaded from other formats, which do allow loading normal, non-masked arrays. So I would expect objects that can be loaded from PP, grib, etc to contain either a non-masked or masked array, but would not expect objects that can only be loaded from netCDF to be a non-masked array.

We encountered this issue in our testing: we construct a synthetic cube with a mesh and this included using a non-masked numpy array for the connectivity. We have now fixed our test mesh generator to use a masked array so that it is more representative of the equivalent data loaded in from a file. This means the behaviour that surprised us has been fixed for our use case: given it surprised us, though, I thought it worth flagging to see if we could prevent others being surprised.

(At the risk of a tangent, in ANTS, we do convert all cubes on load to use masked arrays for the data payload: we mostly use netCDF files, so making the behaviour of non-netCDF files consistent helps to avoid surprises within our processing code. I don't think that's appropriate for iris - other workflows could be using netCDFs rarely or not at all - but I think it may be context for why we were surprised).

because they are considered somewhat niche by NumPy and Dask developers and so there is less optimisation effort when it comes to masks.

Yeah, it's a constant source of frustration ☹

[trexfeathers]

From @scitools/peloton: this is effectively a decision between:

• Confusing downstream users/developers - the current state - confusing because Connectivitys are only non-masked when created from scratch and at no other time.
• Confusing Iris developers - the proposal - confusing because Connectivity would become an exception versus all other Iris dimensional objects (e.g. Cube, DimCoord, etc).

We're undecided what direction to take, particularly since it's hard to be objective when every person is either an Iris developer or a downstream user/developer.

[trexfeathers]

To this end: please any Iris user make some noise if you have an opinion!

There might be some of you out there that need the ability to have non-masked arrays in Connectivity. We don't know unless you tell us.

📢📢📢📢