Sorted coordinates for combining

[Zepeng-Mu]

Hello, I'm very new to xarray. I'm trying to read in many netCDF files as a dataset using open_mfdataset and encountered this error:

ValueError: Coordinate variable rail_id is neither monotonically increasing nor monotonically decreasing on all datasets

I know that open_mfdataset read in each file separately and then combine them, and this error message is from that step I think. But my coordinates are all strings, so I'm not sure how I'm supposed to sort them (see pic).

Also, is it a general rule that coordinates should be sorted so it's easier to combine them?

Thanks!

[andersy005]

@Zepeng-Mu, would .sortby accomplish what you are looking for?

In [10]: ds['x'] = xr.DataArray(np.arange(10).reshape(2, 5), dims=['junc', 'rail_id'], coords={'junc': [1, 2], 'rail_id': ['a', 'd', 'b', 'e', 'c']})

In [11]: ds
Out[11]: 
<xarray.Dataset>
Dimensions:  (junc: 2, rail_id: 5)
Coordinates:
  * junc     (junc) int64 1 2
  * rail_id  (rail_id) <U1 'a' 'd' 'b' 'e' 'c'
Data variables:
    x        (junc, rail_id) int64 0 1 2 3 4 5 6 7 8 9

In [12]: ds.sortby('rail_id')
Out[12]: 
<xarray.Dataset>
Dimensions:  (junc: 2, rail_id: 5)
Coordinates:
  * junc     (junc) int64 1 2
  * rail_id  (rail_id) <U1 'a' 'b' 'c' 'd' 'e'
Data variables:
    x        (junc, rail_id) int64 0 2 4 1 3 5 7 9 6 8

[TomNicholas]

Hi @Zepeng-Mu , thank you for raising this, and apologies for not seeing this earlier.

is it a general rule that coordinates should be sorted so it's easier to combine them?

By using open_mfdataset with it's default values, the combining of datasets is delegated to the function xarray.combine_by_coords. The docstring for combine_by_coords says

"Will attempt to order the datasets such that the values in their dimension coordinates are monotonic along all dimensions. If it cannot determine the order in which to concatenate the datasets, it will raise a ValueError."

That's the source of the ValueError you're seeing. What combine_by_coords wants to do is arrange all your datasets so that along each dimension, the whole resulting concatenated coordinate will be monotonic. That requires the coordinate on each of those datasets to be monotonic individually too. Your rail_id coordinates don't meet that criteria: 14065, 14573, ... , 62732 are clearly not sorted. (And I don't think the fact you stored them as strings really matters here - panda.Index wouldn't consider these to be monotonic whether they are in string or integer form).

There two options:

1. Sort the coordinates on each dataset first and then use combine_by_coords. This is what @andersy005 suggested - sorting the rail_ids on each dataset first will make it possible for combine_by_coords to construct a result where the entire concatenated rail_id coordinate is monotonic.

2. Don't sort them and use combine_nested. If for some reason you wanted to preserve the original order of your rail_ids, you could instead combine manually with combine_nested. This would concatenate your datasets together but not enforce any kind of ordering on the coordinates. It would look something like

result = combine_nested([[ds1, ds2], [ds3, ds4]], concat_dim=['rail_id', 'junc'])

[Zepeng-Mu]

Hi, yes I ended up using combine_nested . Thanks