Problem using sdba.LOCI on ensemble hindcast

[jamaa]

Setup Information

• Xclim version: 0.48.2

Context

I am trying to use the Local Intensity Scaling (LOCI) bias-adjustment method on an ensemble forecast of daily precipitation, by training it on hindcasts. I want the adjustment factors to be computed per lead time (no. of days - step), but across all realizations of the ensemble (number).

Steps To Reproduce

da_hindcast = xr.open_dataarray("hindcast.nc")

xarray.DataArray 'tp_mmd_pos' number: 10 latitude: 6 longitude: 9 time: 940
[507600 values with dtype=float32]
Coordinates:
number (number) int32 1 2 3 4 5 6 7 8 9 10
latitude (latitude) float64 51.6 51.2 50.8 50.4 50.0 49.6
longitude (longitude) float64 8.8 9.2 9.6 10.0 ... 11.2 11.6 12.0
surface ()  float64 ...
time (time) datetime64[ns] 2003-07-10 ... 2022-08-25
step (time)  int64 ...
Indexes: (4) 
Attributes:
  units :  mm d-1

da_observed = xr.open_dataarray("observed.nc")

xarray.DataArray 'pr' time: 940 latitude: 6 longitude: 9
[50760 values with dtype=float32]
Coordinates: 
longitude (longitude)  float64 8.8 9.2 9.6 10.0 ... 11.2 11.6 12.0
latitude (latitude) float64 51.6 51.2 50.8 50.4 50.0 49.6
time (time) datetime64[ns] 2003-07-10 ... 2022-08-25
Indexes: (3)
Attributes:
  standard_name : thickness_of_rainfall_amount
  long_name : Daily Precipitation Sum
  units : mm d-1

group = sdba.Grouper("time.dayofyear", add_dims=['number'])
loci = sdba.LOCI.train(da_observed, da_hindcast, thresh="1 mm/d", group=group)

Result:

ValueError: loci_train failed on block with coords : Coordinates:
  * longitude  (longitude) float64 72B 8.8 9.2 9.6 10.0 10.4 10.8 11.2 11.6 12.0
  * latitude   (latitude) float64 48B 51.6 51.2 50.8 50.4 50.0 49.6
  * time       (time) datetime64[ns] 8kB 2003-07-10 2003-07-11 ... 2022-08-25
  * number     (number) int32 40B 1 2 3 4 5 6 7 8 9 10.

In the code above, "time.dayofyear" is a proxy for lead time, ideally I want to use the coordinate "step" instead, but I am not sure how to achieve that? I've also tried the following:

group = sdba.Grouper(da_hindcast["step"], add_dims=["number"])
loci = sdba.LOCI.train(da_observed, da_hindcast, thresh="1 mm/d", group=group)

which results in

IndexError: dimension coordinate 'time' conflicts between indexed and indexing objects:
<xarray.DataArray 'time' (time: 940)> Size: 8kB
array(['2003-07-10T00:00:00.000000000', '2003-07-11T00:00:00.000000000',
       '2003-07-12T00:00:00.000000000', ..., '2003-08-23T00:00:00.000000000',
       '2003-08-24T00:00:00.000000000', '2003-08-25T00:00:00.000000000'],
      dtype='datetime64[ns]')
Coordinates:
  * time     (time) datetime64[ns] 8kB 2003-07-10 2003-07-11 ... 2003-08-25
vs.
<xarray.IndexVariable 'time' (time: 940)> Size: 8kB
array(['2003-07-10T00:00:00.000000000', '2003-07-11T00:00:00.000000000',
       '2003-07-12T00:00:00.000000000', ..., '2022-08-23T00:00:00.000000000',
       '2022-08-24T00:00:00.000000000', '2022-08-25T00:00:00.000000000'],
      dtype='datetime64[ns]')

I've been fiddling around with this for days now and would appreciate if anybody has some ideas or hints, maybe I am doing it completely wrong?

[aulemahal]

Hi @jamaa,

You are not doing anything wrong, but sadly sdba is currently implemented with datetime grouping in mind. I thought it would still work with an integer coordinate like your step, but some technical assumptions of Grouper makes it fail. It really is an implementation problem where we simply didn't think of that case.

Without modifying xclim, it would work if somehow your step was a datetime property of the time axis, if one could do ds.time.dt.step. Of course making this work would be incredibly hacky. I think we'd need to modify xclim to understand that the grouping index is the "auxiliary" coordinate step or if you can swap time and step, that the grouping is to be done with the values of the step dimension. I don't think we're far off though.

i.e. in a better world you should do able to do Grouper('time.step') or ds.swap_dims(time='step'); Grouper('step').

As a side note, the first error you gave is not the real error, that's just the consequence of an internal error on the map_blocks wrapper. The actual error should be printed higher in the stack. As for the second error, Grouper expects a coordinate name, not the coordinate itself.

[jamaa]

Hi @aulemahal ,

thanks for the quick response. I see. But if I understand you correctly, the following should work (my first attempt from above, similar to the example in the docs):

group = sdba.Grouper("time.dayofyear", add_dims=['number'])
loci = sdba.LOCI.train(da_observed, da_hindcast, thresh="1 mm/d", group=group)

But this also gives me an error.

Here is the full stacktrace

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
File [d:\DEV\venv\cfgrib\lib\site-packages\xclim\sdba\base.py:684](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/base.py:684), in map_blocks.<locals>._decorator.<locals>._map_blocks.<locals>._call_and_transpose_on_exit(dsblock, **f_kwargs)
    [683](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/base.py:683)     _decode_cf_coords(dsblock)
--> [684](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/base.py:684)     func_out = func(dsblock, **f_kwargs).transpose(*all_dims)
    [685](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/base.py:685) except Exception as err:

File [d:\DEV\venv\cfgrib\lib\site-packages\xclim\sdba\_adjustment.py:187](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/_adjustment.py:187), in loci_train(ds, group, thresh)
    [181](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/_adjustment.py:181) """LOCI: Train on one block.
    [182](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/_adjustment.py:182) 
    [183](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/_adjustment.py:183) Dataset variables:
    [184](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/_adjustment.py:184)   ref : training target
    [185](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/_adjustment.py:185)   hist : training data
    [186](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/_adjustment.py:186) """
--> [187](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/_adjustment.py:187) s_thresh = group.apply(
    [188](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/_adjustment.py:188)     u.map_cdf, ds.rename(hist="x", ref="y"), y_value=thresh
    [189](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/_adjustment.py:189) ).isel(x=0)
    [190](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/_adjustment.py:190) sth = u.broadcast(s_thresh, ds.hist, group=group)

File [d:\DEV\venv\cfgrib\lib\site-packages\xclim\sdba\base.py:402](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/base.py:402), in Grouper.apply(self, func, da, main_only, **kwargs)
    [401](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/base.py:401) else:
--> [402](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/base.py:402)     out = grpd.map(func, dim=dims, **kwargs)
    [404](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/base.py:404) # Case where the function wants to return more than one variable.
    [405](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/base.py:405) # and that some have grouped dims and other have the same dimensions as the input.
    [406](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/base.py:406) # In that specific case, groupby broadcasts everything back to the input's dim, copying the grouped data.

File [d:\DEV\venv\cfgrib\lib\site-packages\xarray\core\groupby.py:1827](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/groupby.py:1827), in DatasetGroupByBase.map(self, func, args, shortcut, **kwargs)
   [1798](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/groupby.py:1798) """Apply a function to each Dataset in the group and concatenate them
   [1799](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/groupby.py:1799) together into a new Dataset.
   [1800](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/groupby.py:1800) 
   (...)
   [1825](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/groupby.py:1825)     The result of splitting, applying and combining this dataset.
   [1826](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/groupby.py:1826) """
-> [1827](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/groupby.py:1827) return self._map_maybe_warn(func, args, shortcut, warn_squeeze=True, **kwargs)

File [d:\DEV\venv\cfgrib\lib\site-packages\xarray\core\groupby.py:1839](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/groupby.py:1839), in DatasetGroupByBase._map_maybe_warn(self, func, args, shortcut, warn_squeeze, **kwargs)
   [1838](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/groupby.py:1838) applied = (func(ds, *args, **kwargs) for ds in self._iter_grouped(warn_squeeze))
-> [1839](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/groupby.py:1839) return self._combine(applied)

File [d:\DEV\venv\cfgrib\lib\site-packages\xarray\core\groupby.py:1859](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/groupby.py:1859), in DatasetGroupByBase._combine(self, applied)
   [1858](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/groupby.py:1858) """Recombine the applied objects like the original."""
-> [1859](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/groupby.py:1859) applied_example, applied = peek_at(applied)
   [1860](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/groupby.py:1860) coord, dim, positions = self._infer_concat_args(applied_example)

File [d:\DEV\venv\cfgrib\lib\site-packages\xarray\core\utils.py:205](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/utils.py:205), in peek_at(iterable)
    [204](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/utils.py:204) gen = iter(iterable)
--> [205](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/utils.py:205) peek = next(gen)
    [206](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/utils.py:206) return peek, itertools.chain([peek], gen)

File [d:\DEV\venv\cfgrib\lib\site-packages\xarray\core\groupby.py:1838](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/groupby.py:1838), in <genexpr>(.0)
   [1829](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/groupby.py:1829) def _map_maybe_warn(
   [1830](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/groupby.py:1830)     self,
   [1831](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/groupby.py:1831)     func: Callable[..., Dataset],
   (...)
   [1836](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/groupby.py:1836) ) -> Dataset:
   [1837](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/groupby.py:1837)     # ignore shortcut if set (for now)
-> [1838](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/groupby.py:1838)     applied = (func(ds, *args, **kwargs) for ds in self._iter_grouped(warn_squeeze))
   [1839](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/groupby.py:1839)     return self._combine(applied)

File [d:\DEV\venv\cfgrib\lib\site-packages\xclim\sdba\utils.py:67](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/utils.py:67), in map_cdf(ds, y_value, dim)
     [48](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/utils.py:48) """Return the value in `x` with the same CDF as `y_value` in `y`.
     [49](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/utils.py:49) 
     [50](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/utils.py:50) This function is meant to be wrapped in a `Grouper.apply`.
   (...)
     [65](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/utils.py:65)   Quantile of `x` with the same CDF as `y_value` in `y`.
     [66](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/utils.py:66) """
---> [67](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/utils.py:67) return xr.apply_ufunc(
     [68](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/utils.py:68)     map_cdf_1d,
     [69](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/utils.py:69)     ds.x,
     [70](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/utils.py:70)     ds.y,
     [71](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/utils.py:71)     input_core_dims=[dim] * 2,
     [72](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/utils.py:72)     output_core_dims=[["x"]],
     [73](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/utils.py:73)     vectorize=True,
     [74](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/utils.py:74)     keep_attrs=True,
     [75](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/utils.py:75)     kwargs={"y_value": np.atleast_1d(y_value)},
     [76](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/utils.py:76)     output_dtypes=[ds.x.dtype],
     [77](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/utils.py:77) )

File [d:\DEV\venv\cfgrib\lib\site-packages\xarray\core\computation.py:1270](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/computation.py:1270), in apply_ufunc(func, input_core_dims, output_core_dims, exclude_dims, vectorize, join, dataset_join, dataset_fill_value, keep_attrs, kwargs, dask, output_dtypes, output_sizes, meta, dask_gufunc_kwargs, on_missing_core_dim, *args)
   [1269](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/computation.py:1269) elif any(isinstance(a, DataArray) for a in args):
-> [1270](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/computation.py:1270)     return apply_dataarray_vfunc(
   [1271](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/computation.py:1271)         variables_vfunc,
   [1272](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/computation.py:1272)         *args,
   [1273](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/computation.py:1273)         signature=signature,
   [1274](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/computation.py:1274)         join=join,
   [1275](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/computation.py:1275)         exclude_dims=exclude_dims,
   [1276](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/computation.py:1276)         keep_attrs=keep_attrs,
   [1277](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/computation.py:1277)     )
   [1278](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/computation.py:1278) # feed Variables directly through apply_variable_ufunc

File [d:\DEV\venv\cfgrib\lib\site-packages\xarray\core\computation.py:316](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/computation.py:316), in apply_dataarray_vfunc(func, signature, join, exclude_dims, keep_attrs, *args)
    [315](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/computation.py:315) data_vars = [getattr(a, "variable", a) for a in args]
--> [316](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/computation.py:316) result_var = func(*data_vars)
    [318](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/computation.py:318) out: tuple[DataArray, ...] | DataArray

File [d:\DEV\venv\cfgrib\lib\site-packages\xarray\core\computation.py:734](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/computation.py:734), in apply_variable_ufunc(func, signature, exclude_dims, dask, output_dtypes, vectorize, keep_attrs, dask_gufunc_kwargs, *args)
    [732](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/computation.py:732) output_dims = [broadcast_dims + out for out in signature.output_core_dims]
--> [734](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/computation.py:734) input_data = [
    [735](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/computation.py:735)     (
    [736](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/computation.py:736)         broadcast_compat_data(arg, broadcast_dims, core_dims)
    [737](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/computation.py:737)         if isinstance(arg, Variable)
    [738](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/computation.py:738)         else arg
    [739](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/computation.py:739)     )
    [740](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/computation.py:740)     for arg, core_dims in zip(args, signature.input_core_dims)
    [741](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/computation.py:741) ]
    [743](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/computation.py:743) if any(is_chunked_array(array) for array in input_data):

File [d:\DEV\venv\cfgrib\lib\site-packages\xarray\core\computation.py:736](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/computation.py:736), in <listcomp>(.0)
    [732](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/computation.py:732) output_dims = [broadcast_dims + out for out in signature.output_core_dims]
    [734](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/computation.py:734) input_data = [
    [735](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/computation.py:735)     (
--> [736](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/computation.py:736)         broadcast_compat_data(arg, broadcast_dims, core_dims)
    [737](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/computation.py:737)         if isinstance(arg, Variable)
    [738](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/computation.py:738)         else arg
    [739](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/computation.py:739)     )
    [740](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/computation.py:740)     for arg, core_dims in zip(args, signature.input_core_dims)
    [741](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/computation.py:741) ]
    [743](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/computation.py:743) if any(is_chunked_array(array) for array in input_data):

File [d:\DEV\venv\cfgrib\lib\site-packages\xarray\core\computation.py:681](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/computation.py:681), in broadcast_compat_data(variable, broadcast_dims, core_dims)
    [680](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/computation.py:680) if reordered_dims != old_dims:
--> [681](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/computation.py:681)     order = tuple(old_dims.index(d) for d in reordered_dims)
    [682](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/computation.py:682)     data = duck_array_ops.transpose(data, order)

File [d:\DEV\venv\cfgrib\lib\site-packages\xarray\core\computation.py:681](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/computation.py:681), in <genexpr>(.0)
    [680](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/computation.py:680) if reordered_dims != old_dims:
--> [681](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/computation.py:681)     order = tuple(old_dims.index(d) for d in reordered_dims)
    [682](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/computation.py:682)     data = duck_array_ops.transpose(data, order)

ValueError: tuple.index(x): x not in tuple

The above exception was the direct cause of the following exception:

ValueError                                Traceback (most recent call last)
Cell In [7], [line 1](vscode-notebook-cell:?execution_count=7&line=1)
----> [1](vscode-notebook-cell:?execution_count=7&line=1) loci = sdba.LOCI.train(da_observed, da_hindcast, thresh="1 mm/d", group=group)
      [2](vscode-notebook-cell:?execution_count=7&line=2) loci

File [d:\DEV\venv\cfgrib\lib\site-packages\xclim\sdba\adjustment.py:187](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/adjustment.py:187), in TrainAdjust.train(cls, ref, hist, **kwargs)
    [184](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/adjustment.py:184) else:
    [185](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/adjustment.py:185)     train_units = ""
--> [187](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/adjustment.py:187) ds, params = cls._train(ref, hist, **kwargs)
    [188](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/adjustment.py:188) obj = cls(
    [189](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/adjustment.py:189)     _trained=True,
    [190](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/adjustment.py:190)     hist_calendar=get_calendar(hist),
    [191](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/adjustment.py:191)     train_units=train_units,
    [192](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/adjustment.py:192)     **params,
    [193](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/adjustment.py:193) )
    [194](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/adjustment.py:194) obj.set_dataset(ds)

File [d:\DEV\venv\cfgrib\lib\site-packages\xclim\sdba\adjustment.py:785](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/adjustment.py:785), in LOCI._train(cls, ref, hist, thresh, group)
    [775](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/adjustment.py:775) @classmethod
    [776](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/adjustment.py:776) def _train(
    [777](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/adjustment.py:777)     cls,
   (...)
    [782](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/adjustment.py:782)     group: str | Grouper = "time",
    [783](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/adjustment.py:783) ):
    [784](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/adjustment.py:784)     thresh = convert_units_to(thresh, ref)
--> [785](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/adjustment.py:785)     ds = loci_train(
    [786](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/adjustment.py:786)         xr.Dataset({"ref": ref, "hist": hist}), group=group, thresh=thresh
    [787](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/adjustment.py:787)     )
    [788](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/adjustment.py:788)     ds.af.attrs.update(long_name="LOCI adjustment factors")
    [789](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/adjustment.py:789)     ds.hist_thresh.attrs.update(long_name="Threshold over modeled data")

File <boltons.funcutils.FunctionBuilder-285>:2, in _map_blocks(ds, **kwargs)

File [d:\DEV\venv\cfgrib\lib\site-packages\xclim\sdba\base.py:481](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/base.py:481), in parse_group.<locals>._parse_group(*f_args, **f_kwargs)
    [478](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/base.py:478) @wraps(func)
    [479](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/base.py:479) def _parse_group(*f_args, **f_kwargs):
    [480](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/base.py:480)     f_kwargs = _update_kwargs(f_kwargs, allowed=allow_only)
--> [481](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/base.py:481)     return func(*f_args, **f_kwargs)

File [d:\DEV\venv\cfgrib\lib\site-packages\xclim\sdba\base.py:703](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/base.py:703), in map_blocks.<locals>._decorator.<locals>._map_blocks(ds, **kwargs)
    [700](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/base.py:700) tmpl = tmpl.drop_vars(extra_coords.keys(), errors="ignore")
    [702](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/base.py:702) # Call
--> [703](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/base.py:703) out = ds.map_blocks(
    [704](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/base.py:704)     _call_and_transpose_on_exit, template=tmpl, kwargs=kwargs
    [705](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/base.py:705) )
    [706](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/base.py:706) # Add back the extra coords, but only those which have compatible dimensions (like xarray would have done)
    [707](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/base.py:707) out = out.assign_coords(
    [708](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/base.py:708)     {
    [709](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/base.py:709)         name: crd
   (...)
    [712](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/base.py:712)     }
    [713](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/base.py:713) )

File [d:\DEV\venv\cfgrib\lib\site-packages\xarray\core\dataset.py:8773](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/dataset.py:8773), in Dataset.map_blocks(self, func, args, kwargs, template)
   [8671](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/dataset.py:8671) """
   [8672](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/dataset.py:8672) Apply a function to each block of this Dataset.
   [8673](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/dataset.py:8673) 
   (...)
   [8769](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/dataset.py:8769)     a        (time) float64 192B dask.array<chunksize=(24,), meta=np.ndarray>
   [8770](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/dataset.py:8770) """
   [8771](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/dataset.py:8771) from xarray.core.parallel import map_blocks
-> [8773](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/dataset.py:8773) return map_blocks(func, self, args, kwargs, template)

File [d:\DEV\venv\cfgrib\lib\site-packages\xarray\core\parallel.py:409](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/parallel.py:409), in map_blocks(func, obj, args, kwargs, template)
    [403](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/parallel.py:403)         raise TypeError(
    [404](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/parallel.py:404)             "Cannot pass dask collections in kwargs yet. Please compute or "
    [405](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/parallel.py:405)             "load values before passing to map_blocks."
    [406](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/parallel.py:406)         )
    [408](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/parallel.py:408) if not is_dask_collection(obj):
--> [409](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/parallel.py:409)     return func(obj, *args, **kwargs)
    [411](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/parallel.py:411) try:
    [412](file:///D:/DEV/venv/cfgrib/lib/site-packages/xarray/core/parallel.py:412)     import dask

File [d:\DEV\venv\cfgrib\lib\site-packages\xclim\sdba\base.py:686](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/base.py:686), in map_blocks.<locals>._decorator.<locals>._map_blocks.<locals>._call_and_transpose_on_exit(dsblock, **f_kwargs)
    [684](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/base.py:684)     func_out = func(dsblock, **f_kwargs).transpose(*all_dims)
    [685](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/base.py:685) except Exception as err:
--> [686](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/base.py:686)     raise ValueError(
    [687](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/base.py:687)         f"{func.__name__} failed on block with coords : {dsblock.coords}."
    [688](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/base.py:688)     ) from err
    [689](file:///D:/DEV/venv/cfgrib/lib/site-packages/xclim/sdba/base.py:689) return func_out

ValueError: loci_train failed on block with coords : Coordinates:
  * longitude  (longitude) float64 72B 8.8 9.2 9.6 10.0 10.4 10.8 11.2 11.6 12.0
  * latitude   (latitude) float64 48B 51.6 51.2 50.8 50.4 50.0 49.6
  * time       (time) datetime64[ns] 8kB 2003-07-10 2003-07-11 ... 2022-08-25
  * number     (number) int32 40B 1 2 3 4 5 6 7 8 9 10.

[aulemahal]

Indeed it doesnt work! Well, this is a related but different problem, Grouper kinda expects the add_dims to be present on all data it receives. And of course your observations don't have number. I was able to make LOCI work with minimal changes to the code, I'll try to push a PR soon.

In the mean time, a hack you can do is da_observed = da_observed.expand_dims(number=da_hindcast.number). I think it is mathematically equivalent as the series is simply replicated number times ?

[jamaa]

Thanks, that hack does allow the training to run, and the results do look plausible (ignore the last value, which I think is due to leap years):

loci.ds["af"].isel(latitude=3, longitude=4).plot()

But after that, I am having trouble applying the adjustment on a forecast. This is what my forecast DataArray da_forecast looks like:

xarray.DataArray'tp_mmd_pos' step: 47 number: 50 latitude: 6 longitude: 9
[126900 values with dtype=float32]
Coordinates:
number
(number)
int32
1 2 3 4 5 6 7 ... 45 46 47 48 49 50
step
(step)
timedelta64[ns]
0 days 1 days ... 45 days 46 days
latitude
(latitude)
float64
51.6 51.2 50.8 50.4 50.0 49.6
longitude
(longitude)
float64
8.8 9.2 9.6 10.0 ... 11.2 11.6 12.0
time
()
datetime64[ns]
...
surface
()
float64
...
valid_time
(step)
datetime64[ns]
...
Indexes: (4)
Attributes:
units :
mm/d

when I run

da_forecast_bc = loci.adjust(da_forecast)

I get the error KeyError: 'time'

Full traceback

{
	"name": "ValueError",
	"message": "loci_adjust failed on block with coords : Coordinates:
  * longitude  (longitude) float64 72B 8.8 9.2 9.6 10.0 10.4 10.8 11.2 11.6 12.0
  * latitude   (latitude) float64 48B 51.6 51.2 50.8 50.4 50.0 49.6
  * dayofyear  (dayofyear) int64 384B 191 192 193 194 195 ... 235 236 237 238
  * number     (number) int32 200B 1 2 3 4 5 6 7 8 9 ... 43 44 45 46 47 48 49 50
  * step       (step) timedelta64[ns] 376B 0 days 1 days ... 45 days 46 days.",
	"stack": "---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
File d:\\DEV\\venv\\cfgrib\\lib\\site-packages\\xclim\\sdba\\base.py:684, in map_blocks.<locals>._decorator.<locals>._map_blocks.<locals>._call_and_transpose_on_exit(dsblock, **f_kwargs)
    683     _decode_cf_coords(dsblock)
--> 684     func_out = func(dsblock, **f_kwargs).transpose(*all_dims)
    685 except Exception as err:

File d:\\DEV\\venv\\cfgrib\\lib\\site-packages\\xclim\\sdba\\_adjustment.py:210, in loci_adjust(ds, group, thresh, interp)
    204 \"\"\"LOCI: Adjust on one block.
    205 
    206 Dataset variables:
    207   hist_thresh : Hist's equivalent thresh from ref
    208   sim : Data to adjust
    209 \"\"\"
--> 210 sth = u.broadcast(ds.hist_thresh, ds.sim, group=group, interp=interp)
    211 factor = u.broadcast(ds.af, ds.sim, group=group, interp=interp)

File <boltons.funcutils.FunctionBuilder-261>:2, in broadcast(grouped, x, group, interp, sel)

File d:\\DEV\\venv\\cfgrib\\lib\\site-packages\\xclim\\sdba\\base.py:481, in parse_group.<locals>._parse_group(*f_args, **f_kwargs)
    480 f_kwargs = _update_kwargs(f_kwargs, allowed=allow_only)
--> 481 return func(*f_args, **f_kwargs)

File d:\\DEV\\venv\\cfgrib\\lib\\site-packages\\xclim\\sdba\\utils.py:213, in broadcast(grouped, x, group, interp, sel)
    212 if group.prop != \"group\" and group.prop not in sel:
--> 213     sel.update({group.prop: group.get_index(x, interp=interp != \"nearest\")})
    215 if sel:
    216     # Extract the correct mean factor for each time step.

File d:\\DEV\\venv\\cfgrib\\lib\\site-packages\\xclim\\sdba\\base.py:295, in Grouper.get_index(self, da, interp)
    293     return da[self.dim].rename(\"group\")
--> 295 ind = da.indexes[self.dim]
    296 if self.prop == \"week\":

File d:\\DEV\\venv\\cfgrib\\lib\\site-packages\\xarray\\core\\indexes.py:1662, in Indexes.__getitem__(self, key)
   1661 def __getitem__(self, key) -> T_PandasOrXarrayIndex:
-> 1662     return self._indexes[key]

KeyError: 'time'

The above exception was the direct cause of the following exception:

ValueError                                Traceback (most recent call last)
Cell In [25], line 1
----> 1 da_forecast_bc = loci.adjust(da_forecast)
      2 da_forecast_bc

File d:\\DEV\\venv\\cfgrib\\lib\\site-packages\\xclim\\sdba\\adjustment.py:217, in TrainAdjust.adjust(self, sim, *args, **kwargs)
    214         self._check_inputs(sim, *args, group=self.group)
    216     sim = convert_units_to(sim, self.train_units)
--> 217 out = self._adjust(sim, *args, **kwargs)
    219 if isinstance(out, xr.DataArray):
    220     out = out.rename(\"scen\").to_dataset()

File d:\\DEV\\venv\\cfgrib\\lib\\site-packages\\xclim\\sdba\\adjustment.py:793, in LOCI._adjust(self, sim, interp)
    792 def _adjust(self, sim, interp=\"linear\"):
--> 793     return loci_adjust(
    794         xr.Dataset(
    795             {\"hist_thresh\": self.ds.hist_thresh, \"af\": self.ds.af, \"sim\": sim}
    796         ),
    797         group=self.group,
    798         thresh=self.thresh,
    799         interp=interp,
    800     ).scen

File <boltons.funcutils.FunctionBuilder-286>:2, in _map_blocks(ds, **kwargs)

File d:\\DEV\\venv\\cfgrib\\lib\\site-packages\\xclim\\sdba\\base.py:481, in parse_group.<locals>._parse_group(*f_args, **f_kwargs)
    478 @wraps(func)
    479 def _parse_group(*f_args, **f_kwargs):
    480     f_kwargs = _update_kwargs(f_kwargs, allowed=allow_only)
--> 481     return func(*f_args, **f_kwargs)

File d:\\DEV\\venv\\cfgrib\\lib\\site-packages\\xclim\\sdba\\base.py:703, in map_blocks.<locals>._decorator.<locals>._map_blocks(ds, **kwargs)
    700 tmpl = tmpl.drop_vars(extra_coords.keys(), errors=\"ignore\")
    702 # Call
--> 703 out = ds.map_blocks(
    704     _call_and_transpose_on_exit, template=tmpl, kwargs=kwargs
    705 )
    706 # Add back the extra coords, but only those which have compatible dimensions (like xarray would have done)
    707 out = out.assign_coords(
    708     {
    709         name: crd
   (...)
    712     }
    713 )

File d:\\DEV\\venv\\cfgrib\\lib\\site-packages\\xarray\\core\\dataset.py:8773, in Dataset.map_blocks(self, func, args, kwargs, template)
   8671 \"\"\"
   8672 Apply a function to each block of this Dataset.
   8673 
   (...)
   8769     a        (time) float64 192B dask.array<chunksize=(24,), meta=np.ndarray>
   8770 \"\"\"
   8771 from xarray.core.parallel import map_blocks
-> 8773 return map_blocks(func, self, args, kwargs, template)

File d:\\DEV\\venv\\cfgrib\\lib\\site-packages\\xarray\\core\\parallel.py:409, in map_blocks(func, obj, args, kwargs, template)
    403         raise TypeError(
    404             \"Cannot pass dask collections in kwargs yet. Please compute or \"
    405             \"load values before passing to map_blocks.\"
    406         )
    408 if not is_dask_collection(obj):
--> 409     return func(obj, *args, **kwargs)
    411 try:
    412     import dask

File d:\\DEV\\venv\\cfgrib\\lib\\site-packages\\xclim\\sdba\\base.py:686, in map_blocks.<locals>._decorator.<locals>._map_blocks.<locals>._call_and_transpose_on_exit(dsblock, **f_kwargs)
    684     func_out = func(dsblock, **f_kwargs).transpose(*all_dims)
    685 except Exception as err:
--> 686     raise ValueError(
    687         f\"{func.__name__} failed on block with coords : {dsblock.coords}.\"
    688     ) from err
    689 return func_out

ValueError: loci_adjust failed on block with coords : Coordinates:
  * longitude  (longitude) float64 72B 8.8 9.2 9.6 10.0 10.4 10.8 11.2 11.6 12.0
  * latitude   (latitude) float64 48B 51.6 51.2 50.8 50.4 50.0 49.6
  * dayofyear  (dayofyear) int64 384B 191 192 193 194 195 ... 235 236 237 238
  * number     (number) int32 200B 1 2 3 4 5 6 7 8 9 ... 43 44 45 46 47 48 49 50
  * step       (step) timedelta64[ns] 376B 0 days 1 days ... 45 days 46 days."
}

I have tried removing the time coordinate, which is just the forecast initialization date, and renaming valid_time to time instead, but unfortunately with no luck. Any help would be greatly appreciated.