How to solve the memory issue while using the concat command?

[emaildipen]

data=xr.concat([xr.open_rasterio(i) for i in ENVIFiles], dim='time')
I use the above command but every time I use this code, it shows a memory issue.

[andersy005]

@emaildipen, when you get a moment, can you try loading files lazily via open_mfdataset

data = xr.open_mfdataset(ENVIFiles, engine="rasterio", chunks={}, parallel=True, concat_dim='time')

??

---

Note: if you are using xarray v0.18 or later, xr.open_rasterio has been deprecated: #4697

[emaildipen]

INSTALLED VERSIONS

commit: None
python: 3.7.6 | packaged by conda-forge | (default, Mar 23 2020, 22:22:21) [MSC v.1916 64 bit (AMD64)]
python-bits: 64
OS: Windows
OS-release: 10
machine: AMD64
processor: Intel64 Family 6 Model 142 Stepping 12, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None
LOCALE: None.None
libhdf5: 1.10.5
libnetcdf: None

xarray: 0.15.1
pandas: 1.0.3
numpy: 1.20.2
scipy: 1.3.1
netCDF4: None
pydap: None
h5netcdf: None
h5py: 2.10.0
Nio: None
zarr: None
cftime: None
nc_time_axis: None
PseudoNetCDF: None
rasterio: 1.1.3
cfgrib: None
iris: None
bottleneck: None
dask: 2021.03.1
distributed: 2021.04.0
matplotlib: 3.2.1
cartopy: None
seaborn: 0.11.1
numbagg: None
setuptools: 49.6.0.post20210108
pip: 20.0.2
conda: 4.10.3
pytest: 5.4.1
IPython: 7.22.0
sphinx: 2.4.4

[andersy005]

Ok. do you mind upgrading your xarray version to a more recent version and installing rioxarray (https://corteva.github.io/rioxarray/stable/installation.html)?

[andersy005]

Otherwise, you can try the following

data=xr.concat([xr.open_rasterio(i, chunks={}) for i in ENVIFiles], dim='time')

[emaildipen]

Ok. do you mind upgrading your xarray version to a more recent version and installing rioxarray (https://corteva.github.io/rioxarray/stable/installation.html)?

Will update it and try it. Thanks, much appriciated.

[andersy005]

@emaildipen, did this solution end up working for you?

[emaildipen]

I have 5 variables in xarray.Dataset. One of my variables 'DNB' has '65535' which represents the missing values. I want to change this value to nan. I tried this
NTL.where(NTL['DNB']!=65535), it worked but it changes the values of the other 4 variables as well to nan. Is there any other better way to do this?

[andersy005]

To leave the other 4 variables unaffected, try the following:

NTL['DNB'] = NTL['DNB'].where(NTL['DNB']!=65535)

[emaildipen]

It worked. Thank you!