Anna-Lena Deppenmeier (Jun 22 2021 at 20:43): Anna-Lena Deppenmeier (Jun 22 2021 at 20:43):Hi there,
I am discovering new errors :nerd:
I am calculating the beloved heat budget, and there's this line
budget["DIA_IMPVF_TEMP"][:, 0, :, :] = ((SRF_TEMP_FLUX * dsxgcm.TAREA - dsxgcm.DIA_IMPVF_TEMP.isel(z_w_bot=0) * dsxgcm.TAREA) / dsxgcm.VOL.values[ 0, :, :])
which requires me to load budget["DIA_IMPVF_TEMP"] to be able to assign something. Now that seems to work, but once I want to plot, or just load a single column at a single time I am getting this error that I haven't seen before:
msgpack/_packer.pyx in msgpack._cmsgpack.Packer._pack() ValueError: memoryview is too large
(the upper line repeats a good number of times) .
I was thinking I might have to "reverse" the .load because that's the only memory issue I could come up with (budget.DIA_IMPVF_TEMP is 14GB large)
Pinging @Anderson Banihirwe and @Deepak Cherian as per @Max Grover's suggestion
Anna-Lena Deppenmeier (Jun 22 2021 at 20:54):examples for when the error comes up:
subset = budget.isel(nlon_t=lola_inds["i_140_w"], nlat_t=lola_inds["j_0n"], time=10).load()
or
subset.VDIF.plot(y="z_t", ylim=(300e2, 0), label="VDIF", ax=ax[0])
Anderson Banihirwe (Jun 22 2021 at 21:04):What are the dimension sizes of budget["DIA_IMPVF_TEMP"] and does it contain a dask.array?
Anna-Lena Deppenmeier (Jun 22 2021 at 21:16):('time', 'z_t', 'nlat_t', 'nlon_t') and it's a xarray.DataArray -- presumably because I had to
budget["DIA_IMPVF_TEMP"] = -(gridxgcm.diff(dsxgcm.DIA_IMPVF_TEMP * dsxgcm.TAREA, axis="Z") / dsxgcm.VOL).load()
inorder to make the assignment above
Anna-Lena Deppenmeier (Jun 22 2021 at 21:17):another variable of budget is a dask array:
Anna-Lena Deppenmeier (Jun 22 2021 at 21:17):
Anderson Banihirwe (Jun 22 2021 at 22:12):What's the output of
xr.show_versions()
?
Deepak Cherian (Jun 23 2021 at 17:27):@Anna-Lena Deppenmeier can you provide your dask version? Dask allows assignment , but only with very recent versions: https://docs.dask.org/en/latest/array-assignment.html
Anna-Lena Deppenmeier (Jun 23 2021 at 17:28):'2021.02.0'
Anna-Lena Deppenmeier (Jun 23 2021 at 17:28):What's the output of
xr.show_versions()?
INSTALLED VERSIONS
commit: None
python: 3.8.8 | packaged by conda-forge | (default, Feb 20 2021, 16:22:27)
[GCC 9.3.0]
python-bits: 64
OS: Linux
OS-release: 3.10.0-1160.25.1.el7.x86_64
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: ('en_US', 'UTF-8')
libhdf5: 1.10.6
libnetcdf: 4.7.4
xarray: 0.18.2
pandas: 1.2.2
numpy: 1.20.1
scipy: 1.5.3
netCDF4: 1.5.6
pydap: None
h5netcdf: 0.10.0
h5py: 3.1.0
Nio: None
zarr: 2.6.1
cftime: 1.4.1
nc_time_axis: 1.2.0
PseudoNetCDF: None
rasterio: None
cfgrib: None
iris: 2.4.0
bottleneck: 1.3.2
dask: 2021.02.0
distributed: 2021.02.0
matplotlib: 3.3.4
cartopy: 0.18.0
seaborn: 0.11.1
numbagg: None
pint: None
setuptools: 49.6.0.post20210108
pip: 21.0.1
conda: 4.9.2
pytest: 6.2.2
IPython: 7.21.0
sphinx: 3.5.1
Deepak Cherian (Jun 23 2021 at 17:30):Looks like it went in on FEb 10, and is in 2021.03.0
Anna-Lena Deppenmeier (Jun 23 2021 at 17:35):ok I will update
Anna-Lena Deppenmeier (Jun 23 2021 at 17:40):I update like this conda install -c conda-forge dask==2021.03.0 correct? Asking because it's taking forever to solve environment Solving environment: failed with initial frozen solve. Retrying with flexible solve.
and conda update dask told me all requested packages already installed.
Deepak Cherian (Jun 23 2021 at 17:43):try mamba instead of conda?
Deepak Cherian (Jun 23 2021 at 17:43):@Anderson Banihirwe can you put your usual tricks for this scenario in a FAQ entry?
Anna-Lena Deppenmeier (Jun 23 2021 at 17:55):mamba worked (and looks super fancy) and the assignment also works now :tada:
Deepak Cherian (Jun 23 2021 at 17:56):Nice! Let's add a note to the top of the notebook then. We should consider distributing some environment files.
Anna-Lena Deppenmeier (Jun 23 2021 at 17:56):definitely
Anna-Lena Deppenmeier (Jun 23 2021 at 17:57):will do that after lunch.
Anderson Banihirwe (Jun 23 2021 at 18:30):Anderson Banihirwe can you put your usual tricks for this scenario in a FAQ entry?
Done. The PR resides here: https://github.com/NCAR/esds/pull/71
Deepak Cherian (Jun 23 2021 at 18:32):awesome thanks!
Last updated: May 16 2025 at 17:14 UTC