memoryview is too large error · python-questions

Stream: python-questions

Topic: memoryview is too large error

Anna-Lena Deppenmeier (Jun 22 2021 at 20:43):

Hi there,

I am discovering new errors :nerd:
I am calculating the beloved heat budget, and there's this line

budget["DIA_IMPVF_TEMP"][:, 0, :, :] = ((SRF_TEMP_FLUX * dsxgcm.TAREA
                                         -
                                         dsxgcm.DIA_IMPVF_TEMP.isel(z_w_bot=0) * dsxgcm.TAREA)
                                        /
                                        dsxgcm.VOL.values[ 0, :, :])

which requires me to load budget["DIA_IMPVF_TEMP"] to be able to assign something. Now that seems to work, but once I want to plot, or just load a single column at a single time I am getting this error that I haven't seen before:

msgpack/_packer.pyx in msgpack._cmsgpack.Packer._pack()

ValueError: memoryview is too large

(the upper line repeats a good number of times) .

I was thinking I might have to "reverse" the .load because that's the only memory issue I could come up with (budget.DIA_IMPVF_TEMP is 14GB large)

Pinging @Anderson Banihirwe and @Deepak Cherian as per @Max Grover's suggestion

Anna-Lena Deppenmeier (Jun 22 2021 at 20:54):

examples for when the error comes up:

subset = budget.isel(nlon_t=lola_inds["i_140_w"], nlat_t=lola_inds["j_0n"], time=10).load()

subset.VDIF.plot(y="z_t", ylim=(300e2, 0), label="VDIF", ax=ax[0])

Anderson Banihirwe (Jun 22 2021 at 21:04):

What are the dimension sizes of budget["DIA_IMPVF_TEMP"] and does it contain a dask.array?

Anna-Lena Deppenmeier (Jun 22 2021 at 21:16):

('time', 'z_t', 'nlat_t', 'nlon_t') and it's a xarray.DataArray -- presumably because I had to

budget["DIA_IMPVF_TEMP"] = -(gridxgcm.diff(dsxgcm.DIA_IMPVF_TEMP * dsxgcm.TAREA, axis="Z") / dsxgcm.VOL).load()

inorder to make the assignment above

Anna-Lena Deppenmeier (Jun 22 2021 at 21:17):

another variable of budget is a dask array:

Anna-Lena Deppenmeier (Jun 22 2021 at 21:17):

pasted image

Anderson Banihirwe (Jun 22 2021 at 22:12):

What's the output of

xr.show_versions()

Last updated: Jan 30 2022 at 12:01 UTC