Stream: python-questions

Topic: xarray on cgd system

view this post on Zulip Anna-Lena Deppenmeier (Mar 11 2020 at 19:31):

Is anyone using xr.open_mfdataset() on the cgd machine? It doesn't work for me, failing with an error I haven't seen before:
distributed.protocol.core - CRITICAL - Failed to deserialize

this happens whenever / however I pass more than one files, either with a wildcard or when passing a list of files.
I made sure that my dask / distributed / xarray versions are the same as the ones I use on casper when running this.
any ideas?

view this post on Zulip Anderson Banihirwe (Mar 11 2020 at 20:37):

@Anna-Lena Deppenmeier,

What version of
- xarray
- dask
- distributed

are you using?

view this post on Zulip Anderson Banihirwe (Mar 11 2020 at 20:40):

What is the output of client.get_versions(check=True)?

view this post on Zulip Anna-Lena Deppenmeier (Mar 11 2020 at 20:43):

distrubuted version: 2.7.0
dask version: 2.7.0
xarray version: 0.14.1

view this post on Zulip Anna-Lena Deppenmeier (Mar 11 2020 at 20:43):

{'scheduler': {'host': (('python', '3.7.3.final.0'),
('python-bits', 64),
('OS', 'Linux'),
('OS-release', '3.10.0-1062.12.1.el7.x86_64'),
('machine', 'x86_64'),
('processor', 'x86_64'),
('byteorder', 'little'),
('LC_ALL', 'None'),
('LANG', 'en_US.UTF-8'),
('LOCALE', 'en_US.UTF-8')),
'packages': {'required': (('dask', '2.7.0'),
('distributed', '2.7.0'),
('msgpack', '1.0.0'),
('cloudpickle', '1.3.0'),
('tornado', '5.1.1'),
('toolz', '0.10.0')),
'optional': (('numpy', '1.16.4'),
('pandas', '0.25.0'),
('bokeh', '1.3.1'),
('lz4', '2.1.9'),
('dask_ml', None),
('blosc', '1.8.1'))}},
'workers': {'tcp://127.0.0.1:32838': {'host': (('python', '3.7.3.final.0'),
('python-bits', 64),
('OS', 'Linux'),
('OS-release', '3.10.0-1062.12.1.el7.x86_64'),
('machine', 'x86_64'),
('processor', 'x86_64'),
('byteorder', 'little'),
('LC_ALL', 'None'),
('LANG', 'en_US.UTF-8'),
('LOCALE', 'en_US.UTF-8')),
'packages': {'required': (('dask', '2.7.0'),
('distributed', '2.7.0'),
('msgpack', '1.0.0'),
('cloudpickle', '1.3.0'),
('tornado', '5.1.1'),
('toolz', '0.10.0')),
'optional': (('numpy', '1.16.4'),
('pandas', '0.25.0'),
('bokeh', '1.3.1'),
('lz4', '2.1.9'),
('dask_ml', None),
('blosc', '1.8.1'))}},
'tcp://127.0.0.1:36557': {'host': (('python', '3.7.3.final.0'),
('python-bits', 64),
('OS', 'Linux'),
('OS-release', '3.10.0-1062.12.1.el7.x86_64'),
('machine', 'x86_64'),
('processor', 'x86_64'),
('byteorder', 'little'),
('LC_ALL', 'None'),
('LANG', 'en_US.UTF-8'),
('LOCALE', 'en_US.UTF-8')),
'packages': {'required': (('dask', '2.7.0'),
('distributed', '2.7.0'),
('msgpack', '1.0.0'),
('cloudpickle', '1.3.0'),
('tornado', '5.1.1'),
('toolz', '0.10.0')),
'optional': (('numpy', '1.16.4'),
('pandas', '0.25.0'),
('bokeh', '1.3.1'),
('lz4', '2.1.9'),
('dask_ml', None),
('blosc', '1.8.1'))}},
'tcp://127.0.0.1:37239': {'host': (('python', '3.7.3.final.0'),
('python-bits', 64),
('OS', 'Linux'),
('OS-release', '3.10.0-1062.12.1.el7.x86_64'),
('machine', 'x86_64'),
('processor', 'x86_64'),
('byteorder', 'little'),
('LC_ALL', 'None'),
('LANG', 'en_US.UTF-8'),
('LOCALE', 'en_US.UTF-8')),
'packages': {'required': (('dask', '2.7.0'),
('distributed', '2.7.0'),
('msgpack', '1.0.0'),
('cloudpickle', '1.3.0'),
('tornado', '5.1.1'),
('toolz', '0.10.0')),
'optional': (('numpy', '1.16.4'),
('pandas', '0.25.0'),
('bokeh', '1.3.1'),
('lz4', '2.1.9'),
('dask_ml', None),
('blosc', '1.8.1'))}},
'tcp://127.0.0.1:37465': {'host': (('python', '3.7.3.final.0'),
('python-bits', 64),
('OS', 'Linux'),
('OS-release', '3.10.0-1062.12.1.el7.x86_64'),
('machine', 'x86_64'),
('processor', 'x86_64'),
('byteorder', 'little'),
('LC_ALL', 'None'),
('LANG', 'en_US.UTF-8'),
('LOCALE', 'en_US.UTF-8')),
'packages': {'required': (('dask', '2.7.0'),
('distributed', '2.7.0'),
('msgpack', '1.0.0'),
('cloudpickle', '1.3.0'),
('tornado', '5.1.1'),
('toolz', '0.10.0')),
'optional': (('numpy', '1.16.4'),
('pandas', '0.25.0'),
('bokeh', '1.3.1'),
('lz4', '2.1.9'),
('dask_ml', None),
('blosc', '1.8.1'))}},
'tcp://127.0.0.1:37751': {'host': (('python', '3.7.3.final.0'),
('python-bits', 64),
('OS', 'Linux'),
('OS-release', '3.10.0-1062.12.1.el7.x86_64'),
('machine', 'x86_64'),
('processor', 'x86_64'),
('byteorder', 'little'),
('LC_ALL', 'None'),
('LANG', 'en_US.UTF-8'),
('LOCALE', 'en_US.UTF-8')),
'packages': {'required': (('dask', '2.7.0'),
('distributed', '2.7.0'),
('msgpack', '1.0.0'),
('cloudpickle', '1.3.0'),
('tornado', '5.1.1'),
('toolz', '0.10.0')),
'optional': (('numpy', '1.16.4'),
('pandas', '0.25.0'),
('bokeh', '1.3.1'),
('lz4', '2.1.9'),
('dask_ml', None),
('blosc', '1.8.1'))}},
'tcp://127.0.0.1:38280': {'host': (('python', '3.7.3.final.0'),
('python-bits', 64),
('OS', 'Linux'),
('OS-release', '3.10.0-1062.12.1.el7.x86_64'),
('machine', 'x86_64'),
('processor', 'x86_64'),
('byteorder', 'little'),
('LC_ALL', 'None'),
('LANG', 'en_US.UTF-8'),
('LOCALE', 'en_US.UTF-8')),
'packages': {'required': (('dask', '2.7.0'),
('distributed', '2.7.0'),
('msgpack', '1.0.0'),
('cloudpickle', '1.3.0'),
('tornado', '5.1.1'),
('toolz', '0.10.0')),
'optional': (('numpy', '1.16.4'),
('pandas', '0.25.0'),
('bokeh', '1.3.1'),
('lz4', '2.1.9'),
('dask_ml', None),
('blosc', '1.8.1'))}}},
'client': {'host': [('python', '3.7.3.final.0'),
('python-bits', 64),
('OS', 'Linux'),
('OS-release', '3.10.0-1062.12.1.el7.x86_64'),
('machine', 'x86_64'),
('processor', 'x86_64'),
('byteorder', 'little'),
('LC_ALL', 'None'),
('LANG', 'en_US.UTF-8'),
('LOCALE', 'en_US.UTF-8')],
'packages': {'required': [('dask', '2.7.0'),
('distributed', '2.7.0'),
('msgpack', '1.0.0'),
('cloudpickle', '1.3.0'),
('tornado', '5.1.1'),
('toolz', '0.10.0')],
'optional': [('numpy', '1.16.4'),
('pandas', '0.25.0'),
('bokeh', '1.3.1'),
('lz4', '2.1.9'),
('dask_ml', None),
('blosc', '1.8.1')]}}}

view this post on Zulip Anderson Banihirwe (Mar 11 2020 at 20:51):

Interesting... I see that you are on ('msgpack', '1.0.0'). I suspect that this is where the issue is coming from ( a few weeks ago, this release of mspack had some incompatibility issue with dask)...

view this post on Zulip Anderson Banihirwe (Mar 11 2020 at 20:52):

I recommend upgrading your dask/distributed (even xarray) to the latest versions

view this post on Zulip Anderson Banihirwe (Mar 11 2020 at 20:53):

The latest versions of dask is 2.12.0

view this post on Zulip Anderson Banihirwe (Mar 11 2020 at 20:54):

Another solution is to downgrade msgpack to a version earlier than 1.0.0

view this post on Zulip Anna-Lena Deppenmeier (Mar 11 2020 at 20:55):

OKay, I will try to update everything and see how it goes. The reason I had specified the versions was because there was some problem with a later dask or distributed version, I think 2.9, but I see that that is superseded by now.

view this post on Zulip Anderson Banihirwe (Mar 11 2020 at 20:57):

Great! Let me know how it goes

view this post on Zulip Anna-Lena Deppenmeier (Mar 11 2020 at 21:06):

Thanks @Anderson Banihirwe , it works!

Last updated: Jan 30 2022 at 12:01 UTC