KeyError when retrieving a zarr file · python-questions

Stream: python-questions

Topic: KeyError when retrieving a zarr file

Precious Mongwe (Jun 07 2020 at 16:59):

Has anyone encountered this error when trying to retrieve a stored zarr file:

/ncar/usr/jupyterhub/envs/cmip6-201910/lib/python3.7/site-packages/xarray/backends/zarr.py in open_zarr(store, group, synchronizer, chunks, decode_cf, mask_and_scale, decode_times, concat_characters, decode_coords, drop_variables, consolidated, overwrite_encoded_chunks, **kwargs)
595 synchronizer=synchronizer,
596 group=group,
--> 597 consolidated=consolidated,
598 )
599 ds = maybe_decode_store(zarr_store)

/ncar/usr/jupyterhub/envs/cmip6-201910/lib/python3.7/site-packages/xarray/backends/zarr.py in open_group(cls, store, mode, synchronizer, group, consolidated, consolidate_on_close)
257 if consolidated:
258 # TODO: an option to pass the metadata_key keyword
--> 259 zarr_group = zarr.open_consolidated(store, **open_kwargs)
260 else:
261 zarr_group = zarr.open_group(store, **open_kwargs)

/ncar/usr/jupyterhub/envs/cmip6-201910/lib/python3.7/site-packages/zarr/convenience.py in open_consolidated(store, metadata_key, mode, **kwargs)
1177
1178 # setup metadata sotre
-> 1179 meta_store = ConsolidatedMetadataStore(store, metadata_key=metadata_key)
1180
1181 # pass through

/ncar/usr/jupyterhub/envs/cmip6-201910/lib/python3.7/site-packages/zarr/storage.py in __init__(self, store, metadata_key)
2499
2500 # retrieve consolidated metadata
-> 2501 meta = json_loads(store[metadata_key])
2502
2503 # check format of consolidated metadata

/ncar/usr/jupyterhub/envs/cmip6-201910/lib/python3.7/site-packages/zarr/storage.py in __getitem__(self, key)
789 return self._fromfile(filepath)
790 else:
--> 791 raise KeyError(key)
792
793 def __setitem__(self, key, value):

KeyError: '.zmetadata'

Matt Long (Jun 07 2020 at 19:31):

@Precious Mongwe, can you post the path to the zarr archive you are trying to read? It's possible that error is a result of an incomplete archive.

Precious Mongwe (Jun 07 2020 at 19:35):

@Matt Long "/glade/scratch/pmongwe/LE_spl_proc_avr_data/LE_TEMP_hist_ds_anm_v4.zarr" /LE_TEMP_hist_ds_anm_v2.zarr

Matt Long (Jun 07 2020 at 19:37):

I can read
'/glade/scratch/pmongwe/LE_spl_proc_avr_data/LE_TEMP_hist_ds_anm_v4.zarr'

with no apparent problems. (xarray v0.15.1)

Precious Mongwe (Jun 07 2020 at 19:49):

I just tried this a different environment and it works!, thanks, problem solved!

Anderson Banihirwe (Jun 08 2020 at 05:41):

@Precious Mongwe,

Reading with consolidated=False works

In [2]: import xarray as xr

In [3]: path = "/glade/scratch/pmongwe/LE_spl_proc_avr_data/LE_TEMP_hist_ds_anm_v4.zarr"

In [4]: xr.open_zarr(path)
Out[4]:
<xarray.Dataset>
Dimensions:  (member_id: 34, nlat: 384, nlon: 320, year: 46, z_t: 45)
Coordinates:
    TLAT     (nlat, nlon) float64 dask.array<chunksize=(384, 320), meta=np.ndarray>
    TLONG    (nlat, nlon) float64 dask.array<chunksize=(384, 320), meta=np.ndarray>
  * year     (year) int64 1920 1921 1922 1923 1924 ... 1961 1962 1963 1964 1965
  * z_t      (z_t) float32 500.0 1500.0 2500.0 ... 140049.72 157394.64 176400.33
Dimensions without coordinates: member_id, nlat, nlon
Data variables:
    TEMP     (member_id, year, z_t, nlat, nlon) float32 dask.array<chunksize=(1, 6, 45, 384, 320), meta=np.ndarray>

Opening the store using zarr's consolidated metadata capability fails

In [5]: xr.open_zarr(path, consolidated=True)
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-5-00e040a4d4a9> in <module>
----> 1 xr.open_zarr(path, consolidated=True)

/glade/work/abanihi/softwares/miniconda3/envs/analysis/lib/python3.7/site-packages/xarray/backends/zarr.py in open_zarr(store, group, synchronizer, chunks, decode_cf, mask_and_scale, decode_times, concat_characters, decode_coords, drop_variables, consolidated, overwrite_encoded_chunks, **kwargs)
    595         synchronizer=synchronizer,
    596         group=group,
--> 597         consolidated=consolidated,
    598     )
    599     ds = maybe_decode_store(zarr_store)

/glade/work/abanihi/softwares/miniconda3/envs/analysis/lib/python3.7/site-packages/xarray/backends/zarr.py in open_group(cls, store, mode, synchronizer, group, consolidated, consolidate_on_close)
    257         if consolidated:
    258             # TODO: an option to pass the metadata_key keyword
--> 259             zarr_group = zarr.open_consolidated(store, **open_kwargs)
    260         else:
    261             zarr_group = zarr.open_group(store, **open_kwargs)

/glade/work/abanihi/softwares/miniconda3/envs/analysis/lib/python3.7/site-packages/zarr/convenience.py in open_consolidated(store, metadata_key, mode, **kwargs)
   1177
   1178     # setup metadata sotre
-> 1179     meta_store = ConsolidatedMetadataStore(store, metadata_key=metadata_key)
   1180
   1181     # pass through

/glade/work/abanihi/softwares/miniconda3/envs/analysis/lib/python3.7/site-packages/zarr/storage.py in __init__(self, store, metadata_key)
   2499
   2500         # retrieve consolidated metadata
-> 2501         meta = json_loads(store[metadata_key])
   2502
   2503         # check format of consolidated metadata

/glade/work/abanihi/softwares/miniconda3/envs/analysis/lib/python3.7/site-packages/zarr/storage.py in __getitem__(self, key)
    789             return self._fromfile(filepath)
    790         else:
--> 791             raise KeyError(key)
    792
    793     def __setitem__(self, key, value):

KeyError: '.zmetadata'

So, the error is also hinting at the fact that you probably didn't save the zarr store metadata in a consolidated file i.e. you saved it with ds.to_zarr(ZARR_PATH). To consolidate your zarr store metadata, you need to use ds.to_zarr(ZARR_PATH, consolidated=True)

Last updated: Jan 30 2022 at 12:01 UTC