Seth McGinnis (Mar 13 2020 at 18:44): Seth McGinnis (Mar 13 2020 at 18:44):I'm working on an esm-intake catalog for the NA-CORDEX dataset. We'll use that to convert the dataset to zarr format, then upload it to AWS as a contribution to the Amazon Sustainability Data Initiative.
I also want to develop some pangeo jupyter notebooks for working with the data. I'm currently thinking we'll do some of the standard visualizations that we use for QA and general evaluation, but I'm open to ideas and suggestions.
Seth McGinnis (Mar 13 2020 at 21:40):I cloned the intake-esm-datastore repo on github and have added a first version of the catalog and JSON for the NA-CORDEX dataset: https://github.com/sethmcg/intake-esm-datastore
I think it should be functional, but I'm not sure where to go from here. How do I check and test it? @Anderson Banihirwe ?
Anderson Banihirwe (Mar 13 2020 at 22:09):There seems to be some formatting errors in the json:
In [1]: import intake In [5]: path = "intake-esm-datastore/catalogs/glade-na-cordex.json" In [6]: col = intake.open_esm_datastore(path) --------------------------------------------------------------------------- JSONDecodeError Traceback (most recent call last) <ipython-input-6-a5564d66b493> in <module> ----> 1 col = intake.open_esm_datastore(path) /glade/work/abanihi/softwares/miniconda3/envs/analysis/lib/python3.7/site-packages/intake_esm/core.py in __init__(self, esmcol_path, progressbar, log_level, **kwargs) 75 logger.setLevel(numeric_log_level) 76 self.progressbar = progressbar ---> 77 self._col_data, self.esmcol_path = _fetch_and_parse_json(esmcol_path) 78 self.df = _fetch_catalog(self._col_data, self.esmcol_path) 79 self._entries = {} /glade/work/abanihi/softwares/miniconda3/envs/analysis/lib/python3.7/site-packages/intake_esm/utils.py in _fetch_and_parse_json(input_path) 62 63 except Exception as e: ---> 64 raise e 65 66 return data, input_path /glade/work/abanihi/softwares/miniconda3/envs/analysis/lib/python3.7/site-packages/intake_esm/utils.py in _fetch_and_parse_json(input_path) 59 with open(input_path) as f: 60 logger.info(f'Loading ESMCol from filesystem: {input_path}') ---> 61 data = json.load(f) 62 63 except Exception as e: /glade/work/abanihi/softwares/miniconda3/envs/analysis/lib/python3.7/json/__init__.py in load(fp, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw) 294 cls=cls, object_hook=object_hook, 295 parse_float=parse_float, parse_int=parse_int, --> 296 parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw) 297 298 /glade/work/abanihi/softwares/miniconda3/envs/analysis/lib/python3.7/json/__init__.py in loads(s, encoding, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw) 346 parse_int is None and parse_float is None and 347 parse_constant is None and object_pairs_hook is None and not kw): --> 348 return _default_decoder.decode(s) 349 if cls is None: 350 cls = JSONDecoder /glade/work/abanihi/softwares/miniconda3/envs/analysis/lib/python3.7/json/decoder.py in decode(self, s, _w) 335 336 """ --> 337 obj, end = self.raw_decode(s, idx=_w(s, 0).end()) 338 end = _w(s, end).end() 339 if end != len(s): /glade/work/abanihi/softwares/miniconda3/envs/analysis/lib/python3.7/json/decoder.py in raw_decode(self, s, idx) 351 """ 352 try: --> 353 obj, end = self.scan_once(s, idx) 354 except StopIteration as err: 355 raise JSONDecodeError("Expecting value", s, err.value) from None JSONDecodeError: Expecting ':' delimiter: line 32 column 17 (char 1020) In [7]:
Anderson Banihirwe (Mar 13 2020 at 22:11):To validate the json against the schema, we have a tool (esmcol-validator), you can install it via conda or pip
Anderson Banihirwe (Mar 13 2020 at 22:11):Instructions on how to install it are available here: https://github.com/NCAR/esmcol-validator
Anderson Banihirwe (Mar 13 2020 at 22:12):Once it is installed, you can run
$ esmcol-validator glade-na-cordex.json
Seth McGinnis (Mar 13 2020 at 23:38):Thanks! Looks like it's time for me to learn how to manage pull requests, so I will return to this later.
Last updated: May 16 2025 at 17:14 UTC