Stream: python-questions

Topic: inconsistent files in CMIP6

Jean-Francois Lamarque (Aug 20 2021 at 23:10):

Hi I am trying to read the CMIP6 files on CGD in /project/data/cmip6/historical/day/tas/GFDL-CM4/r1i1p1f1/gr1.
I use xr.open_mfdata: ds=xr.open_mfdataset(files, concat_dim="time")

The problem is that the last file (tas_day_GFDL-CM4_historical_r1i1p1f1_gr1_20100101-20141231.nc) has a variable (height)
not present in the other files. I have tried the options in open_mfdata, including preprocessing but nothing seems
to work. I can't find a way to ignore this additional variable. Thanks!

Deepak Cherian (Aug 20 2021 at 23:14):

You can use the preprocess argument to provide a function that "cleans" up every dataset before the combine stage.

def preprocess(ds):
    return ds.drop_vars("height", errors="ignore")

ds = xr.open_mfdataset(..., preprocess=preprocess)

Jean-Francois Lamarque (Aug 20 2021 at 23:22):

I am getting the error message

Traceback (most recent call last):
File "interp_daily_tas_to_cities.py", line 59, in <module>
ds=xr.open_mfdataset(files, concat_dim="time", preprocess=preprocess)
File "/usr/local/anaconda-3.7.0/lib/python3.7/site-packages/xarray/backends/api.py", line 627, in open_mfdataset
datasets = [preprocess(ds) for ds in datasets]
File "/usr/local/anaconda-3.7.0/lib/python3.7/site-packages/xarray/backends/api.py", line 627, in <listcomp>
datasets = [preprocess(ds) for ds in datasets]
File "interp_daily_tas_to_cities.py", line 21, in preprocess
return ds.drop_vars("height", errors="ignore")
File "/usr/local/anaconda-3.7.0/lib/python3.7/site-packages/xarray/core/common.py", line 177, in __getattr__
(type(self).__name__, name))
AttributeError: 'Dataset' object has no attribute 'drop_vars'

Deepak Cherian (Aug 22 2021 at 16:42):

hmm.. must be an older version. Use .drop instead then

Jean-Francois Lamarque (Aug 23 2021 at 20:13):

Thanks, Deepak! It needed another step for it to work (to ensure that only the files that have "height" are affected by "drop"

def preprocess(ds):
if 'height' in list(ds.keys()):
return ds.drop("height")
else:
return ds

Notification Bot (Jan 26 2022 at 19:43):

This topic was moved here from #jupyterlab-hub > inconsistent files in CMIP6 by Anderson Banihirwe

Last updated: May 16 2025 at 17:14 UTC