Stream: general

Topic: xarray open_mfdataset time problem

Mari Tye (Jan 28 2025 at 23:03):

Hello all,

I created yearly files of daily total precipitation from ERA5. The netCDFs have dimensions:
double time(time) ;
time:standard_name = "time" ;
time:units = "days since 2023-01-01 00:00:00" ;
time:calendar = "proleptic_gregorian" ;
time:axis = "T" ;
double longitude(longitude) ;
longitude:standard_name = "longitude" ;
longitude:long_name = "longitude" ;
longitude:units = "degrees_east" ;
longitude:axis = "X" ;
double latitude(latitude) ;
latitude:standard_name = "latitude" ;
latitude:long_name = "latitude" ;
latitude:units = "degrees_north" ;
latitude:axis = "Y" ;
float TP(time, latitude, longitude) ;
TP:_FillValue = NaNf ;
TP:missing_value = NaNf ;

Reading the files into xarray with open_mfdataset throws a cftime error. But reading them in with decode_cf=False gives

"ValueError: Coordinate variable time is neither monotonically increasing nor monotonically decreasing on all datasets"

Is there a simple work around? Or do I need to recreate the netCDFs with a common base year?

Thanks,
Mari

Katelyn FitzGerald (Jan 28 2025 at 23:07):

Could you share the cftime error?

Mari Tye (Jan 29 2025 at 16:45):

There were two messages:

OverflowError: time values outside range of 64 bit signed integers

During handling of the above exception, another exception occurred:

ValueError: unable to decode time units 'days since 1982-01-01 00:00:00' with "calendar 'proleptic_gregorian'". Try opening your dataset with decode_times=False or installing cftime if it is not installed.

Katelyn FitzGerald (Jan 29 2025 at 17:00):

Thanks!

I think you're right that you do need consistent time units for your time dimension with the multi-file dataset, but you can probably write a preprocess function that handles this (i.e. the adjusting the times to be consistent across files) to pass to open_mfdataset if you'd like to avoid rewriting the files.

Mari Tye (Jan 29 2025 at 17:16):

Thank you! Would the preprocess be something along these lines?:
times = xr.date_range(start='1940-01-01', end='2023-12-31', freq='D')
ds['dates'] = times
ds01 = ds0.assign_coords({"index": ds0.dates.values}).drop("time").rename({"dates":"time"}).drop_dims("index")

Katelyn FitzGerald (Jan 29 2025 at 17:52):

I think you'd want to do something a little different. Do you have couple of sample files you could point me to?

Mari Tye (Jan 29 2025 at 18:21):

I've copied a decade into /glade/derecho/scratch/maritye/ERA5/

Katelyn FitzGerald (Jan 29 2025 at 18:34):

Agh, it looks like I don't have read permissions.

Mari Tye (Jan 29 2025 at 18:42):

just changed the permissions

Last updated: May 16 2025 at 17:14 UTC