Cloud-optimized access to CESM Timeseries netCDF files in the Cloud with kerchunk

Summary

We benchmark reading a subset of the CESM2-Large Ensemble stored as a collection of netCDF files on the cloud (Amazon / AWS) from Casper. We use a single ensemble member historical experiment with daily data from 1850 to 2009, with a total dataset size of 600+ GB, from 13 netCDF4 files.

We read in two ways:

• accessing the netCDF files directly

• using kerchunk as an intermediate layer to enable cloud-optimized access.

File Access Method	Dataset Lazy Read In Wall Time	Visualization Wall Time
s3 netcdf	8-9 s	6 min 21 s
kerchunk	460 ms	43.4 s

The “s3 netcdf” row refers to accessing the data remotely using the traditional s3 api, the interface with Amazon’s cloud storage. There are Google and Microsoft equivalents of this

kerchunk is a new package, developed within the fsspec python community, which aims to improve I/O performance when reading from cloud hosted datasets which are not neccessarily in a “cloud optimized format (ex. netCDF vs. Zarr) where netCDF was engineered to performant on regular POSIX filesystems.

Note that usually you would want to execute some benchmarks and analysis on a cloud instance. The performance benefits here extend to that use case but the gap will be smaller (but still significant). For more see this post and this post.

Tip

The Project Pythia Cookbook on kerchunk is a great resource!

Imports

%load_ext watermark

import glob
import os

import cftime
import dask
import fsspec
import holoviews as hv
import hvplot.xarray
import kerchunk
import s3fs
import ujson  # fast json
import xarray as xr
from distributed import Client
from kerchunk.combine import MultiZarrToZarr
from kerchunk.hdf import SingleHdf5ToZarr
from ncar_jobqueue import NCARCluster

hv.extension('bokeh')

%watermark -iv

xarray   : 2023.2.0
ujson    : 5.7.0
json     : 2.0.9
dask     : 2023.3.0
kerchunk : 0.1.0
s3fs     : 2023.3.0
hvplot   : 0.8.2
cftime   : 1.6.2
fsspec   : 2023.3.0
holoviews: 1.15.4
sys      : 3.11.0 | packaged by conda-forge | (main, Jan 14 2023, 12:27:40) [GCC 11.3.0]

Gather a list of files available from an Amazon s3 bucket

As mentioned previously, s3 is Amazon’s object store file system. For this example, I moved a single ensemble member’s worth of daily historical timeseries output from the CESM2-LE dataset.

We use fsspec to create a filesystem-like representation:

fs = fsspec.filesystem('s3', anon=True, skip_instance_cache=True)

filelist = fs.glob(f'ncar-cesm2-lens/tseries/*nc')
filelist

['ncar-cesm2-lens/tseries/b.e21.BHISTcmip6.f09_g17.LE2-1001.001.cam.h6.T.18500101-18591231.nc',
 'ncar-cesm2-lens/tseries/b.e21.BHISTcmip6.f09_g17.LE2-1001.001.cam.h6.T.18600101-18691231.nc',
 'ncar-cesm2-lens/tseries/b.e21.BHISTcmip6.f09_g17.LE2-1001.001.cam.h6.T.18700101-18791231.nc',
 'ncar-cesm2-lens/tseries/b.e21.BHISTcmip6.f09_g17.LE2-1001.001.cam.h6.T.18800101-18891231.nc',
 'ncar-cesm2-lens/tseries/b.e21.BHISTcmip6.f09_g17.LE2-1001.001.cam.h6.T.18900101-18991231.nc',
 'ncar-cesm2-lens/tseries/b.e21.BHISTcmip6.f09_g17.LE2-1001.001.cam.h6.T.19000101-19091231.nc',
 'ncar-cesm2-lens/tseries/b.e21.BHISTcmip6.f09_g17.LE2-1001.001.cam.h6.T.19100101-19191231.nc',
 'ncar-cesm2-lens/tseries/b.e21.BHISTcmip6.f09_g17.LE2-1001.001.cam.h6.T.19200101-19291231.nc',
 'ncar-cesm2-lens/tseries/b.e21.BHISTcmip6.f09_g17.LE2-1001.001.cam.h6.T.19300101-19391231.nc',
 'ncar-cesm2-lens/tseries/b.e21.BHISTcmip6.f09_g17.LE2-1001.001.cam.h6.T.19400101-19491231.nc',
 'ncar-cesm2-lens/tseries/b.e21.BHISTcmip6.f09_g17.LE2-1001.001.cam.h6.T.19500101-19591231.nc',
 'ncar-cesm2-lens/tseries/b.e21.BHISTcmip6.f09_g17.LE2-1001.001.cam.h6.T.19600101-19691231.nc',
 'ncar-cesm2-lens/tseries/b.e21.BHISTcmip6.f09_g17.LE2-1001.001.cam.h6.T.19700101-19791231.nc',
 'ncar-cesm2-lens/tseries/b.e21.BHISTcmip6.f09_g17.LE2-1001.001.cam.h6.T.19800101-19891231.nc',
 'ncar-cesm2-lens/tseries/b.e21.BHISTcmip6.f09_g17.LE2-1001.001.cam.h6.T.19900101-19991231.nc',
 'ncar-cesm2-lens/tseries/b.e21.BHISTcmip6.f09_g17.LE2-1001.001.cam.h6.T.20000101-20091231.nc',
 'ncar-cesm2-lens/tseries/b.e21.BHISTcmip6.f09_g17.LE2-1001.001.cam.h6.T.20100101-20141231.nc']

We can check the size of a single dataset here

print(fs.size(filelist[0]) / 1e9, 'gb')

37.649093521 gb

We need to add the s3:// portion at the beginning to specify this is from the s3 filesystem.

Note

We’ll use only last 5 files to reduce access time.

filelist = filelist[-5:]
urls = ["s3://" + f for f in filelist]
so = dict(mode='rb', anon=True, default_fill_cache=False, default_cache_type='first')

Approach

kerchunk works by first reading a “reference” file that maps virtual chunks of data to byte ranges stored in specific files. This “reference” file looks like a Zarr dataset, and is read using the Zarr library. Previously, we demonstrated the use of Kerchunk to create virtual aggregate datasets without modifying the files on disk.

Here, the goal is to demonstrate cloud-optimized data access enabled by using Zarr as an intermediate layer to access underlying netCDF files.

The approach is similar:

• We generate a “reference” JSON file, effectively a Zarr representation, for each netCDF file. kerchunk provides a helper function for this. netCDF4 files are HDF5 files so we use that helper function: kerchunk.hdf.SingleHdf5ToZarr

• We combine these into a single reference file representing the aggregate dataset: MultiZarrToZarr

Generate the Mapper Files

Set the output directory of the json mapping files

json_dir = 'cesm2_le_reference_files'

We set up a helper function to help with generating these files

def gen_json(u):
    with fs.open(u, **so) as infile:
        h5chunks = SingleHdf5ToZarr(infile, u, inline_threshold=300)
        p = u.split('/')
        fname = p[-1]
        outf = f'{fname}.json'
        print(outf)
        with open(f"{json_dir}/{outf}", 'wb') as f:
            f.write(ujson.dumps(h5chunks.translate()).encode());

Spin up a Cluster

We will spin up a dask cluster to add distributed/parallel computing - we are spinning up 10 workers, each with 25 GB of RAM and a single processor

cluster = NCARCluster()
cluster.scale(5)
client = Client(cluster)
client

/glade/u/home/dcherian/miniconda3/envs/os-obs/lib/python3.11/site-packages/dask_jobqueue/core.py:255: FutureWarning: job_extra has been renamed to job_extra_directives. You are still using it (even if only set to []; please also check config files). If you did not set job_extra_directives yet, job_extra will be respected for now, but it will be removed in a future release. If you already set job_extra_directives, job_extra is ignored and you can remove it.
  warnings.warn(warn, FutureWarning)
/glade/u/home/dcherian/miniconda3/envs/os-obs/lib/python3.11/site-packages/dask_jobqueue/core.py:274: FutureWarning: env_extra has been renamed to job_script_prologue. You are still using it (even if only set to []; please also check config files). If you did not set job_script_prologue yet, env_extra will be respected for now, but it will be removed in a future release. If you already set job_script_prologue, env_extra is ignored and you can remove it.
  warnings.warn(warn, FutureWarning)
/glade/u/home/dcherian/miniconda3/envs/os-obs/lib/python3.11/site-packages/distributed/node.py:182: UserWarning: Port 8787 is already in use.
Perhaps you already have a cluster running?
Hosting the HTTP server on port 43962 instead
  warnings.warn(

Client

Client-ccd4a03a-c33f-11ed-a1ea-3cecef1b11fa

Connection method: Cluster object	Cluster type: dask_jobqueue.PBSCluster
Dashboard: https://jupyterhub.hpc.ucar.edu/stable/user/dcherian/proxy/43962/status

Cluster Info

PBSCluster

e1214b55

Dashboard: https://jupyterhub.hpc.ucar.edu/stable/user/dcherian/proxy/43962/status	Workers: 0
Total threads: 0	Total memory: 0 B

Scheduler Info

Scheduler

Scheduler-f59b0bfe-ae4f-409a-8454-e622b658eb45

Comm: tcp://10.12.206.54:37597	Workers: 0
Dashboard: https://jupyterhub.hpc.ucar.edu/stable/user/dcherian/proxy/43962/status	Total threads: 0
Started: Just now	Total memory: 0 B

Workers

Create reference file for each netCDF file

We can compute these reference files in parallel!

%%time
dask.compute(*[dask.delayed(gen_json)(u) for u in urls], retries=10);

CPU times: user 7.47 s, sys: 580 ms, total: 8.05 s
Wall time: 3min 25s

Merge the reference files together

The previous process produced a single reference file for each netCDF file - let’s merge these all together to create single store.

reference_files = sorted(glob.glob('cesm2_le_reference_files/*'))
reference_files

['cesm2_le_reference_files/b.e21.BHISTcmip6.f09_g17.LE2-1001.001.cam.h6.T.19700101-19791231.nc.json',
 'cesm2_le_reference_files/b.e21.BHISTcmip6.f09_g17.LE2-1001.001.cam.h6.T.19800101-19891231.nc.json',
 'cesm2_le_reference_files/b.e21.BHISTcmip6.f09_g17.LE2-1001.001.cam.h6.T.19900101-19991231.nc.json',
 'cesm2_le_reference_files/b.e21.BHISTcmip6.f09_g17.LE2-1001.001.cam.h6.T.20000101-20091231.nc.json',
 'cesm2_le_reference_files/b.e21.BHISTcmip6.f09_g17.LE2-1001.001.cam.h6.T.20100101-20141231.nc.json']

Now, we can generate the single store using MultiZarrToZarr! We add a few extra arguments here… no need to worry about the details here.

mzz = MultiZarrToZarr(
    reference_files,
    target_options={'anon': False},
    remote_protocol='s3',
    remote_options={'anon': 'True'},  # JSON files
    concat_dims="time",
)

After merging our reference files together, we can output the new json!

%%time
#%%prun -D multizarr_profile
mzz.translate('cesm2-le-tseries-aws.json')

CPU times: user 2.27 s, sys: 578 ms, total: 2.85 s
Wall time: 3.29 s

---

Read directly from S3 (without kerchunk)

We went through all that work previously, but let’s take a look at what the performance would be without using this new Kerchunk package.

To open the files, use fsspec again and generate a list of URLs.

fs_s3 = s3fs.S3FileSystem(anon=True)

flist = [fs_s3.open(path, mode="rb") for path in urls]

Now read in with open_mfdataset

Warning

parallel=True does not work at the moment.

Our wall time for lazily reading in the datasets isn’t too bad…

%%time
ds_read_directly = xr.open_mfdataset(
    flist,
    engine="h5netcdf",
    chunks={"lat": 20, "lon": 20},
    data_vars="minimal",
    coords="minimal",
    compat="override",
)

CPU times: user 2.69 s, sys: 310 ms, total: 3 s
Wall time: 9.92 s

ds_read_directly

<xarray.Dataset>
Dimensions:       (lat: 192, lon: 288, zlon: 1, nbnd: 2, lev: 32, ilev: 33,
                   time: 16426)
Coordinates:
  * lat           (lat) float64 -90.0 -89.06 -88.12 -87.17 ... 88.12 89.06 90.0
  * lon           (lon) float64 0.0 1.25 2.5 3.75 ... 355.0 356.2 357.5 358.8
  * zlon          (zlon) float64 0.0
  * lev           (lev) float64 3.643 7.595 14.36 24.61 ... 957.5 976.3 992.6
  * ilev          (ilev) float64 2.255 5.032 10.16 18.56 ... 967.5 985.1 1e+03
  * time          (time) object 1970-01-01 00:00:00 ... 2015-01-01 00:00:00
Dimensions without coordinates: nbnd
Data variables: (12/27)
    gw            (lat) float64 dask.array<chunksize=(20,), meta=np.ndarray>
    zlon_bnds     (zlon, nbnd) float64 dask.array<chunksize=(1, 2), meta=np.ndarray>
    hyam          (lev) float64 dask.array<chunksize=(32,), meta=np.ndarray>
    hybm          (lev) float64 dask.array<chunksize=(32,), meta=np.ndarray>
    P0            float64 ...
    hyai          (ilev) float64 dask.array<chunksize=(33,), meta=np.ndarray>
    ...            ...
    n2ovmr        (time) float64 dask.array<chunksize=(3650,), meta=np.ndarray>
    f11vmr        (time) float64 dask.array<chunksize=(3650,), meta=np.ndarray>
    f12vmr        (time) float64 dask.array<chunksize=(3650,), meta=np.ndarray>
    sol_tsi       (time) float64 dask.array<chunksize=(3650,), meta=np.ndarray>
    nsteph        (time) float64 dask.array<chunksize=(3650,), meta=np.ndarray>
    T             (time, lev, lat, lon) float64 dask.array<chunksize=(3650, 32, 20, 20), meta=np.ndarray>
Attributes:
    Conventions:       CF-1.0
    source:            CAM
    case:              b.e21.BHISTcmip6.f09_g17.LE2-1001.001
    logname:           sunseon
    host:              mom2
    initial_file:      b.e21.B1850.f09_g17.CMIP6-piControl.001.cam.i.1001-01-...
    topography_file:   /mnt/lustre/share/CESM/cesm_input/atm/cam/topo/fv_0.9x...
    model_doi_url:     https://doi.org/10.5065/D67H1H0V
    time_period_freq:  day_1

xarray.Dataset

• Dimensions:
  • lat: 192
  • lon: 288
  • zlon: 1
  • nbnd: 2
  • lev: 32
  • ilev: 33
  • time: 16426
• Coordinates: (6)
  • lat
    (lat)
    float64
    -90.0 -89.06 -88.12 ... 89.06 90.0

    long_name :

    latitude

    units :

    degrees_north

    array([-90.      , -89.057592, -88.115183, -87.172775, -86.230366, -85.287958,
           -84.34555 , -83.403141, -82.460733, -81.518325, -80.575916, -79.633508,
           -78.691099, -77.748691, -76.806283, -75.863874, -74.921466, -73.979058,
           -73.036649, -72.094241, -71.151832, -70.209424, -69.267016, -68.324607,
           -67.382199, -66.439791, -65.497382, -64.554974, -63.612565, -62.670157,
           -61.727749, -60.78534 , -59.842932, -58.900524, -57.958115, -57.015707,
           -56.073298, -55.13089 , -54.188482, -53.246073, -52.303665, -51.361257,
           -50.418848, -49.47644 , -48.534031, -47.591623, -46.649215, -45.706806,
           -44.764398, -43.82199 , -42.879581, -41.937173, -40.994764, -40.052356,
           -39.109948, -38.167539, -37.225131, -36.282723, -35.340314, -34.397906,
           -33.455497, -32.513089, -31.570681, -30.628272, -29.685864, -28.743455,
           -27.801047, -26.858639, -25.91623 , -24.973822, -24.031414, -23.089005,
           -22.146597, -21.204188, -20.26178 , -19.319372, -18.376963, -17.434555,
           -16.492147, -15.549738, -14.60733 , -13.664921, -12.722513, -11.780105,
           -10.837696,  -9.895288,  -8.95288 ,  -8.010471,  -7.068063,  -6.125654,
            -5.183246,  -4.240838,  -3.298429,  -2.356021,  -1.413613,  -0.471204,
             0.471204,   1.413613,   2.356021,   3.298429,   4.240838,   5.183246,
             6.125654,   7.068063,   8.010471,   8.95288 ,   9.895288,  10.837696,
            11.780105,  12.722513,  13.664921,  14.60733 ,  15.549738,  16.492147,
            17.434555,  18.376963,  19.319372,  20.26178 ,  21.204188,  22.146597,
            23.089005,  24.031414,  24.973822,  25.91623 ,  26.858639,  27.801047,
            28.743455,  29.685864,  30.628272,  31.570681,  32.513089,  33.455497,
            34.397906,  35.340314,  36.282723,  37.225131,  38.167539,  39.109948,
            40.052356,  40.994764,  41.937173,  42.879581,  43.82199 ,  44.764398,
            45.706806,  46.649215,  47.591623,  48.534031,  49.47644 ,  50.418848,
            51.361257,  52.303665,  53.246073,  54.188482,  55.13089 ,  56.073298,
            57.015707,  57.958115,  58.900524,  59.842932,  60.78534 ,  61.727749,
            62.670157,  63.612565,  64.554974,  65.497382,  66.439791,  67.382199,
            68.324607,  69.267016,  70.209424,  71.151832,  72.094241,  73.036649,
            73.979058,  74.921466,  75.863874,  76.806283,  77.748691,  78.691099,
            79.633508,  80.575916,  81.518325,  82.460733,  83.403141,  84.34555 ,
            85.287958,  86.230366,  87.172775,  88.115183,  89.057592,  90.      ])

  • lon
    (lon)
    float64
    0.0 1.25 2.5 ... 356.2 357.5 358.8

    long_name :

    longitude

    units :

    degrees_east

    array([  0.  ,   1.25,   2.5 , ..., 356.25, 357.5 , 358.75])

  • zlon
    (zlon)
    float64
    0.0

    long_name :

    longitude

    units :

    degrees_east

    bounds :

    zlon_bnds

    array([0.])

  • lev
    (lev)
    float64
    3.643 7.595 14.36 ... 976.3 992.6

    long_name :

    hybrid level at midpoints (1000*(A+B))

    units :

    hPa

    positive :

    down

    standard_name :

    atmosphere_hybrid_sigma_pressure_coordinate

    formula_terms :

    a: hyam b: hybm p0: P0 ps: PS

    array([  3.643466,   7.59482 ,  14.356632,  24.61222 ,  35.92325 ,  43.19375 ,
            51.677499,  61.520498,  73.750958,  87.82123 , 103.317127, 121.547241,
           142.994039, 168.22508 , 197.908087, 232.828619, 273.910817, 322.241902,
           379.100904, 445.992574, 524.687175, 609.778695, 691.38943 , 763.404481,
           820.858369, 859.534767, 887.020249, 912.644547, 936.198398, 957.48548 ,
           976.325407, 992.556095])

  • ilev
    (ilev)
    float64
    2.255 5.032 10.16 ... 985.1 1e+03

    long_name :

    hybrid level at interfaces (1000*(A+B))

    units :

    hPa

    positive :

    down

    standard_name :

    atmosphere_hybrid_sigma_pressure_coordinate

    formula_terms :

    a: hyai b: hybi p0: P0 ps: PS

    array([   2.25524 ,    5.031692,   10.157947,   18.555317,   29.734676,
             39.273001,   47.114499,   56.240499,   66.800497,   80.701418,
             94.941042,  111.693211,  131.401271,  154.586807,  181.863353,
            213.952821,  251.704417,  296.117216,  348.366588,  409.835219,
            482.149929,  567.224421,  652.332969,  730.445892,  796.363071,
            845.353667,  873.715866,  900.324631,  924.964462,  947.432335,
            967.538625,  985.11219 , 1000.      ])

  • time
    (time)
    object
    1970-01-01 00:00:00 ... 2015-01-...

    long_name :

    time

    bounds :

    time_bnds

    array([cftime.DatetimeNoLeap(1970, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1970, 1, 2, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1970, 1, 3, 0, 0, 0, 0, has_year_zero=True), ...,
           cftime.DatetimeNoLeap(2014, 12, 30, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2014, 12, 31, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 1, 1, 0, 0, 0, 0, has_year_zero=True)],
          dtype=object)

• Data variables: (27)
  • gw
    (lat)
    float64
    dask.array<chunksize=(20,), meta=np.ndarray>

    long_name :

    latitude weights

    Array Chunk Bytes 1.50 kiB 160 B Shape (192,) (20,) Dask graph 10 chunks in 2 graph layers Data type float64 numpy.ndarray

  • zlon_bnds
    (zlon, nbnd)
    float64
    dask.array<chunksize=(1, 2), meta=np.ndarray>

    long_name :

    zlon bounds

    units :

    degrees_east

    Array Chunk Bytes 16 B 16 B Shape (1, 2) (1, 2) Dask graph 1 chunks in 2 graph layers Data type float64 numpy.ndarray

  • hyam
    (lev)
    float64
    dask.array<chunksize=(32,), meta=np.ndarray>

    long_name :

    hybrid A coefficient at layer midpoints

    Array Chunk Bytes 256 B 256 B Shape (32,) (32,) Dask graph 1 chunks in 2 graph layers Data type float64 numpy.ndarray

  • hybm
    (lev)
    float64
    dask.array<chunksize=(32,), meta=np.ndarray>

    long_name :

    hybrid B coefficient at layer midpoints

    Array Chunk Bytes 256 B 256 B Shape (32,) (32,) Dask graph 1 chunks in 2 graph layers Data type float64 numpy.ndarray

  • P0
    ()
    float64
    ...

    long_name :

    reference pressure

    units :

    Pa

    [1 values with dtype=float64]

  • hyai
    (ilev)
    float64
    dask.array<chunksize=(33,), meta=np.ndarray>

    long_name :

    hybrid A coefficient at layer interfaces

    Array Chunk Bytes 264 B 264 B Shape (33,) (33,) Dask graph 1 chunks in 2 graph layers Data type float64 numpy.ndarray

  • hybi
    (ilev)
    float64
    dask.array<chunksize=(33,), meta=np.ndarray>

    long_name :

    hybrid B coefficient at layer interfaces

    Array Chunk Bytes 264 B 264 B Shape (33,) (33,) Dask graph 1 chunks in 2 graph layers Data type float64 numpy.ndarray

  • ndbase
    ()
    float64
    ...

    long_name :

    base day

    [1 values with dtype=float64]

  • nsbase
    ()
    float64
    ...

    long_name :

    seconds of base day

    [1 values with dtype=float64]

  • nbdate
    ()
    float64
    ...

    long_name :

    base date (YYYYMMDD)

    [1 values with dtype=float64]

  • nbsec
    ()
    float64
    ...

    long_name :

    seconds of base date

    [1 values with dtype=float64]

  • mdt
    ()
    float64
    ...

    long_name :

    timestep

    units :

    s

    [1 values with dtype=float64]

  • date
    (time)
    float64
    dask.array<chunksize=(3650,), meta=np.ndarray>

    long_name :

    current date (YYYYMMDD)

    Array Chunk Bytes 128.33 kiB 28.52 kiB Shape (16426,) (3650,) Dask graph 5 chunks in 11 graph layers Data type float64 numpy.ndarray

  • datesec
    (time)
    float64
    dask.array<chunksize=(3650,), meta=np.ndarray>

    long_name :

    current seconds of current date

    Array Chunk Bytes 128.33 kiB 28.52 kiB Shape (16426,) (3650,) Dask graph 5 chunks in 11 graph layers Data type float64 numpy.ndarray

  • time_bnds
    (time, nbnd)
    object
    dask.array<chunksize=(3650, 2), meta=np.ndarray>

    long_name :

    time interval endpoints

    Array Chunk Bytes 256.66 kiB 57.03 kiB Shape (16426, 2) (3650, 2) Dask graph 5 chunks in 11 graph layers Data type object numpy.ndarray

  • date_written
    (time)
    object
    dask.array<chunksize=(3650,), meta=np.ndarray>

    Array Chunk Bytes 128.33 kiB 28.52 kiB Shape (16426,) (3650,) Dask graph 5 chunks in 11 graph layers Data type object numpy.ndarray

  • time_written
    (time)
    object
    dask.array<chunksize=(3650,), meta=np.ndarray>

    Array Chunk Bytes 128.33 kiB 28.52 kiB Shape (16426,) (3650,) Dask graph 5 chunks in 11 graph layers Data type object numpy.ndarray

  • ndcur
    (time)
    float64
    dask.array<chunksize=(3650,), meta=np.ndarray>

    long_name :

    current day (from base day)

    Array Chunk Bytes 128.33 kiB 28.52 kiB Shape (16426,) (3650,) Dask graph 5 chunks in 11 graph layers Data type float64 numpy.ndarray

  • nscur
    (time)
    float64
    dask.array<chunksize=(3650,), meta=np.ndarray>

    long_name :

    current seconds of current day

    Array Chunk Bytes 128.33 kiB 28.52 kiB Shape (16426,) (3650,) Dask graph 5 chunks in 11 graph layers Data type float64 numpy.ndarray

  • co2vmr
    (time)
    float64
    dask.array<chunksize=(3650,), meta=np.ndarray>

    long_name :

    co2 volume mixing ratio

    Array Chunk Bytes 128.33 kiB 28.52 kiB Shape (16426,) (3650,) Dask graph 5 chunks in 11 graph layers Data type float64 numpy.ndarray

  • ch4vmr
    (time)
    float64
    dask.array<chunksize=(3650,), meta=np.ndarray>

    long_name :

    ch4 volume mixing ratio

    Array Chunk Bytes 128.33 kiB 28.52 kiB Shape (16426,) (3650,) Dask graph 5 chunks in 11 graph layers Data type float64 numpy.ndarray

  • n2ovmr
    (time)
    float64
    dask.array<chunksize=(3650,), meta=np.ndarray>

    long_name :

    n2o volume mixing ratio

    Array Chunk Bytes 128.33 kiB 28.52 kiB Shape (16426,) (3650,) Dask graph 5 chunks in 11 graph layers Data type float64 numpy.ndarray

  • f11vmr
    (time)
    float64
    dask.array<chunksize=(3650,), meta=np.ndarray>

    long_name :

    f11 volume mixing ratio

    Array Chunk Bytes 128.33 kiB 28.52 kiB Shape (16426,) (3650,) Dask graph 5 chunks in 11 graph layers Data type float64 numpy.ndarray

  • f12vmr
    (time)
    float64
    dask.array<chunksize=(3650,), meta=np.ndarray>

    long_name :

    f12 volume mixing ratio

    Array Chunk Bytes 128.33 kiB 28.52 kiB Shape (16426,) (3650,) Dask graph 5 chunks in 11 graph layers Data type float64 numpy.ndarray

  • sol_tsi
    (time)
    float64
    dask.array<chunksize=(3650,), meta=np.ndarray>

    long_name :

    total solar irradiance

    units :

    W/m2

    Array Chunk Bytes 128.33 kiB 28.52 kiB Shape (16426,) (3650,) Dask graph 5 chunks in 11 graph layers Data type float64 numpy.ndarray

  • nsteph
    (time)
    float64
    dask.array<chunksize=(3650,), meta=np.ndarray>

    long_name :

    current timestep

    Array Chunk Bytes 128.33 kiB 28.52 kiB Shape (16426,) (3650,) Dask graph 5 chunks in 11 graph layers Data type float64 numpy.ndarray

  • T
    (time, lev, lat, lon)
    float64
    dask.array<chunksize=(3650, 32, 20, 20), meta=np.ndarray>

    mdims :

    1

    units :

    K

    long_name :

    Temperature

    cell_methods :

    time: mean

    Array Chunk Bytes 216.55 GiB 356.45 MiB Shape (16426, 32, 192, 288) (3650, 32, 20, 20) Dask graph 750 chunks in 11 graph layers Data type float64 numpy.ndarray

• Indexes: (6)
  • lat
    PandasIndex

    PandasIndex(Float64Index([             -90.0, -89.05759162303664,  -88.1151832460733,
                  -87.17277486910994,  -86.2303664921466, -85.28795811518324,
                   -84.3455497382199, -83.40314136125654, -82.46073298429319,
                  -81.51832460732984,
                  ...
                   81.51832460732984,  82.46073298429319,  83.40314136125654,
                    84.3455497382199,  85.28795811518324,   86.2303664921466,
                   87.17277486910994,   88.1151832460733,  89.05759162303664,
                                90.0],
                 dtype='float64', name='lat', length=192))

  • lon
    PandasIndex

    PandasIndex(Float64Index([   0.0,   1.25,    2.5,   3.75,    5.0,   6.25,    7.5,   8.75,
                    10.0,  11.25,
                  ...
                   347.5, 348.75,  350.0, 351.25,  352.5, 353.75,  355.0, 356.25,
                   357.5, 358.75],
                 dtype='float64', name='lon', length=288))

  • zlon
    PandasIndex

    PandasIndex(Float64Index([0.0], dtype='float64', name='zlon'))

  • lev
    PandasIndex

    PandasIndex(Float64Index([  3.64346569404006,  7.594819646328688, 14.356632251292467,
                  24.612220004200935,  35.92325001955032,   43.1937500834465,
                   51.67749896645546,  61.52049824595451,  73.75095784664154,
                   87.82123029232025, 103.31712663173676, 121.54724076390266,
                  142.99403876066208, 168.22507977485657,  197.9080867022276,
                  232.82861895859241,  273.9108167588711,  322.2419023513794,
                  379.10090386867523,   445.992574095726,  524.6871747076511,
                   609.7786948084831,  691.3894303143024,   763.404481112957,
                   820.8583686500788,  859.5347665250301,  887.0202489197254,
                    912.644546944648,  936.1983984708786,   957.485479535535,
                    976.325407391414,   992.556095123291],
                 dtype='float64', name='lev'))

  • ilev
    PandasIndex

    PandasIndex(Float64Index([ 2.255239523947239,  5.031691864132881,  10.15794742852449,
                   18.55531707406044,  29.73467559512106,  39.27300125360489,
                   47.11449891328812,   56.2404990196228,  66.80049747228622,
                   80.70141822099686,  94.94104236364365, 111.69321089982986,
                  131.40127062797546,  154.5868068933487, 181.86335265636444,
                  213.95282074809074, 251.70441716909409, 296.11721634864807,
                   348.3665883541107, 409.83521938323975,  482.1499288082123,
                     567.22442060709,  652.3329690098763,  730.4458916187286,
                   796.3630706071854,  845.3536666929722,  873.7158663570881,
                   900.3246314823627,  924.9644624069333,  947.4323345348239,
                   967.5386245362461,   985.112190246582,             1000.0],
                 dtype='float64', name='ilev'))

  • time
    PandasIndex

    PandasIndex(CFTimeIndex([1970-01-01 00:00:00, 1970-01-02 00:00:00, 1970-01-03 00:00:00,
                 1970-01-04 00:00:00, 1970-01-05 00:00:00, 1970-01-06 00:00:00,
                 1970-01-07 00:00:00, 1970-01-08 00:00:00, 1970-01-09 00:00:00,
                 1970-01-10 00:00:00,
                 ...
                 2014-12-23 00:00:00, 2014-12-24 00:00:00, 2014-12-25 00:00:00,
                 2014-12-26 00:00:00, 2014-12-27 00:00:00, 2014-12-28 00:00:00,
                 2014-12-29 00:00:00, 2014-12-30 00:00:00, 2014-12-31 00:00:00,
                 2015-01-01 00:00:00],
                dtype='object', length=16426, calendar='noleap', freq='D'))

• Attributes: (9)

  Conventions :

  CF-1.0

  source :

  CAM

  case :

  b.e21.BHISTcmip6.f09_g17.LE2-1001.001

  logname :

  sunseon

  host :

  mom2

  initial_file :

  b.e21.B1850.f09_g17.CMIP6-piControl.001.cam.i.1001-01-01-00000.nc

  topography_file :

  /mnt/lustre/share/CESM/cesm_input/atm/cam/topo/fv_0.9x1.25_nc3000_Nsw042_Nrs008_Co060_Fi001_ZR_sgh30_24km_GRNL_c170103.nc

  model_doi_url :

  https://doi.org/10.5065/D67H1H0V

  time_period_freq :

  day_1

But, if we actually load in some data to plot, you’ll see that it is pretty slow…

%%time
ds_read_directly.T.isel(time=range(1460), lev=0, lat=0, lon=0).hvplot(x='time', grid=True)

WARNING:param.CurvePlot00876: Converting cftime.datetime from a non-standard calendar (noleap) to a standard calendar for plotting. This may lead to subtle errors in formatting dates, for accurate tick formatting switch to the matplotlib backend.

CPU times: user 42.7 s, sys: 3.07 s, total: 45.8 s
Wall time: 17min 12s

Benchmark Reading in from Kerchunk mappings

Now, we can try reading in using our reference files (created from Kerchunk)

fs = fsspec.filesystem(
    "reference",
    fo='cesm2-le-tseries-aws.json',
    remote_protocol='s3',
    remote_options={'anon': True},
)
m = fs.get_mapper("")

Read the files in an zarr instead of netCDF (remember the reference file is a virtual Zarr dataset).

%%time
ds = xr.open_dataset(m, engine="zarr")
ds

<timed exec>:1: RuntimeWarning: Failed to open Zarr store with consolidated metadata, but successfully read with non-consolidated metadata. This is typically much slower for opening a dataset. To silence this warning, consider:
1. Consolidating metadata in this existing store with zarr.consolidate_metadata().
2. Explicitly setting consolidated=False, to avoid trying to read consolidate metadata, or
3. Explicitly setting consolidated=True, to raise an error in this case instead of falling back to try reading non-consolidated metadata.

CPU times: user 212 ms, sys: 22.3 ms, total: 234 ms
Wall time: 484 ms

<xarray.Dataset>
Dimensions:       (time: 16426, lev: 32, lat: 192, lon: 288, ilev: 33, nbnd: 2,
                   zlon: 1)
Coordinates:
  * ilev          (ilev) float64 2.255 5.032 10.16 18.56 ... 967.5 985.1 1e+03
  * lat           (lat) float64 -90.0 -89.06 -88.12 -87.17 ... 88.12 89.06 90.0
  * lev           (lev) float64 3.643 7.595 14.36 24.61 ... 957.5 976.3 992.6
  * lon           (lon) float64 0.0 1.25 2.5 3.75 ... 355.0 356.2 357.5 358.8
  * time          (time) object 1970-01-01 00:00:00 ... 2015-01-01 00:00:00
  * zlon          (zlon) float64 0.0
Dimensions without coordinates: nbnd
Data variables: (12/27)
    P0            (time) float64 ...
    T             (time, lev, lat, lon) float64 ...
    ch4vmr        (time) float64 ...
    co2vmr        (time) float64 ...
    date          (time) float64 ...
    date_written  (time) object ...
    ...            ...
    nscur         (time) float64 ...
    nsteph        (time) float64 ...
    sol_tsi       (time) float64 ...
    time_bnds     (time, nbnd) object ...
    time_written  (time) object ...
    zlon_bnds     (time, zlon, nbnd) float64 ...
Attributes:
    Conventions:       CF-1.0
    case:              b.e21.BHISTcmip6.f09_g17.LE2-1001.001
    host:              mom2
    initial_file:      b.e21.B1850.f09_g17.CMIP6-piControl.001.cam.i.1001-01-...
    logname:           sunseon
    model_doi_url:     https://doi.org/10.5065/D67H1H0V
    source:            CAM
    time_period_freq:  day_1
    topography_file:   /mnt/lustre/share/CESM/cesm_input/atm/cam/topo/fv_0.9x...

xarray.Dataset

• Dimensions:
  • time: 16426
  • lev: 32
  • lat: 192
  • lon: 288
  • ilev: 33
  • nbnd: 2
  • zlon: 1
• Coordinates: (6)
  • ilev
    (ilev)
    float64
    2.255 5.032 10.16 ... 985.1 1e+03

    formula_terms :

    a: hyai b: hybi p0: P0 ps: PS

    long_name :

    hybrid level at interfaces (1000*(A+B))

    positive :

    down

    standard_name :

    atmosphere_hybrid_sigma_pressure_coordinate

    units :

    hPa

    array([   2.25524 ,    5.031692,   10.157947,   18.555317,   29.734676,
             39.273001,   47.114499,   56.240499,   66.800497,   80.701418,
             94.941042,  111.693211,  131.401271,  154.586807,  181.863353,
            213.952821,  251.704417,  296.117216,  348.366588,  409.835219,
            482.149929,  567.224421,  652.332969,  730.445892,  796.363071,
            845.353667,  873.715866,  900.324631,  924.964462,  947.432335,
            967.538625,  985.11219 , 1000.      ])

  • lat
    (lat)
    float64
    -90.0 -89.06 -88.12 ... 89.06 90.0

    long_name :

    latitude

    units :

    degrees_north

    array([-90.      , -89.057592, -88.115183, -87.172775, -86.230366, -85.287958,
           -84.34555 , -83.403141, -82.460733, -81.518325, -80.575916, -79.633508,
           -78.691099, -77.748691, -76.806283, -75.863874, -74.921466, -73.979058,
           -73.036649, -72.094241, -71.151832, -70.209424, -69.267016, -68.324607,
           -67.382199, -66.439791, -65.497382, -64.554974, -63.612565, -62.670157,
           -61.727749, -60.78534 , -59.842932, -58.900524, -57.958115, -57.015707,
           -56.073298, -55.13089 , -54.188482, -53.246073, -52.303665, -51.361257,
           -50.418848, -49.47644 , -48.534031, -47.591623, -46.649215, -45.706806,
           -44.764398, -43.82199 , -42.879581, -41.937173, -40.994764, -40.052356,
           -39.109948, -38.167539, -37.225131, -36.282723, -35.340314, -34.397906,
           -33.455497, -32.513089, -31.570681, -30.628272, -29.685864, -28.743455,
           -27.801047, -26.858639, -25.91623 , -24.973822, -24.031414, -23.089005,
           -22.146597, -21.204188, -20.26178 , -19.319372, -18.376963, -17.434555,
           -16.492147, -15.549738, -14.60733 , -13.664921, -12.722513, -11.780105,
           -10.837696,  -9.895288,  -8.95288 ,  -8.010471,  -7.068063,  -6.125654,
            -5.183246,  -4.240838,  -3.298429,  -2.356021,  -1.413613,  -0.471204,
             0.471204,   1.413613,   2.356021,   3.298429,   4.240838,   5.183246,
             6.125654,   7.068063,   8.010471,   8.95288 ,   9.895288,  10.837696,
            11.780105,  12.722513,  13.664921,  14.60733 ,  15.549738,  16.492147,
            17.434555,  18.376963,  19.319372,  20.26178 ,  21.204188,  22.146597,
            23.089005,  24.031414,  24.973822,  25.91623 ,  26.858639,  27.801047,
            28.743455,  29.685864,  30.628272,  31.570681,  32.513089,  33.455497,
            34.397906,  35.340314,  36.282723,  37.225131,  38.167539,  39.109948,
            40.052356,  40.994764,  41.937173,  42.879581,  43.82199 ,  44.764398,
            45.706806,  46.649215,  47.591623,  48.534031,  49.47644 ,  50.418848,
            51.361257,  52.303665,  53.246073,  54.188482,  55.13089 ,  56.073298,
            57.015707,  57.958115,  58.900524,  59.842932,  60.78534 ,  61.727749,
            62.670157,  63.612565,  64.554974,  65.497382,  66.439791,  67.382199,
            68.324607,  69.267016,  70.209424,  71.151832,  72.094241,  73.036649,
            73.979058,  74.921466,  75.863874,  76.806283,  77.748691,  78.691099,
            79.633508,  80.575916,  81.518325,  82.460733,  83.403141,  84.34555 ,
            85.287958,  86.230366,  87.172775,  88.115183,  89.057592,  90.      ])

  • lev
    (lev)
    float64
    3.643 7.595 14.36 ... 976.3 992.6

    formula_terms :

    a: hyam b: hybm p0: P0 ps: PS

    long_name :

    hybrid level at midpoints (1000*(A+B))

    positive :

    down

    standard_name :

    atmosphere_hybrid_sigma_pressure_coordinate

    units :

    hPa

    array([  3.643466,   7.59482 ,  14.356632,  24.61222 ,  35.92325 ,  43.19375 ,
            51.677499,  61.520498,  73.750958,  87.82123 , 103.317127, 121.547241,
           142.994039, 168.22508 , 197.908087, 232.828619, 273.910817, 322.241902,
           379.100904, 445.992574, 524.687175, 609.778695, 691.38943 , 763.404481,
           820.858369, 859.534767, 887.020249, 912.644547, 936.198398, 957.48548 ,
           976.325407, 992.556095])

  • lon
    (lon)
    float64
    0.0 1.25 2.5 ... 356.2 357.5 358.8

    long_name :

    longitude

    units :

    degrees_east

    array([  0.  ,   1.25,   2.5 , ..., 356.25, 357.5 , 358.75])

  • time
    (time)
    object
    1970-01-01 00:00:00 ... 2015-01-...

    bounds :

    time_bnds

    long_name :

    time

    array([cftime.DatetimeNoLeap(1970, 1, 1, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1970, 1, 2, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1970, 1, 3, 0, 0, 0, 0, has_year_zero=True), ...,
           cftime.DatetimeNoLeap(2014, 12, 30, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2014, 12, 31, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 1, 1, 0, 0, 0, 0, has_year_zero=True)],
          dtype=object)

  • zlon
    (zlon)
    float64
    0.0

    bounds :

    zlon_bnds

    long_name :

    longitude

    units :

    degrees_east

    array([0.])

• Data variables: (27)
  • P0
    (time)
    float64
    ...

    long_name :

    reference pressure

    units :

    Pa

    [16426 values with dtype=float64]

  • T
    (time, lev, lat, lon)
    float64
    ...

    cell_methods :

    time: mean

    long_name :

    Temperature

    mdims :

    1

    units :

    K

    [29065347072 values with dtype=float64]

  • ch4vmr
    (time)
    float64
    ...

    long_name :

    ch4 volume mixing ratio

    [16426 values with dtype=float64]

  • co2vmr
    (time)
    float64
    ...

    long_name :

    co2 volume mixing ratio

    [16426 values with dtype=float64]

  • date
    (time)
    float64
    ...

    long_name :

    current date (YYYYMMDD)

    [16426 values with dtype=float64]

  • date_written
    (time)
    object
    ...

    [16426 values with dtype=object]

  • datesec
    (time)
    float64
    ...

    long_name :

    current seconds of current date

    [16426 values with dtype=float64]

  • f11vmr
    (time)
    float64
    ...

    long_name :

    f11 volume mixing ratio

    [16426 values with dtype=float64]

  • f12vmr
    (time)
    float64
    ...

    long_name :

    f12 volume mixing ratio

    [16426 values with dtype=float64]

  • gw
    (time, lat)
    float64
    ...

    long_name :

    latitude weights

    [3153792 values with dtype=float64]

  • hyai
    (time, ilev)
    float64
    ...

    long_name :

    hybrid A coefficient at layer interfaces

    [542058 values with dtype=float64]

  • hyam
    (time, lev)
    float64
    ...

    long_name :

    hybrid A coefficient at layer midpoints

    [525632 values with dtype=float64]

  • hybi
    (time, ilev)
    float64
    ...

    long_name :

    hybrid B coefficient at layer interfaces

    [542058 values with dtype=float64]

  • hybm
    (time, lev)
    float64
    ...

    long_name :

    hybrid B coefficient at layer midpoints

    [525632 values with dtype=float64]

  • mdt
    (time)
    float64
    ...

    long_name :

    timestep

    units :

    s

    [16426 values with dtype=float64]

  • n2ovmr
    (time)
    float64
    ...

    long_name :

    n2o volume mixing ratio

    [16426 values with dtype=float64]

  • nbdate
    (time)
    float64
    ...

    long_name :

    base date (YYYYMMDD)

    [16426 values with dtype=float64]

  • nbsec
    (time)
    float64
    ...

    long_name :

    seconds of base date

    [16426 values with dtype=float64]

  • ndbase
    (time)
    float64
    ...

    long_name :

    base day

    [16426 values with dtype=float64]

  • ndcur
    (time)
    float64
    ...

    long_name :

    current day (from base day)

    [16426 values with dtype=float64]

  • nsbase
    (time)
    float64
    ...

    long_name :

    seconds of base day

    [16426 values with dtype=float64]

  • nscur
    (time)
    float64
    ...

    long_name :

    current seconds of current day

    [16426 values with dtype=float64]

  • nsteph
    (time)
    float64
    ...

    long_name :

    current timestep

    [16426 values with dtype=float64]

  • sol_tsi
    (time)
    float64
    ...

    long_name :

    total solar irradiance

    units :

    W/m2

    [16426 values with dtype=float64]

  • time_bnds
    (time, nbnd)
    object
    ...

    long_name :

    time interval endpoints

    [32852 values with dtype=object]

  • time_written
    (time)
    object
    ...

    [16426 values with dtype=object]

  • zlon_bnds
    (time, zlon, nbnd)
    float64
    ...

    long_name :

    zlon bounds

    units :

    degrees_east

    [32852 values with dtype=float64]

• Indexes: (6)
  • ilev
    PandasIndex

    PandasIndex(Float64Index([ 2.255239523947239,  5.031691864132881,  10.15794742852449,
                   18.55531707406044,  29.73467559512106,  39.27300125360489,
                   47.11449891328812,   56.2404990196228,  66.80049747228622,
                   80.70141822099686,  94.94104236364365, 111.69321089982986,
                  131.40127062797546,  154.5868068933487, 181.86335265636444,
                  213.95282074809074, 251.70441716909409, 296.11721634864807,
                   348.3665883541107, 409.83521938323975,  482.1499288082123,
                     567.22442060709,  652.3329690098763,  730.4458916187286,
                   796.3630706071854,  845.3536666929722,  873.7158663570881,
                   900.3246314823627,  924.9644624069333,  947.4323345348239,
                   967.5386245362461,   985.112190246582,             1000.0],
                 dtype='float64', name='ilev'))

  • lat
    PandasIndex

    PandasIndex(Float64Index([             -90.0, -89.05759162303664,  -88.1151832460733,
                  -87.17277486910994,  -86.2303664921466, -85.28795811518324,
                   -84.3455497382199, -83.40314136125654, -82.46073298429319,
                  -81.51832460732984,
                  ...
                   81.51832460732984,  82.46073298429319,  83.40314136125654,
                    84.3455497382199,  85.28795811518324,   86.2303664921466,
                   87.17277486910994,   88.1151832460733,  89.05759162303664,
                                90.0],
                 dtype='float64', name='lat', length=192))

  • lev
    PandasIndex

    PandasIndex(Float64Index([  3.64346569404006,  7.594819646328688, 14.356632251292467,
                  24.612220004200935,  35.92325001955032,   43.1937500834465,
                   51.67749896645546,  61.52049824595451,  73.75095784664154,
                   87.82123029232025, 103.31712663173676, 121.54724076390266,
                  142.99403876066208, 168.22507977485657,  197.9080867022276,
                  232.82861895859241,  273.9108167588711,  322.2419023513794,
                  379.10090386867523,   445.992574095726,  524.6871747076511,
                   609.7786948084831,  691.3894303143024,   763.404481112957,
                   820.8583686500788,  859.5347665250301,  887.0202489197254,
                    912.644546944648,  936.1983984708786,   957.485479535535,
                    976.325407391414,   992.556095123291],
                 dtype='float64', name='lev'))

  • lon
    PandasIndex

    PandasIndex(Float64Index([   0.0,   1.25,    2.5,   3.75,    5.0,   6.25,    7.5,   8.75,
                    10.0,  11.25,
                  ...
                   347.5, 348.75,  350.0, 351.25,  352.5, 353.75,  355.0, 356.25,
                   357.5, 358.75],
                 dtype='float64', name='lon', length=288))

  • time
    PandasIndex

    PandasIndex(CFTimeIndex([1970-01-01 00:00:00, 1970-01-02 00:00:00, 1970-01-03 00:00:00,
                 1970-01-04 00:00:00, 1970-01-05 00:00:00, 1970-01-06 00:00:00,
                 1970-01-07 00:00:00, 1970-01-08 00:00:00, 1970-01-09 00:00:00,
                 1970-01-10 00:00:00,
                 ...
                 2014-12-23 00:00:00, 2014-12-24 00:00:00, 2014-12-25 00:00:00,
                 2014-12-26 00:00:00, 2014-12-27 00:00:00, 2014-12-28 00:00:00,
                 2014-12-29 00:00:00, 2014-12-30 00:00:00, 2014-12-31 00:00:00,
                 2015-01-01 00:00:00],
                dtype='object', length=16426, calendar='noleap', freq='D'))

  • zlon
    PandasIndex

    PandasIndex(Float64Index([0.0], dtype='float64', name='zlon'))

• Attributes: (9)

  Conventions :

  CF-1.0

  case :

  b.e21.BHISTcmip6.f09_g17.LE2-1001.001

  host :

  mom2

  initial_file :

  b.e21.B1850.f09_g17.CMIP6-piControl.001.cam.i.1001-01-01-00000.nc

  logname :

  sunseon

  model_doi_url :

  https://doi.org/10.5065/D67H1H0V

  source :

  CAM

  time_period_freq :

  day_1

  topography_file :

  /mnt/lustre/share/CESM/cesm_input/atm/cam/topo/fv_0.9x1.25_nc3000_Nsw042_Nrs008_Co060_Fi001_ZR_sgh30_24km_GRNL_c170103.nc

Plot the output from the Mapper Method

%%time
ds.T.isel(time=range(1460), lev=0, lat=0, lon=0).hvplot(x='time', grid=True)

WARNING:param.CurvePlot00983: Converting cftime.datetime from a non-standard calendar (noleap) to a standard calendar for plotting. This may lead to subtle errors in formatting dates, for accurate tick formatting switch to the matplotlib backend.

CPU times: user 18 s, sys: 2.09 s, total: 20.1 s
Wall time: 44.2 s

Virtual aggregate CESM MOM6 datasets with kerchunk   Recap: Unstructured Grid Collaborative Work Time