CMIP6 Cloud data access · jupyterlab-hub

Hi, I have a jupyter notebook that reads data from the CMIP6 AWS Cloud storage for the CESM2 historical runs. The notebook was working until Friday last week and over the weekend it seems to have run into an error trying to access the URL for the data. Here is the code snippet to query the URL:

# choose where to load data from:
load_data_from = 'cloud'

col_url = "https://raw.githubusercontent.com/NCAR/intake-esm-datastore/master/catalogs/pangeo-cmip6.json"
col = intake.open_esm_datastore(col_url)
file = 'available_data_cloud.txt'

The error message I get is: "HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Max retries exceeded with url: /NCAR/intake-esm-datastore/master/catalogs/pangeo-cmip6.json (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x2b3643dfd240>: Failed to establish a new connection: [Errno 110] Connection timed out'))"

I am able to open the URL on my browser. Could I know if there has been any change on the NCAR jupyterhub that would prevent accessing this URL starting this weekend?

Brian Bonnlander (Nov 23 2020 at 23:56):

If I am not mistaken, NCAR's jupyterhub directs users to either Casper (DAV) or Cheyenne. If you run jupyterhub on Cheyenne, I don't believe that internet access is possible. But this could be outdated information, I'm not sure.

Aneesh Subramanian (Nov 24 2020 at 06:25):

Thanks for the reply. I did have it running on both Cheyenne and Casper DAV jupyterhubs last week. It seems something broke over the weekend.

Matt Long (Nov 24 2020 at 12:55):

Anderson Banihirwe (Nov 24 2020 at 17:33):

Brian is right about this.... The compute nodes on Cheyenne are on a network that is cut off from the public internet....

Can you confirm that your notebook doesn't work on casper: https://jupyterhub.ucar.edu/dav ? I am
asking because I am able to run your code on Casper, and everything seems to be working fine on casper:

abanihi at casper-login2 in ~
$ ipython
Python 3.7.8 | packaged by conda-forge | (default, Jul 31 2020, 02:25:08)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.19.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import intake

In [2]: col_url = "https://raw.githubusercontent.com/NCAR/intake-esm-datastore/master/catalogs/pangeo-cmip6.json"

In [3]: col = intake.open_esm_datastore(col_url)
/glade/work/abanihi/softwares/miniconda3/envs/playground/lib/python3.7/site-packages/IPython/core/interactiveshell.py:3418: DtypeWarning: Columns (10) have mixed types.Specify dtype option on import or set low_memory=False.
  exec(code_obj, self.user_global_ns, self.user_ns)

In [4]: col.df.head()
Out[4]:
  activity_id institution_id source_id experiment_id member_id table_id variable_id grid_label                                             zstore  dcpp_init_year   version
0  AerChemMIP        AS-RCEC   TaiESM1       histSST  r1i1p1f1   AERmon    od550aer         gn  gs://cmip6/AerChemMIP/AS-RCEC/TaiESM1/histSST/...             NaN  20200310
1  AerChemMIP            BCC  BCC-ESM1       histSST  r1i1p1f1   AERmon       mmrbc         gn  gs://cmip6/AerChemMIP/BCC/BCC-ESM1/histSST/r1i...             NaN  20190718
2  AerChemMIP            BCC  BCC-ESM1       histSST  r1i1p1f1   AERmon     mmrdust         gn  gs://cmip6/AerChemMIP/BCC/BCC-ESM1/histSST/r1i...             NaN  20191127
3  AerChemMIP            BCC  BCC-ESM1       histSST  r1i1p1f1   AERmon       mmroa         gn  gs://cmip6/AerChemMIP/BCC/BCC-ESM1/histSST/r1i...             NaN  20190809
4  AerChemMIP            BCC  BCC-ESM1       histSST  r1i1p1f1   AERmon      mmrso4         gn  gs://cmip6/AerChemMIP/BCC/BCC-ESM1/histSST/r1i...             NaN  20191127

Aneesh Subramanian (Nov 24 2020 at 17:47):

@Anderson Banihirwe @Brian Bonnlander , I must be mistaken then. Sorry about that. I thought I had I had it running on Cheyenne as well, but surely not as you say.

I just checked and it does work on the Canvas DAV nodes. Thanks again for the clarification.

Stream: jupyterlab-hub

Topic: CMIP6 Cloud data access

Aneesh Subramanian (Nov 23 2020 at 22:30):

Brian Bonnlander (Nov 23 2020 at 23:56):

Aneesh Subramanian (Nov 24 2020 at 06:25):

Matt Long (Nov 24 2020 at 12:55):

Anderson Banihirwe (Nov 24 2020 at 17:33):

Aneesh Subramanian (Nov 24 2020 at 17:47):

Aneesh Subramanian (Nov 24 2020 at 18:20):