Demo Notebook · jupyterlab-hub · Zulip Chat Archive

Does anyone have a good recommendation for a Jupyter Notebook that would make for a good, short demo of the current NCAR JupyterHub? In particular, I'm looking for something that demonstrates Dask and the Dask Labextension, uses Dask-Jobqueue (or NCAR-Jobqueue), and runs in less than 2 minutes.

Kevin Paul (Jul 23 2020 at 16:13):

...NOTE: I got both "money and time," so if it can scale up to a large number of workers...bonus!

Kevin Paul (Jul 23 2020 at 16:14):

...NOTE: Can work on either Casper or Cheyenne

Kevin Paul (Jul 23 2020 at 23:11):

Hmm. No takers?

Brian Bonnlander (Jul 23 2020 at 23:14):

Anderson did a live demo for SciPy 2019 on Cheyenne, but I'm not sure which notebook it was. It looked cool though while I was watching in the audience. :rock_on:

Kevin Paul (Jul 23 2020 at 23:15):

Cool! @Anderson Banihirwe: Do you mind sharing?

Anderson Banihirwe (Jul 24 2020 at 13:47):

@Kevin Paul,

Here's the notebook I presented at SciPy: https://nbviewer.jupyter.org/github/andersy005/talks/blob/gh-pages/notebooks/dask-jupyter-scipy-2019.ipynb. In the notebook I use the GMET dataset in zarr format. Unfortunately, this zarr dataset appears to have been purged. Let me see I can recreate the zarr version again.

Anderson Banihirwe (Jul 24 2020 at 13:52):

In particular, I'm looking for something that demonstrates Dask and the Dask Labextension, uses Dask-Jobqueue (or NCAR-Jobqueue), and runs in less than 2 minutes.

As far as I can remember, the notebook took between 4 minutes -5 minutes, but with a beefy dask cluster you may be able to run it under 2 minutes.

Kevin Paul (Jul 24 2020 at 14:59):

Thanks, @Anderson Banihirwe! I’ll try it out.

Kevin Paul (Jul 24 2020 at 19:03):

@Anderson Banihirwe: Can you export your environment for this notebook?

Anderson Banihirwe (Jul 24 2020 at 21:16):

You are welcome! The zarr copy of the data resides here: "/glade/scratch/abanihi/data/gmet_v1.zarr"

Regarding the environment, here's the contents of the YAML file (scipy-demo.yml) you can use:

name: scipy-demo
channels:
  - conda-forge
dependencies:
  - xarray
  - zarr
  - dask-jobqueue
  - dask
  - distributed
  - hvplot
  - dask-labextension
  - ipywidgets
  - jupyterlab
  - nodejs

$ conda env update -f scipy-demo.yml
$ conda activate scipy-demo
$ jupyter labextension install --clean \
                            @jupyter-widgets/jupyterlab-manager  dask-labextension @pyviz/jupyterlab_pyviz

Anderson Banihirwe (Jul 24 2020 at 21:51):

It looks like the notebook can easily run in under 2 minutes... I just ran it on casper with the following cluster configuration and it took 1 minute and 20 seconds.

Screen-Shot-2020-07-24-at-3.48.40-PM.png