Using GatewayCluster with Dask

When LocalCluster is no longer sufficient and you are maxing out local resources it’s time to scale your application and create a separate Dask cluster with it’s own dedicated resources. In the NCAR JupyterHub dask_gateway is utilized to provision a Dask GatewayCluster via Python. An example of how to launch Dask Gateway in Python can be seen below.

from dask.distributed import Client
from dask_gateway import GatewayCluster

cluster = GatewayCluster()  # connect to Gateway and create a cluster

cluster.adapt(minimum=2, maximum=20)  # Make an adaptable cluster with a min and max number of workers

client = Client(cluster)  # connect Client to Cluster
client # Display the client information and Dashboard URL

Note

It is best practice to run cluster.close() at the end of your computations. GatewayCluster should close idle clusters automatically.

The Dashboard URL will bring you to a page where Dask cluster resources can be viewed in real time. The URL can also be copied and pasted in to the Dask extension on the left as seen below.

Each box can be dragged in to your workspace and arranged as different tiles alongside your notebook. This enables you to monitor Dask resources while watching your notebooks run.