Using Dask-Jobqueue to deploy Dask on HPC job queue systems

Note

The Dask-Jobqueue package will only work on the HPC JupyterHub. Dask-Gateway is the correct solution for deploying scalable Dask clusters on the NCAR K8s JupyterHub.

Dask-Jobqueue is a Python module that enables an easy way to deploy Dask clusters on HPC job queueing systems like PBS, Slurm, MOAB, SGE, LSF, and HTCondor. An example on how to use dask_jobqueue to provision a Dask cluster on a PBS scheduler can be seen below:

from dask_jobqueue import PBSCluster
from distributed import Client
cluster = PBSCluster(
    job_name = 'my-job-name',
    cores = 1,
    memory = '10GiB',
    processes = 1,
    local_directory = '/scratch',
    queue='queue-name',
    interface = 'ib0'
)
client = Client(cluster)

The Dask-Jobqueue documentation here provides a much greater in depth explanation of each of these values and how to customize them appropriately. There is also a YouTube video on the CISL YouTube channel called “Using Dask on HPC Systems” that is very informative. The beginning of part 2 goes over dask_jobqueue on NCAR HPC resources and can be viewed here