Casper and NCARCluster no longer working? · dask

Up until a few days ago, when I would request a cluster of Dask workers using NCARCluster on Casper, I would see pending requests for those workers using the squeue --me command. Lately, however, I am not seeing the pending request for workers, and I am never granted those workers.

Is anyone aware of the issue? Does SLURM not add worker requests to the queue if it's too full, maybe?

Max Grover (Mar 25 2021 at 18:34):

Are you having trouble accessing the actual JupyterHub? Accessing an environment with ncar_jobqueue installed?

Max Grover (Mar 25 2021 at 18:35):

Brian Bonnlander (Mar 25 2021 at 18:35):

I am using my own conda environment with JupyterLab installed. It has been working great for months until recently.

Max Grover (Mar 25 2021 at 18:37):

So are you able to get into jupyterlab/notebook? Or is this issue of actually running the notebook? When you login, are you able to request the resources you need?

Brian Bonnlander (Mar 25 2021 at 18:37):

My question is about whether others have ever seen Dask worker requests not show up as pending in the SLURM queue. I am running JupyterLab without issues using execdav, as recommended.

Deepak Cherian (Mar 25 2021 at 18:39):

Max Grover (Mar 25 2021 at 18:40):

Brian Bonnlander (Mar 25 2021 at 18:41):

Ah, I commented out the cluster.scale() line! I didn't realize its importance. I take it that this is what actually creates the SLURM reqeusts.

Haiying Xu (Mar 25 2021 at 19:06):

Anderson Banihirwe (Mar 25 2021 at 19:19):

It depends... By default, dask doesn't mind proceeding with work if it has access to one worker at least. So, if your workload requires a minimum amount of resources/workers, you may still need the wait_for_workers() call because there's no guarantee that the requested resources (via .scale()) will be available at the same time.

Haiying Xu (Mar 25 2021 at 19:26):

But recently, I scale to the specific total number of workers, but my codes still stuck at wait_for_workers, then one of the node retired after some timeout time, thus my codes stuck forever.

Anderson Banihirwe (Mar 25 2021 at 19:29):

Could you share a minimal script to reproduce or point me to the code you are currently using? It's hard to know what's going on unless I see your code

Will Wieder (Apr 09 2021 at 16:33):

I feel really daft asking this question, but I'm requesting 10 workers and 400 Gb of memory with the following
I'd assumed the cluster.scale(2) would get me 20 workers and 800 GB of memory, but this isn't happening.
What am I doing wrong?

from dask_jobqueue import PBSCluster
from dask.distributed import Client
cluster = PBSCluster(
cores=ncores, # The number of cores you want
memory=nmem, # Amount of memory
processes=ncores, # How many processes
queue='casper', # The type of queue to utilize (/glade/u/apps/dav/opt/usr/bin/execcasper)
resource_spec='select=1:ncpus='+str(ncores)+':mem='+nmem, # Specify resources
project='P93300641', # Input your project ID here
walltime='1:00:00', # Amount of wall time
interface='ib0', # Interface to use
extra=["--lifetime", "55m", "--lifetime-stagger", "4m"],
)
# Scale up
cluster.scale(2)

Matt Long (Apr 09 2021 at 16:33):

Will Wieder (Apr 09 2021 at 16:42):

Matt Long (Apr 09 2021 at 16:45):

Matt Long (Apr 09 2021 at 16:46):

I have a little wrapper for convenience (since ncar-jobqueue is currently broken):

def get_ClusterClient():
    import dask
    from dask_jobqueue import PBSCluster
    from dask.distributed import Client
    cluster = PBSCluster(
        cores=1,
        memory='25GB',
        processes=1,
        queue='casper',
        local_directory='$TMPDIR',
        log_directory='$TMPDIR',
        resource_spec='select=1:ncpus=1:mem=25GB',
        project='NCGD0011',
        walltime='01:00:00',
        interface='ib0',)

    dask.config.set({
        'distributed.dashboard.link':
        'https://jupyterhub.hpc.ucar.edu/stable/user/{USER}/proxy/{port}/status'
    })
    client = Client(cluster)
    return cluster, client

cluster, client = utils.get_ClusterClient()
cluster.scale(12) #adapt(minimum_jobs=0, maximum_jobs=24)

Max Grover (Apr 09 2021 at 16:58):

So are you requesting 40 GB of memory per worker then? Or are you looking for 400 GB per worker?

Max Grover (Apr 09 2021 at 17:01):

from dask_jobqueue import PBSCluster
from dask.distributed import Client
ncores=1
nmem = '40GB'

cluster = PBSCluster(
    cores=ncores, # The number of cores you want
    memory=nmem, # Amount of memory
    processes=ncores, # How many processes
    queue='casper', # The type of queue to utilize (/glade/u/apps/dav/opt/usr/bin/execcasper)
    resource_spec='select=1:ncpus='+str(ncores)+':mem='+nmem, # Specify resources
    project='NCGD0011', # Input your project ID here
    walltime='1:00:00', # Amount of wall time
    interface='ib0', # Interface to use
)

cluster.scale(10)

results in 10 workers w/ 10 cores and 400 GB of memory. If you increase cluster.scale(n) to n=20, the result is 20 workers with 800 GB of memory.

Will Wieder (Apr 09 2021 at 17:01):

no, 40 GB / worker. Matt's example feems to work fine, if I use .scale(20) I get the 800 GB of memory needed

Max Grover (Apr 09 2021 at 17:03):

I guess the key would be take however much memory you are looking for in total, divided by the number of workers you would like to get the suggested nmem. Thanks for asking this question!

Will Wieder (Apr 09 2021 at 17:21):

John Clyne (Apr 09 2021 at 22:35):

FYI I'm using NCARCluster for the first time, launching a "casper batch" session via https://jupyterhub.hpc.ucar.edu/, and finding that 'identify_host()' cannot identify casper as "casper".

Matt Long (Apr 10 2021 at 14:00):

Anderson Banihirwe (Apr 14 2021 at 22:39):

This issue has been resolved in the latest release of ncar-jobqueue. You can upgrade to the latest version via

conda install -c conda-forge ncar-jobqueue==2021.4.14

python -m pip install ncar-jobqueue --upgrade

Stream: dask

Topic: Casper and NCARCluster no longer working?

Brian Bonnlander (Mar 25 2021 at 18:30):