Brian Dobbins (Aug 20 2024 at 19:50): Brian Dobbins (Aug 20 2024 at 19:50):Hi all,
Has anyone used Dask with full nodes on Derecho, not Casper? When I use Casper, I typically request N workers needing 2-4GB/each, and it works fine (with N ranging from 2 - 200). On Derecho, though, since full nodes are the default allocation, I wasn't sure how best to do this, and am admittedly a Dask neophyte. I'd love any guidance, including whether I should just switch to dask-mpi within a job, or continue to use PBSCluster under dask_jobqueue.
Thank you!
- Brian
Katelyn FitzGerald (Aug 21 2024 at 17:19):I haven't personally, but would be curious to hear more about others experiences.
I know @Negin Sobhani and @Brian Vanderwende wrote the NCAR Dask Tutorial and the ESDS 2024 Xarray and Dask Tutorial, but I think these largely focus on Casper. Not sure if they have thoughts on this and/or other resources.
Negin Sobhani (Aug 21 2024 at 19:22):Hey @Brian Dobbins , we don't suggest running the Dask workflow on Derecho, especially using Casper workflows (i.e. PBSCluster) , since users will be charged for the full node.
Besides, since Dask is designed for me-intensive workflows, it is also more suitable for Casper (data analysis machine), which has more memory per node. Derecho, with its lower memory per core, is better suited for compute-heavy tasks.
That being said, you can indeed use Dask on derecho with different methods of organizing workers and much less flexibility.
I can put together some example dask workflows for Derecho that I can share if you are interested but overall I strongly advice running dask on Derecho. If you are interested I can talk more about dask-jobqueue vs dask-mpi.
Is there any particular reason you are interested in running dask workflows on Derecho?
Brian Dobbins (Aug 21 2024 at 19:32):Thanks for the replies, Katelyn and Negin! My interest in running on Derecho is because we're evaluating a new time-series conversion tool for potential use in CMIP7 -- the current tools run via MPI, on Derecho, but have some limitations. We want to set up a 1:1 comparison since this is a major workflow, to evaluate performance and memory needs of each. Currently, we don't expect (at normal resolutions) the memory / core limitations on Derecho to be a problem.
I think I can just do this with dask-mpi, I just wasn't sure if there were other recommended ways. I think if the tests / performance look good, we'd also consider running these workflows on Casper, but given the scale of the CMIP7 runs, and the amount of data, I'm not yet sure that's the best platform for this, even though it's obviously superior for interactive / smaller-scale data analysis.
Thanks!
John Clyne (Aug 21 2024 at 19:51):@Brian Dobbins is dask-mpi still actively being developed?
Brian Dobbins (Aug 21 2024 at 19:58):@John Clyne That's a good question - I hadn't thought so, but it does seem like Kevin Paul and others are indeed making very occasional commits in Github:
https://github.com/dask/dask-mpi/commits/main/
Ultimately, the method by which we use Dask doesn't matter to us too much; I think for now I'm just interested in setting up the performance and memory tests, and depending on the results of said tests, we can consider how best to deploy it, and where. Casper may indeed be viable, I just don't know - the speed will likely be the big factor there.
Negin Sobhani (Aug 21 2024 at 19:58):Thanks @Brian Dobbins .
Yes, you can indeed use dask-mpi to run dask on Derecho, but please note that there is no swap space on Derecho nodes, and dask rely heavily on swap space to avoid memory errors. I still think casper will be more performant if you want to go through large dataset (i.e. CMIP7 runs) with dask.
@John Clyne : Yes, I think there are some recent development on dask-mpi from the repository but the simplest and preferred method for running dask workflows on HPC systems is dask-jobqueue.
Negin Sobhani (Aug 21 2024 at 20:00):@Brian Dobbins : Please let me know if you'd like to meet to talk more about this. I would be happy to help. :-)
Michael Levy (Aug 21 2024 at 21:36):@Negin SobhaniIs there a way to submit a job on Casper from derecho? If so, can jobs on one machine be held until a job on the other machine completes successfully? The CMIP workflow @Brian Dobbins briefly mentioned is basically: (1) Run CESM, (2) run some post-processing tools, (3) repeat 1 & 2 a bunch of times. It would be pretty cool if those two steps could run on different machines while still running sequentially!
Negin Sobhani (Aug 21 2024 at 22:03):Thanks for explaining the workflow @Michael Levy .
Yes, cross-submission is a cool feature we have recently. Here is how you can do this workflow entirely from Derecho (while running post-processing on casper):
(1) Run CESM on Derecho (job 1: JOBID1)
(2) From Derecho, submit post-processing job to casper (i.e. job 2 that depends on job 1):
qsub -q casper@casper-pbs -W depend=afterok:$JOBID1 postproc.pbs
Notice the syntax for cross submission to casper (i.e. -q casper@casper-pbs )
@Brian Dobbins and @Michael Levy Please check our docs for the cross-system PBS capabilities: https://ncar-hpc-docs.readthedocs.io/en/latest/pbs/
Last updated: May 16 2025 at 17:14 UTC