Holly Olivarez (Sep 28 2023 at 18:44): Holly Olivarez (Sep 28 2023 at 18:44):Hello! I am a PhD student at CU Boulder with access to Casper. I want to compare two decades of several (ocean, monthly) variables available in CESM2-LE and am struggling with enormous file sizes and processing times using Jupyter notebooks. Is there someone who has worked or is working with analyzing output from 100 ensemble members that can share tips to make my experiments more efficient? I can send a path to one notebook for you to peruse, if that helps. Thanks in advance!
Julia Kent (Sep 28 2023 at 18:58):Hi Holly. Are you using Dask?
Holly Olivarez (Sep 28 2023 at 19:02):Hi Julia. Yes, I am.
Julia Kent (Sep 28 2023 at 23:50):My next questions would be: are you sure you've requested multiple workers on Casper? And can you make sure that your Dask chunks are a reasonable size and not in the dimension you are computing over? It still would be helpful for someone else from ESDS to provide some insight to their workflows (as I am not working with huge datasets like this on the regular).
Holly Olivarez (Sep 28 2023 at 23:55):Thank you for thinking on this. I am sure but agree that someone could provide insight in case I've missed anything that might help. I made a virtual appointment with HPC for Monday but feel I still may be searching for someone who is working with these 100 ensemble members. I am going to ask in a couple of grad Slack's I'm in, too. Thanks again!
Katie Dagon (Oct 02 2023 at 18:18):@Holly Olivarez there are a couple of ESDS blog posts using CESM2-LE from a few years back that could be helpful.
It would be great to see if there are any more recent examples from the community that we could feature as blog posts or Pythia cookbooks.
Negin Sobhani (Oct 02 2023 at 20:21):Hey Everyone, I have previously worked with the all LENS2 dataset extensively and I was in the process of creating a cookbook for them. I've met with Holly today and provided some guidelines on how to accelerate the workflow. But this gives me the motivation to finish up a notebook example of dealing with large number of ensembles...
Danielle Touma (Oct 03 2023 at 19:11):I've also been working with CESM2-LE daily data. I've figured out some workflows that work with xarray/dask, but not sure if they are very efficient... I would love to see your notebook example, @Negin Sobhani, to see how I can improve my workflow once you put it together!
Last updated: May 16 2025 at 17:14 UTC