slowness of geocat interp_hybrid_to_pressure · dask

Hello, I'm using interp_hybrid_to_pressure with extrapolate = True. This takes much longer than NCL used to but I'm also noticing a big difference between how long it takes on the CISL machines with a dask cluster verus how long it takes on the CGD machines, with CGD machines being much faster. I'm running the same command on the same data. It's the GeoCat function but i've pulled it out into my own subroutine to mess around with it (so it may also not be entirely up to date).

I had already loaded all the inputs prior to executing this command. It's taking about four times longer on casper. On casper I'm using a dask cluster with the following resources:

I'm assuming these resources are much greater than what we have on the CGD postprocessor machines, so I would have expected it to be faster. But it seems like maybe the fact that this is using a dask cluster with the chunking that entails is making it much slower.

In general I'm finding vertical interpolation to be an extreme bottleneck now with GeoCAT. NCL was a LOT faster. This is not a big dataset. It's 1 degree resolution, 58 years, monthly time resolution. It's relatively small compared to the datasets that we need to interpolate and for it to take 48 minutes with those resources, it's kind of unmanageable. I'm wondering if there's anything we could do to try to speed this up? I'd be happy to brainstorm with someone about ideas or try things out because this is really a critical need for GeoCAT if we can no longer use NCL. Alternatively, if anyone can provide guidance on how I might set things up better on the casper to make it more efficient, I'd be glad to hear it.

Here's my actual code on the CISL machines: /glade/u/home/islas/python/RRAtlantic/DATA_SORT/plev_interp/GOGA/plevinterp_U.ipynb

Anissa Zacharias (Jul 09 2024 at 21:50):

Hi Isla! We recently released some performance improvements for interp_hybrid_to_pressure (PR #592, if you're curious) in v2024.04.0 that I hope can help your issue.

I would try pulling the most recent version of the function and see if that improves your performance.

Isla Simpson (Jul 10 2024 at 13:52):

That's great to here! I'm excited to try this out. However, I'm having trouble updating my environment. I'm not the most experienced with conda. I have updated ny conda installation and then i have run

but it still keeps me at version 2022.07.0. Do you know why? I suppose I could specify the geocat-comp version in my yml file but I'm confused about why this doesn't work.

Katelyn FitzGerald (Jul 10 2024 at 14:40):

Ah, I think that's the newest version on the older "ncar" conda channel. You'll need to switch where you're installing from.

conda remove geocat-comp
conda install -c conda-forge geocat-comp

Negin Sobhani (Jul 10 2024 at 19:06):

Hello @Isla Simpson I have noticed a discrepancy in your dask PBSClusger resources specs.

Here you specify memory = 30 GB while your resouce_specs are requesting 10 GB chunks on each node. The memory parameters provide information to Dask, while the resource_spec provides information to PBS. So essentially this means that your dask resources think they can use up to 30 GB before spilling to disk while PBS only assign 10 GB per worker. If the memory on the node is mostly used, this means that your workers are using swap memory which leads to poor performance ...

Isla Simpson (Jul 11 2024 at 17:04):

@Katelyn FitzGerald @Negin Sobhani . Thanks for both of these suggestions. I'm trying to work with the new version of geocat. I'm now using version 2024.04.0 but whenever I try to run the vertical interpolation code I get the following error. I closed my dask cluster and started again and still got this error. Any thoughts on what do to about it?
Screen-Shot-2024-07-11-at-11.02.44-AM.png

Isla Simpson (Jul 11 2024 at 17:13):

another note on this is that I also just tried doubling the resources I'm using. I first tried cluster.scale(12) and then tried cluster.scale(24) and I got an error for both of them.

Katelyn FitzGerald (Jul 11 2024 at 17:27):

Hrm, I'm not sure offhand and did run a modified version of your code when troubleshooting this the first time around on Casper. Not that something couldn't still be wrong.

Isla Simpson (Jul 11 2024 at 17:32):

Isla Simpson (Jul 11 2024 at 17:33):

I did just try using the older version of the interpolation code and I got the same error. So it seems like it isn't something about the new geocat code, but rather something about my new environment that is not working nicely with it.

Isla Simpson (Jul 11 2024 at 17:51):

Katelyn FitzGerald (Jul 11 2024 at 18:10):

I thought the job might be getting cancelled for some reason, but it looks like that's not the case.

I wonder if you have some older versions of things in your environment that aren't working well (maybe Dask?).

If you want to point me to a script / notebook, I can also try to take a look. I think we're missing a bit of log info here.

Isla Simpson (Jul 11 2024 at 19:28):

@Katelyn FitzGerald thanks for the further suggestions. I updated my environment with the command you suggested and I'm still getting this error. My script is here /glade/u/home/islas/python/RRAtlantic/DATA_SORT/plev_inter/RRAtlantic/plevinterp_U.ipynb. It seems like the cell that opens the dask cluster executes fine without warnings and then it's when the pressure level interpolation cell fails that all the warnings in the screenshot above appear. I'd be interested to see if you find the same thing and if not maybe we can compare environments to see what's different? Thanks

Katelyn FitzGerald (Jul 11 2024 at 19:49):

Thanks for sharing. I’ll take a look. Might be a moment since I’m at a conference, but I should have some time later today.

Isla Simpson (Jul 11 2024 at 19:50):

No rush. I've actually done what I need to do already on the CGD machines. It just concerns me for the future if I can't get this up and running on the CISL machines once the CGD machines go away. Thanks!

Katelyn FitzGerald (Jul 11 2024 at 20:58):

I wasn't able to run a copy of your notebook, because of datafile permissions, but I did confirm my older test notebook (based on what you shared before) is working with the a copy of the NPL2024a env with updated geocat-comp. I'll check to see if there's a new one coming soon that might have the updated geocat-comp.

I also noticed you have some lingering compute and/or load calls in your new notebook (I think those got added in troubleshooting before or as a workaround if I recall correctly). I'd try removing those.

Isla Simpson (Jul 15 2024 at 02:11):

HI Katelyn, Thanks for looking into it further. I've changed my notebook to point toward files that you should be able to see at /glade/u/home/islas/python/RRAtlantic/DATA_SORT/plev_interp/RRAtlantic/plevinterp_U.ipynb. If you were willing to give it a shot with your environment I'd be interested to know whether it works. I just tried creating my environment from scratch again and had the same problem. I then made a very simple environment that only contains the basic things, to avoid there being anything weird about it and I still get the same "FutureCancelledError". I created that simple environment using this yml file /glade/u/home/islas/python/envs/islaenv_simple.yml. Thanks.

Katelyn FitzGerald (Jul 15 2024 at 14:51):

Katelyn FitzGerald (Jul 16 2024 at 23:16):

Just getting to this after some stuff came up yesterday and some JupyterHub issues today. I can't seem to access "/glade/campaign/cgd/cas/observations/ERA5/mon/ua/u_2023.nc" though.

Isla Simpson (Jul 16 2024 at 23:55):

Sorry. I have hopefully fixed the permissions now. But I've also copied that file into the directory that I copied all the other files and changed the path on my original script to point toward that.

Katelyn FitzGerald (Jul 17 2024 at 00:23):

Ok, so I was able to replicate what you're seeing (I'm fairly sure anyway). I think it has to do with Dask and how it's interacting with the function as applied to a larger dataset. This didn't come up with the smaller subset I was working with before, but I did notice some things that might be worth taking a deeper look at now.

Katelyn FitzGerald (Jul 17 2024 at 00:27):

One additional note - it does seem to work if you subset the time dimension some to make the dataset and resulting Dask task graph smaller. Obviously this isn't ideal, but it is a workaround for the moment.

Isla Simpson (Jul 17 2024 at 15:46):

HI Katelyn, I guess I'm glad to know it's not just me! Thanks for looking into it and for the workaround suggestion.

Stream: dask

Topic: slowness of geocat interp_hybrid_to_pressure

Isla Simpson (Jul 09 2024 at 21:31):