Deepak Cherian (Apr 13 2023 at 14:46): Deepak Cherian (Apr 13 2023 at 14:46):Hi I'm looking for examples of challenging computations with xarray and dask that are requiring workarounds like loops to complete. The goal is to get simplified examples to use to inform dask.array development. I think a real use case involving a big regridding, rolling window operations, rolling.construct, or groupby.map would be good examples.
Isla Simpson (Apr 13 2023 at 14:54):I have one that I have been struggling with. It is interpolating atmospheric model data from one set of hybrid coordinates to another e.g., interpolating ERA5 model level data to the CAM grid. The issue is that both the original vertical pressure levels and the new vertical pressure levels are a function of longitude latitude and time. So I end up having to loop over longitude, latitude and time. I can make a cleaned up example if this is an example you want to follow (am out of town right now so can't do it right now).
Deepak Cherian (Apr 13 2023 at 14:56):I would be surprised if this is still an issue with xgcm.transform (https://xgcm.readthedocs.io/en/latest/transform.html).
Isla Simpson (Apr 13 2023 at 21:20):oh ok, I wasn't area of this. I'll look into it. Thanks
Stephen Yeager (Apr 14 2023 at 14:34):Here is an example of a big dask computation that requires looping over time chunks. This notebook computes MOC(sigma) from 0.1° POP output spanning 2006-2100, and requires multiple 3D fields at monthly resolution. We got it working for 10-year chunks at a time.
Deepak Cherian (Apr 15 2023 at 02:27):I got it to work 2006-2050 at one go (well I didn't complete but its looking good). Let's follow up on the repo
Daniel Marsh (Apr 18 2023 at 21:08):Deepak Cherian said:
I would be surprised if this is still an issue with
xgcm.transform(https://xgcm.readthedocs.io/en/latest/transform.html).
Nice! This could be very useful. One of the things we need to do all the time is convert constituent mixing ratios on hybrid pressure to density on altitude grids for comparison to observations. This would involve converting ilev and lev to real pressure (1-D to 3-D) using PS, calculating geometric height by adjusting geopotential height (Z3) to take into account the falloff of gravity and then calculating total density on lev using T, so a fair number of steps before this routine can do the conservative remapping. It would be a nice worked example if I could get it all figured out. It's fairly computationally expensive, so would be good to get it working with DASK.
Deepak Cherian (Apr 22 2023 at 21:28):@Stephen Yeager or anyone else really, do you know if there's a full-depth POP 1/10° dataset on the cloud that I could use to replicate? The Pangeo cloud version seems to have only two depth levels:
Stephen Yeager (Apr 25 2023 at 18:14):@Deepak Cherian I don't know but I'd ask Ryan Abernathy because he was talking to Ping Chang about putting some of the iHESP data on the cloud.
Last updated: May 16 2025 at 17:14 UTC