Deepak Cherian (Sep 13 2022 at 15:16): Deepak Cherian (Sep 13 2022 at 15:16):CarbonPlan - Building open source downscaling pipelines for the cloud
We recently completed a global climate model downscaling project, for which we processed hundreds of terabytes of climate data entirely using cloud infrastructure. We chose to use the cloud for several reasons: scalability, reproducibility, and the ability to access large volumes of data in cloud storage directly, rather than downloading to local machines.
Working in the cloud on a project at this scale also posed several challenges. In this post, we explain how we approached the problem, and then highlight what worked well and where we struggled. We hope the lessons learned might be useful to others working on related efforts.
Last updated: May 16 2025 at 17:14 UTC