Deepak Cherian (Mar 28 2025 at 14:27): Deepak Cherian (Mar 28 2025 at 14:27)::rocket: Solving #NASA ’s cloud data dilemma: Icechunk unlocks 100x faster access to archival data formats
We're thrilled to publish results from our pilot project with NASA and
@developmentseed.org
to enable high-performance cloud-native access for NASA’s 100s of petabytes of Earth observation data.
In the pilot, we used our new open source tensor storage engine #Icechunk and #VirtualiZarr to present archival NetCDF data stored in S3 as a single analysis-ready cloud-optimized (ARCO) dataset.
In the benchmark shown below, we were able to extract a month-long time series from GPM IMERG data stored in S3 in just 3 seconds. In comparison, the previous cloud-based approach takes 5 minutes!
https://earthmover.io/blog/nasa-icechunk
Last updated: May 16 2025 at 17:14 UTC