November 9-10, 2023

Workshop on Correctness and Reproducibility for Climate and Weather Software

Program (Slides & Recordings)


Model simulations are essential tools for understanding weather and climate. As we adapt to our changing climate, simulation codes inform both our understanding and policy decisions. These complex software artifacts are often the result of multiple decades of development. And they are in a state of near-constant development as scientific capabilities advance and high-performance computing (HPC) technologies evolve.

Given the societal importance of these codes, maintaining confidence and preserving code quality and reliability is critical. Yet scientific computing applications are often developed without the use of extensive software verification tools and techniques. Instead, development practices are typically dominated by short-term concerns about performance, resources, and project timelines. Technical challenges in running and evaluating climate and weather models further complicate code verification efforts. Given the scale of these models, a thorough correctness evaluation may be prohibitively expensive. It is also customary to require regression tests to yield bitwise identical results. This requirement is often unmet due to the chaotic nature of climate and weather models and the large variety of hardware/software environments they are run on. When bitwise identical results cannot be sought, field experts are to evaluate model results in a time-consuming and subjective manner.

In short, climate and weather modeling communities are in need of practical and feasible means of ensuring correctness and reproducibility. For example, we are interested in means to easily assess whether changes to a model code result in output that is systematically different or introduce artifacts that could influence scientific conclusions. Such changes may include hardware or software stack infrastructure differences, replacing parts of the model with ML-routines, or applying data compression to the output data. In this workshop, we aim to provide a venue to discuss challenges, opportunities, and recent advances in ensuring software correctness and reproducibility for climate and weather modelers, HPC community members, and industry partners.


Topics of interest include but not limited to:

  • Tools and approaches for software testing, debugging, quality assurance, and continuous integration.
  • Statistical and ensemble-based approaches for evaluating model consistency and software correctness.
  • Software design approaches and development practices for streamlining correctness and reproducibility efforts.
  • Formal methods, abstraction, and logical proof techniques for rigorous verification.
  • Verifying and validating large-scale applications running on HPC clusters, cloud computing systems, heterogeneous systems, GPUs, etc.
  • Other software correctness and reproducibility approaches for facilitating verification and validation.

Submissions may include technical results, approaches, experiences, and opinions involving one or more of the above topics applied to:

  • Climate and weather simulation codes such as drivers, couplers, frameworks, and model components.
  • External libraries and packages used in climate and weather simulation applications.
  • Artificial intelligence techniques, such as machine learning and deep learning, applied to climate and weather software.
  • Diagnostics, post-processing, visualization tools, and libraries.
  • Packaging, environment management, version control, and porting techniques for facilitating reproducibility.
  • Other software development and approaches extensively used within the climate and weather simulation context.


  • Submissions are closed.



  • Abstract submissions due: EXTENDED to August 7, 2023.
  • Notification of acceptance: August 31, 2023.
  • Registration deadline: October 20, 2023 (in person), November 3, 2023 (virtual)
  • Workshop date: November 9-10, 2023.


  • Allison Baker, Computational & Information Systems Lab, National Center for Atmospheric Research
  • Alper Altuntas, Climate & Global Dynamics Lab, National Center for Atmospheric Research
  • Ilene Carpenter, Earth Sciences Segment Manager, Hewlett Packard Enterprise
  • Brian Dobbins, Climate & Global Dynamics Lab, National Center for Atmospheric Research
  • Michael Duda, Mesoscale & Microscale Meteorology Lab, National Center for Atmospheric Research
  • Dorit Hammerling, Applied Mathematics and Statistics, Colorado School of Mines
  • Thomas Hauser, Computational & Information Systems Lab, National Center for Atmospheric Research
  • Karsten Peters-von Gehlen, Department of Data Management, Deutsches Klimarechenzentrum GmbH (DKRZ)


Keynote Speakers:
  • John Baugh: Professor, Civil Engineering and Operations Research; North Carolina State University
  • Peter Dueben: Head of the Earth System Modelling Section, European Centre for Medium Range Weather Forecasts (ECMWF)
  • Steve Easterbrook: Director, School of the Environment; Professor, Department of Computer Science; University of Toronto
  • Dorit Hammerling: Associate Professor, Applied Mathematics and Statistics; Colorado School of Mines
Thursday November 9th, 2023
8:40 - 9:40 KEYNOTE: Models, Data, and Wisdom: How do we know when to trust a climate model? slides Steve Easterbrook
School of the Environment and Department of Computer Science, University of Toronto
9:40 - 10:00 TALK: Component Level Regression Testing in a Hierarchical Architecture. slides Thomas Clune
NASA Goddard Space Flight Center
10:00 - 10:20 TALK: High Performance Climate and Weather Benchmark (HPCW): a framework for reproducible benchmarks of ESM models and mini-applications. slides David Guibert
Center for Excellence in Performance Programming, Eviden
10:20 - 10:50 BREAK
10:50 - 11:10 TALK: Correctness Challenges in HPC and ML. slides Harvey Dam, Ganesh Gopalakrishnan
Department of Computer Science, University of Utah
11:10 - 11:30 TALK: Reliable and reproducible Earth System Model data analysis with ESMValTool. slides Valeriu Predoi*
NCAS-CMS, University of Reading
11:30 - 11:50 TALK: Testing approach for porting legacy 4-mode Modal Aerosol Model (MAM4) to C++/Kokkos. slides Balwinder Singh
Atmospheric Sciences and Global Change Division, Pacific Northwest National Laboratory
11:50 - 12:10 TALK: Verification of the ICON model with the GT4Py dycore - challenges and insights. slides Abishek Gopal*
Institute for Atmospheric and Climate Science, ETH Zurich
12:10 - 1:10 LUNCH Mesa Lab Cafeteria
Included with Registration
1:10 - 2:10 KEYNOTE: Earth system models of the future. slides Peter Dueben*
Earth System Modelling Section, European Centre for Medium Range Weather Forecasts (ECMWF)
2:10 - 2:30 TALK: A Theory of Scientific Programming Efficacy. slides Michael Coblenz
Department of Computer Science, UC San Diego
2:30 - 2:50 TALK: An overview of the MOM6 development cycle. slides Marshall Ward*
Geophysical Fluid Dynamics Lab, NOAA
2:50 - 3:20 BREAK
3:20 - 3:40 TALK: Challenges in Ensuring Reproducibility for Machine Learning Weather Model Training and Deployment. slides David John Gagne
Computational and Information Systems Lab, NCAR
3:40 - 4:00 TALK: METplus: The Long and Winding Road to Unified Verification. slides Tara Jensen*
Research Applications Lab, NCAR
4:00 - 4:20 TALK: Unit Testing NCEPLIBS. slides Edward Hartnett
Friday November 10th, 2023
8:30 - 9:30 KEYNOTE: Lightweight Formal Methods: The What, Why, and How. slides John Baugh*
Civil Engineering and Operations Research, North Carolina State University
9:30 - 9:50 TALK: What could the next 30 years of software verification in climate science look like?. slides Dominic Orchard*
Department of Computer Science and Technology, University of Cambridge and School of Computing, University of Kent
9:50 - 10:10 TALK: Parallel reproducibility of the SHYFEM-MPI model. slides Francesco Carere*
Euro Mediterranean Center on Climate Change Foundation (CMCC Foundation)
10:10 - 10:40 BREAK
10:40 - 11:40 KEYNOTE: Contained Chaos: Quality Assurance for the Community Earth System Model. slides Dorit Hammerling
Applied Mathematics and Statistics, Colorado School of Mines
11:40 - 12:00 TALK: Methods and Tools for the Application of UF-ECT to New Climate Models. slides Teo Price-Broncucia
Department of Computer Science University of Colorado Boulder
12:00 - 12:20 TALK: Ensure the correctness and reproducibility in UFS Weather Model CI. slides Jun Wang
12:20 - 1:20 LUNCH Mesa Lab Cafeteria
Included with Registration
1:20 - 1:40 TALK: Towards Ensuring Statistical Climate Reproducibility of Earth System Models in the Exascale Age. slides Salil Mahajin
Computational Earth Sciences Group, Oakridge National Laboratory
1:40 - 2:00 TALK: Improvements in Reproducibility Testing Through False Discovery Rate Correction. slides Michael Kelleher
Computational Earth Sciences Group, Oakridge National Laboratory
2:00 - 3:30 PANEL: Correctness and verification across platforms
- Ilene Carpenter, Hewlett Packard Enterprise
- Karsten Peters-von Gehlen, Deutsches Klimarechenzentrum GmbH (DKRZ)
- Ganesh Gopalakrishnan, University of Utah
- Aaron Donahue, Livermore National Laboratory
Moderator: Brian Dobbins, NCAR
3:30 - 4:00 BREAK
*Indicates speaker is remote


The workshop will be held in person (with a virtual option) and at the Mesa Laboratory of the National Center for Atmospheric Research. (Helpful things to know for your visit.)

As an architectural landmark, the Mesa Laboratory of the National Center for Atmospheric Research is located atop Table Mesa at the west end of Boulder, Colorado, and is recognized as one of the major works of architect I.M. Pei.

Address: 1850 Table Mesa Dr, Boulder, CO 80305

Note: Virtual Meeting details will be announced later.


  • Hilton Garden Inn (2701 Canyon Boulevard, Boulder, CO 80302)
  • Book by October 19, 2023 ($229/night)
  • Block reservation link
  • Note: We will arrange shared transportation from the hotel to the Mesa Lab each morning and returning each afternoon for guests who need it (more info to come).