November 9-10, 2023

Workshop on Correctness and Reproducibility for Climate and Weather Software

Submit an Abstract

Workshop

Model simulations are essential tools for understanding weather and climate. As we adapt to our changing climate, simulation codes inform both our understanding and policy decisions. These complex software artifacts are often the result of multiple decades of development. And they are in a state of near-constant development as scientific capabilities advance and high-performance computing (HPC) technologies evolve.

Given the societal importance of these codes, maintaining confidence and preserving code quality and reliability is critical. Yet scientific computing applications are often developed without the use of extensive software verification tools and techniques. Instead, development practices are typically dominated by short-term concerns about performance, resources, and project timelines. Technical challenges in running and evaluating climate and weather models further complicate code verification efforts. Given the scale of these models, a thorough correctness evaluation may be prohibitively expensive. It is also customary to require regression tests to yield bitwise identical results. This requirement is often unmet due to the chaotic nature of climate and weather models and the large variety of hardware/software environments they are run on. When bitwise identical results cannot be sought, field experts are to evaluate model results in a time-consuming and subjective manner.

In short, climate and weather modeling communities are in need of practical and feasible means of ensuring correctness and reproducibility. For example, we are interested in means to easily assess whether changes to a model code result in output that is systematically different or introduce artifacts that could influence scientific conclusions. Such changes may include hardware or software stack infrastructure differences, replacing parts of the model with ML-routines, or applying data compression to the output data. In this workshop, we aim to provide a venue to discuss challenges, opportunities, and recent advances in ensuring software correctness and reproducibility for climate and weather modelers, HPC community members, and industry partners.

Scope

Topics of interest include but not limited to:

  • Tools and approaches for software testing, debugging, quality assurance, and continuous integration.
  • Statistical and ensemble-based approaches for evaluating model consistency and software correctness.
  • Software design approaches and development practices for streamlining correctness and reproducibility efforts.
  • Formal methods, abstraction, and logical proof techniques for rigorous verification.
  • Verifying and validating large-scale applications running on HPC clusters, cloud computing systems, heterogeneous systems, GPUs, etc.
  • Other software correctness and reproducibility approaches for facilitating verification and validation.

Submissions may include technical results, approaches, experiences, and opinions involving one or more of the above topics applied to:

  • Climate and weather simulation codes such as drivers, couplers, frameworks, and model components.
  • External libraries and packages used in climate and weather simulation applications.
  • Artificial intelligence techniques, such as machine learning and deep learning, applied to climate and weather software.
  • Diagnostics, post-processing, visualization tools, and libraries.
  • Packaging, environment management, version control, and porting techniques for facilitating reproducibility.
  • Other software development and approaches extensively used within the climate and weather simulation context.

Submissions

Registration

Registration will open in the Summer of 2023.

  • In person fee: $50
  • Virtual fee: $25

Dates

  • Abstract submissions due: August 1, 2023.
  • Notification of acceptance: August 31, 2023.
  • Registration deadline: October 20, 2023 (in person), November 3, 2023 (virtual)
  • Workshop date: November 9-10, 2023.

Organizers

Co-chairs
  • Allison Baker, Computational & Information Systems Lab, National Center for Atmospheric Research
  • Alper Altuntas, Climate & Global Dynamics Lab, National Center for Atmospheric Research
Committee
  • Ilene Carpenter, Earth Sciences Segment Manager, Hewlett Packard Enterprise
  • Brian Dobbins, Climate & Global Dynamics Lab, National Center for Atmospheric Research
  • Michael Duda, Mesoscale & Microscale Meteorology Lab, National Center for Atmospheric Research
  • Dorit Hammerling, Applied Mathematics and Statistics, Colorado School of Mines
  • Thomas Hauser, Computational & Information Systems Lab, National Center for Atmospheric Research
  • Karsten Peters-von Gehlen, Department of Data Management, Deutsches Klimarechenzentrum GmbH (DKRZ)
Administrator

Program

The workshop program will include keynote speakers, invited talks, peer-reviewed submissions, and discussions.

Confirmed Keynote Speakers:

  • Dorit Hammerling: Associate Professor, Applied Mathematics and Statistics; Colorado School of Mines

  • Steve Easterbrook: Director, School of the Environment; Professor, Department of Computer Science; University of Toronto

  • John Baugh: Professor, Civil Engineering and Operations Research; North Carolina State University

Venue

The workshop will be held in person (with a virtual option) and at the Mesa Laboratory of the National Center for Atmospheric Research.


As an architectural landmark, the Mesa Laboratory of the National Center for Atmospheric Research is located atop Table Mesa at the west end of Boulder, Colorado, and is recognized as one of the major works of architect I.M. Pei.

Address: 1850 Table Mesa Dr, Boulder, CO 80305

Note: Virtual Meeting details will be announced later.