We are Xdev!#
More than a year ago, Matthew Long and I started shopping around the idea that a lot of what we do here at NCAR, as well as how we do it, needed a bit of a facelift. Our pitch was simple: As an institution that supports the geoscientific community by providing valuable datasets, supercomputing facilities, and computer models, NCAR needs to exemplify modern best practices in the development and use of these resources. We took inspiration from the remarkably effective Pangeo community, which continues to show enormous success with Open Source development when coupled with modern software best practices, and we started to develop a plan of how to help shift, albeit slowly, both NCAR and—though collaboration with the Pangeo community—the geoscience community toward a better future.
To implement our plan, Matt and I started to see how much benefit to NCAR it could be to have a small, Agile team that was dedicated to prototyping new technologies for the benefit of the rest of the organization and the rest of the geoscience community, as a whole, like the UK Met Office’s Informatics Lab. However, instead of just churning out technology, this group would also serve as an example to the rest of NCAR on how (and why) to use modern best practices (GitHub, continuous integration, test-driven development, etc.) for scientific software development. …and for science itself!
Matt and I dreamed big, and as usually happens when the reality of budgets collides with dreams, we were forced to scale back our vision of a ten-person team to small group of two. Two is still greater than zero, so we considered that a success, and the Experimental Development Team (Xdev) was born! In October of 2018, Anderson Banihirwe joined the team as a member of NCAR’s Computational & Informatics Laboratory (CISL), and in January of 2019, Julia Kent (CISL) joined us, too. By the fall of 2019, we were able to bolster our ranks with part-time support from Brian Bonnlander (CISL), Michael Levy (CGD), and Mohammad Abouali (CISL). Read more about us here.
Currently, the Xdev Team is working to make curiousity-driven analysis of “big” scientific data a day-to-day reality. This is a tall order! Not only does the technology need to advance to make such a dream a reality, but helping users (i.e., scientists) fully understand this new paradigm is extremely challenging. We appreciate that many scientists do not want to become developers themselves, which means that we must promote technology that is expressive, letting the user do more science with less engineering. This means abstracting away computational details, such as parallelism (e.g., with Dask) or I/O (e.g., with Xarray) or complex scientific functions, without abstracting away so much that the user can no longer problem solve when things go wrong. We believe that means using the Python programming language. Python is designed for rapid prototyping, which is the nature of most scientific programming for analysis purposes.
Achieving this goal also means teaching scientists about a new way of performing and thinking about data analysis. And this means not only teaching scientists how to use the technology (e.g., giving tutorials, providing online examples, etc.), but also how the technology can improve science (e.g., by making their workflows reproducible or replicable, by letting scientists ask questions they couldn’t with unscalable legacy tools, etc.).
So, the Xdev Teams works with scientists to develop new technology, promotes existing technology, and teaches scientists how to use this technology for the greatest good. This is who we are, and we are just beginning to scratch the surface on a very big problem.