METdataio

MET tools write verification results as plain-text statistics files. Those files are accurate but awkward to query, plot, or analyze directly. METdataio is the data input/output layer that reads MET output and reshapes it: loading it into a relational database for interactive plotting, or reformatting it into tidy, long-format tables for scripted analysis. It is a Python rewrite of the data-handling capabilities originally built into METviewer.

What METdataio is

The Model Evaluation Tools (MET) produces verification statistics as ASCII (plain-text) files — for example, files with the .stat extension from tools like Point-Stat and Grid-Stat, or .tcst files from tropical-cyclone tools. Each file packs many line types (groups of related statistics) into one document, which is great for the machine that wrote it but hard for a human or a plotting tool to consume.

METdataio is the part of the METplus ecosystem that closes that gap. Rather than asking every downstream tool to re-parse MET’s text format, METdataio reads it once and hands off clean, structured data in two shapes:

A relational database — The METdbLoad utility parses MET output and inserts it into a SQL database, which METviewer and METexpress then query to build plots interactively.

Tidy, long-format tables — METreformat rewrites .stat data into a long, human-readable text file (one line type per file with a column-header row), and METreadnc reads netCDF input straight into a pandas or xarray dataframe for scripted analysis.

Who uses it: anyone who has run MET and now wants to look at the results — either through a point-and-click plotting front end, or in a Python notebook alongside METcalcpy.

Where it sits in a verification workflow

METdataio occupies the layer between MET (which produces statistics) and the tools that present or analyze them. It does not compute new verification scores; it moves and reshapes the scores MET has already written.

Figure 1. METdataio reads MET output once and hands off two shapes of data: a relational database for the plotting front ends, and tidy, long-format tables for scripted analysis.

Mental model

MET answers “how good was the forecast?” and writes the answer to text files. METdataio answers “now get those answers somewhere I can use them” — a queryable database or a clean dataframe. No new statistics are computed here.

Capabilities at a glance

METdataio bundles a small set of focused utilities. The table below summarizes each one’s job, what it reads, and what it produces.

Utility	What it does	Reads	Produces
METdbLoad	Parses MET output and loads it into a relational database, driven by an XML load specification.	.stat (STAT incl. MPR, ORANK line types), MODE, MTD	Rows in a MySQL / MariaDB / Aurora database (METviewer mv_ schema)
METreformat	Reformats .stat / .tcst output into a long, header-labeled ASCII layout, one line type per file, with statistic name/value columns.	.stat / .tcst (Point-Stat, Grid-Stat, Ensemble-Stat, TC-Pairs)	A reformatted text file readable by METplotpy / METcalcpy
METreadnc	Reads netCDF input directly into memory for scripted analysis.	One or more netCDF files	A list of xarray Datasets, or a pandas DataFrame

Scope today

The reformat utility supports a defined set of line types across Point-Stat, Grid-Stat, Ensemble-Stat, and TC-Pairs — not all MET tools or every line type. Unsupported line types raise an error rather than producing partial output. See the supported line types on the reformat deep dive for the full list and the MET tool each comes from.

METdbLoad & the load XML

METdbLoad is the flagship. It walks a set of MET output files, parses the line types it finds, and inserts the rows into a SQL database that follows the METviewer schema (the database name must carry the mv_ prefix). Everything about a run — where the files are, which database to write to, and which data types and line types to load — is described in a single XML load specification file.

Running the loader

1. Stand up a database. METdbLoad targets MySQL, MariaDB, or Aurora. The target database is created from the METviewer SQL schema and named with the mv_ prefix (for example mv_met_data).

2. Write a load specification XML. Describe the connection, point at the input files, and set the load flags (details below).

3. Run the loader. Invoke it from the METdbLoad ush directory: python met_db_load.py /path-to/load_specification.xml

4. Query from a front end. Once loaded, METviewer and METexpress read the populated database to build plots.

Anatomy of the load specification

The XML root element is <load_spec>. Its children fall into a few groups — connection, what to load, where the input lives, and which line types to keep.

Element	Group	Purpose
<connection>	Connection	Wraps the database credentials.
<management_system>	Connection	Optional; selects mysql, mariadb, or aurora.
<host>	Connection	Database host as hostname:port.
<database>	Connection	Target database name (must use the mv_ prefix).
<user> / <password>	Connection	Login credentials.
<load_stat>	What to load	Load STAT data.
<load_mode>	What to load	Load MODE data.
<load_mtd>	What to load	Load MODE Time Domain (MTD) data.
<load_mpr>	What to load	Load matched-pair (MPR) data.
<load_orank>	What to load	Load observation-rank (ORANK) data.
<line_type> / <val>	Filter	Optional; restrict to named line types — if omitted, all line types load.
<folder_tmpl> + <load_val> / <field>	Input	A directory template with placeholders, filled by <field> values.
<load_files> / <file>	Input	Alternative: an explicit list of file paths.
<verbose>	Behavior	Controls output volume.
<insert_size>	Behavior	Rows per SQL INSERT.
<stat_header_db_check>	Behavior	Optional duplicate-header check for STAT (has a performance cost).
<mode_header_db_check> / <mtd_header_db_check>	Behavior	Optional duplicate-header checks for MODE / MTD.
<drop_indexes> / <apply_indexes>	Behavior	Drop indexes before loading and re-apply after, for speed.
<force_dup_file>	Behavior	Force loading of files already seen.
<group> / <description>	Metadata	Database group name and description (used by the front ends).

Note on extensions

For STAT loading, only files carrying the .stat extension are picked up. The other data types — MODE, MTD, MPR, ORANK — are enabled through their respective <load_*> flags in the XML.

Reformat & read netCDF

Not every workflow wants a database. For Python-based analysis, METdataio offers two utilities that produce in-memory or text artifacts you can hand straight to pandas and the rest of the METplus Python stack.

METreformat — tidy, long-format statistics

A single MET .stat file interleaves many line types. METreformat’s write_stat_ascii.py untangles them: it reshapes Point-Stat, Grid-Stat, Ensemble-Stat, and TC-Pairs output into a long, human-readable text file where each line type is grouped with a column header row, and adds explicit columns for statistic names, values, and confidence limits (bootstrap and normal). The result reads cleanly and can be passed to METcalcpy and METplotpy.

The currently supported line types — drawn from Point-Stat, Grid-Stat, Ensemble-Stat, and TC-Pairs — are:

FHO CNT CTC CTS SL1L2 VL1L2 ECNT MCTS VCNT PCT RHIST TCDIAG MPR DMAP

See the reformat deep dive for the MET tool each line type comes from and the plot it feeds. Unsupported line types raise an error rather than producing partial output.

METreadnc — netCDF straight into Python

For netCDF input, the METreadnc.util.read_netcdf module exposes a ReadNetCDF() class. Point it at one file path or a list of them, then choose the output shape:

read_into_xarray(infile) — Returns a list of xarray Dataset objects — convenient for gridded, labeled, multi-dimensional data.

read_into_pandas(infile) — Returns a pandas DataFrame — convenient for tabular analysis.

A minimal session looks like this:

import METreadnc.util.read_netcdf as read_netcdf

file_reader = read_netcdf.ReadNetCDF()
ds = file_reader.read_into_xarray(infile)   # list of xarray Datasets
df = file_reader.read_into_pandas(infile)   # pandas DataFrame

The module depends on xarray, netcdf4, pandas, and pyyaml (versions aligned with the METcalcpy requirements).

Deep dives

Two companion pages go further into the two main paths through METdataio.

METdbLoad: loading into the database → The load specification XML in detail, the supported data types, the mv_ schema, and running the loader from the command line. Database path · METviewer & METexpress

Reformatting & reading netCDF → METreformat's long-format output and supported line types, plus reading netCDF into xarray / pandas with METreadnc. Dataframe path · METcalcpy & METplotpy

Ecosystem links

METdataio is plumbing: its value shows up in the tools downstream of it.

Ecosystem link

METdataio’s METdbLoad populates the relational database that METviewer and METexpress query to build their plots. Its tidy, long-format and netCDF output feeds METcalcpy (and METplotpy) for scripted statistics and figures.

---

A derived, human-readable re-presentation — not official documentation. Sources: METdataio User’s Guide — Overview