Reformat

MET tools write statistics into .stat and .tcst files whose layout changes from one line type to the next, with many columns left unlabelled. METdataio’s METreformat module rearranges that output into named columns the METplotpy plotting scripts (and METcalcpy aggregation code) expect, while its METreadnc module reads NetCDF files straight into Xarray Datasets or a pandas DataFrame.

Why reformatting exists

The MET tools — Point-Stat, Grid-Stat, Ensemble-Stat, and TC-Pairs — write their verification numbers as ASCII columnar data. A single .stat (or .tcst for TC-Pairs) file can hold rows from several different line types at once, where a line type is the category of statistic written on a row (for example contingency-table counts, continuous statistics, or matched pairs). Which line types appear is controlled by settings in the MET configuration file.

Every MET .stat line type shares a block of common stat columns that do have headers — things like model, verification mask, description, and forecast lead. After those, the remaining columns are unlabelled, and their meaning and count differ from line type to line type. That makes raw .stat output awkward to hand to a generic analysis or plotting routine: the same physical column position means different things on different rows.

METreformat rearranges those unlabelled columns into a layout suited to the specific METplotpy plot you want. As the docs put it, the needed shape varies by plot: a histogram plot needs ranks and values, whereas a line plot needs a statistic-name column and a statistic-value column. The reformatter targets these shapes directly.

A note on jargon

Wide format packs many statistics across columns of one row; long (or “tidy”) format gives each statistic its own row. MPR is the matched pair line type — one forecast/observation pair per record. Reformatting generally converts wide rows into longer ones so every statistic value lands in a named column.

Not every line type is supported

Some MET line types do not have reformatter support. When an unsupported line type is requested, METreformat generates an error message rather than silently producing partial output.

From wide rows to named columns

The picture below contrasts a raw .stat row — common labelled columns followed by a run of unlabelled values — with the reformatted “longer” output, where each statistic gets its own row under stat_name / stat_value while the common columns are repeated. The lower track shows the separate METreadnc path: a NetCDF file read straight into an Xarray Dataset or a pandas DataFrame.

Figure 1. A wide .stat row (labelled common columns + unlabelled statistic columns) becomes several tidy long rows under stat_name / stat_value; separately, METreadnc reads NetCDF into an Xarray Dataset or a pandas DataFrame. Statistic names and values shown are illustrative.

Supported line types

METreformat currently supports the line types below. Each comes from one or more MET tools and feeds particular METplotpy plot types. The “from” column lists the MET tools that emit it.

Line type	Meaning	From MET tools
FHO	Forecast, Hit, Observation rates	Point-Stat, Grid-Stat
CNT	Continuous statistics	Point-Stat, Grid-Stat
CTC	Contingency table counts	Point-Stat, Grid-Stat
CTS	Contingency table statistics	Point-Stat, Grid-Stat
SL1L2	Scalar L1L2 partial sums	Point-Stat, Grid-Stat
VL1L2	Vector L1L2 partial sums	Point-Stat, Grid-Stat
ECNT	Ensemble continuous statistics	Point-Stat, Ensemble-Stat
MCTS	Multi-category contingency table statistics	Point-Stat
VCNT	Vector continuous statistics	Point-Stat, Grid-Stat
PCT	Contingency table counts for probabilistic forecasts	Point-Stat, Grid-Stat, Ensemble-Stat
RHIST	Ranked histogram counts	Ensemble-Stat
TCDIAG	Tropical cyclone diagnostics (with TCMPR matched pairs)	TC-Pairs
MPR	Matched pair data	Point-Stat, Grid-Stat
DMAP	Distance map statistics	Grid-Stat

TCMPR is always there

For TC-Pairs output, the TCMPR line is always generated, while the TCDIAG line appears only when diagnostics are requested in the MET config. The reformatter consolidates each TCDIAG line with its corresponding TCMPR line so the pair becomes one record.

The reformatted formats

The reformatted output adds new columns whose shape depends on the plot and line type. The source guide groups these into a handful of categories. The most common is the statistic name / value layout used by line, bar, box, contour, performance-diagram, revision, and Taylor-diagram plots.

By statistic name, value, and confidence limits

Unlabelled columns from the original .stat file are placed under these named columns, in addition to the common stat columns. This is the wide-to-long conversion: one statistic per row, with the common columns repeated.

Column	Meaning
stat_name	Name of the statistic on this row
stat_value	Value of that statistic
stat_ncl	Lower-level normal confidence limit
stat_ncu	Upper-level normal confidence limit
stat_bcl	Lower-level bootstrap confidence limit
stat_bcu	Upper-level bootstrap confidence limit

The common stat columns — model, vx mask, description, forecast lead, and the rest — are shared across all line types for Point-Stat, Grid-Stat, and Ensemble-Stat (see Tables 11.1, 12.1, and 13.1 in the MET User’s Guide). After reformatting, rows that share the same common columns repeat them while differing only by stat_name and stat_value.

By threshold values (PCT → ROC diagram)

The PCT line type has a variable number of unlabelled columns for THRESH_i, OY_i, and ON_i. These are labelled, then reordered into a long shape used by the ROC diagram plot:

Column	Meaning
thresh_i	The ith probability threshold value (repeated)
oy_i	Count of observation=YES events whose forecast probability falls between the ith and i+1th threshold
on_i	Count of observation=NO events whose forecast probability falls between the ith and i+1th threshold
i_value	The value number, i.e. the i of thresh_i, oy_i, and on_i

By ranked data values (RHIST → rank histogram)

For the rank histogram plot, in addition to the common stat columns and the RHIST-specific columns, the reformatted file adds:

Column	Meaning
rank_i	Count for the ith rank-histogram bin (repeated)
i_value	The rank number

By all columns (MPR / DMAP → scatter)

For scatter plots of the MPR or DMAP line types, the unlabelled columns are placed under the line-type-specific column names from the MET User’s Guide. The reformatted file keeps the common stat columns plus all line-type-specific columns, with a leading index value created during reformatting. Here the stat_name, stat_value, and confidence-limit columns are not needed — control this with keep_all_cols (see below).

TCDIAG consolidated with TCMPR

The TCDIAG reformat consolidates each TCDIAG line with its corresponding TCMPR line, producing a record that carries the common TCDIAG stat columns, the TCMPR stat columns, and the TCDIAG stat columns (MET User’s Guide Tables 24.1, 24.2, and 24.3). The TCDIAG INDEX column is renamed INDEX_PAIRS to distinguish it from the TCMPR INDEX, and the N_DIAG column reports how many storm-diagnostic name/value pairs that row carries. These records drive the TCMPR plots in METplotpy.

ECNT for aggregation (METcalcpy)

The ECNT line type (from Ensemble-Stat) can be reformatted to carry all the ECNT statistic values, with the statistics split into stat_name / stat_value plus the same four confidence-limit columns. When the output is destined for METcalcpy agg_stat.py to compute aggregation statistics, the stat_name entries are pre-pended with ECNT_ — so RMSE becomes ECNT_RMSE, SPREAD_PLUS_OERR becomes ECNT_SPREAD_PLUS_OERR, and so on — and the stat_value column is left empty for agg_stat to fill. Reformatting for agg_stat is currently available only for the ECNT line type.

Aggregation vs. not

Not all data needs aggregation — histogram plots, for instance, do not. Read input_stats_aggregated as a decision about what happens next: True = the output is plot-ready; False = run it through the METcalcpy agg_stat.py module first (available for ECNT only). The name describes the input, but the flag gates the downstream aggregation step.

Configuration reference

Running the reformatter from the command line uses a YAML configuration file, reformat_stat.yaml, copied from the source directory into your working directory. It points at the input data, names the output, sets logging, and chooses the line type. The mandatory settings:

input_stats_aggregated — Gates whether downstream aggregation must run. True (default) = the output is plot-ready and can go straight to a METplotpy plotting script — either because the input was already processed by MET Stat-Analysis to compute aggregation statistics, or because the data does not require aggregation. False = aggregation statistics still need to be calculated, so run the output through the METcalcpy agg_stat.py module (ECNT only) before plotting.

input_data_dir — Full path (no environment variables) to the directory holding all the input .stat files from Point-Stat, Grid-Stat, or Ensemble-Stat. Data spread across several directories must be consolidated into one.

output_dir — Full path (no environment variables) to the directory where the reformatted file is saved.

output_filename — Name of the output file. Save with a .data extension if it will be plotted with METplotpy. Re-running without removing an existing file of the same name overwrites it.

log_filename — Name of the log file. Set to STDOUT / stdout (case insensitive) to skip writing a log file.

log_dir — Full path (no environment variables) to the directory for the log file.

log_level — Logging verbosity: INFO, DEBUG, WARNING, or ERROR. As the source guide states, INFO is most verbose and ERROR least — though note that in standard Python logging DEBUG emits the most detail of the four.

line_type — The line type to reformat — one of the supported types: FHO, CNT, CTC, CTS, SL1L2, VL1L2, ECNT, MCTS, VCNT, RHIST, PCT, TCDIAG, MPR, or DMAP.

keep_all_cols — Relevant for the MPR or DMAP line types only. True when reformatting for a scatter plot (all column names are kept and the stat_name / stat_value / confidence columns are not needed), False otherwise.

No environment variables in paths

The path settings do not expand environment variables — give the actual absolute path, not $BASE_DIR or similar, inside the YAML file.

Running the reformatter

The reformatter draws on the MET Stat-Analysis tool for any filtering, and on METdbLoad modules to find and collect data from the individual .stat files into a single structure (which is why the input must live under one directory). A typical run looks like this:

1. Filter first if needed. Use the MET Stat-Analysis tool to filter by model, valid times, regions, and so on. Its output is then the reformatter’s input. If no filtering is needed, the raw .stat files from Point-Stat, Grid-Stat, or Ensemble-Stat can be used directly.
2. Consolidate the data. Place all the .stat / .tcst files of interest into a single directory; reorganize if they are spread across many.
3. Set the environment. Export BASE_DIR (the METdataio source location), WORKING_DIR (where the YAML lives), and a PYTHONPATH that includes METdbLoad and METreformat.
4. Edit reformat_stat.yaml. Set input_stats_aggregated, the input/output directories, the output filename, the line type, the logging settings, and keep_all_cols (for MPR / DMAP).
5. Run write_stat_ascii.py. Invoke python $BASE_DIR/METreformat/write_stat_ascii.py $WORKING_DIR/reformat_stat.yaml. A text file is created in the output directory under the name you set in the YAML.
6. Aggregate if required. When a plot needs aggregation (for example an ensemble spread-skill line plot using ratio lines), run the reformatted ECNT output through the METcalcpy agg_stat.py module before plotting.

Command line vs. METviewer

The reformatted shapes mirror what METviewer’s database query would hand to METplotpy. Generating plots from the command line bypasses METviewer and its database — but then Stat-Analysis must do the filtering, and reformatting must happen before the METplotpy scripts are invoked.

Reading NetCDF with METreadnc

The companion METreadnc module reads NetCDF input files and produces either a list of Xarray Datasets or a pandas DataFrame — useful for pulling MET’s NetCDF output into the same Python analysis world as the reformatted stat data. It requires xarray, netcdf4, pandas, and pyyaml to be installed; the guide recommends matching the package versions pinned in METcalcpy’s requirements files.

From within your own Python code, import the module, create a reader, and call one of two methods. Each accepts infile as either a single file path (string) or a list of file names; give the full path in both cases.

Step	Call	Result
Import	import METreadnc.util.read_netcdf as read_netcdf	—
Create reader	file_reader = read_netcdf.ReadNetCDF()	A reader object
Read to Xarray	ds = file_reader.read_into_xarray(infile)	A list of Xarray Datasets
Read to pandas	df = file_reader.read_into_pandas(infile)	A pandas DataFrame

As with the reformatter, set up BASE_DIR, WORKING_DIR (if used from the command line), and a PYTHONPATH that includes METdataio/METreadnc before importing.

Scope note

The source guide documents METreadnc generically as “read netCDF input files.” It does not, in this section, enumerate specific MET NetCDF point versus gridded output types or the resulting column / variable names — those depend on the file you read.

Where this sits

METreformat is the bridge between MET’s raw ASCII output and the rest of the METplus analysis stack: Stat-Analysis (and optionally METdbLoad) feed it, METcalcpy aggregates what it produces, and METplotpy plots the result. METreadnc handles the NetCDF side of the same pipeline.

Ecosystem links

Reformatted .data files are consumed by METplotpy plotting scripts; aggregation statistics are computed by METcalcpy’s agg_stat.py. Upstream, filtering and collection rely on the MET Stat-Analysis tool and METdbLoad. See the ecosystem overview for how the pieces connect.

---

A derived, human-readable re-presentation — not official documentation. Sources: reformat_stat_data · read_nc