Aggregation

Combining verification statistics across many cases — correctly — with the agg_stat module.

A single forecast run gives you one number. Real verification asks a harder question: how good is the model over hundreds of forecasts? METcalcpy’s aggregation module answers that by combining MET statistics across many cases, fairly equalizing the samples being compared, and attaching bootstrap confidence intervals so you know whether a difference is real or just noise.

What aggregation is

METcalcpy is the Python statistics engine behind the METplus verification ecosystem — the same calculation logic used by METviewer, METexpress, and the METplotpy plotting packages. Its agg_stat module takes MET statistical output and produces aggregated statistics together with confidence intervals.

“Aggregation” here means rolling up the verification of many individual forecast cases — many valid times, lead times, or locations — into summary scores that describe overall model performance. A score like a root-mean-square error is only meaningful when it summarizes a large, well-defined sample. Aggregation is how you build that summary from the per-case pieces that MET writes out.

Prerequisite

Input data must first be reformatted using the METdataio METreformat module so that every column is labeled with its corresponding statistic name. For the ensemble ECNT line type, follow the METreformat instructions for “computing aggregation statistics with METcalcpy agg_stat” before running anything here.

The averaging pitfall — why you don’t just average the scores

The intuitive way to combine, say, ten daily RMSE values is to average them. For most verification statistics, this is wrong. A final score is usually a non-linear function of underlying sums, and the average of a function is not the function of the average.

MET is designed around this fact. Instead of emitting only the final scores, MET writes partial-sum and count line types that carry the raw ingredients of each score:

SL1L2 / SAL1L2 — scalar partial sums — Per-case sums such as the mean forecast, mean observation, mean of squared errors, and the case count. These are the building blocks for continuous scores like ME, RMSE, and correlation. The “A” variant (SAL1L2) carries the anomaly form of the same sums.

Contingency counts (2×2 and multi-category) — Raw hit / miss / false-alarm / correct-negative counts per case, from which categorical scores (POD, FAR, CSI, GSS, and similar) are computed.

The correct procedure is to sum the partial sums across all cases first, and then compute the final statistic once from those combined sums. The aggregated score is weighted naturally by each case’s sample size, and it equals the score you would have gotten by pooling all the raw pairs together.

Do not average final scores

Averaging per-day RMSE (or POD, CSI, correlation, …) is a common mistake. It silently mis-weights small and large samples and gives a number that does not correspond to any real pooled dataset. Always re-aggregate from partial sums and contingency counts, then derive the score. agg_stat is built to do exactly this.

Where this comes from

This partial-sum aggregation practice is the standard procedure MET’s partial-sum and contingency-count line types are designed for; it is described in the MET documentation for those line types (SL1L2 / SAL1L2 and the contingency-table outputs), not on the aggregation source page, which documents the agg_stat interface and its config keys rather than this rationale. We include it here because it is the single most important correctness point when using agg_stat.

Wrong vs. right, visualized

Figure 1. Two ways to combine three days of RMSE. The top path averages the final scores (wrong); the bottom path pools the partial sums and computes the score once (right).

Event equalization — comparing like with like

First, a definition: a series here is one selection of the data — a model or member together with a forecast variable — that is aggregated into a single curve. When you compare two series — for example two model configurations, or two forecast leads — a difference in their aggregated scores is only meaningful if both series summarize the same set of cases. If Series A happened to verify against 100 cases and Series B against only 80 (perhaps because of missing data on some days), then any difference between their scores could be an artifact of the differing samples rather than a real difference in skill.

Event equalization — a standard technique in verification, not unique to this module — removes that confound. Before aggregating, it keeps only the cases present in all series being compared and drops any case that is missing from even one of them. Every series is then aggregated over an identical, matched sample, so a difference in the result reflects the models — not the bookkeeping.

What counts as a “case” is the unit being matched: an instance of the independent variable — typically a valid time at a given forecast lead (and level), i.e. the indy_var / indy_vals key. Equalization matches on that key across series, so knowing your independent variable tells you exactly which rows can be dropped.

In agg_stat this behavior is toggled by the event_equal configuration flag (the example configuration sets event_equal: False; the source lists only that value, without explanatory text, so the semantics above are the general verification meaning). Turn it on when you are comparing series and want the comparison to be fair.

Figure 2. Event equalization drops cases that are not present in every series, so all series are aggregated over the same matched sample.

Bootstrap confidence intervals

An aggregated score is an estimate from a finite sample, so it carries uncertainty. agg_stat attaches a confidence interval to each statistic and writes it alongside the value — the documentation describes the output as containing “aggregated statistics and confidence intervals.”

The intervals are computed by bootstrapping: drawing many resamples of the cases (with replacement), recomputing the aggregated statistic on each resample, and reading the spread of those recomputed values to bound the estimate. The number of resamples drawn is set by num_iterations (the count of bootstrap replicates B); the source lists the key with example value 1 but does not define it, so treat it as the resample count and size it for stability rather than reading anything into the example value. Because verification cases are often serially correlated in time, agg_stat offers a circular block bootstrap, which resamples contiguous blocks of cases rather than individual cases so that autocorrelation is preserved. The block bootstrap also has a block-length parameter — the key tuning knob for serially correlated samples — which the source does not surface in this config example; check the bootstrap code for its name and default.

The same machinery supports judging whether the difference between two series is statistically significant: if the confidence interval on a difference excludes zero at the chosen significance level, the difference is unlikely to be an artifact of sampling.

Key	Example	Role
alpha	0.05	Significance level; with 0.05 the intervals are 95% confidence intervals.
circular_block_bootstrap	True	Use block resampling to respect temporal correlation between cases.
method	perc	Interval construction method — here the percentile method (perc).
num_iterations	1	Number of bootstrap resamples (replicates B) drawn; the example value 1 only exercises the code path.
random_seed	null	Seed for reproducible resampling; null leaves it unseeded.

Bootstrap / confidence-interval settings in config_agg_stat.yaml

Set enough iterations

The shipped example uses num_iterations: 1, which is only enough to exercise the machinery. Meaningful confidence intervals require many bootstrap replicates — raise num_iterations substantially for production runs, and set random_seed if you need reproducible intervals.

Configuration reference

A YAML file (the example is config_agg_stat.yaml) drives a run. It names the input and output, selects the line type and statistics, defines the independent variable and the series, and sets the bootstrap behavior. The example reproduced in the source verifies ensemble ECNT output.

Key	Example value	Meaning
agg_stat_input	…/test/data/rrfs_ecnt_for_agg.data	Full path to the reformatted input data.
agg_stat_output	…/ecnt_aggregated.data	Full path for the aggregated-statistics output file.
append_to_file	null	Whether to append results to an existing file.
line_type	ecnt	MET line type being aggregated (here ensemble continuous, ECNT).
list_stat_1	ECNT_RMSE, ECNT_SPREAD_PLUS_OERR	Statistics to aggregate for series 1.
list_stat_2	[]	Statistics for a second series (empty in the example).
fcst_var_val_1	{TMP: [ECNT_RMSE, ECNT_SPREAD_PLUS_OERR]}	A mapping from a forecast variable (TMP) to the list of ECNT stats requested for it — series 1.
fcst_var_val_2	{}	Forecast variable for series 2 (empty in the example).
series_val_1	{model: [RRFS_GEFS_GF.SPP.SPPT]}	A mapping from a dimension (model) to a one-element list selecting the model / member — series 1.
series_val_2	{}	Series 2 selection (empty in the example).
indy_var	fcst_lead	Independent variable across which results are organized.
indy_vals	['30000', …, '360000']	Independent-variable values (here forecast leads in MET’s HHMMSS encoding — e.g. 030000 = 3 h, 360000 = 36 h — not seconds).
derived_series_1 / _2	[]	Derived-series calculations (empty in the example).
event_equal	False	Toggles event equalization (keep only cases common to all compared series). The source lists only the value; the behavior is the standard verification meaning.
num_threads	-1	Thread count; -1 lets the tool choose automatically.

Selected keys from the example config_agg_stat.yaml

Two-series comparisons

The _1 / _2 pairs (fcst_var_val, series_val, list_stat, derived_series) let you define two series and compare them in one run. The shipped example fills in only series 1; populate the _2 keys, and turn on event_equal, to do a fair head-to-head comparison.

Running agg_stat

1. Reformat the MET output. Run the MET .stat files through the METdataio METreformat module so each column is labeled with its statistic name. For ECNT, follow the reformat instructions for aggregation with agg_stat.

2. Edit the configuration. In config_agg_stat.yaml, set agg_stat_input and agg_stat_output, choose the line_type, list the statistics (list_stat_1), define the series and the independent variable, and set the bootstrap options.

3. Set the environment. Export METCALCPY_BASE and add it to PYTHONPATH so the module and its config can be found.

4. Run the module. Invoke it from the command line:

   python $METCALCPY_BASE/metcalcpy/agg_stat.py path/to/config_agg_stat.yaml

   or call it from Python:

   from metcalcpy.agg_stat import AggStat
   
   AGG_STAT = AggStat(PARAMS)
   AGG_STAT.calculate_stats_and_ci()

   where PARAMS is a dictionary holding the input/output locations and settings.

5. Read the output. The run writes the aggregated-statistics file (for example ecnt_aggregated.data) containing the aggregated statistics and their confidence intervals, ready for plotting or further analysis.

Where this fits

Aggregation is one stage in a verification pipeline: MET produces per-case statistics, METdataio reformats and labels them, METcalcpy aggregates them with confidence intervals, and downstream tools plot or tabulate the result.

Ecosystem links

Inputs come from MET and are prepared by the METdataio METreformat module. The aggregated output is consumed by the METplotpy plotting packages and powers the statistics shown in METviewer and METexpress. Browse the rest of this component from the METcalcpy entrypoint.

• agg_stat.py
• AggStat
• partial sums
• SL1L2 / SAL1L2
• contingency counts
• event equalization
• circular block bootstrap
• confidence intervals
• ECNT

---

A derived, human-readable re-presentation — not official documentation. Sources: METcalcpy User’s Guide — Aggregation