Diagnostic diagrams

Verification scores are easy to compute and hard to feel. These four diagrams turn columns of statistics into a picture you can read at a glance: is my forecast detecting events, is it honestly calibrated, can it tell events from non-events, and does it match the observed climate? METplotpy draws all four from the line-type output that MET’s Point-Stat and Grid-Stat tools already produce.

How to use this page

Each diagram below follows the same four-part shape so you can compare them: the question it answers, the axes (what is plotted against what), how to read good vs. bad, and the MET line type that feeds it. A line type is just the named row format MET writes into its .stat output — for example a CTS row holds contingency-table statistics, a PCT row holds a probability contingency table.

Every diagram is a small theme-reactive schematic, not a real data plot — the curves are illustrative so the geometry is clear. The captions point out the one feature that matters most.

A shared vocabulary

POD = Probability of Detection (hit rate); the PODY column is POD for the event (“yes”) category, so POD and PODY are used interchangeably here. FAR = False Alarm Ratio. SR = Success Ratio = 1 − FAR. POFD = Probability of False Detection. CSI = Critical Success Index. These show up across several diagrams below.

1 · Performance diagram

“Is my categorical forecast catching events without crying wolf?”

The performance diagram packs four categorical statistics into a single square. It shows the relationship between detecting events and avoiding false alarms, and — because of the geometry — it reveals frequency bias and Critical Success Index at the same time without you having to read them off a table.

Axes and reference lines

X-axis — Success Ratio, SR = 1 − FAR — How often a “yes” forecast was right. Higher is better; it runs left (bad) to right (good).

Y-axis — Probability of Detection, PODY — How often an observed event was forecast. Higher is better; it runs bottom (bad) to top (good).

CSI curves — Curves of equal Critical Success Index sweep from the top of the plot to the right side. A point sitting on a higher CSI curve is a better overall forecast.

Frequency-bias lines — Dashed lines radiating from the origin, labeled with their bias value. The 1:1 diagonal is perfect bias (1.0). Lines steeper than it (leaning toward the upper-left, high POD relative to success ratio) are bias > 1 — over-forecasting; lines shallower than it (toward the lower-right) are bias < 1 — under-forecasting.

How to read it

Perfect sits in the upper-right corner — maximum POD and maximum success ratio at once. A point near the diagonal has good bias; a point on the highest CSI curve has good overall skill. Drift down-left and the forecast is both missing events and raising false alarms.

Figure 1. The performance diagram. Solid grey arcs are equal-CSI curves; the dashed teal diagonal is perfect frequency bias (1.0). Forecast A beats B — it sits on a higher CSI curve and closer to the perfect upper-right corner.

What feeds it

The simplest input is MET’s contingency-table statistics output — the CTS line type — which can be produced by many MET tools (Point-Stat, Grid-Stat, and others). METplotpy reads the PODY, FAR, and CSI columns from that columnar text and places each forecast as a point on the square.

2 · Reliability diagram

“When I say 70%, does it happen 70% of the time?”

The reliability diagram is the calibration check for probabilistic forecasts. It plots the conditional bias of those forecasts: for each forecast-probability bin, it asks how often the event actually occurred. An honest forecast system that says “30% chance” should be right about 30% of the time — no more, no less.

Axes and the perfect line

X-axis — forecast probability — The probability the forecast issued, binned (for example 0 through 0.9 in the documentation’s example).

Y-axis — observed relative frequency — The fraction of times the event was actually observed within that probability bin.

The 45° diagonal — perfect reliability — Where forecast probability equals observed frequency. A curve lying on this line is perfectly calibrated.

How to read it

Follow the curve relative to the diagonal. Above the line the event happens more often than forecast (under-confident). Below the line the event happens less often than forecast (over-confident). Resolution is how much the curve rises across the plot — how far the observed frequency spreads from low to high as the forecast probability climbs. A steeply rising curve like the one drawn has good resolution; a flat, near-horizontal curve has little resolution, barely distinguishing high-probability days from low-probability ones.

Figure 2. A reliability diagram. The dashed teal diagonal is perfect calibration; the blue curve is a sample forecast that is under-confident at low probabilities (above the line) and over-confident at high probabilities (below the line).

What feeds it

Reliability is built from MET’s probability contingency table — the PCT line type, read as columnar text output. The METplotpy reliability plotter references statistics such as PSTD_CALIBRATION, PSTD_BASER, and PSTD_NI, and can optionally overlay a no-skill line, a reference line, and a skill line via configuration toggles (add_noskill_line, add_reference_line, add_skill_line).

3 · ROC diagram

“Can my forecast tell events from non-events at all?”

The Receiver Operating Characteristic (ROC) diagram measures discrimination — the forecast’s ability to separate the days an event happens from the days it doesn’t, independent of calibration. The documentation describes it as plotting the true positive rate (sensitivity) against the false positive rate (1 − specificity) for different cut-off points of a parameter.

Axes and the no-skill line

Y-axis — Probability of Detection, POD (true positive rate / sensitivity) — The fraction of events correctly forecast as “yes.”

X-axis — Probability of False Detection, POFD (false positive rate / 1 − specificity) — The fraction of non-events incorrectly forecast as “yes.”

The curve — Each point is one decision cut-off (one probability threshold). Sweeping the threshold from strict to lenient traces the curve from the lower-left toward the upper-right.

How to read it

A curve that bows toward the upper-left is good: high detection with few false alarms. The diagonal is the no-skill line — POD equal to POFD means the forecast is no better than a random guess. The further the curve sits above that diagonal, the more skill; the area between the curve and the no-skill diagonal grows with discrimination skill.

Source note

The METplotpy ROC page itself documents the axes and configuration but does not formally define the no-skill diagonal or the area-under-the-curve summary; those are standard ROC interpretations included here as context. Treat them as background, not as quotes from the source page.

Figure 3. A ROC diagram. The dashed grey diagonal is the no-skill line; the blue curve bows toward the upper-left. Each marker is one probability threshold, and the shaded region between curve and diagonal grows with discrimination skill.

What feeds it

METplotpy can build a ROC curve from two MET line types: the probability contingency table (PCT), where each row’s threshold gives one point on the curve, or contingency table counts (CTC). The plotter exposes roc_pct and roc_ctc boolean flags to choose the input mode. The source notes the data may first need reformatting via the METdataio METreformat module before plotting.

4 · Taylor diagram

“How close is my model to the observed climate, all in one geometry?”

The Taylor diagram is the odd one out — it’s a polar plot for continuous fields, and it folds three statistics into a single point. It quantifies the correspondence between models and a “reference” (the observations) using the Pearson correlation coefficient, the centered RMSE, and the standard deviation, all at once.

The geometry

Angle — Pearson correlation coefficient — The azimuth around the arc encodes correlation: zero angle (along the x-axis) is correlation 1.0, and the angle opens up as correlation falls.

Radius — standard deviation — The distance from the origin is the (normalized) standard deviation of the model field, measuring how much variability it has compared with the observations.

Distance to the reference point — centered RMSE — The straight-line distance from a model point to the reference point equals the centered (pattern) root-mean-square difference — the quantity the Taylor geometry actually encodes (the source page calls it simply the RMSE). The reference sits on the x-axis at the observed standard deviation.

How to read it

The single reference point marks the observations. A model is good when its point lands close to the reference: similar standard deviation (same radius), high correlation (small angle), and therefore small centered RMSE (short distance). Three diagnostics, one glance.

Figure 4. A Taylor diagram drawn as a quarter circle. Radius is standard deviation, angle is correlation. The teal reference point on the x-axis is the observations; the dashed line from the blue model point to the reference is the centered RMSE.

What feeds it

The Taylor plotter reads MET’s continuous-statistics line type CNT from Point-Stat or Grid-Stat (columnar text). The three statistics it needs are FSTDEV (forecast standard deviation), OSTDEV (observation standard deviation), and PR_CORR (Pearson correlation). The taylor_voc option controls whether the plot shows only positive correlation values; when negative correlations are allowed the geometry extends past the vertical (corr = 0) axis into a half-circle, so an anticorrelated model lands to the left of it. A separate taylor_show_gamma toggle draws the standard-deviation arcs.

What feeds each diagram

The diagrams differ in what they ask, but they all start from MET line-type rows. This is the quick cross-reference: pick the diagram, find the line type, look for these columns.

Diagram	Question	MET line type	Key columns / statistics
Performance	Detection vs. false alarms (categorical)	CTS	PODY, FAR, CSI (SR = 1 − FAR)
Reliability	Calibration of probabilities	PCT	PSTD_CALIBRATION, PSTD_BASER, PSTD_NI
ROC	Discrimination across thresholds	PCT or CTC	POD vs. POFD per threshold (roc_pct / roc_ctc)
Taylor	Match to observed climate (continuous)	CNT	FSTDEV, OSTDEV, PR_CORR

Ecosystem link

These line types come from MET’s Point-Stat and Grid-Stat tools and land in .stat files. METplotpy reads them as columnar text; for some diagrams the rows are first reshaped by the METdataio reformat module before plotting. From here, browse the rest of METplotpy.

From .stat to plot

Each diagram is driven the same way: a default YAML the plotter loads automatically, plus a small custom YAML where you point at your data and name your output.

1. Produce the line type. Run Point-Stat or Grid-Stat so the .stat output contains the rows the diagram needs (CTS for performance, PCT for reliability/ROC, CTC for ROC, CNT for Taylor).
2. Get the data into columnar form. METplotpy reads text output in columnar format; for some diagrams the rows are reshaped by the METdataio METreformat module first.
3. Write a custom YAML. Override the diagram’s defaults file — for example set stat_input (the input data path) and plot_filename (where the PNG is written), plus any reference-line or styling toggles.
4. Run the plotter and read the picture. Generate the diagram, then read it with the good-vs-bad rules above: upper-right for performance, on-the-diagonal for reliability, bowed-up for ROC, near-the-reference for Taylor.

Tip

The defaults YAML (for example performance_diagram_defaults.yaml, reliability_defaults.yaml, roc_diagram_defaults.yaml, taylor_diagram_defaults.yaml) is loaded automatically — your custom file only needs the handful of values you want to change.

---

A derived, human-readable re-presentation — not official documentation. Sources: Performance diagram · Reliability diagram · ROC diagram · Taylor diagram