METdbLoad

MET writes its verification results as plain-text ASCII files. That format is easy for MET to produce but awkward to query, filter, and plot at scale. METdbLoad is the loader that reads those files and inserts them into a MySQL-compatible database, so that aggregation and plotting tools — METviewer and METexpress — can pull statistics back out with a query instead of a file scan.

What METdbLoad is

METdbLoad is a component of METdataio. Its job is narrow and important: take the ASCII output that the Model Evaluation Tools (MET) write to disk and insert it into a relational database — a structured store you query with SQL rather than read line by line.

The documentation describes it as a module that “provides support for inserting MET output data (as .stat files) into a relational database (mysql, mariadb, or aurora).” In other words, METdbLoad is the bridge between MET’s verification step and the analysis tools that consume verification results in bulk.

Why a database at all?

A single MET run can emit thousands of statistics lines across many models, valid times, forecast leads, vertical levels, variables, and regions. Answering a question like “show me RMSE for model A versus model B at 24-hour lead over the last month” means filtering across all of those dimensions. A database indexes that metadata so the filter is fast — that is exactly what METdbLoad sets up.

METdbLoad targets MySQL-compatible databases: MySQL, MariaDB, or Amazon Aurora. The target database must use the mv_ name prefix (for example mv_met_data) and must be built from the METviewer schema file mv_mysql.sql. That shared schema is what lets METviewer and METdbLoad agree on table shapes.

What it loads

METdbLoad reads MET’s ASCII output. Only files with the .stat extension are loaded — “only data files with the .stat extension will be loaded.” Within those files MET packs many different line types, each a distinct row format carrying a particular family of statistics. Which families are ingested is controlled by load flags in the specification.

Output families

STAT line types — The general statistics output. A single .stat file interleaves many line types — continuous, categorical, ensemble, and more. Loading them is governed by the <load_stat> flag.

MPR — Matched Pairs — The individual forecast/observation pairs that the summary statistics are derived from. These are voluminous, so they have their own flag, <load_mpr>.

ORANK — Observation Rank — Ensemble rank information used for rank histograms and related ensemble diagnostics. Loaded via <load_orank>.

MODE — object output — Output from MET’s MODE (Method for Object-based Diagnostic Evaluation) tool, which identifies and compares spatial objects in forecast and observed fields. Loaded via <load_mode>.

MTD — MODE Time Domain — Output from MODE-TD, the space-time extension of MODE that tracks objects across time. Loaded via <load_mtd>.

These two levels work together: the <load_*> flags switch whole families on or off, and within the STAT family the individual line types can be restricted with a <line_type> block (more below); if that block is omitted, “all linetypes will be loaded.” A few names — notably MPR and ORANK — are used at both levels: they each have a dedicated load flag and name a STAT line type. Line-type names seen in the documentation’s own examples include CNT, ORANK, SEEPS_MPR, ME, ECNT, VL1L2, and SAL1L2.

A sampling of names that appear in the load documentation. Most are true line types; ME is a statistic within the CNT line type, shown here because it appears as a selectable filter value in the load examples. The full set of MET line types is defined by MET itself, not by METdbLoad.

Line type	Family	Roughly what it carries
CNT	Continuous	Continuous-statistics summary (e.g. error and correlation measures)
ME	Continuous (CNT statistic)	Mean error — a statistic within the CNT line type, not a line type itself; appears in the docs as a selectable filter value
VL1L2	Vector partial sums	Vector wind partial-sum components
SAL1L2	Scalar partial sums	Scalar anomaly partial-sum components
ECNT	Ensemble	Ensemble continuous statistics
ORANK	Ensemble	Observation rank within the ensemble
SEEPS_MPR	Matched pair	SEEPS per-pair output

Stay honest about scope

The MET line-type catalog is large and is owned by MET’s own documentation. The families above are the ones the METdbLoad page itself names or controls with a load flag. For the authoritative meaning of any individual statistic, consult the MET User’s Guide rather than this loader page.

The load pipeline

At a high level METdbLoad sits in the middle of a flow: MET writes ASCII files; an XML load specification tells the loader what to read and where to put it; the loader parses the files and inserts rows; downstream tools query the result.

Figure 1. MET ASCII output and an XML load specification feed the METdbLoad parser, which writes header tables and per-line-type data tables into an mv_ database that METviewer and METexpress then query.

Ecosystem link

The files METdbLoad ingests are produced upstream by MET. The database it populates is read downstream by METviewer and METexpress for aggregation and plotting. If you only need to reshape ASCII output (not load a database), see Reformatting instead.

The XML load specification

METdbLoad is driven entirely by a single XML load specification file. This one document answers three questions: which database to connect to, which files to read, and which options to apply. You pass its path to the loader and nothing else is required.

Conceptually the specification has three regions, and the elements must appear in the order the XML schema defines:

Connection

Where the rows go. The hostname and port, the database name, and the credentials used to connect.

What to load

Either a folder template with variable placeholders, or an explicit list of files — plus an optional line-type filter.

Options

Verbosity, batch size, which families to load, header-duplicate checks, index handling, and metadata to record with the load.

This file holds a password

The specification contains the database credentials in plain text. The documentation warns: “Do not save this XML specification file where it can be read by anyone who should not have access to this information, as this file contains the database password.” Protect it with filesystem permissions.

A worked example ships with the source tree at METdataio/METdbLoad/test/Examples/example_load_specification.xml — the recommended starting point for a new load.

In skeleton form, the three regions nest like this (illustrative values; see the example file for a complete, runnable specification):

<load_spec>
  <connection>
    <host>localhost:3306</host>
    <database>mv_met_data</database>
    <user>myuser</user>
    <password>********</password>
  </connection>

  <load_stat>true</load_stat>
  <load_mode>false</load_mode>
  <verbose>false</verbose>
  <insert_size>1</insert_size>

  <folder_tmpl>/data/met_out/{config}/metprd</folder_tmpl>
  <load_val>
    <field name="config">
      <val>gfs</val>
    </field>
  </load_val>
</load_spec>

Specification reference

The elements below are those the load documentation names. “Mandatory” / “optional” and the stated limits reflect the schema as described on the source page.

Connection

Element	Required	Purpose
<host>	yes	Database hostname including port (3–67 characters)
<database>	yes	Database name, must use the mv_ prefix (3–124 characters)
<user>	yes	Username (3–32 characters)
<password>	yes	Password (3–30 characters)
<management_system>	optional	Engine: mysql, mariadb, or aurora
<local_infile>	optional	True/False — enable local-file loading

Load flags

Element	Required	Controls
<verbose>	yes	True/False — verbose logging
<insert_size>	yes	Positive integer — rows per insert batch
<load_stat>	optional	Load STAT line types
<load_mode>	optional	Load MODE object output
<load_mtd>	optional	Load MODE-TD (MTD) output
<load_mpr>	optional	Load matched-pair (MPR) lines
<load_orank>	optional	Load observation-rank (ORANK) lines
<stat_header_db_check>	optional	Check for duplicate STAT headers in the DB
<mode_header_db_check>	optional	Check for duplicate MODE headers in the DB
<mtd_header_db_check>	optional	Check for duplicate MTD headers in the DB
<drop_indexes>	optional	Drop indexes before loading (faster bulk insert)
<apply_indexes>	optional	Re-apply indexes after loading
<force_dup_file>	optional	Force loading of a file already recorded as loaded

Header DB checks trade time for safety

The header-duplicate checks (stat_header_db_check, mode_header_db_check, mtd_header_db_check) protect against inserting the same header twice. The documentation cautions that enabling them “could significantly increase load time.” For a fresh load into an empty database you can usually leave them off; for incremental loads into a shared database they are the safer choice.

Data location and selection

Element	Required	Purpose
<folder_tmpl>	one of	Path template with {placeholder} variables
<load_files> / <file>	one of	Explicit file list (max 200 entries)
<load_val> / <field>	with template	Defines template variables (max 20 fields)
<val>	within field	Static subdirectory value (max 100 per field)
<date_list>	optional	Datetime range: start, end, inc, format (max 5)
<line_type> / <val>	optional	Restrict to named line types (max 800; all loaded if omitted)

Metadata recorded with the load

Element	Required	Purpose
<group>	optional	Database grouping label (1–300 characters)
<description>	optional	Database description (1–300 characters)
<load_xml>	optional	True/False — store the XML commands in the database
<load_note>	optional	Descriptive note saved with the load (in instance_info)

Templating with folder_tmpl

Real verification output is rarely a flat directory. It is usually a deep tree organized by configuration, initialization time, ensemble member, valid time, and so on. The <folder_tmpl> mechanism lets you describe that tree once, with {placeholder} variables, and then enumerate the values each placeholder may take.

The documentation gives a template of this shape:

<folder_tmpl>/.../met_out/{config}/{fcst_init}/{mem}/{valid_times}/metprd/{met_out}/</folder_tmpl>

Each {variable} in the template needs a matching <field name="variable"> inside <load_val>. A field is filled in one of two ways:

• With one or more <val> elements, for fixed subdirectory names (for example a set of configuration names or ensemble members).
• With a <date_list name="..."/> reference, for date- or time-stamped directories generated from a start/end/inc range.

METdbLoad expands the template against every combination of field values, walks the resulting folders, and loads the .stat files it finds. The alternative — listing each file by hand under <load_files> — is fine for small or irregular sets, but the template is what scales to a large experiment.

The database schema

The destination database is built from the METviewer schema file mv_mysql.sql. At a high level it follows a header + line-data design that mirrors how MET’s ASCII files are structured.

Header tables — Every MET output line begins with a block of header metadata — the descriptors that identify a result, such as the model, the initialization and valid times, the forecast lead, the level, the variable, and the verification region/mask. METdbLoad factors this shared metadata into header tables (the documentation names stat_header, mode_header, and mtd_header). Many data rows reference one header row, so the metadata is stored once and pointed to.

Per-line-type data tables — The numeric payload of each line type lives in its own table, keyed back to a header row. Because each MET line type has a different set of columns, each gets its own table — for example a continuous-statistics table, an observation-rank table, and a matched-pair table. This is why MPR and ORANK have dedicated load flags: they fan out into large, dedicated tables.

File-tracking and instance tables — The loader records which files have been ingested so that re-running a load does not duplicate data — this is what the <force_dup_file> flag overrides. Load-time metadata such as <load_note> is stored in the instance_info table.

On exact column names

The header metadata keys above (model, init/valid time, lead, level, variable, region/mask) describe the kind of information stored in the header tables. The load documentation page does not enumerate the exact column names, so for the precise schema consult mv_mysql.sql or the METviewer documentation directly.

The header-plus-line-data shape is what makes the database efficient to query: tools filter on the small, indexed header tables to find the rows of interest, then join out to the larger data tables only for the matches.

Running a load

A load is run from the loader’s ush/ directory with the XML specification as the only required argument.

1. Create the database. The target database must already exist, with insert, delete, update, and index privileges. Create it with, for example, mysql -u[user] -p[pass] -e'create database [db_name];' — using an mv_-prefixed name.

2. Build the schema. Load the METviewer schema into the new database: mysql -u[user] -p[pass] [db_name] < sql/mv_mysql.sql. This creates the header, line-data, and supporting tables.

3. Write the load specification. Copy example_load_specification.xml and edit the connection block, the <folder_tmpl> or <load_files> data location, the load flags for the families you want, and any metadata.

4. Run the loader. From [install]/METdataio/METdbLoad/ush/ run python met_db_load.py /path-to/load_specification.xml. To keep a record, redirect output: python met_db_load.py spec.xml &> log/run.log &.

5. Verify the result. Confirm rows landed by querying the database, or open it in METviewer/METexpress. For a re-load of already-recorded files, use <force_dup_file>; to (re)build only indexes, use the -index option.

Useful command-line options

-h/--help prints usage. -index processes the index step only and does not load data — handy after a bulk load done with <drop_indexes> enabled. An optional tmpdir argument points the loader at a custom directory for its temporary files.

Where the data goes

Once loaded, the database is a queryable home for MET’s verification results. The two primary consumers are METviewer — the interactive plotting and aggregation web application — and METexpress, which offers guided, pre-built verification plots over the same schema. Both read directly from the tables METdbLoad populated, which is exactly why the database must conform to the shared mv_mysql.sql schema.

Ecosystem link

METdbLoad is the on-ramp into the database half of the ecosystem. After loading, head to METviewer to aggregate and plot, or stay in METdataio to explore the companion reformatting tools that reshape ASCII output without a database.

---

A derived, human-readable re-presentation — not official documentation. Sources: METdataio User’s Guide — Loading Data into a Database