Adding External Forward Operators to an Observation

External, or “precomputed” forward operators are forward operators that are not calculated during filter, DART’s assimilation program. They are calculated externally, written to an observation sequence file, and read in by filter.

This example shows how to add an external forward operator to an observation. We’ll be using observations for the Lorenz 96 tracer model, which is a toy model used to test assimilation algorithms. We are going to create some RAW_TRACER_CONCENTRATION forward operators.

Import the obs_sequence module, and numpy

import os
import pydartdiags.obs_sequence.obs_sequence as obsq
import numpy as np

Read in the observation sequence file. In this example we’ll use the obs_seq.out.tracer file that comes with the pyDARTdiags package. This file only has two observations.

data_dir = os.path.join(os.getcwd(), "../..", "data")
data_file = os.path.join(data_dir, "obs_seq.out.tracer")
obs_seq = obsq.ObsSequence(data_file)

obs_seq.df

	obs_num	observation	truth	Quality_Control	linked_list	location	type	metadata	external_FO	seconds	days	time	obs_err_var
0	1	2.072563	1.648863	0.0	-1 2 -1	0.097632	RAW_TRACER_CONCENTRATION	[]	[]	0	0	2000-01-01	10.01
1	2	0.509401	0.715153	0.0	1 -1 -1	0.111607	RAW_STATE_VARIABLE	[]	[]	0	0	2000-01-01	9.99

The external forward operator is stored in the ‘external_FO’ column. Currently, the ‘external_FO’ column is empty.

obs_seq.df.iloc[0]['external_FO']

[]

Let’s add a forward operator to the RAW_TRACER_CONCENTRATION observation. For this example, we’re going to use random numbers to create our ensemble of forward operators. For a scientific experiment, you would use appropriate forward operator results from your model. We want 80 ensemble members, so we’ll create 80 random numbers, and we want tracer concentrations, so we’ll force the numbers to be positive.

n = 80
random_numbers = np.abs(np.random.normal(loc=0.75, scale=0.1, size=n)).tolist()

DART observation sequence files have a specific format. DART expects the list of forward operators to have ‘external_FO’ followed by the number of ensemble members, followed by the ‘key’ for the forward operator. The ‘key’ is not used by DART, so we will just put 1 for the value.

random_numbers.insert(0, f'external_FO {n} 1')

Now we can add the forward operator to the ‘external_FO’ column for row 0

obs_seq.df.at[0,'external_FO'].extend(random_numbers)

Now the ‘external_FO’ column has the forward operator.

obs_seq.df.iloc[0]['external_FO']

['external_FO 80 1', 0.770747025394175, 0.49178108876779114, 0.8236621506153435, 0.7034172980361586, 0.8015323665629251, 0.7450855870534177, 0.7681337818725311, 0.8658812257642442, 0.6387229205995592, 0.9313514137857292, 0.7635087820114896, 0.8383798983594503, 0.7182785425533248, 0.7045831502170351, 0.8378532087462319, 0.6965168075365662, 0.8524049033871417, 0.7452525108796949, 0.8042818831589562, 0.8468448150118113, 0.7658618811961777, 0.8147736035049766, 0.686262219309751, 0.8451035098359432, 0.7525205683082493, 0.8577114420780927, 0.821278301344055, 0.6022333812378892, 0.8510988990287784, 0.8186495930557087, 0.6800523686935704, 0.9292488367781481, 0.8531835310310032, 0.6699230080415455, 0.6494770158786266, 0.6740092433000364, 0.6631414907538048, 0.877090456199183, 0.8101703406234578, 0.9018970944714588, 0.6605360907326261, 0.864998901256028, 0.5502252176602219, 0.6679768301653666, 0.925986624505758, 0.7116383050919871, 0.7574400860493377, 0.7200045265737023, 0.7107973758390765, 0.7063556087110796, 0.8380542022816366, 0.7679857360659142, 0.5971444848533655, 0.5790437513851419, 0.7705449941116164, 0.795361832058267, 0.8075455491409186, 0.6155447472141063, 0.79447200088488, 0.7670727341924712, 0.7277681117281534, 0.7404972137605033, 0.7820529619266732, 0.7554782495321066, 0.7196103920347728, 0.6242272805836762, 0.8119783094044258, 0.8092134262049568, 0.8043439802122361, 0.7691077659464004, 0.8356623822069393, 0.6769681617425578, 0.720977564588707, 0.7705501003456492, 0.8046932778805609, 0.6353518318605327, 0.7795282066100099, 0.5256158673518339, 0.7662003105678896, 0.7948541324059268]

Let’s write the observation sequence file with the new forward operator.

obs_seq.write_obs_seq('obs_seq.out.tracer_with_external_FO')