FATES-sp simulations at NEON sites

FATES-sp simulations at NEON sites#

Exercise: Use the CESM workflow to run a single point FATES-sp case at a NEON site

Create and run a case called BONA.FATESsp using the:

compset I1PtClm60FatesRsGs, the
resolution CLM_USRDAT, and that
applies USERMODS to set up a NEON site.

The run_tower script we’ve been using is very useful, but it simplifies a number of steps that users may want to take control of in their workflow. We’ll introduce how you can control aspects of your case setup and configuration here. For example, you can use the CESM workflow to configure your case to run FATES case at a supported tower site. We’ll use material from the basic tutorial and customizing your case for this example

Background#

For an SP case, we want to run NEON sites for a bit to let soil water and temperatures the equilibrate before comparing to observations. Luckily, this only takes a few years (in most cases).

A usermod directory (“user modifications” directory) is a bundle of case-specific settings that CLM/CIME can apply when you create or clone a case. It’s a convenient way to start from a known configuration rather than manually setting every option afterward. The usermods may include files such as CLM namelist settings (user_nl_clm), settings for other model components, and XML configuration changes.

You pass the bundle to CIME when creating a case, for example:

./create_newcase ... --user-mods-dir clm/NEON/FATES/BONA

Have a look at the standard usermods we provide in CLM6 in the cime_config directory of your CTSM checkout:

ls /glade/u/home/$USER/code/my_clm_code/cime_config/usermods_dirs/clm

Exercise

Move into the NEON usermods directory:

cd /glade/u/home/$USER/code/my_clm_code/cime_config/usermods_dirs/clm/NEON

List its contents:

ls -1

You should see several directories, including:

ABBY
BARR
BART
BLAN
BONA
CLBJ
CPER
DCFS
defaults
DEJU
DELA
DSNY
FATES
GRSM
...

You’ll notice that FATES has it’s own directory. Have a look inside.

ls FATES

The default directory in NEON/FATES contains shell commands and user_nl_* setting that are shared across all sites. Look and see.

ls -1 FATES/defaults

expected output includes:

shell_commands
user_nl_clm
user_nl_cpl
user_nl_datm
user_nl_datm_streams

You can look to see what default settings you NEON cases will inherit.

Each NEON site also includes custom XML changes. Look and see:

ls -1 FATES/BONA

expected output includes:

include_user_mods
shell_commands

Check Your Understanding

How are NEON site-level details passed to CLM?

Hint: look at the shell_commands in the CLM usermods_dirs for NEON

Get Started#

Let’s create a FATES sp case at a NEON site.

Click here for hints

Is my workspace set up?

Did you already create a cases directory where you will put all new cases

cd /glade/u/home/$USER
mkdir cases

Did you create a new case?

You can create a new case with the command:

./create_newcase --case /glade/u/home/$USER/cases/CASE --res RES --compset COMPSET --user-mods-dir USERMODS --run-unsupported 

Make sure you use the correct information for the CASE, RES, COMPSET, and USERMODS inputs.

How do you customize your case to run a FATES SP experiment?

Specifically you’ll want to make namelist changes:

set up a FATES-SP case.
modify history file output for an SP case.

And XML changes to:

Start from a ‘cold start’ (as we don’t have initial conditions)
Run a quick spinup to initialize the soil moisture state (~4 years should be enough?)
Run for the full observational record

Did you set up the case?

You can invoke case set up as follows:

./case.setup

How do I compile?

You can compile with the command:

qcmd -- ./case.build

How do I check my job status?

You can check job status using the command:

qstat -u $USER

Click here for the solution

Create a new case BONA.FATESsp with the command:

cd /glade/u/home/$USER/code/my_clm_code/cime/scripts

Change the 4-character NEON site below.

export neon_site="BONA"

Then create a new case.

./create_newcase --case /glade/u/home/$USER/cases/${neon_site}.FATESsp \
    --res CLM_USRDAT \
    --compset I1PtClm60FatesRsGs \
    --run-unsupported \
    --user-mods-dir clm/NEON/FATES/${neon_site} \
    --handle-preexisting-dirs r

Case setup:

cd /glade/u/home/$USER/cases/BR-Sa3.test0.ad
./case.setup

Case customization: user_nl_clm changes

echo "use_fates_sp = .true." >> user_nl_clm
echo "hist_fincl2 = 'FCEV','FCTR','FGEV','FIRA','FSA','FSH','FATES_GPP','FATES_GPP_PF','H2OSOI','SNOW_DEPTH','TBOT','TSOI'" >> user_nl_clm

XLM changes First look and see what default settings are for:

Running from a cold start (without initial conditions)
What years of DATM data do we have?
How are long we running the case?

./xmlquery CLM_FORCE_COLDSTART
./xmlquery DATM_YR_START DATM_YR_ALIGN DATM_YR_END
./xmlquery STOP_OPTION STOP_N 

Expected results are:

	CLM_FORCE_COLDSTART: off

	DATM_YR_START: 2018
	DATM_YR_ALIGN: 2018
	DATM_YR_END: 2021

	STOP_OPTION: ndays
	STOP_N: 5

Most NEON sites we have at least 4 complete years of data to work with (2018-2021). Now we’ll change this to:

Force a cold start
Cycle over the full 4 years of DATM data available two times
Start the simulation in 2014, then we’ll have 4 years of spinup and 4 years to compare with the observed data.

./xmlchange CLM_FORCE_COLDSTART=on
./xmlchange STOP_OPTION=nyears
./xmlchange STOP_N=8  
./xmlchange RUN_STARTDATE=2014-01-01 

Now we can build our case:

qcmd -- ./case.build 

And submit:

./case.submit

Check the status:

qstat -u $USER

When the run is completed, look into the archive directory for: BONA.FATESsp.

Check that your archive directory on derecho (The path will be different on other machines):

cd /glade/derecho/scratch/$USER/archive/BONA.FATESsp
ls 

Background

For an SP case, we want to run NEON sites for a bit to let soil water and temperatures the equilibrate before comparing to observations. Luckily, this only takes a few years (in most cases).

What years is the model set up to run over by default? We can look by using ./xmlquery

./xmlquery DATM_YR_START DATM_YR_ALIGN DATM_YR_END
./xmlquery STOP_OPTION STOP_N CLM_FORCE_COLDSTART

Most NEON sites we have 4 complete years of data to work with (2018-2021).

If this is true for your site, we can use xml commands to modify our case

In general, we want to run for twice as many years as we have NEON observations.

CLM will just cycle over the meterological data that NEON provides.
- e.g. data from 2018-2021 will be used for 2014-2017 and then again for 2018-2021
This will allow the second cycle of our our simulation to end with the same dates over which we have NEON observations, making the comaprision with observations easier.

1.2.2 xml changes to modify env_run options#

xml changes are one way you can control your case to customize the simulation

We won’t cover this here, but other tutorials cover more about xml changes.

NOTE RUN_STARTDATE is totally arbitrary. It’s just the model date when the simulation starts.

xml variables like DATM_YR_START and DATM_YR_ALIGN determine the data that’s being read into for your simulation.
CLM_FORCE_COLDSTART makes it so we don’t look for an initial conditions file

./xmlchange RUN_STARTDATE=2014-01-01 
./xmlchange STOP_OPTION=nyears
./xmlchange STOP_N=8  
./xmlchange DATM_YR_END=2021  #this is the last complete year of input data
./xmlchange CLM_FORCE_COLDSTART=on

1.2.3 Modify user_nl_clm#

We also won’t discuss namelist modificaiton here, but they are another way to customize your case.

Specifically these namelist changes:

set up a FATES-SP case.
modifies history file output for an SP case.

#user_nl_clm changes, if sp is true, thne nocomp and fixed_biogeog are too
echo "use_fates_sp = .true." >> user_nl_clm
echo "hist_fincl2 = 'FCEV','FCTR','FGEV','FIRA','FSA','FSH','FATES_GPP','FATES_GPP_PF','H2OSOI','SNOW_DEPTH','TBOT','TSOI'" >> user_nl_clm

1.3 Setup and build your case#

This step configures the model so that it can then compile

1.3.1 case.setup#

The ./case.setup script:

configures the model
creates files to modify input data and run options

./case.setup

./preview_namelists

#### Chekc that you lnd_in file looks OK (e.g. use_fates_sp = .true., fsurdat = *16pfts*, etc)

cat CaseDocs/lnd_in | head -60

1.3.2 Build the executable

This step compiles the model

It also takes a long time, so be patient.

The ./case.build script:

Checks input data
Creates a build/run directory with model executable and namelists

./case.build

This takes some time, and will throw a bunch of errors… don’t worry, just give it time

When the build completes successfully you’ll be notified: MODEL BUILD HAS FINISHED SUCCESSFULLY

You can read on, but before executing any code blocks in the notebook wait for the model to build. This can take a while.

1.4 Submit your case

Now you’re ready to submit the case!

This step submits the model simulation

You’ll also be downloading all the meterological data from NEON for your site, this also takes a little time and prints lots of information to the screen

./case.submit

When you submit a job, you will see confirmation that it successfully submitted:

Congratulations!#

You’ve created and submitted a single point CLM-FATES run from scratch.#

Next, you will probably want to check on the status of your jobs.

TIP: This is dependent on the scheduler that you’re using.

Derecho uses PBS where status is checked with qstat -u $USER
This is also enabled in the cloud for you, try it in the code block below

If you want to stop the simulation, you can do so with qdel here (or on Derecho).

Find your Job ID after typing qstat
Type qdel {Job ID}

#qstat -u $USER

Once your jobs are complete (or show the ‘C’ state under the ‘Use’ column, which means complete), we can check the CaseStatus file to ensure there were no errors and it completed successfully. To do this, we’ll ‘tail’ the end of the CaseStatus file:

tail  ~/scratch/NEON_cases/${neon_site}_FATESsp_test/CaseStatus

You should see several lines, with the middle one saying ‘case.run success’.

Before that you’ll see notifications about xml changes, case.setup, and case.submit, and case.run

Congratulations!

You’ve created a CTSM case for the NEON tower you selected.

We’ll build on these basics in additional tutorials to customize your simulations.

2. Locate model history files#

Your simulation will likely take some time to complete. The information provided next shows where the model output will be located while the model is running and once the simulation is complete. We also provide files from a simulation that is already complete so that you can do the next exercises before your simulation completes.

When your simulation is running history files go to your scratch directory:

~/scratch/NEON_cases/{CASE}

Within this directory you can find /run and /bld subdirectories.

When the simulation is complete, a short-term archive directory is created, and history files are moved here:

~/scratch/NEON_cases/archive/{CASE}/lnd/hist/

Note that files necessary to continue the run are left in the run directory: ~/scratch/NEON_cases/{CASE}/run

2.1 Run directory#

What’s in your run directory?

ls ~/scratch/NEON_cases/${neon_site}_FATESsp_test/run

Do you see any log or history files?

log files look like lnd.log*
history files look like ${neon_site}.transient.clm2.h0.*.nc

You can keep running the cell above until you see log and history files, then the model is running.

2.2 Archive directory#

What’s in your archive directory?

ls ~/scratch/NEON_cases/archive/${neon_site}_FATESsp_test/lnd/hist | head -5

You can drop the | head -5 part of the cell above if you want to see ALL the files that have been archived.

If you don’t see any history files your simulation is likely still running or in the queue (check using squeue or qstat). Check again before you leave today to see if your simulation completed and if the files were transferred to archive. Even if your run isn’t finished, you can move on in this tutorial.

If you’d like to visualize these results you can go back to tutorial Day0c and modify to point to these results from the neon_site you just ran.

3. Create a Clone#

This step is optional, but provides helpful information

Creating and building a new case is slow.

Cloning cases can speed this up!

3.1 Create a clone

Clones use the same resolution, compset, and output root.
Clones can also share the same executable (build), which can save time!

WARNING: If you’re making code modifications be careful using clones that share the same build. The example below is a good way to run the same model configuration at different NEON sites.

# Move back to your source code directory
cd $ctsm_root/cime/scripts

# Change the 4-character new NEON site you want to run.
export neon_site2="TEAK"

# then create a cloned case.
./create_clone --case ~/scratch/NEON_cases/${neon_site2}_FATESsp_test \
               --clone ~/scratch/NEON_cases/${neon_site}_FATESsp_test \
               --user-mods-dirs $ctsm_root/cime_config/usermods_dirs/clm/NEON/FATES/${neon_site2} \
               --keepexe

As before, the code above doesn’t always render properly online. Here’s what we’re doing:

./create_clone --case ~/scratch/NEON_cases/${neon_site2}_FATESsp_test \
               --clone ~/scratch/NEON_cases/${neon_site}_FATESsp_test \
               --user-mods-dirs $ctsm_root/cime_config/usermods_dirs/clm/NEON/FATES/${neon_site2} \
               --keepexe

./create_clone#

NOTE: There is a lot of information that goes into creating a clone.

You can learn more about the options by typing ./create_clone –help on the the command line or in a new code cell.

We’ll briefly go over some of the highlights here.

Required arguments to create a clone#

There are 2 required arguments needed to create a clone. These include:

--case, this is the path and name of the new case your cloning.
--clone, this is the path and name of the existing case you want to clone.

Additional information we provided here were:#

--user-mods-dirs as before, this sets up the configuration of the case with a user modification directory that defines the location of the new site we’re running, but it requires the full path to the user-mods-dir you want to use.
--keepexe Point to original build from the case being cloned.

We’ll also create a non-FATES case clone for comparison

cd $ctsm_root/tools/site_and_regional
./run_tower --neon-sites $neon_site2  \
    --output-root ~/scratch/NEON_cases \
    --setup-only \
    --overwrite

3.2 Customize the clone?

Because you created a clone and point to an existing build we can skip the case.setup and case.build steps

3.2.1 Move to your case directory and compare env_run#

cd ~/scratch/NEON_cases/${neon_site2}_FATESsp_test
diff env_run.xml ../${neon_site2}.transient/env_run.xml

3.2.2 Make xml changes#

In the example above the teak data record only starts in 2019, meaning we have 3 years of meterological data to cycle through

./xmlchange RUN_STARTDATE=2016-01-01 
./xmlchange STOP_OPTION=nyears
./xmlchange STOP_N=6  
./xmlchange DATM_YR_END=2021  #this is the last complete year of input data
./xmlchange CLM_FORCE_COLDSTART=on

# Check that chagnes look as expected from the other FATESsp case
diff env_run.xml ../${neon_site}_FATESsp_test/env_run.xml

3.2.3 Copy user_nl_clm#

We can just copy the user_nl_clm into our cloned case

cp ../${neon_site}_FATESsp_test/user_nl_clm .

3.3 Setup and Submit the case

Because you created a clone and point to an existing build we can skip the case.build step

As before, you’ll be downloading all the meterological data, which takes a little time and prints lots of information to the screen

./case.setup
./case.submit