Checking Your Run#

Batch Job Scheduler Monitoring#

Once CESM has been submitted the status of the run on Cheyenne can be followed through the qstat command. Qstat accesses the information in the Batch Job Scheduler to see the status of all jobs running on Cheyenne. To simplify the list down the -u option can be specified for a particular user.

Result of running qstat with an active job running: 

qstat -u $USER

Output: 

Job ID          Username Queue    Jobname    SessID NDS TSK Memory Time  S Time
--------------- -------- -------- ---------- ------ --- --- ------ ----- - -----
704942.chadmin* $USER regular  run.b.day*   8284  16 576    --  12:00 R 00:00
704945.chadmin* $USER regular  st_archiv*    --    1   1    --  00:20 H   --

Note, there may be a slight delay from submitting CESM until the jobs appear in the queue. Once the jobs have completed they will disappear from the qstat command.

Monitoring Files in the Run and Archive Directories#

As the CESM job runs it will update files in Build / Run Directory. Once the job has completed successfully the archiver will transfer history files to the Archive Directory. One way to keep track of the progress of the run is to monitor the files in these directories.

  • Check the files in the $RUNDIR as the model is running and once it is finished

  • Check the files in the $DOUT_S_ROOT after case.st_archive runs.

CaseStatus file#

All activities for the case are recorded in the CaseStatus file in the CASEROOT directory. By looking through the file the successful or otherwise outcome of each step of the run can be tracked.

Status of the case recorded in the `$CASEROOT` CaseStatus file: 

cd /glade/work/$USER/cases/CASE
more CaseStatus
Click here for example output 
2023-06-04 21:02:40: xmlchange success <command> ./xmlchange JOB_QUEUE=regular  </command>
 ---------------------------------------------------
2023-06-04 21:02:42: case.setup starting 
 ---------------------------------------------------
2023-06-04 21:02:44: case.setup success 
 ---------------------------------------------------
2023-06-04 21:03:23: case.build starting 
 ---------------------------------------------------
CESM version is cesm2.1_tutorial2022
Processing externals description file : Externals.cfg
Processing externals description file : Externals_CLM.cfg
Processing externals description file : Externals_POP.cfg
Processing externals description file : Externals_CISM.cfg
Processing externals description file : Externals_CAM.cfg
Checking status of externals: clm, fates, ptclm, mosart, ww3, cime, cice, pop, cvmix, marbl, cism, source_cism, rtm, cam, clubb, carma, cosp2, chem_proc, 
    ./cime
        clean sandbox, on 0a20d3d82667c30c23de65033bd4df33e46e1a97
    ./components/cam
        clean sandbox, on cam_cesm2_1_rel_55
    ./components/cam/chem_proc
        clean sandbox, on tools/proc_atm/chem_proc/release_tags/chem_proc5_0_03_rel
    ./components/cam/src/physics/carma/base
        clean sandbox, on carma/release_tags/carma3_49_rel
    ./components/cam/src/physics/clubb
        clean sandbox, on vendor_clubb_r8099_n03
     ./components/cam/src/physics/cosp2/src
        clean sandbox, on CFMIP/COSPv2.0/tags/v2.1.4cesm/src
    ./components/cice
        clean sandbox, on cice5_cesm2_1_1_20190321
    ./components/cism
        clean sandbox, on cism-release-cesm2.1.2_02
    ./components/cism/source_cism
        clean sandbox, on release-cism2.1.03
    ./components/clm
        clean sandbox, on release-clm5.0.35
    ./components/clm/src/fates
        clean sandbox, on sci.1.30.0_api.8.0.0
    ./components/clm/tools/PTCLM
        clean sandbox, on PTCLM2_20200902
    ./components/mosart
        clean sandbox, on release-cesm2.0.04
    ./components/pop
        clean sandbox, on pop2_cesm2_1_rel_n14
    ./components/pop/externals/CVMix
        clean sandbox, on v0.93-beta
    ./components/pop/externals/MARBL
        clean sandbox, on cesm2.1-n00
    ./components/rtm
        clean sandbox, on release-cesm2.0.04
    ./components/ww3
        clean sandbox, on ww3_181001
2023-06-04 21:21:29: case.build success 
2023-06-04 23:37:33: case.submit starting 
 ---------------------------------------------------
2023-06-04 23:38:04: case.submit success case.run:704942.chadmin1.ib0.cheyenne.ucar.edu, case.st_archive:704945.chadmin1.ib0.cheyenne.ucar.edu
 ---------------------------------------------------
2023-06-04 23:38:11: case.run starting 
 ---------------------------------------------------
2023-06-04 23:38:18: model execution starting 
 ---------------------------------------------------
2023-06-04 23:40:56: model execution success 
 ---------------------------------------------------
2023-06-04 23:40:56: case.run success 
 ---------------------------------------------------
2023-06-04 23:41:00: st_archive starting 
 ---------------------------------------------------
2023-06-04 23:41:09: st_archive success 
 ---------------------------------------------------