Checking Your Run

Checking Your Run#

Batch Job Scheduler Monitoring#

Once CESM has been submitted the status of the run on derecho can be followed through the qstat command. Qstat accesses the information in the Batch Job Scheduler to see the status of all jobs running on derecho. To simplify the list down the -u option can be specified for a particular user.

Result of running qstat with an active job running: 

qstat -u $USER

Output: 

                                                            Req'd  Req'd   Elap
Job ID          Username Queue    Jobname    SessID NDS TSK Memory Time  S Time
--------------- -------- -------- ---------- ------ --- --- ------ ----- - -----
4743615.desche* $USER    cpu      run.b.day*    --    6 768  1410g 12:00 Q   --  
4743616.desche* $USER    cpu      st_archiv*    --    1   1  235gb 00:20 H   --

Note, there may be a slight delay from submitting CESM until the jobs appear in the queue. Once the jobs have completed they will disappear from the qstat command.

Monitoring Files in the Run and Archive Directories#

As the CESM job runs it will update files in Build / Run Directory. Once the job has completed successfully the archiver will transfer history files to the Archive Directory. One way to keep track of the progress of the run is to monitor the files in these directories.

  • Check the files in the $RUNDIR as the model is running and once it is finished

  • Check the files in the $DOUT_S_ROOT after case.st_archive runs.

CaseStatus file#

All activities for the case are recorded in the CaseStatus file in the CASEROOT directory. By looking through the file the successful or otherwise outcome of each step of the run can be tracked.

Status of the case recorded in the `$CASEROOT` CaseStatus file: 

cd /glade/u/home/$USER/cases/CASE
more CaseStatus
Click here for example output 
2024-06-06 13:41:19: case.setup starting 
 ---------------------------------------------------
2024-06-06 13:41:22: case.setup success 
 ---------------------------------------------------
2024-06-06 13:42:17: case.build starting 
 ---------------------------------------------------
CESM version is release-cesm2.1.5
Processing externals description file : Externals.cfg (/glade/u/home/$USER/my_cesm_code)
Processing externals description file : Externals_CAM.cfg (/glade/u/home/$USER/my_cesm_code/components/cam)
Processing externals description file : Externals_CISM.cfg (/glade/u/home/$USER/my_cesm_code/components/cism)
Processing externals description file : Externals_CLM.cfg (/glade/u/home/$USER/my_cesm_code/components/clm)
Processing externals description file : Externals_POP.cfg (/glade/u/home/$USER/my_cesm_code/components/pop)
Checking local status of required & optional components: cam, chem_proc, carma, clubb, cosp2, cice, cime, cism, source_cism, clm, 
fates, mosart, pop, cvmix, marbl, rtm, ww3, 
    ./cime
        clean sandbox, on cime5.6.49
    ./components/cam
        clean sandbox, on cam_cesm2_1_rel_60
    ./components/cam/chem_proc
        clean sandbox, on tools/proc_atm/chem_proc/release_tags/chem_proc5_0_03_rel
    ./components/cam/src/physics/carma/base
        clean sandbox, on carma/release_tags/carma3_49_rel
    ./components/cam/src/physics/clubb
        clean sandbox, on vendor_clubb_r8099_n03
    ./components/cam/src/physics/cosp2/src
        clean sandbox, on v2.1.4cesm
    ./components/cice
        clean sandbox, on cice5_cesm2_1_1_20231220
    ./components/cism
        clean sandbox, on cism-release-cesm2.1.2_04
    ./components/cism/source_cism
        clean sandbox, on release-cism2.1.04
    ./components/clm
        clean sandbox, on release-clm5.0.37
    ./components/clm/src/fates
        clean sandbox, on sci.1.30.0_api.8.0.0
    ./components/mosart
        clean sandbox, on release-cesm2.0.04
    ./components/pop
        clean sandbox, on pop2_cesm2_1_rel_n15
    ./components/pop/externals/CVMix
        clean sandbox, on v0.93-beta
    ./components/pop/externals/MARBL
        clean sandbox, on cesm2.1-n00
    ./components/rtm
        clean sandbox, on release-cesm2.0.04
    ./components/ww3
        clean sandbox, on ww3_181001
2024-06-06 13:50:57: case.build success 
 ---------------------------------------------------
2024-06-06 13:53:31: case.submit starting 
 ---------------------------------------------------
2024-06-06 13:53:46: case.submit success case.run:4743615.desched1, case.st_archive:4743616.desched1
 ---------------------------------------------------
Contents