6. Testing¶
create_test is the tool that is used to test both CIME and CIME-driven models. It can be used as an easy way to run a single basic test or an entire suite of tests. create_test runs a test suite in parallel for improved performance. It is the driver behind the automated nightly testing of cime-driven models.
More information about CIME testing can be found on CIME: Testing.
6.1. Testname syntax¶
Tests must be named with the following forms, [ ]=optional:
TESTTYPE[_MODIFIERS].GRID.COMPSET[.MACHINE_COMPILER][.GROUP-TESTMODS]
where:
TESTTYPE
defines the general type of test, e.g. SMS. Following is the list of tests that are supported by UFS Medium-Range Weather Application.- SMS: Smoke startup test (default 5 days).
- Do a 5 day initial test. (file suffix: base)
- ERS: Exact restart from startup (default 6 days + 5 days)
- Do an 11 day initial test - write a restart at day 6. (file suffix: base)Do a 5 day restart test, starting from restart at day 6. (file suffix: rest)Compare component history files ‘.base’ and ‘.rest’ at day 11. They should be identical.
- PET: Modified threading OPENMP bit for bit test (default 5 days)
- Do an initial run where all components are threaded by default. (file suffix: base) Do another initial run with NTHRDS=1 for all components. (file suffix: single_thread) Compare base and single_thread.
MODIFIERS
changes to the default settings for the test.GRID
The model grid (can be an alias). Currently,C96
,C192
,C384
andC768
are supported.COMPSET
alias of the compset, or long name, if no--xml
arguments are used. It can beGFSv15p2
orGFSv16beta
.MACHINE
This is optional; if this value is not supplied, create_test will probe the underlying machine.COMPILER
If this value is not supplied, use the default compiler forMACHINE
.GROUP-TESTMODS
This is optional. This points to a directory withuser_nl_xxx
files or ashell_commands
that can be used to make namelist andXML
modifications prior to running a test.
6.2. Query list of supported tests¶
$CIMEROOT/scripts/query_testlists gathers descriptions of the tests and testlists available for CESM, the components, and projects.
The --xml-{compiler,machine,category,testlist}
arguments can be used
as in create_test (above) to focus the search.
The ‘category’ descriptor of a test can be used to run a group of associated tests at the same time.
The available categories, with the tests they encompass, can be listed by:
cd $SRCROOT/cime/scripts
./query_testlists --define-testtypes
The --show-options
argument does the same, but displays the ‘options’ defined for the tests,
such as queue, walltime, etc..
6.3. Using create_test (UFSATM)¶
To run a Smoke startup test:
cd $SRCROOT/cime/scripts
./create_test SMS_D_Lh5.C96.GFSv15p2 --workflow ufs-mrweather_wo_post --test-id try
This will build and run the test in /glade/scratch/$USER/SMS_D_Lh5.C96.GFSv15p2.cheyenne_intel.try
and this directory
is called as CASEROOT. The run directory is in CASEROOT/run and the build is in CASEROOT/bld.
In this case, the C96 resolution model case with CCPP suite version v15p2 is created and runs 5 hours (Lh5) without post-processing step.
To run a test with baseline comparisons against baseline name ‘master’:
cd $SRCROOT/cime/scripts
./create_test SMS_Lh5.C96.GFSv15p2 --workflow ufs-mrweather_wo_post --test-id try --compare master --baseline-root $BASELINE_ROOT
To run a Exact restart test:
cd $SRCROOT/cime/scripts
./create_test ERS_Lh11.C96.GFSv15p2 --workflow ufs-mrweather_wo_post --test-id try
This will build and run the test that includes two runs, first an 11 hour initial run (cold start) with a restart written at hour 6 and then a restart run (warm start) starting from hour 6 and compare the outputs written at hour 11. The output of the runs must be same.
To run a threaded test:
cd $SRCROOT/cime/scripts
./create_test PET_Lh11.C96.GFSv15p2 --workflow ufs-mrweather_wo_post --test-id try
To run entire test suite:
cd $SRCROOT/cime/scripts
./create_test --xml-testlist ../../src/model/FV3/cime/cime_config/testlist.xml --xml-machine MACHINE --generate GENERATE --baseline-root BASELINE_ROOT --workflow ufs-mrweather_wo_post
This will run entire test suite on specified machine MACHINE
such as Stampede2 and generates the baseline under BASELINE_ROOT
directory with a name of GENERATE
.
Note
The baseline directory for supported platforms is placed under $BASELINE_ROOT
directory. This can be queried using ./xmlquery -p BASELINE_ROOT
command once the test is created using ./create.test
command.