2. Workflow Quick Start¶
The workflow for building and running the App is built on the CIME (Common Infrastructure for Modeling Earth) framework. Please refer to the CIME Porting Documentation if CIME has not yet been ported to the target machine.
If you are new to CIME, please consider reading the CIME Case Control System Part 1: Basic Usage after downloading the code. The CIME Users Guide will be easier to follow after the directory structure has been created by the git clone command.
This is the procedure for quickly setting up and running a case of UFS MR Weather App.
Download the UFS MR Weather App
Create a case: use
Setup a case: use
Build a case: use
Run a case: use
Variables presented as
$VAR in this guide typically refer to variables in XML files
that are created in your case directory. From within a case directory, you can determine the value of such a
./xmlquery VAR. In some instances,
$VAR refers to a shell
variable or some other variable; we try to make these exceptions clear.
2.1. Downloading the UFS MR Weather App code and scripts¶
Access to the code requires git. You will need access to the command line clients,
(v1.8 or greater). You can download the latest version of the release
git clone https://github.com/ufs-community/ufs-mrweather-app.git -b ufs-v1.0.0 my_ufs_sandbox cd my_ufs_sandbox
When cloning the ufs-mrweather-app repository, the connection to github may time out. In this
case, resubmit the
git clone command.
To checkout UFS MR Weather Model components, including CIME, run the
checkout_externals script from /path/to/my_ufs_sandbox.
checkout_externals script will read the configuration file
will download the model source and CIME into /path/to/my_ufs_sandbox.
To see more details regarding the checkout_externals script from the command line, type:
To confirm a successful download of all components, you can run
with the status flag to show the status of the externals:
This should show a clean status for all externals, with no characters in the first two columns of output, as in this example:
Checking status of externals: model, stochastic_physics, fv3, ccpp/framework, atmos_cubed_sphere, ccpp/physics, fms, nems, tests/produtil/nceplibs-pyprodutil, fv3gfs_interface, nems_interface, cime, ./cime ./src/model ./src/model/FMS ./src/model/FV3 ./src/model/FV3/atmos_cubed_sphere ./src/model/FV3/ccpp/framework ./src/model/FV3/ccpp/physics ./src/model/FV3/cime ./src/model/NEMS ./src/model/NEMS/cime/ ./src/model/NEMS/tests/produtil/NCEPLIBS-pyprodutil ./src/model/stochastic_physics
You should now have a complete copy of the source code in your /path/to/my_ufs_sandbox.
If there were problems obtaining an external, you might instead see something like:
This might happen if there was an unexpected interruption while downloading.
First try rerunning
If there is still a problem, try running with logging turned on using:
manage_externals.log file to see what errors are reported.
2.2. Model Configurations¶
The UFS MR Weather App can be configured at four out-of-the-box
resolutions with two different compsets,
GFSv16beta. These compsets invoke physics suites that use or not
an ocean-evolving parameterization depending on the initial data
provided. See the Introduction for more information on the physics
suites provided with the release and see the frequently-asked
questions (FAQ) section for more information on compsets,
physics suites, and initial datasets.
Details of available component sets and resolutions are available from the
query_configtool located in the
cd $SRCROOT/cime/scripts ./query_config --help
$SRCROOT is the top directory of the ufs-mrweather-app.
2.2.1. Supported component sets¶
The components of the modeling system can be combined in numerous ways to carry out various scientific or software experiments. A particular mix of components, along with component-specific configuration and/or namelist settings is referred to as component set or “compset”. The UFS MR Weather App has a shorthand naming convention for component sets that are supported out-of-the-box.
To determine what MR Weather App compsets are available in the release, use the following command:
cd $SRCROOT/cime/scripts ./query_config --compsets
This should show a list of available compsets:
Active component: ufsatm -------------------------------------- Compset Alias: Compset Long Name -------------------------------------- GFSv15p2 : FCST_ufsatm%v15p2_SLND_SICE_SOCN_SROF_SGLC_SWAV GFSv16beta : FCST_ufsatm%v16beta_SLND_SICE_SOCN_SROF_SGLC_SWAV
2.2.2. Supported grids¶
CIME has the flexibility to support numerous model resolutions. To see the grids that are currently supported, use the following command
cd $SRCROOT/cime/scripts ./query_config --grids
This should show the a list of available grids for this release.
========================================= GRID naming convention ========================================= The notation for the grid longname is a%name_l%name_oi%name_r%name_m%mask_g%name_w%name where a% => atm, l% => lnd, oi% => ocn/ice, r% => river, m% => mask, g% => glc, w% => wav Supported grid configurations are given via alias specification in the file "config_grids.xml". Each grid alias can also be associated with the following optional attributes ------------------------------------------------------------- default component grids: component compset value ------------------------------------------------------------- atm SATM null lnd SLND null ocnice SOCN null rof SROF null glc SGLC null wav SWAV null iac SIAC null ------------------------------------------------------------- alias: C96 non-default grids are: atm:C96 alias: C192 non-default grids are: atm:C192 alias: C384 non-default grids are: atm:C384 alias: C768 non-default grids are: atm:C768
As can be seen, MR Weather App currently supports four grids with the following nominal resolutions
and all with 64 vertical levels.
2.3. Setup the environment¶
Two environment variables need to be set prior to running the CIME workflow:
export UFS_INPUT=/path/to/inputs export UFS_SCRATCH=/path/to/outputs
UFS_INPUT should be set to the location of a folder where input
data will be accessed. There should be a folder named
ufs_inputdata underneath this folder. The folder
$UFS_INPUT/ufs_inputdata should exist before running the CIME
workflow. This is often a shared location on a platform so that all
users on that platform can access data from the same location.
UFS_SCRATCH should be set to the location of a writeable folder
where output will be written for each case. This is typically a user
scratch space or temporary location with a large allocation available.
The following settings are recommended on the pre-configured platforms:
On platforms that are not pre-configured a script needs to be executed to define a set of environment variables related to the location of NCEPLIBS dependencies.
# SH or BASH shells source $NCEPLIBS_DIR/bin/setenv_nceplibs.sh # CSH or TCSH shells source $NCEPLIBS_DIR/bin/setenv_nceplibs.csh
The recommended best practice to set the
$UFS_INPUT environment variables and source the NCEPLIBS provided
setenv_nceplibs.sh|.csh is to add the above commands
to a startup script such as
$HOME/.bashrc (Bash shell) or
$HOME/.tcshrc (Tcsh shell). These files are executed automatically
when you start a new shell so that you do not need to re-define them
during each login.
On some platforms (in particular Stampede2) this practice is required to ensure the environment variables are properly set on compute nodes accessed by the workflow.
2.4. Create a case¶
The create_newcase command creates a case directory containing the scripts and XML
files to configure a case (see below) for the requested resolution, component set, and
create_newcase has three required arguments:
workflow argument is optional, to select alternate workflow components (see below).
project argument is optional, to set the batch system project account (see below).
create_newcase --help for help).
On machines where a project or account code is needed, you
must either specify the
--project $PROJECT argument in the
create_newcase command, or set the
$PROJECT variable in your shell environment. If this argument is not set, the error message
ERROR: PROJECT_REQUIRED will be reported.
If running on a preconfigured or configurable machine, that machine
will normally be recognized automatically and therefore it is not
required to specify the –machine argument to create_newcase. Generic linux and
macos systems will require the
--machine argument to be used.
create_newcase as follows from the
cd cime/scripts ./create_newcase --case CASENAME --compset COMPSET --res GRID --workflow WORKFLOW
CASENAMEdefines the name of your case (stored in the
$CASEXML variable). This is a very important piece of metadata that will be used in filenames, internal metadata and directory paths.
create_newcasewill create the case directory with the same name as the
CASENAMEis simply a name (not a path), the case directory is created in the
cime/scriptsdirectory where you executed create_newcase. If
CASENAMEis a relative or absolute path, the case directory is created there and the name of the case will be the tail path. The full path to the case directory will be stored in the
COMPSETis the component set and can be
GFSv16beta, which trigger supported Common Community Physics Package (CCPP) suites. If you would like to learn more about CCPP please consider reading the CCPP Overview.
GRIDis the model resolution, which can be
WORKFLOWis the workflow and can be set as
ufs-mrweatherincludes both pre- and post-processing steps, while
ufs-mrweather_wo_postincludes only pre-processing step. In the current version of the UFS MR Weather App, the pre-processing step need to be run to generate initial conditions for the UFS Weather Model.
PROJECTis the project or account code needed to run batch jobs. You may either specify the
--project $PROJECTargument in the
create_newcasecommand, or set the
$PROJECTvariable in your shell environment.
Here is an example on NCAR machine Cheyenne with the
$USER shell environment variable
set to your Cheyenne login name:
cd cime/scripts ./create_newcase --case $UFS_SCRATCH/ufs-mrweather-app-workflow.c96 --compset GFSv15p2 --res C96 --workflow ufs-mrweather
2.5. Setting up the case run script¶
Issuing the case.setup command creates scripts needed to run the model
along with namelist
user_nl_xxx files, where xxx denotes the set of components
for the given case configuration such as
Selected namelist entries can be customized by editing
user_nl_xxx, see FAQ.
cd to the case directory
$UFS_SCRATCH/ufs-mrweather-app-workflow.c96 as shown above:
case.setup, you could modify the
env_mach_pes.xml file in the case directory
using the xmlchange command as needed for the experiment (optional). (Note: To edit any of
the env xml files, use the xmlchange command.
xmlchange --help can be used for help.)
Please also be aware that you need to provide consistent
write_groups namelist options to the model when total number of PEs are changed.
The CIME commands
./case.submit examine and modify
the CIME case, and so, are linked into the directory specified by
--case when the
./create_newcase is run. They should be run from this case directory.
2.6. Build the executable using the case.build command¶
Modify build settings in
Run the build script.
Users of the NCAR cheyenne system should consider using qcmd to compile UFS Weather Model on a compute node as follows:
qcmd -- ./case.build
The UFS Weather Model executable (named as
ufs.exe) will appear in the directory given by the
$EXEROOT, which can be queried using:
2.7. Run the case¶
Modify runtime settings in
env_run.xml (optional). Two settings you may want to change
Run length: By default, the model is set to run for 5 days based on the
These default settings can be useful in troubleshooting runtime problems before submitting for a longer time or a production runs. For example, following setting can be used to set the simulation lenght to 36-hours. Please, also be aware that
STOP_OPTIONare not supported in the UFS MR Weather App.
You can set the
$DOUT_Svariable to FALSE to turn off short term archiving:
The default job wall clock time, which is set to 12-hours, can be changed for relatively short and low-resolution simulations. For example, following commands sets the job wall clock time to 30-minutes.
./xmlchange JOB_WALLCLOCK_TIME=00:30:00 ./xmlchange USER_REQUESTED_WALLTIME=00:30:00
The default start date (2019-08-29, 00 UTC) can be also changed by following commands
./xmlchange RUN_STARTDATE=YYYY-MM-DD ./xmlchange START_TOD=AS_SECOND
RUN_STARTDATEis the start date and need to be given in YYYY-MM-DD format such as 2020-01-15
START_TODis the time of day in seconds such as 12 UTC need to be given as 43200 seconds.
Submit the job to the batch queue using the
Based on the selected workflow (
command submits a chain of jobs that their dependency is automatically set. For example,
workflow submits a job array with three seperate jobs that will run in an order: pre-processing, simulation
and post-processing. The first ten characters of the job names will be
When the jobs are complete, most output will NOT be written under the case directory, but
instead under some other directories (defined by $UFS_SCRATCH).
Review the following directories and files, whose
locations can be found with
xmlquery can be run with a list of
comma separated names and no spaces):
This directory is set in the
env_run.xmlfile. This is the location where UFS MR Weather App was run. Log files for each stage of the workflow can be found here.
“PROGRAM nems HAS ENDED”
“PROGRAM UNIFIED_POST HAS ENDED”
$DOUT_S_ROOTrefers to the short term archive path location on local disk. This path is used by the case.st_archive script when
$DOUT_S = TRUE.
$DOUT_S_ROOT/$CASEis the short term archive directory for this case. If
$DOUT_Sis FALSE, then no archive directory should exist. If
$DOUT_Sis TRUE, then log, history, and restart files should have been copied into a directory tree here.
The log files should have been copied into this directory if the run completed successfully and the short-term archiver is turned on with
$DOUT_S = TRUE. Otherwise, the log files are in the
There could be standard out and/or standard error files output from the batch system.
The case namelist files are copied into this directory from the