# 2. Workflow Quick Start¶

The following quick start guide is applicable to versions of the UFS Medium-Range Weather Application that are on preconfigured or configurable machines as listed here.

The workflow for building and running the App is built on the CIME (Common Infrastructure for Modeling Earth) framework. Please refer to the CIME Porting Documentation if CIME has not yet been ported to the target machine.

If you are new to CIME, please consider reading the CIME Case Control System Part 1: Basic Usage after downloading the code. The CIME Users Guide will be easier to follow after the directory structure has been created by the git clone command.

This is the procedure for quickly setting up and running a case of UFS MR Weather App.

• Create a case: use create_newcase

• Setup a case: use case.setup

• Build a case: use case.build

• Run a case: use case.submit

Note

Variables presented as $VAR in this guide typically refer to variables in XML files that are created in your case directory. From within a case directory, you can determine the value of such a variable with ./xmlquery VAR. In some instances, $VAR refers to a shell variable or some other variable; we try to make these exceptions clear.

Access to the code requires git. You will need access to the command line clients, git (v1.8 or greater). You can download the latest version of the release code:

git clone https://github.com/ufs-community/ufs-mrweather-app.git -b ufs-v1.0.0 my_ufs_sandbox
cd my_ufs_sandbox


Note

When cloning the ufs-mrweather-app repository, the connection to github may time out. In this case, resubmit the git clone command.

To checkout UFS MR Weather Model components, including CIME, run the checkout_externals script from /path/to/my_ufs_sandbox.

./manage_externals/checkout_externals


The checkout_externals script will read the configuration file Externals.cfg and will download the model source and CIME into /path/to/my_ufs_sandbox.

To see more details regarding the checkout_externals script from the command line, type:

./manage_externals/checkout_externals --help


To confirm a successful download of all components, you can run checkout_externals with the status flag to show the status of the externals:

./manage_externals/checkout_externals -S


This should show a clean status for all externals, with no characters in the first two columns of output, as in this example:

Checking status of externals: model, stochastic_physics, fv3, ccpp/framework, atmos_cubed_sphere, ccpp/physics, fms, nems, tests/produtil/nceplibs-pyprodutil, fv3gfs_interface, nems_interface, cime,
./cime
./src/model
./src/model/FMS
./src/model/FV3
./src/model/FV3/atmos_cubed_sphere
./src/model/FV3/ccpp/framework
./src/model/FV3/ccpp/physics
./src/model/FV3/cime
./src/model/NEMS
./src/model/NEMS/cime/
./src/model/NEMS/tests/produtil/NCEPLIBS-pyprodutil
./src/model/stochastic_physics


You should now have a complete copy of the source code in your /path/to/my_ufs_sandbox.

If there were problems obtaining an external, you might instead see something like:

e-  ./src/model/FV3


This might happen if there was an unexpected interruption while downloading. First try rerunning ./manage_externals/checkout_externals. If there is still a problem, try running with logging turned on using:

./manage_externals/checkout_externals --logging


Check the manage_externals.log file to see what errors are reported.

## 2.2. Model Configurations¶

The UFS MR Weather App can be configured at four out-of-the-box resolutions with two different compsets, GFSv15p2 or GFSv16beta. These compsets invoke physics suites that use or not an ocean-evolving parameterization depending on the initial data provided. See the Introduction for more information on the physics suites provided with the release and see the frequently-asked questions (FAQ) section for more information on compsets, physics suites, and initial datasets.

• Details of available component sets and resolutions are available from the query_config tool located in the cime/scripts directory

cd $SRCROOT/cime/scripts ./query_config --help  where $SRCROOT is the top directory of the ufs-mrweather-app.

### 2.2.1. Supported component sets¶

The components of the modeling system can be combined in numerous ways to carry out various scientific or software experiments. A particular mix of components, along with component-specific configuration and/or namelist settings is referred to as component set or “compset”. The UFS MR Weather App has a shorthand naming convention for component sets that are supported out-of-the-box.

To determine what MR Weather App compsets are available in the release, use the following command:

cd $SRCROOT/cime/scripts ./query_config --compsets  This should show a list of available compsets: Active component: ufsatm -------------------------------------- Compset Alias: Compset Long Name -------------------------------------- GFSv15p2 : FCST_ufsatm%v15p2_SLND_SICE_SOCN_SROF_SGLC_SWAV GFSv16beta : FCST_ufsatm%v16beta_SLND_SICE_SOCN_SROF_SGLC_SWAV  ### 2.2.2. Supported grids¶ CIME has the flexibility to support numerous model resolutions. To see the grids that are currently supported, use the following command cd$SRCROOT/cime/scripts
./query_config --grids


This should show the a list of available grids for this release.

=========================================
GRID naming convention
=========================================
The notation for the grid longname is
where
a% => atm, l% => lnd, oi% => ocn/ice, r% => river, m% => mask, g% => glc, w% => wav

Supported grid configurations are given via alias specification in
the file "config_grids.xml". Each grid alias can also be associated  with the
following optional attributes

-------------------------------------------------------------
default component grids:

component         compset       value
-------------------------------------------------------------
atm      SATM              null
lnd      SLND              null
ocnice   SOCN              null
rof      SROF              null
glc      SGLC              null
wav      SWAV              null
iac      SIAC              null
-------------------------------------------------------------

alias: C96
non-default grids are: atm:C96

alias: C192
non-default grids are: atm:C192

alias: C384
non-default grids are: atm:C384

alias: C768
non-default grids are: atm:C768


As can be seen, MR Weather App currently supports four grids with the following nominal resolutions

• C96 (~100km)

• C192 (~50km),

• C384 (~25km)

• C768 (~13km),

and all with 64 vertical levels.

## 2.3. Setup the environment¶

Two environment variables need to be set prior to running the CIME workflow:

export UFS_INPUT=/path/to/inputs
export UFS_SCRATCH=/path/to/outputs


UFS_INPUT should be set to the location of a folder where input data will be accessed. There should be a folder named ufs_inputdata underneath this folder. The folder $UFS_INPUT/ufs_inputdata should exist before running the CIME workflow. This is often a shared location on a platform so that all users on that platform can access data from the same location. UFS_SCRATCH should be set to the location of a writeable folder where output will be written for each case. This is typically a user scratch space or temporary location with a large allocation available. The following settings are recommended on the pre-configured platforms: Table 2.1 Path settings for pre-configured platforms. Platform$UFS_INPUT

$UFS_SCRATCH NCAR cheyenne$CESMDATAROOT

/glade/scratch/$USER NOAA hera /scratch1/NCEPDEV/stmp2/CIME_UFS <my-project-dir>/$USER

On platforms that are not pre-configured a script needs to be executed to define a set of environment variables related to the location of NCEPLIBS dependencies.

# SH or BASH shells
source $NCEPLIBS_DIR/bin/setenv_nceplibs.sh # CSH or TCSH shells source$NCEPLIBS_DIR/bin/setenv_nceplibs.csh


The recommended best practice to set the $UFS_SCRATCH and $UFS_INPUT environment variables and source the NCEPLIBS provided shell script setenv_nceplibs.sh|.csh is to add the above commands to a startup script such as $HOME/.bashrc (Bash shell) or $HOME/.tcshrc (Tcsh shell). These files are executed automatically when you start a new shell so that you do not need to re-define them during each login.

Important

On some platforms (in particular Stampede2) this practice is required to ensure the environment variables are properly set on compute nodes accessed by the workflow.

## 2.4. Create a case¶

The create_newcase command creates a case directory containing the scripts and XML files to configure a case (see below) for the requested resolution, component set, and machine. create_newcase has three required arguments: --case, --compset and --res. The workflow argument is optional, to select alternate workflow components (see below). The project argument is optional, to set the batch system project account (see below). (invoke create_newcase --help for help).

On machines where a project or account code is needed, you must either specify the --project $PROJECT argument in the create_newcase command, or set the $PROJECT variable in your shell environment. If this argument is not set, the error message ERROR: PROJECT_REQUIRED will be reported.

If running on a preconfigured or configurable machine, that machine will normally be recognized automatically and therefore it is not required to specify the –machine argument to create_newcase. Generic linux and macos systems will require the --machine argument to be used.

Invoke create_newcase as follows from the cime/scripts directory:

cd cime/scripts
./create_newcase --case CASENAME --compset COMPSET --res GRID --workflow WORKFLOW


where:

• CASENAME defines the name of your case (stored in the $CASE XML variable). This is a very important piece of metadata that will be used in filenames, internal metadata and directory paths. create_newcase will create the case directory with the same name as the CASENAME. If CASENAME is simply a name (not a path), the case directory is created in the cime/scripts directory where you executed create_newcase. If CASENAME is a relative or absolute path, the case directory is created there and the name of the case will be the tail path. The full path to the case directory will be stored in the $CASEROOT XML variable.

• COMPSET is the component set and can be GFSv15p2 or GFSv16beta, which trigger supported Common Community Physics Package (CCPP) suites. If you would like to learn more about CCPP please consider reading the CCPP Overview.

• GRID is the model resolution, which can be C96, C192, C384 and C768.

• WORKFLOW is the workflow and can be set as ufs-mrweather or ufs-mrweather_wo_post. The ufs-mrweather includes both pre- and post-processing steps, while ufs-mrweather_wo_post includes only pre-processing step. In the current version of the UFS MR Weather App, the pre-processing step need to be run to generate initial conditions for the UFS Weather Model.

• PROJECT is the project or account code needed to run batch jobs. You may either specify the --project $PROJECT argument in the create_newcase command, or set the $PROJECT variable in your shell environment.

Here is an example on NCAR machine Cheyenne with the $USER shell environment variable set to your Cheyenne login name: cd cime/scripts ./create_newcase --case$UFS_SCRATCH/ufs-mrweather-app-workflow.c96 --compset GFSv15p2 --res C96 --workflow ufs-mrweather


## 2.5. Setting up the case run script¶

Issuing the case.setup command creates scripts needed to run the model along with namelist user_nl_xxx files, where xxx denotes the set of components for the given case configuration such as ufsatm and cpl. Selected namelist entries can be customized by editing user_nl_xxx, see FAQ.

cd to the case directory $UFS_SCRATCH/ufs-mrweather-app-workflow.c96 as shown above: cd /glade/scratch/$USER/cases/ufs-mrweather-app-workflow.c96


Before invoking case.setup, you could modify the env_mach_pes.xml file in the case directory using the xmlchange command as needed for the experiment (optional). (Note: To edit any of the env xml files, use the xmlchange command. xmlchange --help can be used for help.)

Please also be aware that you need to provide consistent layout, write_tasks_per_group and write_groups namelist options to the model when total number of PEs are changed.

Invoke the case.setup command.

./case.setup


Note

The CIME commands ./xmlquery, ./case.setup, ./case.build, ./case.submit examine and modify the CIME case, and so, are linked into the directory specified by --case when the ./create_newcase is run. They should be run from this case directory.

## 2.6. Build the executable using the case.build command¶

Modify build settings in env_build.xml (optional).

Run the build script.

./case.build


Users of the NCAR cheyenne system should consider using qcmd to compile UFS Weather Model on a compute node as follows:

qcmd -- ./case.build


The UFS Weather Model executable (named as ufs.exe) will appear in the directory given by the XML variable $EXEROOT, which can be queried using: ./xmlquery EXEROOT  ## 2.7. Run the case¶ Modify runtime settings in env_run.xml (optional). Two settings you may want to change now are: 1. Run length: By default, the model is set to run for 5 days based on the $STOP_N and $STOP_OPTION variables: ./xmlquery STOP_OPTION,STOP_N  These default settings can be useful in troubleshooting runtime problems before submitting for a longer time or a production runs. For example, following setting can be used to set the simulation lenght to 36-hours. Please, also be aware that nyears, nmonths and nsteps options for STOP_OPTION are not supported in the UFS MR Weather App. ./xmlchange STOP_OPTION=nhours,STOP_N=36  2. You can set the $DOUT_S variable to FALSE to turn off short term archiving:

./xmlchange DOUT_S=FALSE

3. The default job wall clock time, which is set to 12-hours, can be changed for relatively short and low-resolution simulations. For example, following commands sets the job wall clock time to 30-minutes.

./xmlchange JOB_WALLCLOCK_TIME=00:30:00
./xmlchange USER_REQUESTED_WALLTIME=00:30:00

4. The default start date (2019-08-29, 00 UTC) can be also changed by following commands

./xmlchange RUN_STARTDATE=YYYY-MM-DD
./xmlchange START_TOD=AS_SECOND


where:

• RUN_STARTDATE is the start date and need to be given in YYYY-MM-DD format such as 2020-01-15

• START_TOD is the time of day in seconds such as 12 UTC need to be given as 43200 seconds.

Submit the job to the batch queue using the case.submit command.

./case.submit


Based on the selected workflow (ufs-mrweather or ufs-mrweather_wo_post), the case.submit command submits a chain of jobs that their dependency is automatically set. For example, ufs-mrweather workflow submits a job array with three seperate jobs that will run in an order: pre-processing, simulation and post-processing. The first ten characters of the job names will be chgres.ufs, run.ufs-mr, and gfs_post.u, respectively.

When the jobs are complete, most output will NOT be written under the case directory, but instead under some other directories (defined by $UFS_SCRATCH). Review the following directories and files, whose locations can be found with xmlquery (note: xmlquery can be run with a list of comma separated names and no spaces): ./xmlquery RUNDIR,CASE,CASEROOT,DOUT_S,DOUT_S_ROOT  • $RUNDIR

This directory is set in the env_run.xml file. This is the location where UFS MR Weather App was run. Log files for each stage of the workflow can be found here.

Table 2.2 Log files

Component

File Name

Look for…

chgres.ufs

chgres_cube.yymmdd-hhmmss.log

“DONE”

run.ufs-mr

ufs.log.<jobid>.yymmdd-hhmmss

“PROGRAM nems HAS ENDED”

gfs_post.ufs

oi.hhh

“PROGRAM UNIFIED_POST HAS ENDED”

• $DOUT_S_ROOT/$CASE

$DOUT_S_ROOT refers to the short term archive path location on local disk. This path is used by the case.st_archive script when $DOUT_S = TRUE.

$DOUT_S_ROOT/$CASE is the short term archive directory for this case. If $DOUT_S is FALSE, then no archive directory should exist. If $DOUT_S is TRUE, then log, history, and restart files should have been copied into a directory tree here.

• $DOUT_S_ROOT/$CASE/logs

The log files should have been copied into this directory if the run completed successfully and the short-term archiver is turned on with $DOUT_S = TRUE. Otherwise, the log files are in the $RUNDIR.

• $CASEROOT There could be standard out and/or standard error files output from the batch system. • $CASEROOT/CaseDocs

The case namelist files are copied into this directory from the \$RUNDIR.