1. Introduction¶
The Unified Forecast System (UFS) is a community-based, coupled, comprehensive Earth modeling system. NOAA’s operational model suite for numerical weather prediction (NWP) is quickly transitioning to the UFS from a number of different modeling systems. The UFS enables research, development, and contribution opportunities within the broader Weather Enterprise (e.g., government, industry, and academia). For more information about the UFS, visit the UFS Portal.
The UFS includes multiple applications that span local to global domains and a range of predictive time scales. This documentation describes the UFS Medium-Range Weather (MRW) Application (App), which targets predictions of atmospheric behavior out to about two weeks. The MRW App includes a prognostic atmospheric model, pre- and post-processing tools, and a community workflow. These components are documented within this User’s Guide and supported through a community forum. Additionally, the MRW App has transitioned from a CIME-based workflow to the Global Workflow. New and improved capabilities for the upcoming release include the option to run in coupled or uncoupled mode, the addition of a verification package (METplus) for both deterministic and ensemble simulations, and support for the updated GFS_v17_p8 physics scheme and stochastic physics. Future work will expand the capabilities of the application to include data assimilation (DA) and a forecast restart/cycling capability.
The MRW App is available on GitHub and is designed to be code that the community can run and improve. It is portable to a set of commonly used platforms. A limited set of configurations, such as specific model resolutions and physics options, are documented and supported. This documentation provides a Quick Start Guide and will soon provide a detailed guide for running the MRW Application on supported platforms. It also provides an overview of the release components and details on how to customize or modify different portions of the workflow.
The MRW App v1.1.0 citation is as follows and should be used when presenting results based on research conducted with the App:
UFS Development Team. (2020, March 11). Unified Forecast System (UFS) Medium-Range Weather (MRW) Application (Version v1.1.0).
1.1. How to Use This Document¶
This guide instructs both novice and experienced users on downloading, building, and running the MRW Application. Please post questions in the UFS Forums.
Throughout the guide, this presentation style indicates shell commands and options,
code examples, etc.
Variables presented as $VAR or Dir123 in this User’s Guide refer to environment variables, variables in scripts, names of files, or directories.
File paths or code that include angle brackets (e.g., <platform>.env) indicate that users should insert options appropriate to their MRW App configuration (e.g., HERA.env).
Hint
To get started running the MRW App, see the Quick Start Guide for beginners or refer to the in-depth chapter on Configuring a New Platform.
For an outline of MRW App components, see section Section 1.3 below or refer to Chapter 3 for a more in-depth treatment.
For background information on the MRW App code repositories and directory structure, see Section 1.4 below.
1.2. Prerequisites for Using the MRW Application¶
1.2.1. Background Knowledge Prerequisites¶
The instructions in this documentation assume that users have certain background knowledge:
Familiarity with LINUX/UNIX systems
Command line basics
System configuration knowledge (e.g., compilers, environment variables, paths, etc.)
Numerical Weather Prediction
Meteorology
Additional background knowledge in the following areas could be helpful:
High-Performance Computing (HPC) Systems for those running the MRW App on an HPC system
Programming (particularly Python) for those interested in contributing to the MRW App code
Creating an SSH Tunnel to access HPC systems from the command line
Containerization
Workflow Managers/Rocoto
1.2.2. Software/Operating System Requirements¶
The UFS MRW Application has been designed so that any sufficiently up-to-date machine with a UNIX-based operating system should be capable of running the application. NOAA Level 1 & 2 systems already have these prerequisites installed. However, users working on other systems must ensure that the following requirements are installed on their system:
Minimum Platform Requirements:
UNIX style operating system such as CNL, AIX, Linux, Mac
>44 GB disk space
18 GB input data from GFS for “out-of-the-box” MRW App case
8 GB for HPC-Stack full installation
3 GB for
ufs-mrweather-appinstallation and build11 GB for 120hr/5-day forecast
Python 3.7+
Perl 5
Git client (1.8+)
Fortran compiler released since 2018
gfortran v9+ or ifort v18+ are the only ones tested, but others may work.
C compiler compatible with the Fortran compiler
gcc v9+, ifort v18+, and clang v9+ (macOS, native Apple clang or LLVM clang) have been tested
Lmod
The following software is also required to run the MRW Application, but HPC-Stack (which contains the software libraries necessary for building and running the MRW App) can be configured to build these requirements:
MPI (MPICH, OpenMPI, or other implementation)
Only MPICH or OpenMPI can be built with HPC-Stack. Other options must be installed separately by the user (if desired).
HPC-Stack, which includes:
NCEPLIBS-external (includes ESMF)
Optional but recommended prerequisites for all systems:
Conda for installing/managing Python packages
Bash v4+
Rocoto Workflow Management System (1.3.1)
Python packages
matplotlib,numpy,cartopy, andnetCDF4for graphics
After installing these prerequisites, users may continue on to build the MRW App as documented in the Workflow Quick Start.
1.3. MRW App Components Overview¶
1.3.1. Build System and Workflow¶
The MRW Application has a portable CMake-based build system that packages together all the components required to build the MRW Application. Once built, users can generate the Rocoto-based Global Workflow, which will run each task in the proper sequence. (See Rocoto documentation for more on workflow management.)
The MRW Application has been tested on a variety of platforms widely used by researchers, including NOAA High-Performance Computing (HPC) systems (e.g., Hera, Jet), cloud environments, and generic Linux and macOS systems. Four levels of support have been defined for the MRW Application. Preconfigured (Level 1) systems already have the required software libraries available in a central location via the HPC-Stack. The MRW Application is expected to build and run out-of-the-box on these systems, and users can download the MRW App code without first installing prerequisites. On other platforms (Levels 2-4), the required libraries will need to be installed as part of the MRW Application build process. On Level 2 platforms, installation should be straightforward, and the MRW App should build and run successfully. On Level 3 & 4 platforms, users may need to perform additional troubleshooting since little or no pre-release testing has been conducted on these systems.
1.3.2. Data and Pre-Processing Utilities¶
The MRW App requires input model data in GRIB2, NEMSIO, or netCDF format. The chgres_cube pre-processing software, which is part of the UFS_UTILS pre-processing utilities package, uses these files to initialize and prepare the model. Additional information about the pre-processor utilities can be found in Section 3.3, in the UFS_UTILS Technical Documentation, and in the UFS_UTILS Scientific Documentation.
1.3.3. Forecast Model¶
1.3.3.1. Atmospheric Model¶
The prognostic atmospheric model in the UFS MRW Application uses the Finite-Volume Cubed-Sphere (FV3) dynamical core. The dynamical core is the computational part of a model that solves the equations of fluid motion for the atmospheric component of the UFS Weather Model. A User’s Guide for the UFS Weather Model can be found here. Additional information about the FV3 dynamical core can be found in the scientific documentation and the technical documentation.
1.3.3.2. Common Community Physics Package¶
The Common Community Physics Package (CCPP) supports interoperable atmospheric physics and land surface model options. Atmospheric physics are a set of numerical methods describing small-scale processes such as clouds, turbulence, radiation, and their interactions. The MRW App currently includes the GFS_v17_p8 physics suite as well as stochastic options to represent model uncertainty.
1.3.4. Unified Post-Processor¶
The Medium-Range Weather (MRW) Application is distributed with a post-processing tool, the Unified Post Processor (UPP). The UPP converts the native netCDF output from the model to GRIB2 format on standard isobaric coordinates in the vertical direction. The UPP can also be used to compute a variety of useful diagnostic fields, as described in the UPP User’s Guide. The UPP output can be used with visualization, plotting and verification packages, or for further downstream post-processing (e.g., statistical post-processing techniques).
1.3.5. METplus Verification Suite¶
The Model Evaluation Tools (MET) package is a set of statistical verification tools developed by the Developmental Testbed Center (DTC) for use by the NWP community to help them assess and evaluate the performance of numerical weather predictions. MET is the core component of the enhanced METplus verification framework. METplus spans a wide range of temporal and spatial scales. It is intended to be extensible through additional capabilities developed by the community. More details about METplus can be found in Chapter 3.6 and on the METplus website.
1.3.6. Visualization Example¶
The MRW Application includes Python scripts to create basic visualizations of the model output. The scripts may be used to complete a visual check to verify that the application is producing reasonable results.
1.4. Code Repositories and Directory Structure¶
1.4.1. Hierarchical Repository Structure¶
The umbrella repository for the MRW Application is named ufs-mrweather-app. It is available on GitHub at https://github.com/ufs-community/ufs-mrweather-app. An umbrella repository is a repository that pulls in external code, called “externals,” from additional repositories. The MRW Application includes the manage_externals tool and a configuration file called Externals.cfg, which tags the appropriate versions of the external repositories associated with the MRW App (see Table 1.1).
Repository Description |
Authoritative repository URL |
|---|---|
UFS Medium-Range Weather Application Umbrella Repository |
|
Repository for the Global Workflow |
The Global Workflow checkout.sh script then checks out the repositories listed in Table 1.2.
Repository Description |
Authoritative repository URL |
|---|---|
UFS Weather Model Repository |
|
Repository for UFS utilities, including pre-processing, chgres_cube, and more |
|
Unified Post Processor (UPP) Repository |
|
Verification package using MET and METplus |
The UFS Weather Model is itself an umbrella repository and contains a number of subcomponent repositories, which are documented here.
Note
The MRW Application prerequisite libraries (including NCEP Libraries and external libraries) are not included in the MRW App repository. The HPC-Stack repository assembles these prerequisite libraries. HPC-Stack has already been built on preconfigured (Level 1) platforms. However, it must be built on other systems. Users can view the HPC-Stack documentation here.
1.4.2. Directory Structure¶
The ufs-mrweather-app umbrella repository structure is determined by the local_path settings contained within the Externals.cfg file. After manage_externals/checkout_externals is run (see Chapter 2), the specific GitHub repositories described in Table 1.1 are cloned into the target subdirectories shown below. Directories that will be created later as part of the build process appear in parentheses and will not be visible until after the build is complete. Some files and directories have been removed for brevity.
ufs-mrweather-app/
├── build_global-workflow.sh
├── describe_version
├── docs
│ └── UsersGuide
├── Externals.cfg
├── global-workflow
│ ├── docs
│ ├── ecf
│ ├── env
│ │ ├── gfs.ver
│ │ ├── HERA.env
│ │ ├── JET.env
│ │ └── ORION.env
│ ├── (exec)
│ ├── Externals.cfg
│ ├── fix
│ ├── gempak
│ │ ├── dictionaries
│ │ ├── fix
│ │ └── ush
│ ├── jobs
│ │ ├── JGDAS_<JOBS> # multiple scripts
│ │ ├── JGFS_<JOBS> # multiple scripts
│ │ ├── JGLOBAL_<JOBS> # multiple scripts
│ │ └── rocoto
│ ├── modulefiles
│ │ ├── module_base.<platform>.lua
│ │ ├── modulefile.ww3.<platform>.lua
│ │ ├── module-setup.csh.inc
│ │ ├── module-setup.sh.inc
│ │ └── workflow_utils.<platform>.lua
│ ├── parm
│ │ ├── chem
│ │ ├── config
│ │ ├── mom6
│ │ ├── parm_fv3diag
│ │ ├── parm_wave
│ │ ├── post
│ │ ├── product
│ │ ├── relo
│ │ ├── transfer
│ │ ├── wave
│ │ └── wmo
│ ├── README.md
│ ├── scripts
│ │ ├── exgdas_<name>.py # multiple python scripts
│ │ ├── exgdas_<name>.sh # multiple shell scripts
│ │ ├── exgfs_aero_init_aerosol.py
│ │ ├── exgfs_<name>.sh # multiple shell scripts
│ │ ├── exglobal_<name>.sh # multiple shell scripts
│ │ └── run_<name>.sh # multiple shell scripts
│ ├── sorc
│ │ ├── build_<name>.sh # multiple shell scripts
│ │ ├── checkout.sh
│ │ ├── cmake
│ │ ├── CMakeLists.txt
│ │ ├── enkf_chgres_recenter.fd
│ │ ├── enkf_chgres_recenter_nc.fd
│ │ ├── fbwndgfs.fd
│ │ ├── fv3nc2nemsio.fd
│ │ ├── gaussian_sfcanl.fd
│ │ ├── gfs_bufr.fd
│ │ ├── gfs_build.cfg
│ │ ├── install
│ │ ├── link_workflow.sh
│ │ ├── logs
│ │ ├── machine-setup.sh
│ │ ├── ncl.setup
│ │ ├── partial_build.sh
│ │ ├── reg2grb2.fd
│ │ ├── regrid_nemsio.fd
│ │ ├── supvit.fd
│ │ ├── syndat_getjtbul.fd
│ │ ├── syndat_maksynrc.fd
│ │ ├── syndat_qctropcy.fd
│ │ ├── tave.fd
│ │ ├── tocsbufr.fd
│ │ ├── ufs_model.fd
│ │ ├── ufs_utils.fd
│ │ ├── verif-global.fd
│ │ └── vint.fd
│ ├── test
│ ├── ush
│ ├── util
│ │ ├── modulefiles
│ │ ├── sorc
│ │ └── ush
│ └── workflow
│ │ └── rocoto
├── LICENSE.md
├── manage_externals
│ ├── checkout_externals
│ ├── LICENSE.txt
│ ├── manic
│ └── README.md
├── plotting_scripts
│ ├── plot_mrw_cloud_diff.py
│ ├── plot_mrw.py
│ ├── python_plotting_documentation.txt
│ └── sample_output.pdf
└── README.md
1.5. User Support, Documentation, and Contributing Development¶
A forum-based online support system organized by topic provides a centralized location for UFS users and developers to post questions and exchange information. The forum complements the distributed documentation, summarized here for ease of use.
Documentation |
Location |
|---|---|
MRW App User’s Guide |
|
UFS_UTILS Technical Documentation |
|
UFS_UTILS Scientific Documentation |
|
UFS Weather Model User’s Guide |
|
Global Workflow User’s Guide |
https://github.com/NOAA-EMC/global-workflow/wiki/Run-Global-Workflow |
FV3 Scientific Documentation |
|
FV3 Technical Documentation |
https://noaa-emc.github.io/FV3_Dycore_ufs-v2.0.0/html/index.html |
CCPP Scientific Documentation |
|
CCPP Technical Documentation |
|
Stochastic Physics Documentation |
https://stochastic-physics.readthedocs.io/en/release-public-v3/ |
Unified Post Processor |
|
HPC-Stack Documentation |
|
Spack-stack Documentation |
|
ESMF manual |
https://earthsystemmodeling.org/docs/release/latest/ESMF_usrdoc/ |
The UFS community is encouraged to contribute to the development effort of all related utilities, model code, and infrastructure. Users can post issues in the related GitHub repositories to report bugs or to announce upcoming contributions to the code base. For code to be accepted in the authoritative repositories, users must follow the code management rules of each UFS component repository, which are outlined in the respective User’s Guides listed in Table 1.3 or on the repository’s wiki. In particular, innovations involving the UFS Weather Model need to be tested using the regression tests described in its User’s Guide. These tests are part of the official NOAA policy on accepting innovations into its code base, whereas the MRW App tests are meant as a sanity check for users.
1.6. Future Direction¶
Users can expect to see incremental improvements and additional capabilities in upcoming releases of the MRW Application to enhance research opportunities and support operational forecast implementations.
Planned advancements include addition of:
component models for other Earth domains (such as oceans and sea ice)
cycled data assimilation for model initialization
expansion of supported platforms