Parallel Programming Tools

There are several obstacles to the operational use of current distributed memory HPC systems: portability, programmability, and performance. Historically NWP models have been large codes that did not run efficiently on HPC systems without some restructuring. Typically a message passing library is used to handle inter-process communication, process synchronization, global operations, data decomposition, and I/O required to run in distributed memory environments. The development of an efficient Message Passing Interface (MPI) library, supported by most vendors has improved the portability of models on distributed memory computers. However, MPI is sufficiently low level in nature that it can be difficult to use. To speed code parallelization, the ACS has developed a high level tool called the Scalable Modeling System (SMS) that simplifies the task required to port and run NWS models on HPC's while offering good scalable performance.

The Scalable Modeling System

The SMS is a directive-based parallelization tool that translates Fortran code into a parallel version that runs efficiently on both shared and distributed memory systems. This software has been used sucessfully since 1993 to parallelize and run many oceanic and atmospheric models.

SMS was significantly updated in 2008 to support icosahedral grids used in the FIM and NIM models. During this upgrade significant portions of legacy code were removed and our list of supported codes was reduced. Codes listed in italics are legacy codes that may not work under SMS version 3.0.0 or higher. Historically, we have used SMS on most HPC systems, however, the upgraded version has only been tested on linux clusters (intel). We anticipate porting SMS version 3 to IBM SP systems at NCEP in late 2009 or early 2010 and to other systems as the need arises.

Models Parallelized using SMS

Atmospheric Models

  • Flow-following Finite volume Icosahedral Model (FIM)
    • Earth System Research Laboratory (ESRL)
  • Non-hydrostatic Icosahedral Model (NIM)
    • Earth System Research Laboratory (ESRL)
  • Rapid Update Cycle Model (RUC)
    • Earth System Research Laboratory (ESRL)
  • 32 km ETA Model
    • National Centers for Environmental Prediction (NCEP)
  • Global Forecast System (GFS)
    • Central Weather Bureau (CWB) - Taiwan
  • Typhoon Forecast System (TFS)
    • Central Weather Bureau (CWB) - Taiwan
  • Non-Hydrostatic Forecast System (NFS)
    • Central Weather Bureau (CWB) - Taiwan
  • Atmospheric Chemistry Model (NALROM)
    • Earth System Research Laboratory (ESRL)

Oceanic Models

  • Princeton Ocean Model (POM)
    • NASA Goddard
  • Regional Ocean Modeling System (ROMS)
    • Rutgers University
    • National Institute of Water and Atmospheric Research (NIWA)    - New Zealand
    • University of Alaska at Fairbanks (UAF)
    • NOAA / Pacific Marine Environmental Laboratory (NOAA/PMEL)
  • Hybrid Coordinate Ocean Model (HYCOM)
    • Los Alamos National Laboratory

These models contain structured regular grids that are resolved using a finite difference approximation. As SMS has matured, the time and effort required to parallelize codes for MPPs has been reduced significantly. Code parallelization has become simpler because SMS provides support for advanced operations including incremental parallelization and parallel debugging.

High Performance: SMS provide a number of performance optimizations. The SMS run-time libraries have been optimized to speed inter-processor communications using techniques such as aggregation. Array aggregation permits multiple model variables to be combined into a single communications call to reduce message-passing latency. SMS also allows the user to perform computations in the halo region to reduce communications. High performance I/O is also provided by SMS. Since models typically output forecasts several times during a model run, SMS can output these data asynchronous to model execution. These optimization can lead to significantly faster execution times.

Distributed Memory Performance

A performance comparison was done between the hand-coded MPI (32 km version) Eta model running operationally at NCEP on 88 processors, and the same Eta model parallelized using SMS. The MPI Eta model was considered a good candidate for fair comparison since it was (1) parallelized by IBM for NCEP, (2) is an operational model used to produce daily weather forecasts for the U.S. National Weather Service and (3) has been optimized for high performance on the IBM SP2. Fewer than 200 directives were added to the 19,000 line Eta model during SMS parallelization. Results of this study show that SMS Eta is 7 percent faster than MPI Eta on 88 processors of NCEP's IBM-SP2.

Shared Memory Performance

Another performance study compared a shared memory version of the Regional Ocean Modelling System (ROMS) to an SMS version. In this study we found that the SMS version provided equivalent performance to the shared-memory version at low numbers of processors and scaled better to higher numbers of processors.

Documentation

  • User's Guide (~134 pages, 530Kb) explains how to use SMS to parallelize Fortran codes.
  • Reference Manual (~38 pages, 150 Kb) provides complete details about each SMS Directive.

Publications

The Scalable Modeling System: Directive-Based Code Parallelization for Distributed and Shared Memory Computers
-   August 2003:  Journal of Parallel Computing volume/issue: 29/8 pp995-1020.

The Parallel Pre-Processor: a Compiler for Shared and Distributed Memory Computers
-   December 2002:  Accepted for the 8th International Workshop on High-Level Parallel Programming Models and Supportive Environments

Performance Analysis of the Scalable Modeling System
-   November 2002: Tenth ECMWF Workshop on the use of Parallel Processors in Meteorology.

SMS: A high level alternative to MPI
-   November 2000: Ninth ECMWF Workshop on the use of Parallel Processors in Meteorology.

SMS Software Downloads

SMS software is freely available but there are some restrictions.  To install, gunzip the downloaded file, un-tar it, cd to the unloaded SMS directory and then follow the instructions in file INSTALL.

  • SMS Version 2.9.0 - 6.7 Mbytes - Release Notes
  • SMS Version 3.1.0 - Updated in 2009 to support Icosahedral Horizontal Grids used in FIM and NIM.  We also upgraded the I/O, and removed a significant amount of legacy code.

Once you begin using SMS to parallelize code, the following information may be useful.

All questions regarding SMS may be directed to sms-info.fsl@noaa.gov

Date of last update:    October 26, 2010