Earth System Modeling Framework (ESMF) and ESMF-based initiatives
The Earth System Modeling Framework (ESMF) was motivated by the growing complexity of developing Earth system models. These are constructed of separate software components representing physical domains and processes; for example, atmosphere, ocean, and sea ice. The components, which are often developed at separate sites, are coupled into integrated systems to create realistic simulations. An additional complication is that the models are computationally intensive and must be designed to run on supercomputers with parallel processors. ESMF defines a set of standard software interfaces and a set of high-performance tools for common functions, such as grid interpolation. Now in its fifth year, it has transitioned from NASA funding to multi-agency support and is the technical basis for the DoD Battlespace Environments Institute, the NASA Modeling Analysis and Prediction Program, and a host of smaller projects. More information appears on the ESMF website.
 |
|||
|
|
 |
|
|
|
This image shows some of the first ultra-high-resolution results (2/3-degree longitude by 1/2-degree latitude) from the GEOS-5 atmospheric general circulation model coupled via ESMF to a sophisticated NASA stratospheric chemistry package (STRAT-CHEM). The GEOS-5 modeling and data assimilation system developed at NASA's Global Modeling and Assimilation Office consists of over two dozen ESMF gridded components that can be coupled through the framework for a variety of applications, from atmospheric reanalysis and weather prediction to coupled climate modeling. The coupling of GEOS-5 and STRAT-CHEM enables scientists to perform calculations with a variety of interactions between the chemistry and the atmospheric radiation, large-scale dynamics, and sub-grid parameterizations. (Image courtesy of the GEOS-5 team at the NASA Global Modeling and Assimilation Office.) |
|
|
|||
By creating a common modeling infrastructure that serves the climate and weather domains and enables the integration of Earth systems with space weather models, hydrological components, and other related domains, ESMF is a full realization of the NCAR strategic priority of "Developing community models for weather, climate, atmospheric chemistry, and solar-terrestrial research." As it moves toward integrating the suite of scientific software at NCAR to better support science workflows, ESMF is fulfilling the NCAR strategic priority of "Creating an Earth system knowledge environment." Looking beyond NCAR, ESMF is a realization of the goals expressed by the broader community to develop cross-institutional and cross-agency infrastructure that enables them to collectively reuse software and exchange components.
Project description
ESMF components are arranged in a hierarchical structure to form applications. Exchanges are facilitated both for large, composite components, such as an atmospheric model, and for smaller components such as particular physics parameterizations.
The current ESMF distribution contains:
- Tools for building scientific components and couplers, and a set of low-level utilities for common modeling functions such as calendar management and data redistribution
- Concurrent or sequential execution, single or multiple executable modes
- Support for configuring ensemble members sequentially or concurrently
- Fortran interfaces and complete documentation, some C++ interfaces
ESMF supports IBM, Cray X1, SGI, Linux, Mac, and other platforms.
Timeframe for the project
ESMF is now in its fifth year and has started its second funding cycle under NOAA, NASA, DoD, and NSF sponsors. In the next three years, the ESMF group anticipates two major release increments: ESMF 4, focusing on rework and extensions to the grid representation and regridding software, and ESMF 5, focusing on consistency and standardization of interfaces.
Accomplishments in brief for FY2007
The number of ESMF science components in the community is an important metric, since more standard components mean more options for researchers creating coupled systems. The adoption of ESMF grew steadily this year, with the number of available science components growing from 36 at end FY2006 to 58 at end FY2007.
One of the major accomplishments of the year was successful completion of a beta test of a coupled, ESMF-based COAMPS-NCOM application. The atmospheric portion of COAMPS (Coupled Ocean Atmosphere Mesoscale Prediction System) is in operational use at the Navy Fleet Numerical Meteorology and Oceanography Center (FNMOC) and Regional Centers. Coupling it to NCOM (NRL Coastal Ocean Model) improves the predictive capability of the system at high resolution. The coupling was accomplished within the ESMF component architecture, using the ESMF sparse matrix multiply for regridding. It is anticipated that the coupled system will be transitioned to FNMOC beginning in FY2008.
Another application milestone for FY2007 was that ESMF successfully passed Stage 1 of its Community Climate System Model (CCSM) evaluation in May 2007, demonstrating minimal component changes relative to the base code, less than 5% overhead in time to solution, and comparable memory usage for standard configurations. The Stage 2 plan was approved by the CCSM Scientific Steering Committee in Septemmber 2007.
An initial implementation of a more general Grid class, capable of representing curvilinear and (with effort) multi-tile grids, was included in ESMF beta version 3.0.3, released in July 2007. Designs for unstructured grids and a generalized regrid operation have been through their first reviews and have been successfully prototyped. Many elements across the framework were standardized, including treatment of datatypes, aspects of error handling, and initializations. The team removed extraneous files and documentation files from the distribution, which made it significantly more compact (about 1/10 its previous size).
On the programmatic front, NOAA joined NASA, DoD, and NSF as a sponsor of the ESMF Core Team. A proposal funding a NOAA ESMF-based Modeling Architecture (NEMA), with support for both development and applications, was awarded in June 2007 with $600K over three years for the development team. The DoD High Performance Computing Modernization Office, sponsors of the Battlespace Environments Institute (BEI), has provided $1.5M in additional funding over the next three years for the ESMF team through BEI. Efforts will initially focus on developing C interfaces and integrating newly redesigned, more general grid and array classes with higher-level classes in the framework.
Plans in brief for FY2008
ESMF will continue to track adoption efforts carefully and will put user surveys online so information can be easily and regularly collected from customers. Progress is expected on the CCSM Stage 2 evaluation plan, but it is unlikely to be completed during FY2008.
The ESMF team will focus on completing the implementation of structured grids, releasing a general, parallel, online regridding capability, and delivering software for the representation of unstructured grids. The location stream (observational data stream) software contributed by NASA Goddard will be integrated into the new ESMF grid software. Usability goals include decreasing the number of open bugs by half and developing C interfaces.
Detailed accomplishments for FY2007
During FY2007, the ESMF team made significant advances in community adoption; training, support, and outreach; software development; and organizational maturity.
Advance community adoption
Means, strategies, and actions:
Assist in the development of new, coupled ESMF applications and the
transition of these applications to routine use and operations.
Measures of success:
Number of ESMF components available.
Number of ESMF applications in routine use or operations.
Actual performance, outputs, and outcomes:
There are now more than 58 ESMF components available, a significant
increase over the 36 available at last reporting. Components are
listed on the ESMF website at
ESMF components.
There are now four major coupled codes that have put ESMF into extensive and routine use:
- The NASA GEOS-5 atmospheric general circulation model based on ESMF has been in production since FY2005, and it has added numerous components including a set of chemistry package options.
- The NCEP Global Forecasting System using ESMF was put into operation on August 22, 2006.
- The coupled HYCOM and CICE model using ESMF is now in routine use at the Naval Research Laboratory.
- The coupled COAMPS and NCOM application using ESMF components and coupling passed its beta test in September 2007 (identified as an adoption goal last year).
ESMF passed the CCSM Stage 1 Evaluation Plan in May 2007, identified as a goal last year. Relative to the base code, the evaluation demonstrated minimal component changes, less than 5% overhead in time to solution, and comparable memory usage for standard configurations. The Stage 2 plan was approved by the CCSM Scientific Steering Committee in September 2007.
Progress was made on a coupled ADCIRC-pWASH123 application, but an alpha test was not completed. It is expected to be completed in 1Q FY2008. A coupled SWAN-ADCIRC application identified as a goal last year was replaced by a different application and shifted to a later delivery date.
Provide ESMF training and support
Means, strategies, and actions:Continue development of ESMF tutorials and documentation. Improve presentation of introductory and "big-picture" concepts in the ESMF documentation. Develop system tests and/or demos showing a variety of configurations of ESMF applications, including ensembles and alternative methods of time management and sequencing. Update the ESMF Developer's Guide, which outlines conventions and processes used in framework development. It was initially written when ESMF was a NASA project, and parts of it still treat ESMF as a single-sponsor, milestone-driven activity.
Measures of success:
Number of training classes.
Number of individuals trained.
Actual performance, outputs, and outcomes:
The ESMF Core Team presented one framework training classes during
FY2007: NCAR, June 1, 2007, Boulder, Colorado. This class contained
about 30 students. A total of about 160 students have been trained
since project inception.
The ESMF website was thoroughly reworked and now includes additional introductory material, including a Frequently Asked Questions section useful for beginners. New big-picture sections include project vision, feature overview, metrics, and science impacts. An ensemble system test was started but is not yet complete; it is expected to be in an early FY2008 release. The team did not complete examples showing alternative methods of time sequencing. The Developer's Guide was updated to reflect the current status of the project.
Software development
Goal: Further implementation of data structures that represent unstructured grids, grids with curvilinear coordinates, and observational data streams.
Means, strategies, and actions:
Complete the implementation of classes that represent data blocks and
their distribution (array and distributed grid classes). These classes
were redesigned in FY2006 to support unstructured grids, multi-patch
curvilinear coordinate grids, and observational data streams. Methods
to be implemented include additional communication operations. Rework
the grid and field classes to provide better support for the
representation and manipulation of these more complex grids. Key
abilities to be developed are the ability to read in a grid
specification from a file and to generate interpolation weights for
regridding within ESMF. ESMF can currently only generate interpolation
weights internally for single-patch rectilinear grids.
Measures of success:
Delivery of an ESMF release with a full implementation of the data
block and distribution class interface, the ability to read in grid
specifications for unstructured and multi-patch curvilinear
coordinate grids, and the ability to generate interpolation weights
for regridding such grids.
Actual performance, outputs and outcomes:
Most methods for data block and distribution classes were completed. A
completely redesigned grid class capable of representing curvilinear
grids and, in index space, multi-patch grids, was released in July
2007. Work began on integrating these new capabilities into fields,
bundles of fields, and components, and the first release with this
improvement is expected in November 2007. No file specification
capability was completed, since users indicated that they preferred
an argument list interface for grid creation to a grid specification
from file. A design for the comprehensive support of unstructured
grids began review, and a prototype was completed. Code for the
representation and manipulation of observational data streams was
contributed by staff at NASA GMAO.
Methods were added and extended to provide better support for regridding unstructured grids with arbitrary distributions in index space. Optimizations for memory and performance of the array sparse matrix multiply used in these operations were introduced.
A general regridding strategy for the internal generation of regridding weights, in 2D and 3D, based on a parallel rendezvous method, was successfully prototyped. These results were presented at the ESMF Annual Community Meeting on May 30-31, 2007 and at the CCSM Annual Workshop on June 19-21, 2007. A release is expected in FY2008.
Goal: Improve usability.
Means, strategies, and actions:
ESMF has many unaddressed bug reports and feature requests, gaps in
testing, and inconsistencies in interface and implementation. These
issues make it harder for users to adopt the framework, and can
hinder and confuse developers. In FY2007 the ESMF team worked to
address these concerns.
Measures of success:
Substantial reduction in the number of open bug reports, support requests,
and untested interfaces and features. Delivery of a release that includes
improvements in consistency in the treatment of basic data types, method
argument conventions, and inter-language interfaces.
Actual performance, outputs, and outcomes:
The ESMF team achieved a substantial reduction in the number of open bug
reports and support requests. As of September 2007, about 30% of open bug
reports were closed, from a high of 229 down to 169. More than half of the
open support requests were closed, from a high of 62 to 15. Resolving
these issues represents a substantial increase in the robustness of the
ESMF software and the ability of users to make progress.
The ESMF team did not reduce the number of feature requests—these continued to come in faster than they can be resolved. The number of open feature requests stands at 86.
To improve usability, this year the ESMF team also improved consistency across the framework by implementing the following:
- Standard treatment of real and integer data types in framework internal code and in method arguments
- A policy and standard to address auto-promotion of data types via compiler directives
- Standard code for initialization of variables across the framework, for consistent behavior across platforms that do/do not support automatic initialization
- Standard error handling across the framework (partly complete)
- A convention for standard treatment of optional arguments in the C interface, contributed by NRL Stennis
New standards were documented in the ESMF Developer's Guide. The first four of these standards were included in FY2007 releases. The last requires significant development effort to propagate C interfaces across the whole framework.
Goal: Development of frequently used utilities.
Means, strategies, and actions:
The implementation of more complex grids will require improvements
to or replacement of the current I/O system. This was scheduled for
FY2006 and not delivered. It was rescheduled for FY2007. Ongoing
improvements are expected in low-level utilities such as the time
manager, configuration attributes, and message logging.
Measures of success:
Delivery of an I/O system to support complex grid implementation.
Delivery of a public ESMF release that includes new features for existing
utilities. These will be prioritized in response to user input.
Actual performance, outputs, and outcomes:
An I/O system was not delivered this year. The ESMF Change Review Board,
which sets project priorities, did not allocate resources for this, and
instead channeled available resources to grid development and
addressing usability issues. The rationale was that I/O capabilities
are secondary in importance to the central concerns of grids and user
progress.
Several utilities in the framework, including LogErr and Config, had minor new features implemented (ability to mask specific errors; ability to turn off Log completely at the ESMF Initialization call; addition of a Config Set method). There was substantial makefile rework this year to improve ease of compiling and linking the ESMF software.
Goal: Framework optimization and porting.
Means, strategies, and actions:
Benchmark component overhead, regridding methods, redistribution, and
middle- and low-level communications on a number of platforms. Include
some evaluations at high processor counts (1,000-5,000 processors).
Measures of success:
Minimal performance burden for working code. (Target <5% overhead
in component overhead and regridding methods.)
Actual performance, outputs, and outcomes:
The ESMF team benchmarked framework component overhead and its
redistribution method on large processor counts. Performance results
are posted on the ESMF website under the Metrics tab. Component overhead
was shown to be small even for high processor counts (∼30 microseconds
at ∼2,000 processors on a Cray XT3). Redistribution results for up to
256 processors showed comparable performance (<5%) between ESMF and
a native package (from the Community Climate System Model) on an IBM
POWER5 and a Cray X1. At larger processor counts the results for
redistribution are still ambiguous. The XT3 that was used to time
performance overhead on ∼2,000 processors was upgraded to an XT3/4.
Multiple processors with different speeds on that machine and hardware
instability have led to a wide range of measurements for time to solution
for the same configuration. ESMF staff will continue to try to obtain
conclusive measurements.
Organizational maturity
Means, strategies, and actions:
Continue to monitor processes and effectiveness of the Change Review
Board and other project bodies. Establish a mechanism for tracking and
reporting sponsorship and progress of application projects adopting ESMF.
The number of these has multiplied quickly, and they have proven
difficult to track.
Promote a paradigm for community development in which stakeholders have direct input into priorities and schedules.
Measures of success:
Broad sponsorship.
Broad representation.
Continuity and regular project meetings.
Actual performance, outputs, and outcomes:
NOAA became a new sponsor of ESMF during FY2007, with the initiation
of the NOAA ESMF-Based Modeling Architecture (NEMA) and a $600K
three-year award. ESMF also received a $1.5M, three-year supplement
from DoD in FY2007 to its Battlespace Environments Institute effort.
The ESMF Change Review Board, the multi-agency group that sets
priorities and schedules, has met regularly with full or close-to-full
attendance. New members have been added to the ESMF Boards representing
COLA, the Air Force Weather Agency, the NASA Land Information System
project, and NCAR. ESMF held a well-attended annual meeting at NCAR in
May-June 2007. A database for tracking ESMF sponsorship, application
projects, and level of adoption was established and customers are being
surveyed about their level of framework use (through email contact)
twice yearly, with results entered into the database.
Project plan evaluation measures for FY2008
Advance community adoption
Means, strategies, and actions:
Continue to focus on development of new, coupled ESMF applications and
the transition of these applications to routine use and operations.
Measures of success:
Number of ESMF components available.
Number of ESMF applications in routine use or operations.
Provide ESMF training and support
Means, strategies, and actions:
Continue development of ESMF tutorials and documentation. Bring
ESMF documentation up to date with new grid development. Improve
presentation of code examples and demos in the ESMF written
documentation and the website. Develop system tests and/or demos
showing a variety of configurations of ESMF applications, including
ensembles and alternative methods of time management and sequencing.
Develop an online survey to assess, on an ongoing basis, user
satisfaction with the ESMF software, documentation, and instructional
materials. Submit a journal paper on the ESMF design or processes.
Measures of success:
Number of training classes. (Goal is two for FY2008.)
Number of individuals trained.
User comments, surveys, and/or evaluations showing that ESMF customers
are satisfied with support, training, and documentation.
Produce a journal paper on ESMF.
Software development
Goal: Further implementation of data structures that represent unstructured grids, grids with curvilinear coordinates, and observational data streams.
Means, strategies, and actions:
Continue rework of structured grids and development of unstructured
grids. The goal for this year, carried over from last year, is to
enable users to represent single and multi-patch structured grids,
to generate regridding weights internally, and to perform regridding
operations using those weights. In addition, the ESMF team plans to
provide a natural representation of unstructured grids using ESMF
data structures. NASA-contributed code that supports manipulation of
observational data streams will be updated so that it works with the
new grid software.
Measures of success:
Delivery of an ESMF release with the capability to represent and
regrid structured grids, to represent unstructured grids, and to
support the manipulation of observational data streams.
Goal: Improve usability.
Means, strategies, and actions:
Reduce the number of outstanding bug reports by half, and achieve 100%
test coverage of public interfaces. The standardization work that began
in FY2007 will continue. Anticipated work will address internal
inter-language interfaces, error handling, public C/C++ interfaces,
and logical data types.
Measures of success:
Reduction by half of outstanding bug reports.
Delivery of a release that further improves consistency of behavior
and interfaces.
Development of C interfaces.
Goal: Develop frequently used utilities.
Means, strategies, and actions:
Delivery of a public ESMF release that includes new features for
existing utilities. These will be prioritized in response to user
input and grid development requirements.
Measures of success:
Ongoing improvements are expected in low-level utilities such as the
time manager, configuration attributes, and message logging. I/O
capabilities will be developed as they are required to support
grid development.
Goal: Framework optimization and porting.
Means, strategies, and actions:
Benchmark regridding methods for a range of processor counts
(1,000-5,000 processors). Ensure that continued framework development
does not degrade redistribution performance or increase component
overhead.
Measures of success:
Minimal performance burden for working code. (Target <5% overhead
for regridding methods vs native packages.)
Organizational maturity
Means, strategies, and actions:
Establish consistency and routine in the management structures and
administration of the ESMF project throughout rapid growth periods.
Work with the operational community to help establish effective
communications among their members and effective communications
between their community and others who are working with and are
invested in ESMF.
Measures of success:
Broad sponsorship.
Broad representation and participation in project bodies.
Continuity and regular project meetings.
Impact of the project
Widespread use of ESMF indicates a paradigm shift in the way weather and climate models are constructed. ESMF is beginning to make model development easier through building model codes using components, increased code interoperability, community building, and standard tools. This approach is also facilitating new, multi-agency science collaborations.
This is producing an Earth science community better equipped to explore basic research issues and to answer questions about the impacts of Earth science on society.
Sponsors
Core ESMF development is sponsored by the National Science Foundation, the National Aeronautics and Space Agency, the National Oceanic and Atmospheric Administration, and the Department of Defense. In addition, a variety of ESMF-based application adoption projects have been sponsored by NASA, NOAA, the U.S. Geological Survey/Department of the Interior, and other agencies and institutions.