Earth System Modeling Framework (ESMF) and ESMF-based Initiatives

	This image illustrates the improvement in predicted sea ice concentrations achieved at the Naval Research Laboratory by taking a standalone sea ice model and coupling it to an ocean model using the Earth System Modeling Framework (ESMF), then comparing the output of the standalone and coupled versions with observations. ESMF, a software package based at NCAR and supported and developed by a multi-agency collaboration, defines standard software interfaces so that models from different research and operational sites can be combined into a variety of integrated systems. The image demonstrates that integrated modeling systems created using ESMF are being used to explore aspects of the Earth system that have far-reaching and profound social consequences.
	This diagram shows the structure of the GEOS-5 atmospheric general circulation model running at the NASA Goddard Global Modeling and Assimilation Office. This model was developed from the ground up using ESMF. Each box is an ESMF component, and every component has a standard interface so that it is swappable. This indicates ESMF's value for a broad range of new modeling challenges: models can be configured and adapted to simulate many different physical processes with increasing ease.

The Earth System Modeling Framework (ESMF) was motivated by the growing complexity of developing Earth system models. These are constructed of separate software components representing physical domains and processes; for example, atmosphere, ocean, and sea ice. The components, which are often developed at separate sites, are coupled together into integrated systems to create realistic simulations. An additional complication is that the models are computationally intensive and must be designed to run on supercomputers with parallel processors. ESMF defines a set of standard software interfaces and a set of high-performance tools for common functions, such as grid interpolation.

ESMF components are arranged in a hierarchical structure to form applications. The lower figure shows an example: it diagrams the architecture of the GEOS-5 AGCM developed at NASA Goddard. This model was built from the ground up using ESMF. Each box, including the couplers, is an ESMF component with a standard interface. The GEOS modeling system can be systematically extended or reconfigured. Exchanges are facilitated both for large, composite components, such as the atmospheric dynamics package, and at the level of a particular physics parameterization.

The ESMF 3 distribution contains:

• Tools for building components and couplers, and a set of low-level utilities for common modeling functions such as regridding, calendar management, and data redistribution
• Concurrent or sequential execution, single or multiple executable modes
• Support for configuring ensembles
• Fortran interfaces and complete documentation, some C++ interfaces
• Support for Linux, IBM, Cray X1, SGI, Mac, and other platforms

Funding and Timeframe

ESMF has transitioned to multi-agency funding support and is the technical basis for an increasing number of modeling projects and programs. These include the DoD Battlespace Environments Institute, the NASA Modeling Analysis and Prediction Program, the NOPP Community Sediment Transport Model, multiple space weather initiatives, and a host of smaller projects.

ESMF is now in its fourth year and has started its second funding cycle. The project has four more years of funding under NASA and Department of Defense sponsorship. ESMF is also supported through NCAR Core funds, which have no term limit. In the next four years, the ESMF group anticipates two more major releases: ESMF 4, focusing on rework and extensions to the grid representation and regridding software, and ESMF 5, focusing on consistency and standardization of interfaces.

Rationale

ESMF is a full realization of the NCAR strategic goal to create a common modeling infrastructure that serves the climate and weather domain and enables the integration of Earth systems with space weather models, hydrological components, and other related domains. It is a key element of NCAR's strategic priority of "Creating an Earth system knowledge environment" that will integrate the suite of scientific software at NCAR to better support science workflows. Looking beyond NCAR, ESMF is a realization of the goals expressed by the broader community to develop cross-institutional and cross-agency infrastructure that enables them to collectively reuse software and exchange components.

Accomplishments in Brief

During FY 2006, the ESMF team made significant advances in community adoption, software development, and organizational maturity.

Community adoption: There are now more than three dozen ESMF components available, representing physical domains such as ocean and land, processes such as radiation and chemistry, and computational functions such as data assimilation. The entire Space Weather Modeling Framework also provides ESMF interfaces to enable interoperation of space and Earth components. Numerous applications built using these components have transitioned to routine use, public distribution, and operations. These include the NASA GEOS-5 atmospheric general circulation model, the NCEP Global Forecast System, and the coupled HYCOM ocean and CICE sea ice models.

Software development: The ESMF team completely redesigned its central data structures for additional flexibility in supporting unstructured, curvilinear, and multi-patch grids. This capability was released in the ESMF 3.0.0 beta version in May 2006.

Organizational maturity: It is essential to the ESMF vision that development priorities are set by the community on an ongoing and regular basis. To this end, the ESMF Change Review Board was created and has met quarterly since August 2005. It consists of representatives from the ESMF customer base who are charged with setting priorities and schedules for the project.

Plans in Brief

In FY 2007 the ESMF team will continue to work toward the transition of prototype components and applications into working research and operational codes. Software development will focus on optimization for very large processor counts, redesign of the framework grid interfaces and implementation, and increasing consistency of behavior. The ESMF team will be working toward integration into the Earth System Knowledge Environment through collaborations with the Earth System Curator, Earth System Grid, and Community Data Portal projects.

Detailed Accomplishments

During FY 2006, the ESMF team made significant advances in community adoption; training, support, and outreach; software development; and organizational development.

Community Adoption Goal

Goal: Advance community adoption
Means, strategies, and actions:
The ESMF development team works with the community to develop and implement adoption plans and to support application groups through the adoption process.

Measures of success:
Number of ESMF components available
Number of ESMF applications in routine use or operations

Actual performance, outputs, and outcomes:
There are now more than 36 ESMF components available, representing physical domains such as ocean and land, processes such as radiation and chemistry, and computational functions such as data assimilation. The entire Space Weather Modeling Framework also provides ESMF interfaces to enable interoperation of space and Earth components. A number of applications built using these components have transitioned to routine use, public distribution, and operations.

There are now three major coupled codes that have put ESMF into extensive and routine use:

• The NASA GEOS-5 atmospheric general circulation model based on ESMF has been in production since FY 2005
• The NCEP Global Forecasting System using ESMF was put into operations on August 22, 2006
• The coupled HYCOM and CICE model using ESMF is now in routine use at the Naval Research Laboratory

Training and Support Goal

Goal: Provide ESMF training and support
Means, strategies, and actions:
Develop and deliver a comprehensive and standard ESMF training class, and continue development of ESMF tutorials and documentation. Training and ongoing support is fundamental to ESMF adoption and the development of the whole-earth environment.

Measures of success:
Number of training classes
Number of individuals trained

Actual performance, outputs, and outcomes:
The ESMF Core Team presented four framework training classes:

• MIT, July 22, 2005 Cambridge, Massachusetts
• Fall/Winter AGU, December 7, 2005 San Francisco, California
• NRL MRY, January 18, 2006 Monterey, CaliforniaA
• AGU Joint Assembly, May 24, 2006 Baltimore, Maryland

Each of these classes contained about 30 students, for a total of 120 students trained.

Tutorial materials evolved over the past year in response to course evaluations and other student feedback. The Core Team found that tutorial students wanted more hands-on programming time and less background on project motivation, history, processes, etc.

A training web site was developed to describe the ESMF training program and distribute training materials. During FY 2006, the ESMF team created and used a set of hands-on exercises to supplement tutorial slides. Substantial new user and developer documentation was added to the ESMF Reference Manual in the area of grids.

Software Development Goals

Goal: Prepare ESMF software for grid generalizations
Means, strategies, and actions:
Consolidate and rework mid-level communications and existing grid constructs to prepare for new grid and related development.

Measures of success:
Delivery of public ESMF release with clear and extensible implementation of data communications at multiple levels, from raw arrays to fields.

Actual performance, outputs, and outcomes:
ESMF developed and released an initial implementation of general, reworked data structures and communications in May 2006 (ESMF Version 3.0.0, "beta" release). This work was the result of an iterative, user-oriented design cycle that began in January 2005. The new data block, decomposition and grid constructs are general and will support a wide variety of grids. An important capability that is included in the Version 3.0.0 release is the ability to perform remappings between grids that have multiple patches and general curvilinear coordinates (e.g. displaced pole, cubed sphere, tripole, and others) or are unstructured. There is currently a limitation that the user must supply an interpolation weight matrix for such remappings by using a package such as SCRIP or custom code. This work represents a major step forward for ESMF, which previously had data block, decomposition, and grid constructs that could only support rectilinear grids. The full implementation of the new data structures will continue in subsequent releases. Additions will include communication methods such as gather, scatter, redistribution, and halo.

Goal: Create an unstructured grid type and associated methods
Means, strategies, and actions:
ESMF usability is currently limited mainly by the restrictions on supported grids and regridding methods. Currently, ESMF supports logically rectangular grids with regular and irregular distributions, and regridding with bilinear and first-order conservative schemes. However, target applications are based on many other grids including unstructured, semi-structured (e.g. cubed spline) and nested. The unstructured grid type may also be used to represent some semi-structured grids. To further extend ESMF usability, we will implement grid specification read/write and interpolation weight read/write, so that applications can use regridding schemes not directly supported by ESMF. Implement grid masks and grid merging.

Measures of success:
Delivery of public ESMF release that includes support for an unstructured grid type, masks and merging, and read/write options for grid specification and interpolation weights.

Actual performance, outputs, and outcomes:
The ESMF Core Team delivered an initial implementation of new data structures that support the regridding of unstructured grids, with the caveat that the user must supply their own interpolation weight matrix (Version 3.0.0). The Core Team did not deliver an implementation of masks and merging, or a read/write option for grid specification. These were pushed off the development timeline by a combination of other tasks taking longer than scheduled, and by the Change Review Board considering these tasks to be lower priority than other items.

Goal: Development of frequently used utilities
Means, strategies, and actions:
The implementation of grid specification read/write and interpolation weight read/write will require extensions of the current I/O system. Modest improvements are also expected in low-level utilities such as time manager, configuration attributes, and message logging.

Measures of success:
Delivery of a public ESMF release that includes new features for existing utilities. These will be prioritized in response to user input.

Actual performance, outputs, and outcomes:
A variety of bug fixes and feature additions were made to ESMF utilities in response to user requests. These are detailed in release notes on the ESMF web site. Changes and additions include the following:

• A suite of non-blocking low-level communication calls
• Performance optimization of the message-logging utility
• Resource files handle logical values, and items in them can be checked for validity and uniqueness
• Clocks and alarms can run in reverse to support data assimilation codes
• Julian calendar support

Goal: Framework optimization and porting
Means, strategies, and actions:
Benchmark regridding methods, redistribution, and middle- and low-level communications on a number of platforms.

Measures of success:
Minimal performance burden for working code (Target <5% overhead in component overhead and regridding methods).

Actual performance, outputs, and outcomes:
The Core Team performed a variety of benchmarking and optimization efforts:

• Redistribution methods on the IBM and X1 platforms. Less than 5% overhead for these methods and platforms is required to satisfy the requirements of the Community Climate System Model (CCSM) as well as BEI codes. The requirements are part of an evaluation of ESMF that the CCSM group is currently performing, but the results and optimizations are important for BEI codes as well.
• Optimization of sends that involve multiple fields for mid-level communications. Here the Core Team implemented a variety of packing and sending strategies for use with different platforms and data arrangements and sizes.
• Implementation of non-blocking sends for low-level communications.

Measurements of total application overhead using ESMF came mainly from application groups.

Results from benchmarking efforts and application groups indicate that the <5% targets are being met.

Organizational Development Goal

Goal: Organizational Maturity
Means, strategies, and actions:
It is essential to the ESMF vision that development priorities are set by the community on an ongoing and regular basis. To this end, the ESMF Change Review Board was created. It is managed by a lead external to NCAR to encourage a broader view, and it is populated by stakeholders from multiple agencies.

Measures of success:
Broad representation
Attendance
Continuity and regular meetings

Actual performance, outputs, and outcomes:
The CRB has met quarterly since August 2005. It consists of about 10 representatives from the ESMF customer base who are charged with setting priorities and schedules for the project. Attendance has been complete for meetings held to date. The ESMF organization has been advanced as an excellent model for community governance by project sponsors.

Project Plan Evaluation Measures for FY 2007

ESMF will continue to be evaluated using categories and metrics outlined for FY 2006.

Advance Community Adoption Goals

Goal: Successfully pass the CCSM-ESMF Stage 1 evaluation plan
Means, strategies, and actions:
Support CCSM through evaluation of the ESMF.

Measures of success, outputs, and outcomes:
CCSM will recognize that ESMF meets the criteria described in the CCSM-ESMF Stage 1 evaluation plan. If successful, CCSM will incorporate ESMF into its software for component representation and data redistribution. Participate in the formulation of the Stage 2 plan.

Goal: Transition HYCOM-CICE to operational platform at NAVO IBM SP5
Means, strategies, and actions:
Successful Alpha and Beta tests were targeted for completion by 4Q FY 2006. Through the Navy's Administrative Modeling Overview Panel (AMOP), the coupled HYCOM-CICE application will undergo OPTEST on the IBM SP5.

Measures of success, outputs, and outcomes:
Successful OPTEST of coupled HYCOM-CICE application (performed by NAVO). Documentation of test cases, installation instructions, and a User Manual for the coupled application will be provided.

Goal: Develop coupled COAMPS-NCOM application
Means, strategies, and actions:
Test and validate two-way coupling between COAMPS atmospheric model and the NCOM ocean model.

Measures of success, outputs, and outcomes:
Successful Beta test of two-way coupled COAMPS-NCOM system by 3Q FY 2007. Test will demonstrate that the forecast properties of the atmospheric boundary layer (e.g., winds, temperature) and of the ocean boundary layer (e.g., currents, temperature) compare to observations as well as, or better than, corresponding one-way coupled simulations.

Goal: Develop coupled COAMPS-SWAN application.
Means, strategies, and actions:
Test coupler to work with both COAMPS and SWAN modeling systems. Identify environmental data (e.g., 10-m winds) that will be exchanged between models. Coupled system will be tested on multiple HPC platforms. System should demonstrate both one- and two-way capability.

Measures of success, outputs, and outcomes:
Successful Alpha test of coupled COAMPS-SWAN system by 3Q FY 2007.

Goal: Develop coupled WASH123D-ADCIRC application.
Means, strategies, and actions:
Coordinate with NRL-SSC and NCAR on EMSF unstructured grid requirements. Develop coupler between WASH123D and ADCIRC. Identify environmental fields to be exchanged (e.g., river discharge, currents, tide levels) between models.

Measures of success, outputs, and outcomes:
Successful Alpha test of pWASH123D-ADCIRC.

Provide ESMF Training and Support Goal

Goal: Provide ESMF training and support
Means, strategies, and actions:
Continue development of ESMF tutorials and documentation. Training and ongoing support is fundamental to ESMF adoption. Improve presentation of introductory and "big-picture" concepts in the ESMF documentation. Develop system tests and/or demos showing a variety of configurations of ESMF applications, including ensembles and alternative methods of time management and sequencing. Update the ESMF Developer's Guide, which outlines conventions and processes used in framework development. It was initially written when ESMF was a NASA project, and parts of it still treat ESMF as a single-sponsor, milestone-driven activity.

Measures of success, outputs, and outcomes:
ESMF training class presented at one or more of next year's AGU meetings. ESMF tutorial student, user, and code contributor comments and evaluations indicate greater satisfaction with training materials and documentation.

Software Development Goals

Goal: Further implementation of data structures that represent unstructured grids, grids with curvilinear coordinates, and observational data streams
Means, strategies, and actions:
Complete the implementation of the specification for classes that represent data blocks and their distribution (array and distributed grid classes). These classes were redesigned in FY 2006 to support unstructured grids, multi-patch curvilinear coordinate grids, and observational data streams. Methods to be implemented include additional communication operations. Rework the grid and field classes to provide better support for the representation and manipulation of these more complex grids. Key abilities to be developed are the ability to read in a grid specification from file and to generate interpolation weights for regridding within ESMF. ESMF can currently only generate interpolation weights internally for single-patch rectilinear grids.

Measures of success, outputs, and outcomes:
Delivery of an ESMF release with a full implementation of the data block and distribution class interface, the ability to read in grid specifications for unstructured and multi-patch curvilinear coordinate grids, and the ability to generate interpolation weights for regridding such grids.

Goal: Improve usability
Means, strategies, and actions:
The implementation of a regridding method that could handle unstructured and multi-patch, curvilinear coordinate grids in FY 2006 removed one of the major barriers to ESMF usability. However, usability issues remain. ESMF has many unaddressed bug reports and feature requests, gaps in testing, and inconsistencies in interface and implementation. These issues make it harder for users to adopt the framework, and can hinder and confuse developers. In FY 2007, the development team will work to address these concerns.

Measures of success, outputs, and outcomes:
Substantial reduction in the number of open bug reports, support requests, and untested interfaces and features. Delivery of a release that includes improvements in consistency in the treatment of basic data types, method argument conventions, and inter-language interfaces.

Goal: Development of frequently used utilities
Means, strategies, and actions:
The implementation of more complex grids will require improvements to or replacement of the current I/O system. This was scheduled for FY 2006 and not delivered, and has been rescheduled for FY 2007. Ongoing improvements are expected in low-level utilities such as the time manager, configuration attributes, and message logging.

Measures of success, outputs, and outcomes:
Delivery of an I/O system to support complex grid implementation. Delivery of a public ESMF release that includes new features for existing utilities. These will be prioritized in response to user input.

Goal: Framework optimization and porting
Means, strategies, and actions:
Benchmark component overhead, regridding methods, redistribution, and middle- and low-level communications on a number of platforms. Include some evaluations at very high processor counts (5,000-10,000 processors).

Measures of success, outputs, and outcomes:
Minimal performance burden for working code (Target <5% overhead in component overhead and regridding methods).

Organizational Development Goal

Goal: Organizational Maturity
Means, strategies, and actions:
Continue to monitor processes and effectiveness of Change Review Board and other project bodies. Establish a mechanism for tracking and reporting sponsorship and progress of application projects adopting ESMF. The number of these has multiplied quickly, and they have proven difficult to track.

Measures of success, outputs, and outcomes:
Broad representation and participation in project bodies
Continuity and regular project meetings
Effective tracking mechanism for ESMF-based application projects

Impact of the Project

Widespread use of ESMF represents a paradigm shift in the way weather and climate models are constructed. Through the composition of modeling codes using components, and increased code interoperability, community building, and standard tools, ESMF is beginning to make model development easier and to facilitate new, multi-agency science collaborations. The end result is an Earth science community better equipped to explore basic research issues and better equipped to answer questions about the impacts of Earth science on society.

Sponsors

Core ESMF development is sponsored by the National Science Foundation, the Department of Defense, and NASA. In addition, a variety of ESMF-based application adoption projects have been sponsored by NASA, NOAA, the USGS/Department of the Interior, and other agencies and institutions.