Overview of the Distributed CSL Plans and Requirements

Version 1.1

March 31, 1995

Objective

The NSF Opportunity Fund Award enables GEO and CISE to pursue our vision of enabling, for the first time, explicit collaborations between the GCRP and the HPCC in the area of distributed, high-performance computing. Specifically, we will explore a potentially powerful climate modeling capability by utilizing the following emerging GCRP and HPCC assets within NSF:

The NCAR Climate Simulation Laboratory (CSL) which is funded by the GCRP and the HPCC;
the NCAR Climate System Model (CSM) which is funded by the GCRP and uses message-passing technology to communicate fluxes between major model components that are running concurrently on either the same computer or on several computers linked by high speed communications;
the NSF experimental Very High-Speed Backbone Network Service (vBNS) which is funded by HPCC;
one of the CISE supercomputer centers which are funded by the HPCC;
the National Laboratory for Applied Network Research (NLANR) which is funded by CISE and which can contribute to the understanding of the performance of the vBNS in the context of the DCSL proposal (see Appendix II); and
other GEO/CISE assets such as mass storage and visualization systems.

We propose to distribute the NCAR Climate System Model (CSM) between the NCAR CSL and one CISE center. We will refer to this as the Distributed Climate System Model (DCSM). All model output will be transmitted to NCAR and stored on a high-performance shared filesystem. Our primary objective will be to evaluate the potential of using all or most of the NSF metacenter high performance systems to provide the amount of computing power required to undertake climate simulations that are simply impossible on any single computing system. We plan to demonstrate this potential by carrying out a four phase project that builds on the aforementioned assets and culminates with "real science" climate simulations consuming roughly 300 Y-MP-1 hours of computing resources. These experiments will require the storage and distributed visualization of the associated model output data. The critical technological milestones that must be accomplished to realize this objective are:

to assess vBNS performance for model component intercommunications;
to assess vBNS stability for production runs;
to assess vBNS performance in support of transmission of model output data to NCAR;
to assess the performance of a high-speed shared file system relative to the task of storing large model output datasets and making them available for post-processing, visualization, and remote access.
to develop and demonstrate high-performance distributed visualization of model results;
to develop job management software to temporarily create a meta-computer which can run reliably and predictably in a production environment.

Technical Overview

The DCSM consists of four principle model components: an atmospheric model, a global ocean model, a sea-ice model, and the flux coupler. The coupling of the component models in DCSM (ocean,atmosphere,sea-ice) is achieved through the exchange of flux information via PVM. This exchange is mediated via the flux coupler. The flux coupler performs the interpolations between different grids and performs the flux computations itself. Because of the physical time scales in the problem, the flux coupler communicates with the atmosphere much more often than with the ocean. For this reason current vBNS bandwidth constraints will require the flux coupler and the atmospheric model to run tightly coupled at one site, and the ocean model at the other.

Good machine-to-machine communication will be needed to support flux exchange via PVM over the vBNS. While it is desirable for the coupled machines to both integrate forward in model time at the same wall clock rate and thus achieve perfect load balance, we recognize that this will be difficult to achieve in practice. We expect the most numerically intensive model components to determine the rate at which the overall simulation proceeds and we refer to the system upon which they run as the "pacing machine". The model components on the non-pacing machine will spend some part of their execution time waiting to receive data from the pacing machine. In all cases, the pacing machine will be at NCAR. We will tune the amount of computing resources allocated to the non-pacing machine so as to achieve as close to overall load balance as possible. We will also need to transport large volumes of model output back to the high-performance shared filesystem at NCAR, via the vBNS. While this is a "background" process, there may be model and system tuning issues involved in order to avoid overflowing the data buffers at both sites. See Appendix I for a detailed description of the DCSM model components and their characteristics.

Figure 1. - Conceptual diagram of project

Distributed climate modeling introduces a number of difficulties into the traditional visualization process, most of which center around the size and number of the datasets, and their wide-area distribution on the network. As processing power continues to grow and outstrip network bandwidth, visualization processes may necessarily become distributed: instead of moving data around we instead calculate imagery, geometry, and compressed movies close to the data, transmit the information over the net and then visualize it. The visualization environment may need to construct visualizations from multiple data sources.

The overall objective of the visualization component of the DCSL is to develop and demonstrate interactive visualization and analysis of distributed very-large (.1TB) atmospheric and oceanic datasets using the compute/storage/visualization resources at two centers and the vBNS as a high-speed network interconnect. Visualization techniques will range from browsing functions to state-of-the-art environments for visualizing atmospheric and related phenomena. Focus areas will include:

Practical and effective 2D/3D visualization and analysis for very large distributed datasets, focusing on browsing capability, data management, and general tool integration.
Time-evolving 3D visualizations of both structure and flow.
Distributed visualization where data, imagery, movies, and/or geometry are computed at remote sites (close to the relevant datastores), shared via the vBNS, and finally integrated as composite visualizations of multiple model components. Also, distributed filesystems.
Demonstration of advanced visualization environments (CAVE, NII Testbed) for atmospheric and oceanic data.

Schedules permitting, first results of this collaborative effort will be demonstrated at SuperComputing 95 in San Diego at the NSF Metacenter Booth.

Appendix I: Descriptions of DCSM Model Components

1. Flux Coupler

The flux coupler performs the interpolations between different model grids and computes the fluxes that couple the model components together. The flux coupler passes these fluxes between the model components using the PVM message passing library, communicating much more often with the atmosphere than with the ocean or sea ice models. The following machine resource estimates assume that the atmospheric component is running at T42 (2.5 degree) horizontal resolution and that the ocean and sea ice component is running at a 360x180 (1 degree) resolution.

a) Core Memory Required: 5.3 Megawords (42.4 Mbytes)
b) Performance: 21 seconds/model-day on 1 Y-MP processor.
c) Data Archiving Requirement: Not applicable
d) Portability: Flux coupler runs in single processor mode on Cray Y-MP and on Sun and IBM workstations.
e) Parallelism: Flux coupler has no multiprocessing capability.

2. DCSM Candidate Ocean Models

The following machine resource estimates assume that the candidate ocean model component is running at 1 degree resolution (360 x 180 horizontal grid).

a) Core Memory Required: 49 Megawords (390 MBytes)
b) Performance: 110 seconds/model-day on 1 Y-MP head.
c) Data Archiving Requirement: The worst case data archiving estimate for the ocean model is saving 6 state variables every model month with no data packing.; History file size is 68MBytes.; Estimated archiving rate is .020 Mbytes/sec.
e) Portability: The POP candidate ocean model is written in Fortran-90, and runs on the CM-5, T3D, IBM and Cray Y-MP systems. The CSM ocean model, MOM, is written in Fortran-77 and runs on SUN and IBM workstations and on the Cray Y-MP.
f) Parallelism: The POP candidate ocean model runs on a single processor of the Cray Y-MP and SUN and IBM workstations and has a multiprocessor capability on the CM-5 and T3D MPP systems. POP could easily be ported to other MPP's using PVM. The current CSM ocean component model, MOM, runs on multiple processors of the Cray Y-MP.

2. Sea Ice Model

The following machine resource estimates assume that the DCSM sea ice model component is running at 1 degree resolution (360 x 180 horizontal grid).

a) Core Memory Required: 12 Megawords (96 MBytes)
b) Performance: 16 seconds/model-day on 1 Y-MP head.
c) Data Archiving Requirement: The worst case data archiving estimate for the sea-ice model is saving every model month with no data packing.; History file size is 12 MBytes.; Estimated archiving rate is .005 Mbytes/sec.
e) Portability: The sea ice model is written in Fortran-77, and runs only on a single processor of a Cray Y-MP system.
f) Parallelism: The sea ice model does not currently run in parallel on any platform.

4. CSM Atmospheric Model

The following machine resource estimates assume that the atmospheric component is running at T42L18 (128 x 64 horizontal grid) with 18 levels.

a) Core Memory Required: 20 Megawords (160 Mbytes)
b) Performance (seconds/model-day on 1 Y-MP processor): 300 sec/model-day
c) Data Archiving Requirement: Worst case is full daily atmospheric history files without packing; The CSM atmospheric model history file size is 30 Mbytes.; Estimated Sustained Archiving Rates (T42L18) 0.1 Mbytes/sec for the atmospheric component running on 1 Y-MP processor.
d) Portability: The atmospheric component is written in Fortran 77. The message passing version uses PVM.
e) Parallelism: The CSM atmospheric model's multiprocessor capability has been tested on the Cray Y-MP, Cray-3, T3D, and clusters of Sun and IBM workstations. The PVM version of the CSM atmospheric component supports the use of up to 64 processors. The atmospheric model also runs on single processors of SUN and IBM workstations.

Appendix II: The National Library for Applied Network Research (NLANR)

The National Laboratory for Applied Network Research (NLANR) has as one of its research roles:

"... a Metacomputer Interconnect Laboratory to develop advanced distributed applications and to transition advanced capabilities, such as those developed in gigabit testbeds, toward robust services delivered on the vBNS and on the larger Internet as bandwidth increases over time."

@(#)index.html 1.3 95/11/29 don@ncar.ucar.edu