What's Happening Abroad in Computing Systems at Some Major
Forecast and Climate Modeling Centers

by
Bill Buzbee, Ph.D.
Director, Scientific Computing Division
National Center for Atmospheric Research
February 6, 1998

I. Introduction

The National Center for Atmospheric Research (NCAR) and the community it serves currently enjoy world leadership in several areas of atmospheric sciences research that depend on high performance computing. In order to maintain this leadership, NCAR must have computing capabilities that are comparable to peer organizations throughout the world. The most powerful computer that NCAR has today is the Cray C90/16 and NCAR will soon install a 128 processor Distributed Shared Memory (DSM) microprocessor system. Neither of these systems will sustain more than 5 Gflops on a single application. However, NCAR's peer centers in Australia, Canada, England, and elsewhere, are installing systems that by January '98 will sustain from 20-100 Gflops on a single application. With these systems, they can, and they are, conducting research that is far beyond the ability of their U.S. counterparts.

Section one of this paper summarizes the computing capabilities of a small number of forecast and climate modeling centers around the world. Sections two and three discuss future plans at some of these centers. Section four summarizes computing capability at a small number of universities in Japan and Europe. Section five discusses the impact on U.S. atmospheric science. Overall, this paper shows that earth systems modelers outside of the U.S. have a substantial computational advantage over their U.S. colleagues and are likely to enjoy such for several years.

II. Systems Currently Installed

Table 1 lists some of NCAR's peer organizations and their associated computing systems that are capable of sustaining 20-100 Gigaflops on a single application.
 
Table 1: What's Happening Abroad?

 
 
Center
System
# of Processors
Capability
Gflops
ECMWF
Fujitsu/VPP
116
80 - 100
Canada
NEC/SX-4
64
40 - 50
UK Met
Cray T3E
700
~ 35
France
Fujitsu/VPP
26
20
Denmark
NEC/SX4
16
12
US GFDL
Cray T90
26
15
Australia
NEC/SX-4
32
20 - 25
In 1995, the European Center for Medium Range Weather Forecasting (ECMWF) selected the Fujitsu Vector Parallel Processor (VPP) system via competitive procurement. As of August 1997, the system has 116 processors, each of which sustains about 0.75 Gflops, giving the possibility of sustaining 80-100 Gflops on a single application. ECMWF is using the VPP to run the climate version of their forecast model (used in seasonal forecasts) at T63L50 resolution [1]. In contrast, the NCAR Community Climate Model, Version 3 (CCM3), is typically run at T42L18. To move the UCAR CCM3 to a configuration similar to that being used at ECMWF would require a machine that can sustain at least 20 Gflops.

The Canadian Meteorological Center (CMC) has a 32 processor NEC SX-4. The CMC set a milestone recently by completing a 24-hour forecast over North America at 10-km resolution in about forty minutes of wallclock time [2]. CMC was able to do this because the SX-4 sustains about 24 Gflops when executing the MC2 forecast model, thus CMC plans to reduce its operational forecast grid size to 10-15 km [3]. By January of 1998, CMC will have two SX-4/32s and by January of 2000 they will have four SX-4/32s that can be clustered into a single 128 processor system via NEC's fiber optic Internode Crossbar Switch [4] giving them an 80-100 Gflop capability. These machines will also be used for climate modeling [3].

In the spring of 1996, the UK Meteorological Office (UK Met) selected the Cray T3E with 696 processors but has not yet put it into operational use. They plan to dedicate 144 processors to the global operational forecast and 144 to the regional forecast. The remaining 408 processors are to be used for research, including climate modeling [5]. This equipment is also used by the Hadley Centre.

Meteo-France has selected the Fujitsu VPP and currently has a system with 26 processors capable of sustaining 20 Gflops on a single model.

The Danish Meteorological Institute has two NEC SX-4s, one with sixteen processors and one with four. The sixteen processor system sustains approximately 12 Gflops. Twenty percent of the wallclock time on this machine is used for forecasting, the remaining eighty-percent and the four processor system are used for research including climate modeling [6].

The most powerful system in the U.S. that is used for climate modeling is a Cray T90 with twenty-six processors at the Geophysical Fluid Dynamics Laboratory (GFDL) in Princeton, New Jersey. A single processor of the T90 sustains about 0.6 Gflops when executing the NCAR CCM; thus the GFDL machine is capable of approximately 15 Gflops.

The Australian Bureau of Meteorology has selected the NEC SX-4 [7]. The current system has sixteen processors, but will be upgraded to thirty-two processors in February 1998. A second SX-4 with twenty processors will be acquired in the third quarter of 1999. The two systems will be clustered via NEC's Internode Crossbar Switch, thus giving a 30-40 Gflop capability.

III. Future Developments Abroad

By 1999 the next generation of Japanese vector systems will probably be available with processors that may be more than twice as fast as the current generation. If so, it will be possible to sustain 80-100 Gflops with fewer than 50 processors and, obviously, implementing and managing models over 30-50 processors is much easier than over hundreds of processors.

The Japanese Science and Technology Agency has established an "Earth Simulator" project [8]. The project was launched in April 1997 with funding of approximately $400 million over five years. The project includes development of a high performance parallel computer with a sustained performance of one or more Teraflops by 2001. This system will be provided by either NEC or Fujitsu. For example, if the next generation Fujitsu VPP has a sustained performance of 2-3 Gflops per processor, then a few hundred of these processors could sustain one or more Teraflops.

IV. A Sample of Computing Systems in Universities Abroad

The National Science Foundation provides university scientists, including atmospheric scientists, with access to high performance computers. The most powerful computer supported by NSF is a 7.5 Gflop (twelve processor) Cray T90 located at San Diego. In contrast, the University of Stuttgart, the Swiss Center for Scientific Computing, and Osaka University have large SX-4s. The University of Tokyo, Nagoya University, and Kyushu University all have Fujitsu VPPs with at least forty processors. Thus, all of these universities have systems that are capable of 20 Gflops or more.

V. Impact on U.S. Atmospheric Science

U.S. atmospheric science modelers currently enjoy global leadership in several areas of research that depend on high performance computers. To maintain that leadership, they need computing capabilities that are comparable to their international peers. For example, a 1-km regional forecast using 4DVAR with full physics adjoint is feasible, but to use such in time critical (less than one hour) forecasting will probably require a machine that can sustain at least 50 Gflops [9]. Another example is a recently developed NCAR global chemistry model (MOZART) -- in order to complete 100-year simulations of the climate within a reasonable timeframe, this model needs a computer that can sustain 20 to 40 Gigaflops [10].

The situation is particularly acute in climate modeling and is exemplified by the computational requirements of the NCAR Coupled System Model (CSM). Now that the CSM project has successfully completed a 350-year control run, there are two major studies that it would like to undertake:

  1. simulate the past 120 years of climate under at least six scenarios and four sensitivity studies per scenario, and
  2. simulate the next 200 years of climate under at least three scenarios and four sensitivity studies per scenario.

    3. The total years to be simulated in 1) and 2) is 5280. At present, the flagship computer of the NCAR Climate Simulation Laboratory (CSL) is a Cray C90 that sustains 5 Gflops and that serves nine USGCRP projects including the CSM. On average, the CSM project can complete 100 years/month using the CSL C90. Thus, to complete 1) and 2) would require more than four calendar years, which is unacceptable relative to progress being made by our international peers.

    The CSM project also plans future improvements to the model such as semi-Lagrangian dynamics, prediction of cloud water, and a sulfate aerosol model. These improvements are expected to quadruple the amount of computation required per simulated year. Thus, a 20 Gflop machine will be required to maintain the current average of 100 years/month.

    For ease of reference, we denote 1) and 2) as Part A of the CSM science plan. Similarly, we denote development and execution of the next generation of CSM as Part B of the CSM science plan.

    Now that the U.S. Department of Commerce has issued an antidumping order against Japanese high performance computers, NCAR plans to continue operating the CSL C90 in FY98-99 and to install a 128 microprocessor, Distributed Shared Memory (DSM) system in the CSL in mid-FY98. Based on measured performance of two leading-edge 128 processor DSM systems executing the NCAR CCM (Community Climate Model) and POP (Parallel Ocean Model), we estimate that 128 processor DSMs will sustain about 5.0 Gflops on the CSM by mid-FY98. If so, then Part A of the CSM science plan can probably be completed by end of FY99. However, we believe that it will be FY99-00 before 256 processor DSMs can approach 20 Gflops. Thus, the following are not possible in the near term:

    VI. Summary

    Meteorological organizations outside the U.S. either have or soon will have computing systems that can sustain 20-100 Gflops on climate simulations, high resolution forecasts, etc. With these systems, they can and they are conducting research that is far beyond the ability of their U.S. counterparts.

    The bottom line -- earth systems modelers outside the U.S. have a substantial computational advantage over their U.S. colleagues and are likely to enjoy such for several years.

    1. James Hack, NCAR CGD Division, personal communication, November 1997.
    2. NEC SX-4'S REAL-TIME 24HR 10KM RES FORECAST DETAILED, HPCwire, August 1, 1997.
    3. CMC PLANS TO REDUCE FORECASTING GRID SIZE WITH NEC SX-4, HPCWIRE, November 20, 1997.
    4. CMC UPGRADES NEC SX-4 TO IMPROVE FORECASTING, HPCWIRE, November 21, 1997.
    5. Dr. Paul Cluley, UK Met, personal communication, November 1997.
    6. Dr. Leif Laursen, Danish Meteorological Institute, "Technical Advances in Short Range Weather Forecasting", RCI European Member Management Symposium X, Rome, Italy, October 1997.
    7. AUSTRALIAN METEOROLOGISTS, RESEARCHERS TO RECEIVE NEC SX-4, HPCwire, July 25, 1997.
    8. Hisashi Nakamura, Director of Research for Computational Earth Science, Research Organization for Information Science and Technology, Tokyo, personal communication, November 1997.
    9. Bill Kuo, NCAR MMM Division, personal communication, November 1997.
    10. Stacy Walters, NCAR ACD Division, personal communication, November 1997.  

      11. February 6, 1998