Recent computing capacity upgrades

Due to the recent installation of two new supercomputers, the computing systems in NCAR's supercomputing center are sustaining about 10 billion floating-point operations per second (10 GFLOPS) as they run models that simulate atmospheric, oceanic, and other geophysical processes. This is more than 100 times the computing power that NCAR provided to researchers in 1985.

Back then, NCAR's Scientific Computing Division (SCD) operated two Cray 1-A computers that represented that era's state of the art in supercomputing. When added together, their average productivity was about 7/10ths of that of a single Cray Y-MP processor (a standard SCD uses to compare computer performance). Today, SCD operates six Cray computers that average about 80 times the productivity of a Y-MP processor. This article describes the supercomputers in the national center, the process through which the two new machines were acquired, and the other major components of the supercomputing center.

By definition, supercomputers are the most powerful computing systems available, and they are used primarily to solve problems that cannot be handled by any other class of equipment. As shown in the functional diagram of the NCAR computing facility, the six supercomputers at NCAR are divided into two groups: community computing and dedicated climate simulation.

The long road to acquiring new computing power

Projects using the Climate Simulation Laboratory (CSL), such as the NCAR Coupled Climate Model Project, needed a machine that could sustain at least 5 GFLOPS by October 1996. Also, by October 1998, CSL users will need at least one machine that can sustain approximately 20 GFLOPS to decrease the completion time of centuries-long climate simulations, incorporate more physical processes into the models, and eventually increase the models' spatial resolution.

In 1995, NCAR initiated a procurement effort to fulfill the needs of the Climate Simulation Laboratory and upgrade community computing power. In the spring of 1996, after extensive performance testing of the computing systems that qualified under the procurement's request for proposals, UCAR announced that equipment from NEC was selected. Of the machines that NCAR has tested, the NEC SX-4 is the only machine that has demonstrated the ability to sustain approximately 20 GFLOPS on the class of simulations that will be addressed by CSL models in 1998 and beyond.

The NEC computers were to have been installed in phases to meet CSL and community needs. In the first phase, a 32-processor NEC SX-4 would have been used to increase community computing resources while fulfilling the CSL's 5 GFLOPS need. In October 1997 a second 32-processor SX-4 would have been installed, allowing one SX-4 to be dedicated to the CSL and the other to the community. In October 1998 a third and fourth 32-processor SX-4 would have been installed to provide the CSL and the community with two SX-4s each. The aggregate sustained power of these four NEC computers would have been 80 GFLOPS, and the projected cost was within the budget for the procurement.

However, acquisition of an SX-4 has been delayed due to a dumping investigation in which it was claimed that the cost to manufacture and market the NEC SX-4 significantly exceeded the price offered to NCAR, and that this could cause material injury to a U.S. industry. The claim is being investigated, and the situation is not expected to be resolved until well into 1997. Consequently, in the fall of 1996, NCAR upgraded the Cray Y-MP in the CSL to a Cray C90. The C90 provides the 5 GFLOPS capability needed by the Coupled Climate Model and other CSL models in 1997.

SCD's Fiscal Year 1997 budget includes over $1 million of funding for an upgrade to community resources. Had SCD been successful in acquiring the SX-4 in October 1996, the community would have been allocated about 12% of that machine. After the procurement was delayed, SCD discussed several options with its advisory panel for using that funding to provide community users with an interim upgrade. The result was a recommendation to acquire a Cray J90se with 20 processors and 1 billion words of memory.

If the dumping investigation does not find inappropriate pricing, then NCAR will renegotiate the contract for the SX-4. If NSF approves the new contract, then installation of an SX-4 would likely occur early in Fiscal Year 1998.

Overview of the Climate Simulation Laboratory

This group of equipment supports a small number of activities that need large amounts of computing, such as coupled climate systems models. This equipment is used by the U.S. Global Change Research Program (USGCRP) to support the multi-agency Climate Simulation Laboratory (CSL). The CSL completed its first year of operation on July 31, 1996, and during this first year nearly 100,000 equivalent Y-MP hours were delivered to 11 projects selected by the CSL Allocations Panel. The 11 projects have used this computer time to carry out multidecadal simulations and to develop and/or evaluate components for use in coupled systems models.

One of NCAR's two new supercomputers, a Cray C90 named antero, is dedicated to the CSL. This system replaces one of NCAR's Cray Y-MPs that had also been named antero. The complete set of CSL computers provides about 7 GFLOPS of sustained computing power:

• antero, a Cray C90 with 16 processors, averages approximately 5 GFLOPS
• aztec, a Cray J90 with 20 processors, averages approximately 1.25 GFLOPS
• the Cray T3D, a massively parallel processing system with 64 processing elements, averages approximately 0.65 GFLOPS

The Cray C90 named antero arrived at NCAR during November 1996 and went into production a few weeks later. With 16 processors and 256 megawords of memory plus 1 gigaword of solid-state memory, it provides better than a factor of 4 increase in computing power over the former antero (an 8-processor Cray Y-MP with 64 megawords of memory plus 256 megawords of solid-state memory). "On the new antero, the Climate System Model is simulating five years per day of wall-clock time instead of one year per day on the former antero," notes NCAR researcher Byron Boville. The Climate System Model calculates physical processes in the atmosphere, ocean, land, and sea ice to simulate the evolution of climate. The current run of this model started on August 1, 1996 (on the Y-MP antero).

Overview of community computing

The community computing resources provide reliable, general-purpose computing power to a wide range of users. This equipment is used by university scientists and NCAR scientists to address questions in the following areas:
• Climate
• Mesoscale and microscale meteorology
• Oceanography
• Chemistry
• Upper atmosphere processes
• Basic fluid dynamics
• Cloud physics
• Astrophysics

NCAR's other new supercomputer, a Cray J90se named ouray, is dedicated to community computing. The complete set of community computers provides about 3 GFLOPS of sustained computing power:

• shavano, a Cray Y-MP with 8 processors, averages 0.9 to 1.2 GFLOPS
• ouray, a Cray J90 with 20 processors, is expected to at least match and probably exceed shavano's performance
• paiute, a Cray J90 with 16 processors averages 0.8 to 1.1 GFLOPS

With 20 processors and 1 gigaword of memory, the Cray J90se named ouray provides the large memory needed by many models. Its large memory allows community computing users to run much larger jobs than they can now run on shavano. The addition of ouray significantly enhances the power and flexibility of the community computing center.

Other critical components of the center

There is a lot more to NCAR's computing facility than the computing engines. The five major functional components include high-performance computers, mass storage system, local and wide area networks, distributed computing services, and visualization laboratories.

Data storage

SCD develops and manages an archival mass storage facility that is balanced in performance and capacity relative to the computational resources. This 86-terabyte data storage facility has to keep growing to keep pace with the data being generated by the increasingly powerful computers. Its current growth rate exceeds 1.5 trillion bytes (terabytes) per month. In March 1997, SCD will make available a new robotic storage device from StorageTek that will store up to 300 terabytes of data when it is eventually filled with tape cartridges. The system uses helical-scan technology to read and write tapes, with each tape holding up to 50 billion bytes of data. With the addition of this storage device to NCAR's Mass Storage System, users will be able to read and write files of up to 6 billion bytes in size.

Networks

To move all of those data to and from the computers, SCD also designs and maintains high-speed network and data communication capabilities that are balanced with respect to the computational facilities, storage facilities, and the needs of a national and international user community. For most scientists, desktop computers and/or departmental computers provide the primary means of software development and analysis of model output. SCD's network connections allow these distributed systems to communicate with the supercomputers and the Mass Storage System.

Distributed computing services

By isolating functions to special servers, each server can be optimized to the function it provides, and changes can be made with minimal disruption to other servers. Functions that SCD has distributed to special servers include job submission to supercomputers, weather data ingestion and distribution, user authentication, electronic mail, and World Wide Web services.

Visualization laboratories

As scientists execute increasingly complex and high-resolution 3D simulations, high-performance exploratory visualization and animation can play an important role in understanding the resulting data. SCD operates two visualization labs to address this area. The Foothills Visualization Lab is a self-service production facility used directly by scientists and their support staff to produce and record animations of their model output. The Mesa Visualization Lab is a state-of-the-art complex supporting R&D in the areas of visualization of very large datasets, volume visualization, distributed visualization, digital media development, High-Definition Television (HDTV), advanced visualization environments including virtual reality technologies, and the application of advanced networking technologies.

For more information

SCD provides a wealth of information about the NCAR supercomputing center via the World Wide Web. SCD also reports its yearly progress to the National Science Foundation with an online Annual Scientific Report. Here are some of the websites that you can visit to learn more about computing at NCAR.
• SCD home page

• Information about the supercomputers

• SCD's services for computer users

• NCAR's data archive

• NCAR's mathematical and statistical libraries

• NCAR Graphics scientific visualization software

• SCD's Visualization Laboratories

• Report on SCD's networking progress during 1996

• Report on SCD's computing, mass storage, and distributed services progress during 1996

• SCD's 1996 Annual Scientific Report

• NCAR's 1996 Annual Scientific Report