Go to SCD News table of contents

SCD News > Feature: July 15, 2004

SCD takes delivery on new Linux supercomputer

"Lightning" is 30-40% faster per processor than bluesky

Lightning, the new Linux cluster

"We expect this system to be a big hit, but we must prove it to ourselves and our users."
— SCD director Al Kellie

Slide show

On 12 July 2004, SCD took delivery of "lightning," a large-scale, high-performance Linux cluster manufactured by IBM. (See photos, below.) The acquisition is part of SCD's new five-year Strategic Plan to evaluate new technologies and find ways for SCD to deliver more cost-effective tools for advancing the frontiers of science at NCAR.

The 1.1-teraflop system is considerably faster on a per-processor basis than bluesky, NCAR's IBM Cluster 1600. In benchmark tests, the Community Atmosphere Model (CAM) tests ran 30% faster, while the Parallel Ocean Program (POP) ran 40% faster. One reason for this is that lightning's Opteron processors, built by Advanced MicroDevices (AMD), have much better memory bandwidth than bluesky's Power4 processors.

"I believe the time is right for NCAR to evaluate the applicability of commodity-based Linux clusters for their readiness in an enterprise computing environment like SCD's," says SCD director Al Kellie. "The press is full of success stories, but very few of them stress the operating environment, equipment performance, and reliability as much as we do. We expect this system to be a big hit, but we must prove it to ourselves and our users."

Many of NCAR's university partners who want to use models such as the Community Climate System Model (CCSM) or the Weather Research and Forecast (WRF) model have Linux-based systems. Lightning will give them, for the first time, the chance to build, test, and evaluate these codes in a full-scale Linux computing environment similar to their own.

Dick Valent, assistant manager of SCD's User Support Section, concurs. "This system will give us valuable experience with the new type of Linux platforms that are going to be showing up at more and more places."

Mesa Lab loading dockMoving a cluster frame onto the dockMcLaughlin and New help move a frameAndersen puts down plywood to protect floorSCD staff watches frames roll through hall

Fuentes checks on progressMoving a frame into the Computer RoomCaravan of frames through Computer RoomIndicating position of first frameMoving a frameLooking at space for Linux cluster

Monitoring system operationsJockeying first frame into placeMoving next frame into placeAll five frames are in positionFuentes looks inside a frame

IBM starts hooking up the clusterInside a nodeInternal configuration diagramIBM engineers at workGombosi prepares ATP

Delivery and configuration

Lightning was assembled and tested by IBM in its facility in Guadalajara, Mexico. It was shipped to Boulder by truck and arrived at NCAR's Mesa Laboratory the morning of Monday, 12 July. SCD and IBM staff worked with Eagle Global Logistics, the moving company, to unload the five frames (total weight: 9,870 pounds) and wheel them through the halls to the Computing Room.

Once the frames were in place, IBM engineers began hooking up nodes, attaching wiring, and connecting power. SCD and IBM are currently working on system configuration and software installation, monitoring the equipment as it runs to check vital signs such as temperature and air flow.

In the days to come, Jim Tuccillo is slated to run benchmark tests on behalf of IBM, ensuring that lightning meets performance specifications. Rory Kelly of SCD's Computational Science Section will then rerun the benchmarks on behalf of NCAR. Once the benchmarks have been validated by both IBM and NCAR, Steve Gombosi of SCD's User Support Section will begin acceptance testing, running codes 24 hours a day for a week to subject the machine to heavy loads and check for correct answers. If the machine is accepted, friendly-user testing will follow.

Lightning is expected to be made available to the larger user community sometime this Fall.

A divisional effort

Project manager Georges Fuentes, head of SCD's Supercomputer Systems Group, notes that the installation and support of the new Linux cluster is a divisional effort involving the hard work and contributions of many SCD staff. Core project team members and the sections they represent are:

  • Director's Office: Tom Bettge, Tom Engel, Rich Loft
  • Computational Science: Rory Kelly, Henry Tufo
  • High-Performance Systems: George Fuentes, Mark Genty, Gene Harano, Michael Oberg
  • Network Engineering and Technologies: Marla Meehl, David Mitchell
  • Operations and Infrastructure Support: Aaron Anderson, Gary New
  • User Support: Ginger Caldwell, Bo Connell, Dick Valent, Steve Gombosi

Lynda Lester

SCD News   ||  UCAR  ||  NCAR   ||   SCD   ||   Contact us   ||  Search
NCAR is managed by UCAR and sponsored by the National Science Foundation