SCD News: September 19, 2005
 |
Gary New, assistant manager of SCD's Enterprise Services Section, gestures toward the new, improved cooling tower, which dissipates heat from the NCAR Mesa Laboratory's Computer Room. More |
This month, SCD completed a massive upgrade of the Computer Room at NCAR's Mesa Laboratory. See the photo gallery
Welders install new chilled-water pipes during the upgrade of the NCAR Mesa Lab's Computer Room. More photos |
The upgrade will make it possible for the computing facility, designed 40 years ago for machines that needed minimal power and cooling by today's standards, to support the latest generation of NCAR supercomputers. These supercomputers, built of thousands of processors that run blazingly fast and hot, include bluesky, bluevista, frost, lightning, tempest, pegasus, coral, and a yet-to-be-determined system that will arrive in 2007.
From planning to completion, the upgrade of the Computer Room infrastructure was a three-phase process that started in 1997.
The first two phases saw the installation of new uninterruptible power supply (UPS) units (2001) and standby generators (2004), which provide power to NCAR supercomputers in the event of electrical outages. The final phase involved an extensive redesign and rebuilding of the Computer Room's chilled-water cooling system. (For a closer look at the system, see "Behind the scenes: Chillers, pumps, and pipes.")
“There hasn't been an upgrade this big since 1986, when 4,000 square feet were added to the computing facility.” says Aaron Andersen, manager of SCD's Enterprise Services Section (ESS).
The driving force behind the upgrade was the NCAR research community's voracious appetite for high-end computing. In the last decade, computer simulations have become just as important as theory and experiment in the atmospheric and related sciences. Each year, models run at higher resolutions and grow more complex, requiring more compute cycles.
Modern supercomputers meet this ever-increasing demand for compute cycles by using faster processor clocks, more sophisticated CPUs, larger data caches, and inexpensive commodity parts.
But as supercomputers get faster, better, and cheaper, they also use more electricity. Each time a transistor flips from one to zero or back again, it consumes energy. The faster these flips occur, the more transistors a CPU contains, and the more densely CPUs are packed into cabinets, the more energy is consumed. For cluster systems such as NCAR uses, power usage has been increasing dramatically. Only five years ago, the consumption of 4 kilowatts by a single processing cabinet was considered high. Today, 20 kilowatts is common. (Indeed, NCAR's newest supercomputer, bluevista, needs 276 kilowatts of power to operate, compared to the 10.5 kilowatts used by the average U.S. home.)
And, notes Aaron, as supercomputers use more electricity, they get hotter. “There's a saying in the data center business: Power in, heat out. For every kilowatt that goes into a computing system, almost an equivalent kilowatt is needed to remove the waste heat.”
One look at the temperature sensors on bluesky, NCAR's lead supercomputer since 2002, shows that it runs hot. The air that cools it is 54 degrees when it enters and 100 degrees when it exits, just a few seconds later.
The trend toward power-hungry, hotter computers means that the electrical and cooling infrastructure is now a dominant factor in operating NCAR's Computer Room.
Power outages shorten the life of high-end equipment, while a cooling failure can result in non-operational temperatures and permanent damage in just seven to ten minutes. Such outages and failures adversely impact computational support for field projects, real-time weather forecasts, and important climate runs.
By providing stable and dependable electrical power and cooling for NCAR supercomputers, the Computer Room upgrade supports scientific advances by NCAR's international research community.
A crane lowers a new chiller into the Mesa Lab. More photos |
While renovation of the computing facility's electrical infrastructure was completed last year, renovation of the cooling infrastructure began in August 2003.
“We felt it was time to redo the entire chilled-water system,” says Gary New, ESS assistant manager. “The old system was a lot more complex than it needed to be. We wanted to increase its capacity and decrease its complexity in an effort to arrive at a more reliable system.”
After specifications and designs were approved, construction began in September 2004. Both of the Computer Room's two chillers were replaced with higher-capacity units. Water pipes were torn out and replaced, new pump motors and transmissions were installed, and the cooling-tower fans were improved.
“It was a nail-biting exercise—like rebuilding a plane while it was in the air,” says Gary New, ESS assistant manager. “We had to swap out old chillers, pipes, pumps, and bring in new ones—in many cases while the Computer Room was online.” But due to careful planning, monitoring, and execution, the project went off without a hitch.
The new system is rated at 450 tons of cooling capacity. (One ton is the equivalent of 12,000 British Thermal Units worth of heat removed per hour. 450 tons is the equivalent of 5,400,000 British Thermal Units—the amount of heat it would take to melt a ton of ice in 24 hours.) (See "Behind the scenes: Chillers, pumps, and pipes" for details on how the system works.)
An added benefit of the new system is that it will divert waste heat from the Computer Room to warm the Mesa Lab in colder months.
Once the new cooling infrastructure was in place, SCD wanted to test it to make sure it worked as designed. This was a challenge. Although the system is rated at 450 tons of cooling capacity, current equipment on the Computer Room floor delivers just 250 tons of load.
The problem was how to test the system at maximum capacity without a full load on the floor. Adding more heat to the Computer Room would be risky. It might damage the supercomputers, which are in constant use, and would be impossible to control if anything dangerous happened.
NCAR's Physical Plant Services came up with an ingenious solution: to route hot water from the boilers that heat the Mesa Lab's driveway snowmelt system to the chillers. By installing two temporary hoses from the boilers to the chillers, they added 150 tons of heat load to the load already produced by the supercomputers. This allowed the test to be consistent and controllable.
“We did failover tests, control logic tests, capacity tests, stress tests,” says Gary. “A failover test ensures that if one chiller fails, the other automatically starts up and takes the load. The control logic test ensures that the sensors and the control components are working properly. The capacity test brings the chiller to 100% capacity and holds it there for 24 hours. The stress test is a capacity test over an extended period of time.”
The last step in evaluating the new cooling system was to bring in an outside firm to review the test data. The firm used ultrasound flow meters and temperature sensors to verify that the numbers were correct and the system worked as it should.
SCD's Gary New, Aaron Andersen, Stan McLaughlin, and Julie Harris collaborated with NCAR Physical Plant Services and numerous outside contractors on the Mesa Lab's Computer Room upgrade. More photos |
The chilled-water expansion is now complete, thanks to the teamwork of many individuals and organizations. SCD collaborated closely with UCAR Physical Plant Services, who were “awesome” in their dedication, says Gary.
SCD staff working on the upgrade were Aaron Andersen, Julie Harris, Janice Kauvar, Stan McLaughlin, and Gary New. Physical Plant Services staff were John Adamson, Steven Haynes (project manager), Dave Heckel, Gordon Kinn, David Maddy, Brian McMillan, and Dave Patterson. Also contributing their expertise were Roseann Suer and Bill Hammel from the UCAR Contracts Office.
Outside contractors included Engineered Solutions, U.S. Engineering, Siemens Building Automation, Encore Electric, McQuay, and E-Cube.
NCAR is operated by the University Corporation for Atmospheric Research (UCAR) under the primary sponsorship of the National Science Foundation.
|
NCAR is managed
by UCAR and sponsored by the National Science Foundation |
