Weathering the forecast: How AMPS powers Antarctic operations

In June 2016, during the southern hemisphere’s brutally cold winter, Storm Knight, an HPC System Administrator at NSF NCAR's Computational and Information Systems Lab (CISL), noticed a news report. "I saw on the news that a rescue mission was going on in Antarctica," Knight recalled. "It should have been fine, but there was always a possibility of something going wrong, and it was important to address that risk."

 

"AMPS is a critical function. Everything 
depends on it."  

—  Jordan Powers, AMPS Scientist

Boulder, Colorado, and Antarctica’s McMurdo Station are separated by nearly 9,000 miles. Despite that vast distance, NSF NCAR and its Derecho supercomputer provide a critical service that ensures the safety of the scientists and logistics teams that work and travel to these remote Antarctic research outposts. Developed and operated by NSF NCAR with support from the NSF Office of Polar Programs (OPP) for the past quarter century, AMPS generates twice-daily forecasts for all of Antarctica using advanced weather models like WRF and MPAS running on Derecho. These forecasts provide crucial real-time weather information for the United States Antarctic Program (USAP), Antarctic science, and international efforts. 

The risky medical evacuation was unfolding in 2016 just before planned maintenance on Yellowstone, the NSF NCAR computer running the Antarctic Mesoscale Prediction System (AMPS) forecasts.
 

CISL ultimately postponed the routine maintenance by a month. This decision allowed AMPS to continue providing critical weather and atmospheric information without interruption during the rescue mission, supporting essential forecasts and enabling authorities to make key operational decisions. 

CISL’s proactive choice directly facilitated a life-saving operation. “The forecasters were very appreciative of having that forecast because it was a critical situation for them and they really needed that information,” stated Kevin Manning, a Scientist with NSF NCAR’s Mesoscale & Microscale Meteorology (MMM) Laboratory. 
 

"Our models have been tuned to the Antarctic environment in ways that global models have not been." 

—  Kevin Manning, AMPS Scientist

Together, Manning and Jordan Powers, an MMM Scientist, jointly manage the AMPS program. “This has happened repeatedly throughout AMPS's long history,” said Powers. “About every 18 months, some emergency evacuation and special operation has to go on. At AMPS, we are happy to create a specialized product for forecasters for a specific region of Antarctica where something is happening, so they can visualize the weather conditions in that precise location. We are committed to helping them in that way.”
 

Powers explained how AMPS functions: "Twice a day, the weather models generate output with forecast information and the future state of the Antarctic atmosphere, in the form of graphical plots, tables, charts, maps, and more." Forecasters, primarily civilians paid by the Naval Information Warfare Center (NIWC) and overseen by the Navy, use the AMPS web page extensively for flight and ground operations.

 

"AMPS is one of our VIPs." 

—  Storm Knight, HPC System Administrator

Powers also described the challenges Antarctic operations face: "The weather in Antarctica can get so bad that the planes can't land." This is critical, as some aircraft lack the fuel range to return to their origin (typically New Zealand) if they cannot land, risking being stranded. AMPS predicts these conditions, especially for air transport. Severe weather also impacts ground operations at the three NSF-managed U.S. bases: McMurdo, Amundsen-Scott, and Palmer. 

Antarctic scientists also rely on these forecasts, particularly during the southern hemisphere's summer (December to February) when many scientists have the opportunity to go Antarctica for field research. Accurate, high-resolution AMPS forecasts are essential for safety and logistics.

Not limited to rescue missions or specific events, AMPS offers unique everyday value: "I’d like to emphasize AMPS’ significance to the NSF and U.S. Antarctic programs and logistics,” said Manning. “They rely on AMPS quite a lot. The forecasters use the AMPS data very heavily and on a daily basis to make decisions." 

Thanks to Derecho, AMPS is able to provide unique, high-resolution output tailored for Antarctic needs, regularly generating custom graphics upon request—an unusual collaboration between forecasters and model developers. 

 

"We rely on the expertise at CISL to actually know how to get things running stably and efficiently. That's been very, very helpful for us. The CISL folks have been very supportive of AMPS from the start." 

—  Kevin Manning

Powers noted that many AMPS users are ex-military forecasters accustomed to receiving a generic set of outputs intended for a broad range of users. In contrast, AMPS offers a strong, collaborative relationship, frequently developing custom solutions like weather plot specification  changes or data focused on specific latitudes and longitudes, often within a day. This tailoring was a key motivation for AMPS' creation in 2000: the need to address a previous lack of high-resolution, tailored forecasts for Antarctica.

AMPS also promotes international collaboration, with many international groups, including bases from Britain, Australia, Italy, China, Korea, Brazil, and Argentina, that openly access its data online. This aligns with the Antarctic Treaty's spirit of cooperation. Powers added, "Antarctica is really a great example of international cooperation in a good spirit. And from the get-go, AMPS was in a position to provide this information to all the countries."

CISL ensures AMPS runs twice a day, every day, without fail. While scientists like Powers and Manning focus on model development, the CISL team resolves issues, optimizes performance, and provides cloud computing via Rescale HPC Cloud as a backup.

Key CISL staff include Ben Kirk (Manager, Consulting Services Group), Rory Kelly (HPC Consultant), and Knight. 
 

"CISL gets a lot out of this partnership in terms of understanding cloud solutions generally, with AMPS as a good test case." 

— Rory Kelly, HPC Consultant

Kelly stated their main work is "figuring out their backup plan for when our main compute is down," involving extensive planning for outages and optimizing cloud performance and cost.

 Kirk noted that the collaboration is a “two-way street”: AMPS' consistent output checks help CISL find system errors quickly, especially during new system pivots. Even vendors like Rescale benefit from insights into complex weather forecasting workloads. 

Running AMPS presents unique challenges, said Kirk: "When an issue arises, it's usually weird, and it takes a lot of creativity to resolve. New problems are always fun."  

 

"Any new tech issues that arise with AMPS are usually unusual and fun to resolve." 

—  Ben Kirk, CSG Manager 

The CISL team manages congestion on shared machines without wasting resources. AMPS' jobs must reliably run within specific daily timeframes. Knight offered an analogy: "The way the big system runs is a lot like a restaurant where you've got people coming and going—a lot of small parties, but also a lot of big parties. AMPS is one of our VIPs," said Knight. "We need to figure out a way to get them seated, but we don't want to lose money or turn people away or anger our regular customers by holding a big section of seats open." This balancing act ensures AMPS' reliability without impacting other users.

AMPS' computing infrastructure evolved from an NSF community machine to dedicated, NSF-funded computers at CISL (then SCD), and now uses NSF-funded allocations on CISL HPC, eliminating independent machine management. The transition to the more powerful Derecho supercomputer in late 2023 directly improved forecast accuracy. Manning noted, "We had the computing power to put in a more complex microphysics scheme. This immediately addressed the dry bias that we had historically seen in AMPS." Derecho also boosted ensemble forecast members, providing valuable information on weather predictability and uncertainty.
 
 

"It's real people, it’s real weather, it all matters." 

—  Rory Kelly

The real-world impact of AMPS is profound. Kelly finds satisfaction in knowing the forecasts directly benefit people in Antarctica: "It's the only thing I work on that's a real-time, 365-day-a-year project where people depend on the output of that model to make decisions about their day."

Manning reflected on CISL's support for AMPS's life-saving efforts during the 2016 rescue mission, stating, "It was gratifying to see that the news sparked an idea in the CISL team. They realized, 'Hey, we can actually make a difference here. We can help.'"

For more information, visit the AMPS homepage.