CISL 2007 annual report banner

Breakthrough Science using blueice

 
 
Evolution of a salinity field

Visualizations from a large-scale simulation in which double diffusive convective motions are organized by ambient shear into "salt sheets"—alternating planar regions of rising and sinking fluid that contain opposed horizontal flow patterns parallel to the sheets. (Also see 26 MB video in .mov format.) This process depends heavily on the vast difference between the molecular diffusivities of heat and salt in water, which renders the simulation a computational grand challenge. Using 320,000 processor hours on blueice, these simulations were the first to take full account of that difference and evolve to a fully turbulent state. It is therefore the first simulation of the important and largely unexplored natural phenomenon of saltwater turbulence. (Image produced using VAPOR, a visual data discovery environment developed at NCAR for very large data sets. Image courtesy W.D. Smyth, Oregon State University.)
 
BTS projects on blueice

This chart shows how blueice resources were used during the Breakthrough Science initiative, by NSF program and NCAR.
 

In fall 2006, NCAR and NSF took an opportunity to provide very large allocations of computational resources to a small number of geoscience researchers prior to the full allocation of blueice, NCAR's newest and most capable supercomputer. Proposals that could use very large amounts of computer time were solicited for computing allocations available during February, March, and April 2007.

NSF program managers in OCE and EAR were invited to nominate projects for their NSF-sponsored research, including researchers who may not have used NCAR supercomputing resources in the past. The CISL HPC Advisory Panel (CHAP) invited researchers who had a successful record of requesting and using large amounts of processor hours. NCAR solicited proposals from NCAR researchers who could use this opportunity to move their science forward.

CISL reviewed all submissions for readiness to use a very large allocation on the IBM POWER5+ system. NSF selected five university proposals, the CHAP selected one university proposal, and NCAR selected two proposals.

This initiative supports NCAR's strategic goal to "Provide robust, accessible, and innovative information services and tools" to the university and NCAR communities. In addition, the research performed supports NCAR's strategic goal to "Improve understanding of the atmosphere, the Earth System, and the Sun."

Because of its potential for discoveries through simulation, this initiative is named Breakthrough Science (BTS).


Outcomes

Six of the eight projects successfully used their large allocations from February 2007 through May 2007, an extension of one additional month. Three university projects requested additional resources to continue their computations in June and July. By the end of the BTS initiative, six projects had consumed 2.6 million GAUs or almost 3 million processor hours on blueice. As shown in the pie chart, university research sponsored by ATM and OCE programs consumed 23% of the resources, while EAR-sponsored research consumed 17%. A total of 63% of the BTS computational resources were delivered to NSF-supported university research.

Two of the six projects had submitted papers for publication by the end of FY2007, and three others have multiple papers in progress. The sixth project is continuring to analyze the massive amounts of data they generated during BTS in preparation for publication. University researchers are reporting that their use of these very large allocations is changing their expectations for the pace of scientific progress that can be made through numerical simulations.

Among NCAR scientists the allocation of very large computational resources to a few research projects created a lot of interest. As a result, in April 2007 the NCAR Executive Committee decided to devote 6% of NCAR's supercomputer resources (out of NCAR's allocation) to one computational project at a time, usually for 4-5 months.

Also because of the BTS successes, the university panel (CHAP) began a pilot experiment of allocating a large number of GAUs to a single project at a time in April 2007.

CHAP will review the accomplishments of the BTS university projects when it meets in October 2007 and will evaluate the possibility of continuing these very large allocations to university geoscience researchers who are performing NSF-supported research.

The BTS initiative has shown:

  1. CISL can effectively provide very large amounts of computer time to individual projects while continuing to serve its diverse research community.

  2. University researchers are ready and able to effectively use very large allocations in the geosciences.

  3. Invitations to participate in the BTS encouraged university geoscience researchers to think "big" in their simulations and propose computational experiments that take five to ten times more computational resources.

  4. BTS researchers generated large amounts of data and needed CISL's mass storage and visualization resources to analyze and manage the output of their simulations.

 

Project plan for FY2008-09

In FY2008 and FY2009, CISL will again offer a few very large allocations for accelerated scientific discovery on its next installed supercomputer. This program will be offered to university and NCAR scientists under the program name "Breakthrough Computations" (BTC). Some of the key parameters of the BTC program for FY2008-09 include:

  1. The next CISL supercomputer will be an IBM POWER6 computer named bluefire. The BTC initiative will run from September through November 2008.

  2. The BTC computational resources will be split between university and NCAR researchers in the atmospheric and closely related sciences. The CISL HPC Advisory Panel (CHAP) will review the university requests and select the top proposals. NCAR management will select the NCAR projects. The evaluation criteria will include computational efficiency and readiness.

  3. Researchers will be given two to three months' advance notice so they can prepare and tune their models. Researchers will be expected to benchmark their models on the IBM POWER5+, blueice, in the spring and on the new IBM POWER6 as soon as it is available. CISL's Consulting Support Group will assign individual consultants to each selected project. This support model worked well in FY2007.

  4. At the conclusion of the BTC initiative, researchers will be expected to provide detailed progress reports on their scientific accomplishments and publications that are being prepared. CISL will follow up with projects later in the year on the specific papers that were published. These progress reports, publications, and other outcomes will be included in future annual reports.

 

Sponsorship

The Breakthrough Science initiative was sponsored the National Science Foundation through the provision of computational and support resources. The university researchers who used these resources were funded by NSF awards from ATM, OCE, and EAR. NCAR researchers utilizing the BTS computational resources were also sponsored by the NSF.