CSL Computing Upgrades and FY05 Plans
CSL Computing Resources
CSL GAU Usage
CSL Project Highlights

CSL Computing Upgrades and FY05 Plans

1.      INTRODUCTION

Since 1994, NCAR, in collaboration with the NSF Atmospheric Sciences, Ocean Sciences, and Mathematical and Physical Sciences Divisions, has operated a special-use dedicated climate system modeling computing facility known as the Climate Simulation Laboratory (CSL).  The CSL provides high performance computing, data storage, and data analysis systems to support large, long-running simulations of the earth's climate system that need to be completed in a short calendar period. 

The laboratory is open to all principal investigators funded or supported by a U.S. university or U.S. federal or U.S. private not-for-profit laboratory agency, including their international collaborators, who are performing research in support of the multi-agency U.S. Climate Change Science Program.  Allocation of the CSL resources is done via a peer review process that includes a panel of experts in climate and/or large, long-running simulations.  Large, collective group efforts, preferably interdisciplinary teams that address broadly posed sets of questions, such as the Intergovernmental Plan on Climate Change (IPCC) issues, are particularly encouraged to apply.  Also encouraged are collaborations with scientists involved with the development of policy, including impacts, mitigation, and adaptation options.

The significant increase in resources made available to the CSL in FY 2003 has meant that the Community Climate System Model (CCSM) project was allocated 104% of their request during the current allocation period through August 31, 2004; giving CCSM 70% of the total CSL resources.

A new Announcement of Opportunity for CSL computational resources will be released in April 2004.  SCD expects to provide 116K GAUs a month for the CSL awardees. The CSL Allocation Panel (CSLAP) will meet in July 2004 to review these proposals and make recommendations for awards.   After input from various agencies, the new CSL awards will be made effective September 2004. 

2. CURRENT YEAR FY04 PLANS

1. High Performance Computers  
   In June 2004 SCD will acquire a new 256-processor Linux system.  The expected peak performance of the Linux system will be about 1.0 Tflops (about half that of IBM SP Power 3 system).  This system will help SCD evaluate the feasibility of deploying Linux Clusters as supercomputing solutions.  Additionally, Platform Computing's LSF batch scheduling product will be evaluated for possible deployment across all HPS supported supercomputing servers to present a common batch interface and to potentially interface into a Grid Computing environment.  System upgrades will continue in FY 2004 to all supercomputers as will operator training, documentation, and tool development.  
   
   At the completion of the IPCC initiative, targeted for late FY 2004, the 14 additional p690 nodes added to bluesky at the beginning of  FY 2004, will be used to augment the existing bluesky capacity for the CSL.
   
   Approximately fifty percent of SCD's computational resources, as shown below, are for CSL projects. 
   

2. Mass Storage Systems (MSS)  
   The Storage Manager Disk Cache project will continue. The old MSS diskfarm was replaced in FY 2003 with the initial deployment of the disk cache. Initially, the disk cache capacity will be increased to support file sizes of up to 50 million bytes. Additional enhancements, subject to available funding, will increase the size of both the disk cache and the files stored on the disk cache. The Storage Manager will be enhanced to support StorageTek's 9940B tape drives. They will provide a 3.3-fold increase in capacity over the current 9940A cartridges (60 GB per cartridge increasing to 200 GB per cartridge), increasing the total capacity of the existing StorageTek robotic tape libraries (silos, and provide a 3-fold increase in data transfer rates (10 MB/sec increasing to 30 MB/sec). Other MSS projects include continuing development of a new MSS metadata server, web MSS metadata tools, MSS server upgrades, operator training, and documentation.
   

3. Data Analysis Support
   SCD provides separate data analysis systems as noted above.  During FY 2004 SCD will develop a plan for integrated systems that support terascale data management, access, analysis, and visualization.
   

4. Operations and Infrastructure
   The continued upward pressure of power consumption and the corresponding heat dissipation continue to stretch the computing facilities infrastructure. In FY 2004 SCD will take a two-pronged approach, first addressing very near term needs by augmenting the chilled water capability to match a maximum connected load of 1.2 MW. Additionally, SCD is looking to the future and will be initiating feasibility studies to build a second data center needed to meet future computing requirements.
    
   In April 2004 SCD implemented a one time password system to provide additional security for SCD's supercomputers. All active CSL users were issued a CRYPTOCard to provide one time password access.
   

5. User Support
   SCD will continue to provide support to researchers via phone and email. These services will be coordinated through the development of Service Level Agreements.  SCD will continue to focus on code performance, especially for widely-used codes.  A big project during the coming year will be integrating the new Linux cluster into the SCD computing environment and bringing this system to a production level for SCD's users.  The first group of users will be the CCSM development staff.  SCD will work collaboratively to test, document, and resolve user issues for this new system.

The SCD Portal was deployed for user access in late FY 2003 and offered job submission and job status. In FY 2004 work will be centered on adding information on accumulated charges for computing resources.  SCD will begin redesigning the resource accounting database that tracks user projects and usage. The current database was designed for a far simpler computing environment.  The new database model will emphasize adaptability to meet unanticipated needs.

3. FY 2005-06 PLANS

In late 2004, SCD will release a discussion draft of a new strategic plan that calls for upgrades in computing, petascale storage and an integration of data analysis activities with computing for a simpler workflow for the science community. During FY2005 SCD plans to decommission the IBM RS6000 SP (blackforest), will continue to explore the feasibility of deploying Linux Clusters as a supercomputing solution for CSL, and a low power Blue Gene platform will be tested in the NCAR environment. Additionally, during 2005, SCD will examine the question of whether or not to exercise the remaining option in the ARCS contract with IBM or to mount a new competitive procurement for an FY 2006 upgrade.  In addition, the environmental constraints of the mesa lab in terms of space power and cooling are dictating a strategic review of the facility and an examination of potential options.

CSL Computing Resources

Approximately 50% of the computational resources, shown below, are for CSL projects . The total system configuration for blackforest and bluesky includes interactive share nodes, as well as, nodes devoted to special project initiatives, which are not shown.

¹ WH2 = Winterhawk 2 (POWER3) nodes with 4-processors per node
² NH2 = Nighthawk 2 (POWER3) nodes with 16-processors per node 
³ POWER4 nodes with 8 processors per node or LPAR (Logical PARtition)
⁴ POWER4 nodes with 32 processors per node or LPAR (Logical PARtition)

Additional computers available to CSL users where use is not charged: 

• DAVE - a 16-processor IBM SP NH2 computer with 32 gigabytes memory; replaced by a 8-processor IBM Power4 node in December 2004; devoted to data analysis

• dataproc - a 16-processor SGI Origin2000 devoted to data analysis; upgrade planned for August 2004; devoted to data analysis

• babyblue - a 64-processor IBM SP with 32 gigabytes memory; available until December 2004

• chinookfe - an 8-processor Origin2100; an interactive front-end to chinook

CSL GAU Usage July 2002 Through April 2004

One GAU is equal to 10 blackforest CPU hours, 10 chinook CPU hours, or 4.667 bluesky CPU hours

CSL Project Highlights

PI:  Raymond Arritt
Title: Ensemble Simulations of Regional Climates

In the past year we used CSL resources to generate a 15.5-year RCM simulation of contemporary climate for the continental US and adjoining regions, which is the longest regional climate model simulation ever performed for the contiguous United States.  This simulation was performed as part of the PIRCS-1C experiment, which we are coordinating in collaboration with other regional climate modeling groups in North America, Europe, and Australia.  We also continued to analyze results from previous simulations, yielding several published papers, conference presentations, and a Ph.D. dissertation. 

CSL Accomplishments Report 2003-2004
Detailed Charts of CSL Usage

PI: Byron Boville
Title: Whole Atmosphere Community Climate Model 

The development of the interactive chemical-dynamical model (WACCM2) continued during 2003 and the first part of 2004, with several simulations both in the context of model development and science studies. From the chemistry point of view, the resolution of the cold tropical tropopause problem has allowed a more realistic simulation of ozone and water chemistry in the stratosphere and mesosphere. Considerable effort went into the chemistry components, particularly the interactive calculation of heating rates shortward of 200 nm and ion chemistry. From a dynamics/physics point of view, the gravity wave parameterization has been examined in great detail, both in the spectral and in the orographic component. Several physical parameterizations (molecular diffusion, nitric oxide cooling) have been updated or added to the model.

CSL Accomplishments Report 2003-2004  
Detailed Charts of CSL Usage

PI: Peter Hess
Title: Simulations of Tropospheric Chemical Variability

Now for the first time computing resources are sufficient to run multi-year integrations of atmospheric chemistry and transport models. Moreover, increased computational resources coupled with better measurements and numerical parameterizations allow a sophisticated coupling between chemical and climate systems models. Coupling and feedbacks between chemistry and other components of the earth system are now included in simulations through increasingly sophisticated parameterizations of biogenic emissions, biomass burning, oceanic emissions, and aerosol models. Using these capabilities we have begun to gauge the long term simulated chemical response to interannual variations in meteorology, emissions and climate against the available measurements; to assess the importance and magnitude of feedbacks between atmospheric chemistry, climate and the biosphere; to explore the variability of the coupled climate-chemistry system and to understand possible changes in atmospheric chemistry under different climate scenarios.

CSL Accomplishments Report 2003-2004
Detailed Charts of CSL Usage

PI: Jeffrey Kiehl
Title: Community Climate System Model (CCSM) - Development

The Kiehl Development Proposal supported the work of the CCSM Atmosphere Model Working Group, the Biogeochemical Working Group, the Ocean Model Working Group, the Land Model Working Group, the Polar Climate Working Group, and the Software Engineering Working Group. This report documents the results achieved by these working groups. The Scientific Steering Committee directed a significant effort toward the remediation of persistent biases to have the best possible simulation capability in support of the Fourth Assessment Report (FAR) of the Intergovernmental Panel on Climate Change (IPCC). Much development of the physics and dynamics was conducted, so that the model produced credible climates at multiple resolutions. Additional work was conducted to have the capability to do integrations as specified by the IPCC. It was necessary to conduct and complete development tests with the highest resolution configuration, T85x1, as this was deemed essential for the IPCC runs. The results of this development effort have been used in the IPCC integrations under way and will be released to the public as CCSM3 on June 23, 2004.

CSL Accomplishments Report 2003-2004
Detailed Charts of CSL Usage

PI: Jeffrey Kiehl
Title: Community Climate System Model (CCSM) - Production

The Kiehl Production Proposal supported all of the working groups in the CCSM project. The allocation during this period was 37400 GAUs/month until February 2004, when the allocation was renormalized to 44880 GAUs/month. This document reports on the scientific results obtained during the period May 1, 2003, to April 30, 2004. The CCSM3 control runs are “owned” by the CCSM as a whole and are reported under the Software Engineering Working Group. Considerable work on the meridional overturning circulation was done with CCSM2, but much of the work has been to produce and validate the simulated climates from the various configurations of CCSM3 including T31x3, T42x1, and T85x1. A major result is a comprehensive suite of control and special integrations in support of the public release of CCSM3 and for the conduct of the special experiments for the Fourth Assessment Report (FAR) of the Intergovernmental Panel on Climate Change (IPCC).

CSL Accomplishments Report 2003-2004
Detailed Charts of CSL Usage

PI: John Kutzbach
Title: Coupled Climate Model Simulations of Climate and Climate Variability: Present, Past, and Future Climates

We have been continuing studies of climate and vegetation/climate interactions for present, future and past climates using climate models. We have been using climate models in studies of present-day climate/vegetation variability, potential future changes of vegetation and climate, past changes of vegetation and climate in the mid-Holocene, and past changes in climate associated with the Last Glacial Maximum. Many of these studies are in collaboration with Otto-Bliesner at NCAR and Jacob at Argonne National Lab. We have also been collaborating with and Bonan at NCAR in comparing results from two models with interactive vegetation. Most of our simulations cover several centuries. We have started exploring techniques for making simulations of several millennia duration. We have about 35 papers recently published, in press, or in preparation

CSL Accomplishments Report 2003-2004
Detailed Charts of CSL Usage

PI:  Carlos Mechoso
Title: Climate Variability in the Atlantic Basin

The overall goal of this project is to gain a better understanding of the climate and climate variability of the Atlantic basin and surrounding continents. We concentrated on three aspects of the larger problem: 1) Impact of African orography on the development of stratocumulus along the Namibian coast, 2) Role of subtropical Atlantic SST anomalies in the winter climate variability of western Europe, and 3) Contribution of atmosphere-ocean interaction processes to variability in the tropical region. To investigate these aspects, we performed several ensemble simulations with the UCLA atmospheric general circulation model (AGCM). A comparison between simulations with and without African orography showed that orography contributes significantly to the persistence of stratocumulus incidence off the coast of Namibia. Ensemble simulations with prescribed SSTs corresponding to the winter of 1995/96 suggested that anomalies in the Atlantic suffice to produce climate anomalies over western Europe with a magnitude comparable to those observed. Ensemble simulations with the AGCM coupled to an oceanic mixed layer model showed that several modes of coupled variability were captured despite the lack of ocean dynamics.

CSL Accomplishments Report 2003-2004
Detailed Charts of CSL Usage

PI: Niklas Schneider
Title: Forecasting Climate Changes over North America from Predictions of Ocean Mixed Layer Anomalies.

The role of the upper-ocean heat-flux convergences in the Kuroshio-Oyashio Extension (KOE) region in modulation of low-frequency climate anomalies over North Pacific/North America was investigated using CCM3/SOM coupled model. Predictable heat flux convergence anomalies over tropical and western North Pacific were obtained from the Ocean GCM hindcast and were applied as forcing to the ocean of the coupled model. Two five-member 40 years long ensemble integrations (total of 400 years) were performed using CSL facilities. In first experiment the coupled model was forced with the heat flux convergence anomalies over tropical Pacific and KOE regions. In the second experiment, the model was forced with the tropical forcing only. The forcing over the KOE region caused changes in the free atmosphere as well as in the surface variables over the United States. All the experiments were performed on the “blackforest” IBM/SP. One year of the simulated time on 16 nodes with 4 tasks per node required 14 GAUs.

CSL Accomplishments Report 2003-2004
Detailed Charts of CSL Usage

PI: Amy Solomon
Title: Pacific Decadal Variability due to Tropical-Extratropical Interactions

The scientific objective of this CSL project is to study Pacific Decadal Variability (PDV) due to tropical-extratropical interactions. We have pursued this objective by considering two complementary modeling approaches, namely, 1) the development of an intermediate coupled model (ICM) consisting of a intermediate Pacific Ocean model anomaly-coupled to the CAM2, and 2) coupled climate model simulations using the CCSM2 modified to prescribe sea surface temperature (SST) in key tropical regions. During this year, the ICM model development was completed and a 200-year control integration was run. This control integration closely simulates observed interannual-to-decadal tropical variability, as well as, meridional overturning of Pacific Subtropical Cells and re-emergence in the North Pacific. In addition, 5 150-year ICM simulations with air-sea coupling in limited regions of the Pacific basin were completed. These simulations are currently being analyzed to identify the role of extratropical forcing of the tropical Pacific through the oceanic bridge on tropical decadal variability. The CCSM2 coupled integrations with observed SST over the period 1948-2002 are being run at this time.

CSL Accomplishments Report 2003-2004
Detailed Charts of CSL Usage

PI: Detlef Stammer
Title: Ocean State Estimation

This project has several separate components that are related to the global ECCO assimilation activity. Those components include: (1) global ocean state estimation as part of a WOCE, CLIVAR and GODAE synthesis activity and (2) regional eddy resolving modeling of the North Atlantic.

Since July 2003, the work related to this project has largely focused on producing the best possible global synthesis for the period 1952 through 2002. This synthesis will be presented at the International CLIVAR Conference in Baltimore in June 2003 as one of the very few global ocean reanalyses paralleling the NCEP and ECMWF reanalysis efforts but in the ocean. It has build in a complete mixed layer model (Large et al., 1994) and an eddy parameterization scheme (Gent and McWilliams, 1990). In parallel we are performing eddy-resolving assimilation in the Pacific and Atlantic oceans.

Now ongoing computations move toward the estimation of mixing coefficients by including them into the control vector of the 50-year estimate with 1 degree spatial resolution. It is anticipated that, in two to three years, the project will be able to address the US CLIVAR and GODAE related objective of depicting the time-evolving ocean state with spatial resolution up to 1/4 degree globally and with substantially higher resolution in nested regional approaches which are required for quantitative studies of the ocean circulation.

CSL Accomplishments Report 2003-2004
Detailed Charts of CSL Usage

PI: David M. Straus, Center for Ocean-Land-Atmosphere Studies
Title: Seasonal Time Scale Realizable Predictability of the Coupled Atmosphere-Biosphere System

PI: Robin Tokmakian
Title: Interannual to decadal ocean variability over 40 years at 0.2 degree resolution

A 40 (1959-1998) year, high resolution simulation has finished and is currently being extended (1999-2002) as resources allow. The simulation allows for the examination of the interannual to decadal variability of global ocean circulation and will help in our understanding of the realism of such a simulation and its variability. The simulation has used the Los Alamos Parallel Ocean Program (POP) primitive equation model in a global configuration with a rotated North Pole at a resolution of 0.2°. High quality atmospheric data for 1958 through 1998 forced the hindcast run and the analyses show that the model reproduces the low frequency variability reasonably well. For example, the sea surface heights show variability consistent with observed tide gauge measurements and small scale eddy features similar to what is seen in composite altimeter sea surface height maps. The decadal change in the mix layer depths is consistent with similarly sampled observational maps.

CSL Accomplishments Report 2003-2004
Detailed Charts of CSL Usage

PI: Stephen Zebiak
Title: Development and Application of Seasonal Climate Predictions

We have continued our previous work as outlined in our CSL proposal. The dominant portion of our current and previous CSL allocation has been used for the development and improvement of SST forecasting systems which utilize an OGCM component model coupled to either an AGCM or a statistical atmospheric model.As part of this project we have developed and implemented a near real-time directly coupled AGCM-OGCM forecasting system consisting of the ECHAM4.5 AGCM and the MOM3 OGCM. A major deficiency with this forecasting system and other similar systems is the bias in the OGCM SST simulation.Our work in the past year using the CSL allocation has concentrated on the exploration of different bias correction formulations for the density (primarily temperature) simulation and especially the SST simulation in the MOM3 OGCM component model. The goal is to improve the SST forecast skill in the tropics by reducing the model bias. In a second project, we have been conducting regional climate model predictability experiments for the Greater Horn of Africa using an ensemble of simulations from the Regional Spectral Model for a long period, i.e. the 1970 to present period.

CSL Accomplishments Report 2003-2004
Detailed Charts of CSL Usage

| CSL home page | | SCD home page 

Questions? Send e-mail to: Ginger Caldwell (cal@ucar.edu)
Other Problems or Concerns? Send e-mail to: Michelle Smart (msmart@ucar.edu)

LAST UPDATED: Monday, October 18, 2004
© Copyright 2002-03 University Corporation for Atmospheric Research (UCAR). All Rights Reserved