NCAR Scientific Computing Division FY2001 Annual Scientific Report

Message from SCD Director Al Kellie

Another year of great challenge and change for the Scientific Computing Division has run its course. During FY2001, SCD has managed significant and lasting achievements, which I am pleased to recognize in this Annual Scientific Report. Given the nature of our efforts in providing high performance computing support for NCAR and the atmospheric sciences community around the world, technological change is an inevitable feature of our planning and operational landscape. We may not always relish the upheavals that accompany such changes, but we always seek out and embrace the associated opportunities to support and advance the research agenda of NCAR's constituent research community.

Looking forward to the future:
A new generation of computing at NCAR

NCAR's new-generation computer observed by America's new generation of scie
ntists

Student visitors to NCAR viewing blackforest, the Advanced Research Computing System housed in SCD's spotless computer room.   -- Photo by Lynda Lester, SCD

This was a watershed year for SCD in terms of providing services to researchers, computational cycles, research and development progress, state-of-the-art enabling technologies, and irreplaceable research data.

During FY2001, SCD has:

In addition, SCD successfully participated in an intensive NSF review of our past five years' accomplishments. NSF's review panel found that SCD is not only highly supportive of NCAR, but that SCD is crucial to the overall mission of NCAR and the advancement of atmospheric science research. SCD staff are justifiably proud of these findings and the accomplishments underlying them. I urge you to read the details of our progress in this year's Annual Scientific Report.

And as you read through our report of FY2001 activities and accomplishments, I hope you will take away some sense of our excitement at the possibilities that lie before us. We are indeed embarking upon a new, bold, and challenging future, fueled by enhanced capabilities in computing, research, data storage, networking, analysis, and visualization.

We look forward to this future as we reflect on our past accomplishments. As always, we seek to provide the finest in computing resources, teamed with a dedicated and talented staff, to help advance the understanding of our complex climate system.


SCD FY2001 ASR - Table of contents

Message from SCD Director Al Kellie
Significant accomplishment highlights

The NCAR Advanced Research Computing System procurement
Balanced, substantive progress by all sections toward fulfilling SCD's mission
The procurement of ARCS, NCAR's Advanced Research Computing System
High performance computing
Maintaining NCAR's production supercomputer environment
Supercomputing systems deployed at NCAR
FY2001 production system overview
Production system performance and utilization statistics
End-FY2001 production supercomputer systems
Key maintenance activities
Computational science research and development
Research
Software libraries and toolkits
Frameworks and modeling infrastructure
Proposals
Technology tracking
Data archiving and management system: The MSS
MSS history
MSS today
MSS accomplishments for FY2001
MSS growth
Future plans for the NCAR MSS
Network engineering and telecommunications
Introduction to NETS
Networking research projects and technology tracking
    Networking research projects
    Network technology tracking and transfer
Local Area Network projects
    NCAR/UCAR LAN projects
    UCAR LAN engineering projects
    SCD LAN projects
Metropolitan Area Network projects
Wide Area Network projects
Other network projects
Research data stewardship
DSS main accomplishments during FY2001
Summary and outline of DSS projects
DSS external linkages
Visualization and enabling technologies
Facilities for analysis, visualization, and collaboration
Community software for data analysis and visualization
Enterprise web services
Research and special projects in data and visualization
Collaborative efforts
Outreach and education
Technical presentations and educational outreach
Visitors
Publications
Assistance and support for NCAR's research community
Technical Consulting Group
    Consulting services
    Production computing support
    Security perimeter changes
    System testing
    ARCS procurement support
Digital Information Group
    Providing online information services using web-based technologies
    SCD website design and architecture
    Web-based information and user documentation
    Services provided to SCD
    Website maintenance and development
    DIG outreach, education, and training
Distributed Systems Group
    Overview of distributed services provided by DSG
    Expansion and upgrade of servers and services
    Clustered and fault-tolerant system configurations
    Expansion of Linux and other operating systems
    Machine room reorganization of consoles and servers
    Desktop systems activity and accomplishments
Database Services Group
    Managing computing accounts for researchers
    Survey of researchers for NSF review
UCAR Security Administrator
    Maintaining and enhancing UCAR's security perimeter
    Enhancing UCAR's security incident detection and response
    Establishing assured connection integrity and privacy
Enabling infrastruture
Business continuity and disaster recovery plan
Network monitoring and NCAB monitoring host policy
Mass Storage System expansion
Application support services
Research and new technology advances
Staffing changes
Decommissioned machines
Community service activities
Educational activities
Colorado Computational Science Fair
Computing grants to classrooms
SC2000 education program
Papers and publications
Publications, refereed
Papers and publications, non-refereed
Staff, visitors, and collaborators
SCD staff
Visitors
Collaborators


Significant accomplishment highlights

SCD's two most significant accomplishments in FY2001 were 1) the procurement of ARCS, the Advanced Research Computing System which doubled the capacity of our largest computer, and 2) the balanced, substantive progress made by all of SCD's sections toward fulfilling the SCD mission.

The NCAR Advanced Research Computing System procurement

The successful ARCS procurement is SCD's top achievement for FY2001. The ARCS system will provide a phased introduction of new computational, storage, and communications technologies through the life of the contract. This will allow NCAR's Scientific Computing Division to maintain a stable, state-of-the-art production facility for the next three to five years.

ARCS - The NCAR Advanced Research Computing System

The initial delivery augments the existing blackforest system by more than doubling its computational capacity, from 0.9 to 2.0 peak TFLOPS, and provides a five-fold increase in disk storage capacity. A second delivery, in September 2002, will introduce IBM's next-generation processor (POWER4), node (Regatta), and switch (Colony) technologies, adding almost 5 peak TFLOPS, upgraded switch communications, and 21 TB of new disk storage. In the fall of 2003, the Colony switch will be replaced with IBM's next-generation Federation switch technology, which provides much lower latency and higher bandwidth than does the Colony switch.

If NCAR chooses to exercise the two-year contract extension option, in the fall of 2004 the system will be upgraded with an additional 4 peak TFLOPS and 32 TB of new disk storage.

IBM and SCD have agreed to work together to improve the user environment and user support services that will be provided to NCAR and CSL. This agreement covers many aspects of the ARCS, including on-site IBM applications specialists, training in advanced programming, performance analysis and tuning techniques, and a more efficient process for reporting, escalating, and resolving compiler and tools problems.

Additionally, the agreement with IBM will provide the opportunity for NCAR to participate in IBM's "Blue Light" HPC project. Blue Light is an exploratory effort of IBM's Exploratory Server Systems department at IBM Research to develop future PetaFLOPs supercomputer systems. NCAR's collaboration with IBM in Blue Light holds the promise of significant and revolutionary advancements in climate, weather, and Earth systems models, and will provide IBM with valuable input on hardware and software design.

Balanced, substantive progress made by all of SCD's sections toward fulfilling the SCD mission

SCD's internal organization is shaped by its mission. SCD's structure makes each of its management units primarily responsible for one line item in SCD's mission. While some line items in the mission are shared, the purpose of each management unit is to focus on and execute one specific part of the mission. This organization places accountability and credit for mission-critical projects on specific people in the division.

Our second highlight for FY2001 is the balanced progress we achieved in each fundamental area of our mission, which is to provide:

  1. High-performance computing and expertise needed for the development and execution of large, long-running numerical simulations
  2. A data archiving and management system that is balanced in performance and capacity relative to computational resources
  3. High-speed network and data communication capabilities that are balanced with respect to computational facilities, storage facilities, and the requirements of a national and international community
  4. Research datasets and expertise needed by atmospheric and related sciences
  5. A computing environment and support services that emphasize user productivity and cost-effectiveness
  6. Education and training in computing and related technologies with an emphasis on under-represented groups
  7. Transfer of appropriate NCAR technology to the private sector in collaboration with the UCAR Foundation

These are the stories of our most significant achievements in each of these areas, organized by the way SCD's management units relate to the line items in our mission:

High performance computing

Primary responsibility for high performance computing is managed by the Supercomputing Systems Group (SSG) to support part (a) of our mission. SSG FY2001 highlights include:

NCAR batch scheduler enhancements
SSG developed and maintains the batch scheduling software that sits on top of the resource management software provided by the vendors. The resource management software controls allocation of resources within each of the supercomputers. The batch scheduling software implements the NCAR business logic for how best to allocate the resources. Many enhancements were made to the scheduler this year, including enhanced error checking with feedback and externalization of the control logic.

When a scientist submits a job to the supercomputers, there are a number of directives that can be specified in the job script. Often times these directives are confusing and conflicting, which results in the job not properly running or not running at all. Changes have been made to the scheduler to provide enhanced filtering for incomplete, incorrect, or conflicting directives. When a job is submitted that has one of these problems, it is rejected and an email is sent to the scientist explaining the problem and recommending possible solutions.

The externalization of the control logic involved parameterizing things like maximum wallclock time for the queues, privilege granting for special queues, and special project authorizations. These items have been removed from user code and centralized into a configuration file. The business office now owns and controls this file, removing this system administration burden from users.

New end-user tools
This year, SSG developed a series of tools that enable scientists, consultants, and systems staff to easily obtain job status information from the supercomputers. These tools provide a global view of the job status from each of the supercomputers. They include things like the status of running jobs, queued jobs, completed jobs, load average of the system, outstanding requests for data from the Mass Storage System, and information about the inner logic of the scheduling algorithm. The output from these tools is in the form of a summary report, and the job status information is also being posted (every five minutes) to the SCD website. A follow-on activity that is currently underway is the conversion of this information into a graphical, point-and-click format that will enable a more timely delivery of this information from within a web browser.

Computational science research and development

Primary responsibility for computational science and computational research and development is managed by the Computational Science Section (CSS) to support part (a) of our mission. CSS FY2001 highlights include:

Terascale spectral element dynamical core for atmospheric General Circulation Models
Climate modeling is a grand challenge problem where scientific progress is measured not in terms of the largest problem that can be solved but by the highest achievable integration rate. Loft, Thomas, and Dennis have developed a scalable spectral element atmospheric model that achieves a high percentage of peak on microprocessors. A semi-implicit time-stepping scheme accelerates the integration rate relative to an explicit model by a factor of two.

SI SEAM performance comparison

The MPI implementation outperforms hybrid MPI/Open MP on the IBM SP. Simulation rates have been measured for the standard shallow water equation benchmarks using up to O (105) horizontal degrees of freedom. A sustained 370 GFLOPS was achieved at NERSC IBM. This work is a finalist for the annual Gordon Bell Award announced at the annual ACM Supercomputing Conference each year.

Earth System Modeling Framework
Over the last few years, the need for software infrastructure for Earth system modeling has grown increasingly apparent. Models and the computational platforms that they run on are becoming extremely complex, leading to excessive time and resources dedicated to solving computational rather than scientific problems. In September 2000, the NASA High Performance Computing and Communications (HPCC) Earth and Space Science (ESS) Project released a Cooperative Agreement Notice (CAN) entitled "Increasing Interoperability and Performance of Grand Challenge Applications in the Earth, Space, Life, and Microgravity Sciences." The NASA CAN calls for the development of an "Earth System Modeling Framework (ESMF)." In response to this NASA announcement, a collaboration led by Cecelia DeLuca submitted a coordinated set of three proposals to develop an Earth System Modeling Framework. The ESMF will allow diverse scientific groups to leverage common software to solve routine computational problems, and it will provide an interface specification so groups working at different institutions and in different disciplines can generate interoperable software components. NASA has selected the proposals from this collaboration for funding, and there will be a three-year effort initiated in FY2002.

Data archiving and management system

Primary responsibility for NCAR's Mass Storage System is managed by the Mass Storage System Group (MSSG) to support part (b) of our mission. MSSG FY2001 highlights include:

SD-3 (Redwood) tape migration
Between June and September 2001, the Mass Storage Systems Group (MSSG) undertook and completed the migration of approximately 75 TB of data from SD-3 (Redwood) media to newer media types. Although it held great promise when introduced in the mid-1990s, Redwood is now considered an "end-of-life" technology, and the vendor (StorageTek) has consequently imposed significant increases in maintenance costs for Redwood tape drives. The migration of this 75 TB of MSS data allowed the MSS group to decommission 10 of its Redwood drives. (The remaining two drives are used only to read secondary copies of MSS files in those rare occasions that the primary copy--stored on non-Redwood media--cannot be read.) The decommissioning of these 10 Redwood drives represents a potential savings of approximately $16,000 per month in maintenance costs. In addition, as part of this project, key users were lobbied to remove as many Redwood-resident MSS files as they could: this resulted in the removal of an estimated 9 TB of unneeded data from the MSS.

9940 media deployment
As part of its ongoing commitment to make the best use of new tape technology in the Mass Storage System, MSSG also deployed 9940 media in FY2001. 9940 is a high-capacity media that uses the same recording technology as 9840 media, which has been in production in the MSS since 1999 and has proven itself to be extremely reliable. The new media has a capacity of 60 GB per cartridge. Higher capacities using the same 60-GB cartridges are planned for the next 12-18 months. In the first 16 weeks of production, over 150 TB of data were stored on 9940 media.

University of Illinois NCDM collaboration
Together with Lawrence Buja of CGD, MSSG set up and hosted a "Data Space" server cluster for the National Center for Data Mining (NCDM). NCDM is part of the Laboratory for Advanced Computing at the University of Illinois at Chicago (UIC). The server cluster consists of three Linux systems providing access to climate model data produced by NCAR via the NCDM's Data Space transfer protocol. The system was showcased at SC2000, demonstrating real-time data access from the Dallas show floor to the server housed in the SCD computer room.

Network engineering and telecommunications

Primary responsibility for developing and maintaining UCAR's networking infrastructure is managed by the Network Engineering and Telecommunications Section (NETS) to support part (c) of our mission. NETS FY2001 highlights include:

Front Range GigaPOP (FRGP)
The Front Range GigaPOP (FRGP) is a consortium of universities, nonprofit corporations, and government agencies that are cooperating in a regional network aggregation point called the FRGP to share the costs of Wide Area Networking (WAN) services. The current FRGP partners are the Boulder Point of Presence (BPOP), Colorado State University (CSU), CU-Boulder, CU-Denver, CU-HSC, CU-CS, CSM, DU, the University of Wyoming, and Fort Lewis College. Additional partners, including the State of Colorado and the University of Northern Colorado, are likely to join soon. There are similar gigapops throughout the U.S. There are a number of advantages gained by sharing services through such a gigapop. Costs for WAN services are reduced for each partner, expertise among partners can be shared, a higher level of services can be purchased than individual institutions could afford, there is more buying power among a consortium, and there are great economies of scale.

Front Range Gigapop members
(Click image for detailed view.)

NCAR/UCAR has provided the engineering and NOC support for the FRGP, with the service costs incurred by NCAR/UCAR being shared by all members. NETS believes that the greater service and bandwidth obtained through the FRGP are important enough for NCAR/UCAR to participate and provide the engineering and NOC services. FRGP has agreed that NETS has the most qualified engineering and NOC staff to provide the very best engineering and NOC services for the FRGP.

This is a critical service for the UCAR/NCAR staff as well as all the other partners, and it has proved to be an extremely successful technical project and an excellent collaboration with the Colorado research community. The FRGP provides NCAR/UCAR's primary WAN connectivity including Abilene connectivity. For more information, see http://www.frgp.net/

Web100
Web100 is a major project. The Web100 project is an initiative proposed by NCAR, PSC, and NCSA to fix some well-known problems with the Unix (and other) operating systems that are currently inhibiting effective utilization of national high-performance networks such as vBNS and Abilene. One of the biggest problems is the current need to manually calculate the optimal bandwidth delay product to specify a TCP window size that is large enough to avoid prematurely halting data transmission between TCP acknowledgment packets.

This issue generally isn't important for LANs, but it is important for high-performance WANs. It is difficult to determine the "bandwidth" part of the product, and right now the only effective way to obtain this is to have knowledge of the network topology, which usually means consulting with a network engineer. Furthermore, most applications don't provide a means for the user to specify this information even if it was available. The Web100 Project is seeking to solve this problem and some other related ones and has received funding from the NSF for a three-year research proposal.

For more information on Web100, see http://www.web100.org/

Research data stewardship

Primary responsibility for collecting, correcting, and distributing valuable research datasets is managed by the Data Support Section (DSS) to support part (d) of our mission. DSS FY2001 highlights include:

Upgraded data and data presentation on the Research Data web server
We implemented a large-scale upgrade and improvement to the information interface for the SCD Research Data Archive website.

The new interface covers all aspects of the Data Support Section web presence. It is not only superior for the data users, but it is also easy and efficient for DSS staff to use and maintain. This information system has more than 2,500 html-formatted pages that are updated automatically. The data available continue to grow as data and metadata are routinely added by DSS staff. The new DSS web interface significantly improves researchers' ability to access and use NCAR's Research Data Archive.

  Services page from new DSS website
(Click image for detailed view.)

New additions to COADS and other ocean datasets
In 2001, a major milestone for COADS was achieved. New early data sources had been recovered through data archaeology efforts around the world. These sources and other new digital sources have been added to the collection for the period prior to 1950. This update and updates from previous years now form the complete replacement and extension for Release 1 COADS (1985). The new archive now covers 1784-1997. The COADS project, a collaborative effort between NOAA/CDC, NOAA/NCDC, and NCAR/SCD, is the world's dataset for describing conditions at the surface of the ocean (air temperature, wind direction and velocity, water temperature, etc.). COADS is a critical resource for studies of climate trends and global weather interactions, as well as for data reanalysis projects.

Visualization and enabling technologies

Primary responsibility for helping researchers visualize, interact with, and understand complex geophysical data is managed by the Visualization and Enabling Technologies Section (VETS) to support part (e) of our mission. VETS FY2001 highlights include:

NCAR's new Visualization Lab
Terascale visualization, collaboration, and the AccessGrid
Internet and web technologies coupled with high-bandwidth networks have served as the substrate for wonderful new opportunities in scientific endeavor and collaboration. While the desktop is still the day-to-day environment of choice for the individual, group meetings are more important than ever. From research organizations to businesses to universities, there is an enhanced focus on sophisticated, technology-mediated meeting spaces that facilitate information flow and enable virtual encounters.

SCD has recently completed the development of its new Visualization Lab, a physical facility that blends visual supercomputing, virtual reality, large-screen tiled display, and advanced collaboration technology. Backed by an array of large-scale computational and storage resources, the lab facilitates group exploration of terascale scientific data.

  New Visualization Lab

Building upon the AccessGrid, a human-scale group-to-group collaboration environment, it also opens up opportunities for group participation in presentations, symposia, and workshops as well as collaborative research.

The community data portal
Sustainable strategies for enabling both providers and consumers of earth system data
Scientific data are at the heart of most of our research activities, and we need to share these data among ourselves and with a geographically distributed community.

Working with divisions and programs across UCAR and NCAR, SCD has initiated a forward-looking pilot project called the Community Data Portal (CDP). The CDP is targeted directly at elevating our organization's collective ability to function as a data provider with a coherent web-based presence.

  CDP user interface

During FY2001, several pilot sub-projects were undertaken, including the ACACIA ARCAS system (ACACIA), Reanalysis-2 data (SCD/DSS), CCM diagnostic tools (CGD), TIME-GCM data (HAO), vegetation/ecosystem data (VEMAP), and distributed climate data analysis (COLA). Our efforts here have been extremely well received and have now grown into the role of a formal NCAR Strategic Initiative.

The Earth System Grid
In 1999, SCD joined with several DOE labs in a DOE-sponsored research project called The Earth System Grid (ESG). This effort was aimed at developing Grid-based technologies that facilitated management and high-speed access to large-scale distributed climate model data. During FY2001, we put some of the ESG technology into production operation for the PCM project, and it now supports a sustained transfer of data from NCAR to NERSC at data rates much higher than previously possible. We joined again this year with several collaborators to submit a new proposal for The Earth System Grid II. Working with Argonne National Laboratory, Lawrence Livermore National Laboratory, the University of Southern California, Oak Ridge National Laboratory, and Lawrence Berkeley National Laboratory, we successfully secured a new research contract to develop and deploy an operational ESG in support of terascale/petascale climate research. The project is a significant opportunity to advance research and computation, and it has already drawn substantial interest that extends into the international community.

Community data analysis and visualization software
Prior to this year, SCD has distributed NCAR Graphics and NCL on a cost-recovery basis. In FY2000, SCD management made the decision to move to an Open Source distribution model for NCAR Graphics and a "free availability" model for NCL (with Open Source planned for the future), and we began the process of implementing the decision. We actually implemented the new distribution modes this year, and since October 2000, there have been roughly 9,000 downloads of NCAR Graphics and 1,000 downloads of NCL. While it represented a significant loss of revenue for SCD, this move was extremely popular with our community and has brought many new users on board. We also continued to work with the new Weather and Research Forecast (WRF) model team to extend NCL's usefulness for the WRF community. New features included enhanced support for the increasingly popular HDF-5 format and completion of a significant portion of the development cycle for a new high-resolution map database for NCL. In a similar vein, we began the process of integrating our enhanced version of the popular Vis5D software into a new Open Source framework for community sharing, development, and usage. Complementing all of this was a substantial amount of experimental work with the scripting language Python and the development of new visualization software for educational uses under the auspices of our Visual Geophysical Exploration Environment (VGEE).

A new architecture for terascale data access, analysis, and visualization
Late in the year, SCD management convened a team tasked with re-examining the resources we provide to deal with data. This included web-based data access, future efforts in data portals, research projects in distributed data, data visualization, and post-computation processing and analysis.

The strategic planning process that ensued led to the definition of a new architecture that integrates a number of functions and systems, and moves various testbed efforts (SANs, MSS Proxy) into a new production phase. The new architecture is built on the concept of large shared data objects (1 TB or more) and speaks directly to user productivity, efficient use of computational and storage resources, and the support of new efforts in data analysis and visualization.

  New visualization architecture

This new effort positions SCD as a leader in providing a powerful, balanced, and--most of all--productive environment. Acquisition of hardware and software began this year, while integration and deployment will happen in FY2002.

Assistance and support for NCAR's research community

Primary responsibility for helping researchers efficiently produce valid simulations on NCAR supercomputers is managed by the User Support Section (USS) to support part (e) of our mission. USS also supports researchers and SCD staff by providing supercomputer usage statistics and web publication services, and by supporting onsite servers, workstations, and application software at the NCAR Mesa Lab. USS FY2001 highlights include:

Extensive model code conversions facilitated by the SCD Consultants
Many researchers using NCAR supercomputers run models that were designed for the parallel vector processing systems such as the Cray Research X-MP, Y-MP, C90, and J90 systems. As the last two Cray systems at NCAR are being decommissioned in FY2002, the code that simulates atmospheric and related physics and chemistry on these vector systems has to be converted to run on one of the newer symmetric multiprocessor architectures.

The most promising of these platforms is the IBM SP system. In the past five years, the SCD Consulting Office guided the programmers and scientists on more than 19 major community models in converting from the old architecture to the new. This radical difference in architectures requires a huge investment in programming time and effort, and with a staff of five software engineers, the Technical Consulting Group has successfully handled a shift from straightforward usage questions to more complex development and design questions in the past year.

In addition, the consultants developed extensive user documentation and organized numerous training classes and workshops to help researchers work productively on the IBM SP systems blackforest and babyblue. As of September 2001, 65% of researchers using SCD computers were performing more than 90% of their computational work on the IBM SP (blackforest) or the SGI Origin 2000 (ute) rather than the Cray computers.

Enabling infrastruture

Primary responsibility for maintaining and operating NCAR's supercomputing environment is managed by the Operations and Infrastructure Support Section (OIS) to support part (e) of our mission. OIS FY2001 highlights include:

Infrastructure upgrades to support ARCS
In November 2000, OIS began to identify and upgrade those portions of the computing center infrastructure that would need to be augmented to support the new ARCS equipment. This specification of the needed infrastructure equipment was a moving target, as the possible machine configurations were in constant flux. After significant analysis, SCD identified the upper boundary of possible equipment to be delivered. OIS then determined that the existing air conditioning was sufficient, but the power distribution was not. OIS then began specifying, procuring, and installing all the necessary equipment so the facility could be ready for the ARCS delivery.

After nine months of constant work and some overtime, the power distribution system passed the startup inspection and went online the first of October, just in time for the delivery of the first wave of ARCS equipment. The project was estimated to cost slightly more than $600,000, and it came in slightly over $500,000.

  Power distribution equipment

These upgrades position the electrical distribution system to support the next three years of the ARCS contract, and they form the cornerstone of a solid infrastructure for SCD to continue providing reliable, production-oriented services and equipment as tools for science.

SCD portal genesis
The Applications Group within OIS has started on an ambitious multi-year project to architect, implement, and deploy an application portal within SCD. This portal will provide web-based access to SCD's suite of resources and services. In addition, the portal is designed to be extensible and customizable so researchers can better manage the flow of information needed for their area of research. The project, while in its early stages, accomplished a great deal in the past year. The portal has moved from strictly a vision to early proof-of-concept systems. Several decisions were made, including identifying key technologies and the development approach. Currently, a preliminary job-submission utility is working, as well as an interface to some portions of the Mass Storage System. In the coming year, these early web-enabled services will be provided to researchers to solicit feedback and evolve the portal.


The procurement of ARCS, NCAR's Advanced Research Computing System

In February 2000, SCD began the process of drafting the technical requirements for new supercomputing equipment to support the NCAR Community and the Climate Simulation Laboratory (CSL) in preparation for issuing an open and competitive procurement. SCD engaged the UCAR Contracts office, and requested that each of the NCAR division directors appoint a technical representative who could participate with SCD in drafting the Request For Proposal (RFP). The RFP2000 project, as it was known then, was formally inaugurated on 10 March 2000 with the first meeting of the Technical Committee. The committee was comprised of 18 SCD staff members, 8 scientific advisors from NCAR's science divisions, and 4 members from UCAR Contracts. Later in the process, three members of the NCAR community were added as an External Review Team.

The committee soon agreed to provide the procurement effort with a more appropriate name: the NCAR Advanced Research Computing System (ARCS). The initial draft of the ARCS technical requirements was assembled by SCD; these were the requirements it believed would best serve the users of NCAR's computing facility. In addition, requirements were examined from the 1995 NCAR Accelerated Computing Environment (ACE) RFP, and recent RFPs from a number of peer centers in North America and Europe, including the Geophysical Fluid Dynamics Lab (GFDL), the National Center for Environmental Prediction (NCEP), and the NOAA Forecast Systems Laboratory (FSL). Throughout the summer of 2000, the ARCS committee evaluated the scientific needs of the NCAR Community and the CSL and forged those into explicit requirements for the ARCS RFP.

In the meantime, the ARCS benchmark suite, including performance assessment kernels, system and I/O subsystem tests, and representative models run at NCAR, was being assembled and tested in preparation for release with the RFP. A Vendor Review Draft of the RFP was released for comment in late August 2000. This attracted the interest of 14 prospective offerors, all of whom were put under a UCAR non-disclosure agreement. Of those interested offerors, 9 responded with comments.

As the final revisions were made to the RFP documents and the benchmark suite was being completed, SCD established a secure website for the distribution of the RFP documents to prospective offerors; without the website, these materials and the ARCS benchmark suite would have required 5 CD-ROMs and over 200 pages of printed material per interested offeror. On November 1, 2000, the ARCS RFP was released with a due date for proposals of 9 January 2001. The RFP requested three- and five-year proposals, and stated: "The ARCS should provide a highly productive computational environment for the development and execution of complex, long-running, computationally intensive earth system models. The paramount objective is to advance atmospheric and related-science research across broad fronts (e.g., turbulence, micro- and mesoscale meteorology, weather, atmospheric chemistry, ocean modeling, climate prediction, paleoclimate, upper atmosphere, solar-terrestrial interactions, the solar atmosphere and its interior, etc.). NCAR seeks the highest level of computing capacity and capability to address this central objective.

The due date for the initial proposals was extended for two weeks, to January 23, 2001, upon the request of several offerors and the consideration of SCD and the ARCS committee for the Christmas and New Year holidays. On December 6, 2000, SCD hosted a Pre-Proposal Vendor Conference at NCAR's Mesa Lab to provide the opportunity for prospective offerors to view the computer room infrastructure and physical plant and to provide a uniform set of questions and answers. All formal communications with prospective offerors, including amendments to the RFP and benchmark suite, additional questions posed by the offerors and formal responses by the committee, were conducted via the secure website. We estimated that the secure website medium provided a uniform mechanism for communications, and expedited the process significantly. NCAR was commended by a number of prospective offerors for utilizing this information technology as a mechanism for delivering information to them and for allowing them to submit their proposals.

While awaiting proposal submission, SCD developed spreadsheets to be used by the three ARCS subcommittees, or teams (business, price, and technical), to objectively assess each of the proposals. These spreadsheets incorporated an automatic scoring mechanism tailored to the evaluation specifications provided by UCAR Contracts. The technical evaluation spreadsheet included all 416 attributes of the technical requirements of the RFP.

Of the 19 prospective offerors, 3 submitted proposals for the ARCS system on January 23, 2001. Those three offerors were Compaq, IBM, and SGI. SCD had developed a second secure website for the exclusive use of ARCS committee members to electronically access all offeror-supplied information, including proposals, benchmark results, pricing and business information. Access control mechanisms were put in place so that during the proposal evaluation and assessment phase, the technical team could access only the technical components of the proposals, thus preventing the business and price proposals from influencing the technical proposal evaluations.

Each of the 14 members of the ARCS technical team conducted a thorough assessment of each of the technical proposals, and used the technical evaluation spreadsheet to submit their individual assessments of the proposals. These 42 spreadsheets were electronically combined and analyzed to produce summary reports of the strengths, weaknesses, and deficiencies of the proposals. Additionally, the benchmark results were analyzed and used to generate performance projections for each model included in the benchmark suite for each of the three proposals. In a series of meetings, these results were presented and considered by the ARCS committee.

The ARCS committee unanimously agreed that all proposals were relatively disappointing and sought a second round of proposals, termed BAFOs (Best And Final Offers), due March 26, 2001. While awaiting the BAFO proposals, the ARCS committee continued its analysis of the initial proposals and benchmark results and submitted sets of clarification questions to the offerors.

Upon receipt of the BAFO proposals, the ARCS committee reiterated the evaluation and assessment process conducted on the previous proposals. New performance projections were generated, presented, and considered by the committee. The committee also reviewed offeror responses to clarification questions and issued new sets of clarification questions to the offerors. Though the BAFO proposals were more favorable, the committee still felt that the offered equipment fell disappointingly shy of expectations, with too little equipment delivered early and major upgrades delivered too late in the contract window. Thus a third set of proposals were solicited from the three offerors with common guidance to the offerors that we sought to at least double computational capacity with the first equipment drop and achieve a sustained performance exceeding 1 TFLOP by 2005. The due date for these Supplemental Proposals was set at May 17, 2001.

Two of the three offerors provided revised proposals on May 17, while the third advised that their final proposal is unchanged from their BAFO proposal. The ARCS committee again evaluated the proposals, reviewed new performance projections based on them, and in a series of meetings during May and June debated the relative merits of the supplemental proposals. SCD presented the results to the NCAR Directors and members of the SCD Advisory Panel in June, and a formal recommendation was presented to UCAR President Rick Anthes, NCAR Director Tim Killeen, and the NCAR Directors on June 21, 2001. SCD compiled an ARCS Evaluation and Recommendation Report and submitted it as supplementary supporting information on July 5. NCAR and UCAR management concurred with the ARCS committees and SCD's recommendations, approving that contract negotiations first be entered with IBM. These negotiations took place in late July, with final contractual terms and conditions completed on August 14. UCAR Contracts submitted the IBM ARCS Contract to the National Science Foundation on September 14, which approved the contract on October 5, 2001.

The ARCS RFP results

The ARCS system, as contracted with IBM, will provide a phased introduction of new computational, storage, and communications technologies through the life of the contract. This will allow NCAR's Scientific Computing Division to maintain a stable, state-of-the-art production facility for the next three to five years. The initial delivery augments the current blackforest system by more than doubling its computational capacity, from 0.9 to 2.0 peak TFLOPS, and provides a five-fold increase in disk storage capacity. A second delivery, in September 2002, will introduce IBM's next-generation processor (POWER4), node (Regatta), and switch (Colony) technologies, adding almost 5 peak TFLOPS, upgraded switch communications, and 21 TB of new disk storage. This system will be called bluesky. In the fall of 2003, the Colony switch will be replaced with IBM's next-generation Federation switch technology, which provides much lower latency and higher bandwidth than does the Colony switch.

If NCAR chooses to exercise the two-year contract extension option, in the fall of 2004 the bluesky system will be upgraded with an additional 4 peak TFLOPS and 32 TB of new disk storage. The following tables summarize the major attributes of the ARCS systems through the contract lifetime:

Date

System

Node type

Processor

Three-year contract

Sept. 2001 blackforest upgrade Winterhawk-2 Nighthawk-2 375 MHz POWER3-II
Sept. 2002 bluesky with Colony/PCI switch Regatta ~1.35 GHz POWER4
Sept. -
Dec. 2003
Federation Switch upgrade (blackforest removed after Federation acceptance)
Two-year extension option

Sept. -
Dec. 2004
bluesky upgrade Regatta/Armada ~2.0 GHz POWER4-GP

Date

System

Total disk capacity (TB)

Total memory (TB)

Total peak TFLOPS

Three-year contract

Sept. 2001 blackforest upgrade 10.5 0.75 1.1
Sept. 2002 bluesky with Colony/PCI switch 33 2.8 5.81+
Sept. - Dec. 2003 Federation Switch upgrade (blackforest removed after Federation acceptance)
Two-year extension option

Sept. - Dec. 2004 bluesky upgrade 65 3.8 8.75+


The "+"es in the TFLOPs column table indicate that negotiated capability performance requirements for the ARCS may require the installation of additional capacity.

The contract established minimum capability performance requirements and a continuing process for maintaining current versions of NCAR models as part of the suite of codes to be used to measure the system performance. The minimum model capability requirements, which can also be thought of as model speed-up relative to blackforest, are 1.0x for the blackforest upgrade, 3.1x for bluesky, and 4.6x for the bluesky upgrade. Failure of IBM to meet these capability requirements will result in a corresponding additional equipment delivery that will increase the total capacity, thus peak TFLOPS, of the system. IBM and SCD have agreed to work together to improve the user environment and user support services that will be provided to NCAR and CSL. This agreement covers many aspects of the ARCS, including on-site IBM applications specialists, training in advanced programming, performance analysis and tuning techniques, and a more efficient process for reporting, escalating, and resolving compiler and tools problems.

The agreement with IBM also includes early access to new hardware and software technologies, Live Test Demonstrations of those technologies prior to equipment delivery, new software feature and tools development, training for systems engineers and operators, and specialized training tailored to the needs of the user communities served by NCAR.

Additionally, the agreement with IBM will provide the opportunity for NCAR to participate in IBM's Blue Light HPC project. Blue Light, like IBM's Blue Gene project, is an exploratory effort of IBM's Exploratory Server Systems department at IBM Research to develop future PetaFLOPS supercomputing systems. NCAR's collaboration with IBM in Blue Light holds the promise of significant and revolutionary advancements in climate, weather, and Earth systems models, and will provide IBM with valuable input on hardware and software design.

The first ARCS equipment delivery, the blackforest upgrade, occurred on Friday, 5 October 2001 with the delivery of 13 SP frames and pallets of cables and support equipment. By SCD's estimate, once the blackforest upgrade equipment is online, NCAR's IBM SP system will be ranked at the number 8 position on the TOP500 Supercomputer Sites list (see http://www.top500.org/lists/2001/06/top500_0106.pdf). While SCD expects NCAR to slip out of the top ten by the end of 2001 due to other centers upgrading their systems, the installation of the bluesky system in the second phase of ARCS, with IBM's POWER4-based processors, has the potential of placing NCAR within the top five.

SCD looks forward to the future of high-performance computing and maintaining a state-of-the-art computational and storage facility through the next three to five years. It is SCD's intent that the ARCS will not only maintain, but also enhance, NCAR's leadership role in Earth science research.


High performance computing

Maintaining NCAR's production supercomputer environment

The production supercomputer environment managed by SCD for NCAR has evolved over the years. During the last 17 years, SCD has brought NCAR's science into the multiprocessing supercomputer world. Prior to the introduction of the four-CPU Cray X-MP in October 1986, all modeling was performed with serial codes. Since then, the focus has been on redeveloping codes to harness the power of multiple CPUs in a single system, and most recently, in multiple systems.

During the last 17 years, SCD has deployed a series of parallel Vector Processor (PVP) systems ranging from a 2-CPU Cray Y-MP to a pair of 24-CPU Cray J90se systems. Massively Parallel Processing (MPP) systems included the Cray T3D with 128 processors and the Thinking Machines CM2 and CM5 systems. Most recently, Distributed Shared Memory (DSM) systems have been deployed; these include the Hewlett-Packard SPP-2000 and Silicon Graphics Origin2000, the Compaq ES40 cluster, and the IBM SPs.

The following diagram shows the systems that SCD has deployed for NCAR's use since its inception. The systems shown with blue bars are those deployed for production purposes, those shown in red were (are) considered experimental systems.

history of supercomputing at NCAR

Supercomputing systems deployed at NCAR

In 1986, with the first multiprocessor system (the Cray X-MP/4) on NCAR's floor, SCD could deliver on average approximately 0.25 GFLOPS of sustainable computing capacity to NCAR's science. In the roughly 17 years since, that sustained computing capacity has grown significantly.

Computing power at NCAR

FY2001 production system overview

There were no major acquisitions for the production supercomputer environment during FY2001. There were major system software upgrades performed on all major supercomputers through the year.

Supercomputer systems maintained during FY2001

DSM systems:

PVP systems:

Production system performance and utilization statistics

At the end of FY2001, the production supercomputer environment managed by SCD for NCAR includes two Cray supercomputers, two IBM supercomputers, four SGI supercomputers, and one Compaq nine-node ES40 cluster. The following tables provide average utilization and performance statistics for the production supercomputer systems SCD operated in FY2001.

In addition, SCD publishes monthly usage reports at http://www.scd.ucar.edu/dbsg/dbs/ These reports provide summary information on system usage, project allocations, and General Accounting Unit (GAU) use.

Production systems for FY2001: Average performance and utilization statistics

System name

Hardware/ #PEs

Notes

GFLOPS

Utilz'n

User

Idle

System

WaitIO

IOfs

IOswp

babyblue IBM SP/40 CSL and Community -- 37.6% 37.8% 59.1% 0.3% 0.1% -- --
blackforest IBM SP/512 CSL and Community -- 67.1% 68.0% 31.0% 0.6% 0.3% -- --
prospect Compaq ES40/36 Community -- 59.5% 60.2% 38.1% 1.7% -- -- --
chipeta Cray J90se/24 Community 1.547 89.9% 91.0% 5.9% 3.0% 0.4% -- --
dataproc SGI O2K/16 CSL and Community -- 28.5% 28.6% 61.6% 5.4% 4.1% 72.7% 0.0%
mouache SGI O2K/4 CSL and Community -- 16.5% 16.6% 81.8% 1.2% 0.3% 16.3% 7.4%
ouray Cray J90se/24 Community 1.499 85.3% 86.1% 10.6% 3.3% 2.0% -- --
ute SGI O2K/128 CSL -- 79.8% 80.2% 15.5% 3.1% 1.2% 76.2% 4.1%
utefe SGI O2K/8 CSL -- 46.8% 47.0% 44.9% 4.8% 3.0% 53.9% 0.0%
 

Note: "GFLOPS" is the average number of floating point operations per second (in billions) during the measuring period; "Utilz'n" is the average user utilization of the system (system downtime counts against utilization); "User" is the percent of uptime occupied in performing computation for user processes; "Idle" is the percent of uptime spent idle; "System" is the percent of uptime consumed in system overhead; "WaitIO" is the percent of uptime spent awaiting I/O completion; "IOfs" is the percent of the WaitIO time spent in performing user filesystem I/O; and "IOswp" is the percent of the WaitIO time spent in performing process swapping/paging.

 

End-FY2001 production supercomputer systems

The SCD supercomputer resources are comprised of two separate computational facilities: the Climate Simulation Laboratory (CSL) and Community Computing facilities. Some systems, such as the new IBM SP systems and the "dataproc" system are shared between these two facilities. The following sections describe the supercomputer systems available in these two facilities.

CSL facility

The Climate Simulation Laboratory facility provided the following supercomputer resources at the end of FY2001:

Climate Simulation Lab facility, FY2001 configuration

  System

# CPUs

GB
memory

Peak
GFLOPS

Notes

Dedicated: IBM SP (blackforest) 256 128 384.0 512 total system batch CPUs; 256 dedicated to CSL
Dedicated: SGI Origin2000 (ute) 128 16 64.0 --
Dedicated: SGI Origin2000 (utefe) 8 8 4.0 Front-end system for ute
Shared: IBM SP (babyblue) 48 24 72.0 Shared new-release test platform; available for user use
Shared: SGI Origin2000 (dataproc) 16 32 8.0 Shared with Community for data analysis and post-processing applications

Community Computing facility

The Community Computing facility had the following supercomputer resources available at the end of FY2001:

Community Computing facility, FY2001 configuration

  System

CPUs

GB
memory

Peak
GFLOPS

Notes

Dedicated: IBM SP (blackforest) 256 128 384.0 512 total system batch CPUs; 256 dedicated to Community
Dedicated: Cray J90se (chipeta) 24 8 4.8 --
Dedicated: Cray J90se (ouray) 24 8 4.8 --
Dedicated: Compaq ES40 cluster (prospect) 36 36 72.0 --
Shared: IBM SP (babyblue) 48 24 72.0 Shared new-release test platform; available for user use
Shared: SGI Origin2000 (dataproc) 16 16 8.0 Shared with CSL for data analysis and post-processing applications

Key maintenance activities

During FY2001, SCD provided ongoing maintenance activities to ensure the integrity and reliability of existing computational systems. Some of the key areas were:

Maintain supercomputer operating systems
SCD stays apprised of major software releases from Cray Research and will carefully schedule upgrades to the production system and product set software based on the judged stability of those upgrades in the NCAR production environment. The Cray systems are considered to be in "maintenance mode," thus no significant enhancements or software upgrades were undertaken in FY2001. SCD also continued to provide major system support for SGI Origin2000, IBM SPs, and the Compaq ES40 cluster systems.

Maintain stability and reliability of systems
One of the most significant attributes of the NCAR computational environment is its overall stability and reliability. For instance, the NCAR Mass Storage System has a reputation for reliability, and SCD has in the last year deployed a number of high-availability fileserver systems. This reliability and stability does not come easily; it stems from a combination of choosing reliable, stable vendor products and using proven, fail-safe system administration and maintenance techniques. SCD will continue to focus on ensuring, in whatever ways possible, highly stable and reliable systems and systems operations.

System monitoring
Over the years, SCD has developed a large number of system monitoring procedures, techniques, and tools. SCD continued to enhance and utilize its collective experience to maintain the stability of the existing production systems through this proactive monitoring. In addition, SCD continued to enhance its monitoring tools, techniques, and procedures, and SCD automated a number of procedures for detecting system failure or trouble. This automation was integrated with commercial alphanumeric paging technology to provide more rapid alert mechanisms to SCD operations and systems staff and thus reduce the amount of time that systems are unavailable to the NCAR user community when they do fail.


Computational science research and development

The SCD Computational Science Section (CSS) conducts research and development in computational science, algorithms, numerical methods, and software engineering in support of the NCAR and SCD mission. The results of our work are shared through direct publications, presentations, collaborations, shared software development, and mathematical libraries. Our efforts are focused in the following five areas:

The section engages in collaborative research and development projects with groups within NCAR and from other institutions and agencies. These projects benefit the broader NCAR community. Currently, CSS is involved in projects in conjunction with the Climate and Global Dynamics (CGD) Division of NCAR and with support from the Department of Energy's (DOE) Climate Change and Prediction Program (CCPP), to develop a state-of-the-art coupled climate model framework that will run efficiently and effectively on distributed-memory parallel computers. We are also working on collaborations to develop a direct numerical simulation turbulence code and studying coronal mass ejection. These projects are described below.

CSS staff are leading a large national collaboration to develop a robust, flexible set of software tools to enhance ease of use, performance portability, interoperability, and reuse in Earth system models.

CSS staff develop software packages to make use of our research results and our numerical and computational expertise. We are in the process of developing highly efficient numerical routines for use by the atmospheric and related sciences community. These codes are optimized for use on RISC-based microprocessor systems.

Finally, CSS leads NCAR's technology tracking (performance monitoring and benchmarking studies) of both hardware and software. This work evaluates and ensures the efficient use of future computing resources and is critical in selecting the most appropriate computers for the future production computing needs of NCAR and the university community. On the software side, staff play an active role evaluating programming languages, programming environments, and paradigms. A new thrust for us is evaluating the new IA-64 architecture.

Research

Spectral element dynamical core

Climate modeling is a grand challenge problem where scientific progress is measured not in terms of the largest problem that can be solved but by the highest achievable integration rate. These models have been notably absent in previous Gordon Bell competitions due to their inability to scale to large processor counts. Rich Loft, Steve Thomas, and John Dennis have developed a scalable and efficient spectral element climate dynamical core. A new semi-implicit time-stepping scheme accelerates the integration rate relative to an explicit model by a factor of two, achieving 130 years per day at T63L30 equivalent resolution. Simulations have been performed for the standard shallow water and Held-Suarez climate benchmarks on IBM SP clusters. The explicit T170-equivalent multi-layer shallow-water model sustains 343 GFLOPS at NERSC, 206 GFLOPS at NPACI (SDSC) and 127 GFLOPS at NCAR. An explicit Held-Suarez integration sustains 369 GFLOPS on 128 16-way IBM nodes at NERSC. More recently the code sustained 465 GFLOPS on the Compaq ES45 cluster at PSC.

This work was recognized with a $500 award and honorable mention (second place) in the "Special" category of the IEEE/Gordon Bell Prize competition at SC01. The IEEE/Gordon Bell Prizes were established to reward practical uses of parallel processing and are given for the best performance improvement in an application. They are considered to be the most prestigious awards in supercomputing.

CSS's prototype spectral element dynamical core, SEAM, has achieved several important milestones. An MPI-2-based restart capability has been developed and tested with the primitive equations. The correctness of the explicit primitive equation core has been validated on both the 1,200-day integrations of the Held-Suarez test case and 30-day integrations of the Baroclinic Instability test case of Williamson and Jablonowski.

The future development plans for SEAM are well defined for the next year. These are as follows:

An efficient spectral dynamical core for distributed memory computers

Building a dynamical core that would run efficiently on low-cost commodity clusters is a challenge. The small caches often present on these types of machines require a careful implementation to guarantee efficiency.

In this code, called BOB ("Build On Beowulf"), developed by R. Loft and L. Rivier, the user can choose to either use the shallow water model or the dry primitive equation model to compute. Several specificities guarantee BOB's efficiency in its implementation of the spectral method. They are principally a one-dimensional decomposition and transposition method insuring load balancing among processes, a cache-blocked implementation of the Legendre transform, a "compute on the fly" of the associated Legendre polynomials producing a lower memory footprint at high resolution. These features enable high-resolution runs on relatively small-memory machines. The code's performance was compared to two well-known existing codes: PSTSWM (Worley and Toonen 1995) and CCM3 dynamical core (Acker et al. 1996), using standard test cases of atmospheric dynamics. Compared to PSTSWM, BOB shows better timing results, particularly at increasing resolutions where the cache effects become important (L. Rivier, R. Loft, L. M. Polvani, to appear in Monthly Weather Review).

The code is currently being packaged in a "user-friendly" way to make it accessible on the Web to share this useful tool with researchers in the climate and geophysical turbulence community who want to participate in the development of extensions of BOB's capabilities or for those who want to use it for their own scientific studies.

References:
L. Rivier, R. Loft, L.M. Polvani "An efficient spectral dynamical core for distributed memory computers" to appear in Monthly Weather Review.
Worley P. H. and B. Toonen, 1995: A User's Guide to PSTSWM.
Acker T. et al., 1996: User's Guide to NCAR CCM3. NCAR/TN-421+IA.

Turbulence

Rodney James is completing the development a new parallel direct numerical simulation (DNS) turbulence research code in collaboration with Yoshi Kimura (MMM and Nagoya University). The new code is completely object-oriented, written in C++, and uses the FFTs developed for the Spectral Toolkit. Currently, the code is being used to study stratified decaying turbulence at 5123 using the CSS server K2 (a 24-processor Sun 6500). Within the next six months, we plan to perform experiment at 10243 and 20483 on IBM and Compaq parallel systems.

The turbulence code is designed to run very efficiently on scalar processor based servers and parallel supercomputers, and approaches 100 GFLOPS on the NCAR IBM SP, blackforest. The kernel of this code is the parallel real 3D FFT from the Spectral Toolkit, in both multi-threaded shared memory and distributed memory versions.

An efficient spectral transform method for solving the shallow water equations on the sphere

William F. Spotz and Paul N. Swarztrauber developed a new, faster, more memory-efficient spectral model for the shallow water equations on the sphere. The model is based on a true double Fourier expansion of variables, meaning the transformation between spectral and physical space can be accomplished with fast Fourier transforms rather than the slower associated Legendre transforms. It is well-known that this model, by itself, is unstable due to non-isotropic representation of waves near the poles. We solve this problem by projecting the prognostic variables onto the space of spherical harmonics at the end of every time step. For the shallow water equations, this reduces the number of associated Legendre transforms per time step from nine to six, and concentrates the transforms into a projection operator that can be further optimized, in two significant ways. First, the memory requirement is improved by a factor of N (where N is the number of latitude points) by introducing and using a complementary space of basis functions orthogonal to the traditional associated Legendre functions. Second, the operation count for the projection operator is cut roughly in half by using an orthogonal complement representation of the projection matrix, which leads to a faster algorithm for half the zonal wave numbers. The projection is still an O(N3) algorithm, but is twice as fast as doing a traditional forward transform followed by a backward transform.

The Eulerian version of the double Fourier model that makes use of the projec tion is remarkably like the traditional spectral transform method. It solves the vorticity-divergence form of the shallow water equations by advancing spectral coefficients forward in time. All of its communication on parallel computers is concentrated in nine transposes per time step of quantities between spectral coefficients decomposed by zonal wave number and physical grid quantities decomposed by latitude. Differences between the double Fourier and traditional spectral model include an equally spaced latitude grid with points at the poles; fast Fourier transforms in the meridional direction; simple tridiagonal solvers required for both time-stepping and the solution of elliptic equations required for velocity calculations and semi-implicit time-stepping; and the projection at the end of the time-step. There is a significant savings in the memory used, which allows higher-resolution models. Experimental timings demonstrate a savings comparable to the improved operation count over the traditional spectral model, while maintaining the accuracy, stability, and parallel efficiency of the spherical harmonics.

A semi-Lagrangian (SL) version of the double Fourier model has also been developed in collaboration with Anita Layton of the Advanced Study Program. The primary hope for the SL model was that the projection might not be necessary in combination with the semi-Lagrangian advection and the damping introduced by the interpolation required at the departure points, thus leading to an order-of-magnitude improvement in the operation count. Currently, however, this has not been shown to be the case, and projections have been necessary to stabilize the model. As with other models, though, the SL approach does allow for a much longer time step.

References:
Paul N. Swarztrauber and William F. Spotz, "Generalized discrete spherical harmonic transforms," Journal of Computational Physics, 159(2), 10 April 2000, pp. 213-230.
William F. Spotz and Paul N. Swarztrauber, "A Performance Comparison of Associated Legendre Projections," Journal of Computational Physics, 168(2), 10 April 2001, pp. 339-355.
Anita T. Layton and William F. Spotz, "A Semi-Lagrangian Double Fourier Method for the Shallow Water Equations on the Sphere," in preparation.

A performance comparison of associated Legendre projections

The associated Legendre projection provides a means for accelerating the dynamical core of a global weather or climate model. Therefore, the goal is to determine the fastest possible projection algorithm, of which this paper compares four: the standard method which computes the projection using a forward and backward associated Legendre transform; the direct method, which uses a single projection matrix when this approach results in fewer operations; the fast multipole method; and the weighted orthogonal complement method. Timing results indicate that all the projections behave like O(N3) algorithms up to at least N=200 spectral truncation. For this range of resolutions, the weighted orthogonal complement has the lowest operation count, best cache utilization, and best overall timings.

Convergence rates of the spectral method for initital-boundary value problems

This work by Swarztrauber establishes the likelihood of algebraic convergence (even fractional) of spectral methods for solving initial-boundary-value problems (IBVPs). Approximate spectral solutions are shown to converge algebraically and nonuniformly. These studies lead to the conclusion that, in practice, spectral methods for IBVPs are likely to be algebraically convergent, and that classical stability and convergence analyses must be revisited in the context of initial boundary-value problems. The general theory of compatibility conditions for the differentiability of solutions to initial-boundary-value problems is well known. This paper introduces the application of that theory to numerical solutions of PDEs and its ramifications on the performance of high-order methods. Explicit application of boundary conditions that are independent of the initial condition results in the compatibility conditions not being satisfied. Since this is the case in most science and engineering applications, it is shown that not only does the error in a spectral method, as measured in the max norm, converge algebraically, but the accuracy of finite differences is also reduced. For the heat equation with a parabolic initial condition and Dirichlet boundary conditions, we prove that Fourier method converges quadratically in the neighborhood of t=0 and the boundaries and quartically at later times when the first-order compatibility conditions are violated.

For the same problem, the Chebyshev method yields quartic convergence in the neighborhood of t=0 and exponential convergence for time bounded away from zero. In contrast, the wave equation subject to the same conditions results in inferior convergence rates with all spectral methods yielding quadratic convergence for all t. These results naturally direct attention to finite difference methods that are also algebraically convergent. In the case of the wave equation, we prove that a second-order finite difference method is reduced to 4/3-order convergence and numerically show that a fourth-order finite difference scheme is apparently reduced to 3/2-order. Lastly for the wave equation subject to general initial conditions and zero boundary conditions, we give a conjecture on the error for a second-order finite difference scheme, showing that log N divided by N squared convergence is possible.

The Chebyshev method for computing transient heat flow

In an earlier paper (above), nonuniform algebraic convergence for the spectral method was established for very large class of initial-boundary-value problems. This work by Swarztrauber provides a method for computing boundary layers that are encountered throughout geophysics without performing an analytic asymptotic analysis. The fast Chebyshev pseudospectral method is used to solve the time-dependent plane heat source problem; where a hot plate comes into contact with a cold half plane.

The goal is to efficiently compute the temperature boundary layer that forms instantaneously for time greater than zero. The computations are complicated by (a) the initial condition does not have a Fourier series representation and (b) an increased computing resource is required for a decreasing integration interval T where the boundary layer is more prominent. A .0001 boundary layer at a nanosec is computed accurate to ten decimal digits. These results have application to computing geophysical boundary layers. Finally, a solution at T=.1 is computed quickly and accurate to six digits starting with an initial condition that has O(1) error. This demonstrates that the boundary layer can be ignored by substituting a suitable initial condition that may differ substantially from that given in the statement of the problem.

On computing the points and weights for Gauss-Legendre quadrature

This work by Swarztrauber increases the reliability of the points and weights software for SPHEREPACK. Where earlier software was showing a loss of accuracy for current and projected high resolutions, the new software is now capable of computing up to a million points and weights. The Gauss points cluster quadratically resulting in an O(n) error growth in the points and order n squared error growth in the weights. In 1996, building on the work of Lether, Yakimiw improved this result significantly by "flattening" the weight function w(x) in the neighborhood of the points, which reduced the error in weights induced by an error in the points. This, together with a high-order Newton method, resulted in an O(1) error in the points and an O(n) error in the weights. Here we achieve comparable results by applying Newton's method to the transformed points that are semi-equally spaced particularly near the equator, which provide an accurate initial estimate for Newton's method. Subsequent estimates are obtained by extrapolation. Error tables for the three methods are presented.

Modes of the linear balance equations

Beginning 40 years ago and continuing in current literature, research has focused on computing bounded solutions to the linear balance equation. This work identifies a fundamental problem associated with solving the linear balance equation; namely, the equation has unbounded as well as discontinuous modes. The eigenfunctions are represented as a series in the normalized associated Legendre functions. The coefficients in the series are given as the eigenvectors of an infinite symmetric tridiagonal system of equations. With the singular exception corresponding to the constant function, the modes are neither symmetric or antisymmetric about the equator. The modes are orthogonal and occur in pairs corresponding to eigenvalues with opposite sign. The eigenfunctions are "singular" in the sense that their series representations converge algebraically, resulting in discontinuous modes and modes that are unbounded at the equator.

Work with MMM on time-stepping schemes and solvers

In collaboration with Piotr Smolarkiewicz in NCAR's MMM division, we are examining more efficient spectral preconditioning strategies for elliptic solvers in NWP models.

The elliptic problems for semi-implicit nonhydrostatic NWP models contain cross derivative terms and are nonseparable when a curvilinear coordinate system is implemented. The resulting linear system can be solved using a generalized conjugate residual (GCR) method. A preconditioner must be found that reduces the computation time. Line relaxation schemes have proven to be effective preconditioners, and several authors have reported on their efficiency. We have developed a horizontal spectral preconditioner whereby a separable operator is obtained by dropping cross-derivative terms and averaging metric terms over the computational domain. Neumann boundary conditions are required to close the discrete equations, and thus a discrete cosine transform (DCT) is applied in a three-time-level semi-implicit semi-Lagrangian atmospheric model. The preconditioner is also applicable to non-oscillatory forward-in-time (NFT) schemes with variable coefficients. Real data and mountain wave and convective flows have been used to establish the accuracy and efficiency of the spectral preconditioner. In particular, small-scale forcing is introduced to establish that the solver is robust.

A publication has been submitted based on this work:
S. J. Thomas, J. P. Hacker, P. K. Smolarkiewicz and R. B. Stull, Spectral preconditioners for elliptic problems in nonhydrostatic NWP models, Journal of Computational Physics, submitted, 2001.

Coronal mass ejection

In collaboration with B.C. Low of HAO, we are modeling static structures in solar atmospheres that are in unstable equilibrium. Solutions are to be specified as initial conditions for a full magneto-hydrodynamic (MHD) simulation to better understand coronal mass ejections (CMEs). These are thought to be time-dependent perturbations of the equilibrium state. The resulting elliptic problem leads to several numerical difficulties including a semi-infinite domain where the solution decays linearly in the radial direction. A highly nonlinear near-field solution must be combined with a potential far-field solution of Laplace's equation. A conformal mapping strategy has been adopted along with a Fourier-Chebyshev discretisation and a far-field radiation boundary condition implemented in spectral space.

Software libraries and toolkits

The development and testing phase of the Fourier transform portion of the Spectral Toolkit was completed in FY2001. The FFTs available include both real and complex FFTs, multi-threaded 2D and 3D real and complex FFTs, and distributed 3D real and complex FFTs. The FFTs performance exceeds 800 MFLOPS for a single processor on the latest generation of super-scalar RISC and EPIC (Intel IA-64) processors, achieving 25-40% of the peak rated float-point capability on all microprocessors tested.

The multi-dimensional transforms are based on the transpose method, and use generic, blocked transpose algorithms that are a part of the Spectral Toolkit. Although the current transforms use the Fourier basis, they can be easily adapted to other basis functions when they are available though the use of generic programming techniques incorporated in the Spectral Toolkit.

The first version of the Spectral Toolkit will be available in the first quarter of calendar year 2002.

Frameworks and modeling infrastructure

Over the last few years, the need for software infrastructure for Earth system modeling has grown increasingly apparent. Models and the computational platforms that they run on are becoming extremely complex, leading to excessive time and resources dedicated to solving computational rather than scientific problems. CSS staff are integrally involved in a number of modeling infrastructure projects.

In September 2000, the NASA Office of Earth and Space Science (ESS) released a Cooperative Agreement Notice (CAN) titled "Increasing Interoperability and Performance of Grand Challenge Applications in the Earth, Space, Life, and Microgravity Sciences." The NASA CAN calls for the development of an "Earth System Modeling Framework (ESMF)." In response to this NASA announcement, a collaboration led by Cecelia DeLuca of CSS submitted a coordinated set of three proposals to develop an Earth System Modeling Framework. The participants include NCAR-SCD, NCAR-CGD, NCAR-MMM, NOAA/NCEP, NOAA/GFDL, NASA/GSFC-DAO, NASA/GSFC-NSIPP, DOE/ANL, DOE/LANL, MIT, and the University of Michigan. ESMF application testbeds encompass a wide range of research and production Earth system models and data assimilation systems, including the Weather Research and Forecast Model (WRF), the Community Climate System Model (CCSM), and the NCEP forecasting and analysis suite.

The ESMF will allow diverse scientific groups to leverage common utility and coupling software to solve routine computational problems, and it will provide an interface specification so groups working at different institutions and in different disciplines can generate interoperable software components. NASA has selected the proposals from this collaboration for funding, and there will be a three-year effort initiated in FY2002.

Cecelia DeLuca collaborated with Byron Boville of NCAR CGD on an Opportunity Fund project to develop a set of five low-level utilities for the Community Atmospheric Model (CAM). This project provided seed funding to establish software prototypes and a development environment for the ESMF. Time Management utilities completed during FY2001 were integrated into the CAM and will be bundled with subsequent distributions of the CAM and the Community Climate System Model.

Cecelia DeLuca is a collaborator on the funded SciDAC proposal "Center for Component Technology for Terascale Simulation Software," more commonly known as the Common Component Architecture (CCA) project. The CCA is a large DOE collaboration led by Rob Armstrong of Sandia National Laboratory that is creating a comprehensive interface specification and toolkit for developing interoperable, multi-component applications on HPC platforms. Cecelia is serving as a liaison between the ESMF project, the SciDAC NCAR/DOE CCSM development project described below, the CCSM, and the CCA. The CCA project is a five-year effort that will begin in FY2002. A joint organizational meeting was held at NCAR this year with representatives from each of the projects involved.

Cecelia DeLuca is a co-investigator on the funded SciDAC proposal "Collaborative Design and Development of the Community Climate System Model for Terascale Computers." This project will expand the scope of the 18-month DOE ACPI "Avant Garde" pilot project that is nearing completion. These efforts are restructuring the CCSM for better performance on highly parallel microprocessor-based systems and will increase the modularity and thereby the flexibility of the code. They are also intended to improve CCSM software development practices. During FY2001 Cecelia was involved in the restructuring and documentation of software procedures for the CCSM group, and served as a liaison to the ESMF project.

Proposals

In addition to the NASA CAN effort above, CSS staff led or participated in a wide variety of proposals submitted for outside funding and led the Division's record year for proposal writing. Proposals seeking approximately $30M were submitted, whereas in a typical year there are proposals worth approximately $3M submitted through SCD. In particular, Steve Hammond led an NCAR effort to develop a visionary proposal to NSF in response to the Information Technology Research initiative. This work involved a collaboration of 17 professors from 10 universities plus 12 senior NCAR staff.

A pre-proposal was submitted in November 2000, and then a full proposal was developed and submitted on April 23, 2001. This five-year, $15M effort proposed to develop a Knowledge Environment for the Geosciences (KEG) and defines what the digital revolution means to the Geosciences. The KEG combines the best features of collaboratories, knowledge systems, data mining, and problem-solving environments to meet the challenges faced by geoscience research, education, and policymaking. It addresses the needs of a distributed, multidisciplinary research community analyzing large datasets and executing simulation models at multiple computational centers. This was a very important NCAR effort and an instance of how NCAR is implementing the ideas in the High Performance Simulation Strategic Plan. This proposal was highly regarded and has excellent reviews but unfortunately did not receive funding.

Technology tracking

The Computational Science Section (CSS) tracks developments in high-performance computing technology for SCD and NCAR. CSS staff assess the capabilities of the latest systems (hardware and software) available from vendors and evaluate the performance of these systems on problems typically found in the atmospheric sciences community. These systems, from workstations to supercomputers, are evaluated primarily through hands-on work and benchmarks.

The NCAR benchmark suite was initially developed for use in SCD's 1995 ACE open procurement. These benchmarks have evolved over time to reflect changes in both the systems available and the changes in models. They are representative of the applications currently being run at NCAR and planned to be run in the future, and they are typical of the anticipated NCAR computing workload. They have been used in every major procurement since 1994. Most recently the benchmark suite was used in the ARCS procurement completed this fiscal year. The NCAR Benchmark Suite is thus a comprehensive benchmark suite that is well tested, documented, and portable. Currently there are over 20 benchmarks including 7 full application codes (some of them run in multiple configurations and varying processor counts).

Description of the NCAR benchmark suite

The NCAR benchmark codes are categorized as either kernels or applications. The kernels are a set of relatively simple codes that measure important aspects of the system such as CPU performance, memory bandwidth, efficiency of intrinsic functions, bandwidth and latency of the interconnection network, I/O, and network capabilities.

In addition, the NCAR benchmark suite includes codes that serve as more extensive tests and some full applications. Full applications include the Los Alamos-developed ocean model called POP, the NCAR/Penn State University mesoscale code MM5, the NCAR Community Climate Model CCM3, the coupled climate model PCM, an atmospheric chemistry model MOZART, a 3D magnetohydrodynamics code MHD3D, and a prototype next-generation weather forecasting model WRF.

New technology

U.S. high performance computer manufacturers are predominantly focusing on building their products from commodity processors and offering systems that are clusters of shared-memory nodes. Toward the end of FY2001, CSS acquired a new four-processor SMP with IA-64 processors made by HP. This new Itanium server has been installed and is in the process of being evaluated. The evaluation includes both performance and usability under both HP-UX and Linux. While the software for both OSes is immature and not ready for production computing, the system demonstrates considerable floating-point performance on cache-based codes. For example, the best single-processor performance posted for the shallow water benchmark by the Itanium is 801 MFLOPS compared to 478 MFLOPS for the IBM Power3 processor in blackforest.


Data archiving and management system: The MSS

The NCAR Mass Storage System (MSS) is a large-scale data archive that stores data used and generated by climate models and other programs executed on NCAR's supercomputers and compute servers. At the end of FY2001, the NCAR MSS managed more than 10.9 million files containing a total of over 379 terabytes (TB) of stored data, and the net growth rate of data in the MSS was approximately 10 TB per month. On average, 105,000 cartridges are being mounted per month, approximately 10% (10,000) of these by operators and 90% (95,000) in the StorageTek Powderhorn Automated Cartridge Subsystems (ACS). The StorageTek Powderhorn ACS systems (also called "silos") use robotics to mount and dismount cartridges. On a daily basis, the MSS handles approximately 20,000 requests resulting in the movement of over 1,200 GB of data. During FY2001, data transfers to and from the MSS exceeded 440 TB.

While some of the data stored on the NCAR MSS originate from field experiments and observations, the bulk of the data is generated by global climate-simulation models and other earth-science models that run on supercomputers. SCD therefore faces an increasing demand to archive data from ever-faster supercomputers. Essentially, the faster the supercomputer, the more data there are to be archived. Ever-greater demands for archiving data will result from the growing use of coupled atmospheric/oceanic simulation models.

MSS history

The NCAR Mass Storage System has evolved over the last 15 years. Prior to late 1989, mass storage at NCAR was comprised strictly of offline, manual-mount media. In November 1989, the first STK Powderhorn "silo" was acquired, commencing a new era of mass storage at NCAR. The following figure shows the various technologies that have been used to store critical datasets throughout NCAR's history.

MSS history timeline

During FY2001, the NCAR Mass Storage System grew from 8,329,517 files with a total of 274.02 TB to 10,950,158 files with a total of 379.28 TB. This was an average net growth rate of 8.77 TB per month during FY2001.

MSS today

MSS access methods

The NCAR MSS provides direct storage-device access via a High-Performance Data Fabric (HPDF). The data fabric consists of host computer High Performance Parallel Interface (HiPPI) channel interfaces, non-blocking HiPPI switches capable of supporting multiple bi-directional 100 MB/sec data transfers, and protocol converters that connect the HiPPI data fabric to the IBM-style device control units. The data fabric provides data paths directly between the MSS storage devices and the client compute servers. To utilize this data fabric, SCD has written a file-transport type of interface to enable users to copy files between their host systems and the MSS. The data fabric can support 16 independent file-transfer operations between the storage devices and the compute servers, with 4 transfers sustaining 3 MB/sec each and 12 transfers sustaining 10 MB/sec each, for an aggregate total of 132 MB/sec.

HiPPI technology continues to be deployed only in a niche market. It has not shown signs of spreading into the commodity marketplace, and as a result the cost of HiPPI technology has remained high and the number of HiPPI vendors is dwindling. The lack of availability of and support for HiPPI technology is becoming a critical issue to the continued operation of the MSS. Replacement technologies are on the horizon, but they are not yet widely available nor are they functional enough to immediately replace HiPPI. Promising replacement technologies are Fibre Channel and Network Attached Storage Devices. Fibre-Channel-attached RAID units are available today at extremely attractive costs. Over the next few years, the number and types of available Fibre-Channel-attached devices are expected to grow and include tape storage. Once tape devices can be Fibre-Channel attached, SCD will evaluate the replacement of our HiPPI fabric with Fibre Channel.

Network Attached Storage Devices (NASD) is another emerging technology that is being closely tracked by SCD. Today a handful of vendors supply Network File System (NFS)-based NASD devices. Some vendors are developing "local-disk"-attached NASD products using Fibre Channel and HiPPI connections. SCD's current strategy is to deploy a Fibre Channel infrastructure (starting with equipment purchased for the Fibre Channel testbed), and possibly add NASD technology at a later time, provided that suitable devices become available from vendors. The diskfarm replacement project, which is scheduled for completion in FY2002, will be the first production use of Fibre Channel RAID as a component of the MSS. While MSS hosts will still use HiPPI for accessing MSS tape devices, access to the FC-RAID diskfarm will not require HiPPI. Hosts will initially access the diskfarm via TCP/IP connections (Gig-E or 100baseT), but the MSS servers will be able to access the diskfarm directly via Fibre Channel. Later on, this capability may be extended to other MSS host machines.

Ultimately, we plan to eliminate the need for HiPPI altogether. This will require not only an alternate path to the diskfarm, but also an alternate path to tape devices from the MSS hosts. This will allow us to phase out our existing ESCON and BlockMux tape devices, and replace them with newer technologies capable of transferring data at much higher data rates than what is currently possible using our existing HiPPI-based HPDF.

MSS storage hierarchy

The NCAR MSS currently uses two levels of storage: online and offline. The most frequently accessed data are kept on the fastest storage media, which is the online storage devices: 180 GB of IBM 3390 Model 3 disks (to be replaced in FY2002 with a Fibre Channel RAID unit), and five StorageTek Powderhorn ACSes. The Powderhorn ACSes use StorageTek 9840 as well as StorageTek 9940 technology. Currently, the NCAR MSS has five ACSes providing a total online capacity of approximately 1 petabyte. Lower in the storage hierarchy is a 3490E offline cartridge tape library holding 116,000 cartridges that can be staged with one of the 16 external IBM 3490E manually mounted cartridge drives. These media are in read-only mode and a migration off this media was started in FY2000. StorageTek 9840 and 9940 drives have been added to the offline storage level for providing secondary copies of the Powderhorn-resident files. In addition, there are almost 5,000 manual-mount STK Redwood cartridges that contain secondary copies of Powderhorn-resident files. A migration off these cartridges will be initiated in FY2002.

"Data ooze" refers to the massive task of transferring tens of terabytes of data from old media to modern media before the equipment that uses the old media becomes obsolete. This task by itself is straightforward; however, this data ooze must be handled as a background task while the processing and storage components of the system remain fully dedicated to supplying prompt, 24-hour-per-day service to users. When the ooze is complete, the total capacity of the offline archive (assuming no reduction in the offline archive's available floor space in the SCD machine room) will exceed 1 petabyte. In FY2000, a data ooze from StorageTek 4490 media to 9840 media was completed.

In FY2001, all data on silo-resident StorageTek Redwood media was oozed to 9840 and 9940 media, and all of the silo Redwood drives were decommissioned (see SD-3 (Redwood) tape migration). In FY2002, the MSS Group is planning to begin oozing all data currently on IBM 3490 and STK manual-mount Redwood media to 9840 and 9940 media.

Expansion of the MSS storage hierarchy is planned over the next five years with the introduction of new tape technologies, new ACSes, and with the integration of a front-end file server having its own HSM to offload active and temporary data. The MSS archive will become a back-end store for the file server accessed only by the front-end HSM. A single global name space will be provided for all data managed by SCD. In order to accomplish this, the MSS Group will continue to evaluate hardware and software solutions being developed by vendors throughout FY2002.

MSS import/export capability

Another important capability of the NCAR MSS is the ability to import and export data to and from external portable media. Importing data involves copying data from portable media to the MSS data archive, while exporting data involves copying data from the MSS data archive to portable media. Import/export allow users to bring data to NCAR with them, as well as take data away. Import also allows data from field experiments to be copied to the NCAR MSS archive.

Options to exchange data with smaller satellite storage systems are being investigated. Using this technique, data generated at NCAR could be transferred to remote sites for further analysis. The NCAR SCD storage model would thus be geographically distributed, rather than centrally located and administered.

In addition to 3480 and 3490E cartridge tapes and 9-track round tapes, the NCAR MSS also offers import/export to single and double-density 8mm Exabyte cartridge tapes. The deployment of an MSS-IV Import/Export server in FY2000 provided the ability to support many more device types, such as CD-ROM, DAT, and newer Exabyte media, to name a few.

MSS accomplishments for FY2001

Ongoing projects

SD-3 (Redwood) tape migration

Between June and September 2001, the Mass Storage Systems Group (MSSG) undertook and completed the migration of approximately 75 TB of data from SD-3 (Redwood) media to newer media types. Although it held great promise when introduced in the mid-1990s, Redwood is now considered an "end-of-life" technology, and the vendor (StorageTek) has consequently imposed significant increases in maintenance costs for Redwood tape drives. The migration of this 75 TB of MSS data allowed the MSS group to decommission 10 of its Redwood drives. (The remaining two drives are used only to read secondary copies of MSS files in those rare occasions that the primary copy--stored on non-Redwood media--cannot be read.) The decommissioning of these 10 Redwood drives represents a potential savings of approximately $16,000 per month in maintenance costs. In addition, as part of this project, key users were lobbied to remove as many Redwood-resident MSS files as they could: this resulted in the removal of an estimated 9 TB of unneeded data from the MSS.

University of Illinois NCDM collaboration

Together with Lawrence Buja of CGD, MSSG set up and hosted a "Data Space" server cluster for the National Center for Data Mining (NCDM). NCDM is part of the Laboratory for Advanced Computing at the University of Illinois at Chicago (UIC). The server cluster consists of three Linux systems providing access to climate model data produced by NCAR via the NCDM's Data Space transfer protocol. The system was showcased at SC2000, demonstrating real-time data access from the Dallas show floor to the server housed in the SCD computer room.

Remote MSS host support

Two remote host machines at Foothills Lab (one owned by MMM, the other by ATD) were connected to the HPDF via Serial HiPPI, using the BRAN network. This was a new capability that gave the two machines direct access to the MSS, even though they are over 10 km away from the Mesa Lab.

Disk cache simulator

A simulator was built that uses actual MSS logs from the past year to evaluate the effect of putting various-sized disk caches in front of the MSS tape archive. This was a critical part of the diskfarm replacement/expansion project.

Beta tests

The MSS Group ran beta tests of two products in FY2001 - the StorageTek SN6000 (disk version) was evaluated, and the "Sledgehammer" NAS device from Maximum Throughput was also evaluated.

FTP server

An FTP server was prototyped in FY2001 that demonstrated the capability of serving MSS files via FTP (for internal clients only). In FY2002, a secure version of this server will be tested to allow file transfers from clients outside UCAR.

Web-based accounting tools

An initial version of the MSS accounting tools was prototyped and readied for deployment in FY2002.

Enhancements and upgrades completed in FY2001

Deployment of STK 9940 technology

This was a critical step toward decommissioning the StorageTek SD-3 (Redwood) media and drives, which will continue in FY2002.

As part of its ongoing commitment to make the best use of new tape technology in the Mass Storage System, MSSG also deployed 9940 media in FY2001. 9940 is a high-capacity media that uses the same recording technology as 9840 media, which has been in production in the MSS since 1999 and has proven itself to be extremely reliable. The new media has a capacity of 60 GB per cartridge. Higher capacities using the same 60-GB cartridges are planned for the next 12-18 months. In the first 16 weeks of production, over 150 TB of data were stored on 9940 media.

New MSS hosts

Several Sun servers (running Solaris 7 and 8) were integrated into the HPDF, using serial-HiPPI connections.

Decommissioning of FDDI

Two new BusTech Network Appliance devices were put into production to provide a direct 100BaseT TCP/IP connection to the Mass Storage Control Processor (massive). This allowed us to decommission the FDDI Network Systems "LPS-Gateway" adapters.

MSS growth

NCAR Mass Storage System growth during FY2001 increased substantially over FY2000. Average net data transfer rate during FY2000 was 5.2 TB per month, whereas the average net data transfer rate during FY2001 was 8.8 TB per month. This increase in the growth rate can be attributed to several factors, such as new MSS hosts coming online, increased amounts of local disk storage on several machines (which increases the size and number of MSS backup files), and the increased usage of the new 160-node IBM SP (blackforest). Further increases in the net growth rate are expected in FY2002 as the utilization of the new ARCS machine starts to ramp up. Projecting this growth into the future, it is not difficult to realize that new storage paradigms and user education will be required, since without this the growth in just three to five years will be untenable.

The following table compares year-end statistics for FY1997, FY1998, FY1999, FY2000, and projected statistics for year-end FY2005. The FY2005 estimates assume a flat budget for supercomputing, historical data storage trends at NCAR, and Moore's Law growth in computer performance per unit cost. Even with the most optimistic vendor projections for storage densities and costs, these estimates indicate that the NCAR MSS would require between one and two dozen ACSes and the annual MSS budget will exceed that for supercomp uters.

MSS growth statistics and expectations

  eFY1998

eFY1999

eFY2000

eFY2001

eFY20054

Total storage (TB): 150 212 273 379 5,700
Total files (millions): 5.1 6.9 8.3 10.9 190
Net growth (TB per month) at eFY: 5.0 5.0 5.2 10 220
Data read/written (TB per month): 20 20 25 37 500
Data migrated internally (TB per month): 20 20 25 74 500
Manual tape mounts (number per month): 37,000 28,000 18,000 10,000 1,0005
Robotic tape mounts (number per month): 37,000 46,000 54,000 95,000 900,0005
Offline cartridge count: 169,0003 169,000 142,000 126,000 85,0006
GFLOPS on NCAR computing floor: ~20 ~36 ~83 ~83 ~1,000
Notes:
1. 16 TB per month = 5 MB/sec
2. All on IBM 3490 cartridge media
3. Mixture of 166,700 3490 cartridge media and 2,300 SD-3 cartridge media
4. Projected assuming a flat computational budget through 2005
5. Assumes one copy of all data is under robotic mount control
6. Assumes size of existing offline archive will decrease as media densities increase and existing data is oozed to those higher density media

Future plans for the NCAR MSS

Key issues to be addressed over the next four years include:


Network engineering and telecommunications

Introduction to NETS

The Network Engineering and Telecommunications Section (NETS) is responsible for determining networking research, strategy, planning, engineering, installation, operation, and maintenance of state-of-the-art networking and data communications facilities for NCAR/UCAR.

Support of these facilities requires NETS staff to:

In summary, NETS provides a vital service to the atmospheric and oceanographic research communities by linking supercomputing resources (including mass storage systems and other data communication resources) to scientists at NCAR and throughout the university research community. These activities are essential to the effective use of NCAR/UCAR's scientific resources, and they foster the overall advancement of scientific inquiry.

The primary NETS accomplishments in FY2001 include FRGP, Web100, MINT-FL, and MALT.

More information about NETS is available at the NETS website: http://www.scd.ucar.edu/nets/

The rest of the NETS FY2001 Annual Scientific Report is organized into these sections:

Networking research projects and technology tracking

Networking research projects

The Networking Engineering and Telecommunications Section (NETS) performs empirical research in networking and data communications technologies, configurations, and equipment. In addition, NETS hosts seminars and makes presentations about networking futures.

Web100 project

The Web100 project is a major project. The Web100 project is an initiative proposed by NCAR, PSC, and NCSA to fix some well-known problems with the Unix (and other) operating systems that are currently inhibiting effective utilization of national high-performance networks such as vBNS and Abilene. One of the biggest problems is the current need to manually calculate the optimal bandwidth delay product to specify a TCP window size that is large enough to avoid prematurely halting data transmission between TCP acknowledgment packets.

This issue generally isn't important for LANs, but it is important for high-performance WANs. It is difficult to determine the "bandwidth" part of the product, and right now the only effective way to obtain this is to have knowledge of the network topology, which usually means consulting with a network engineer. Furthermore, most applications don't provide a means for the user to specify this information even if it were available. The Web100 Project seeks to solve this and some related problems, and it has received funding from the NSF for a three-year research proposal.

NETS hosted the first Web100 developers group at NCAR in July 2001.

For more information on Web100: http://www.web100.org/

Net100 project

The Net100 project is creating software that allows computer operating systems to adjust dynamically to network conditions. Net100 is funded by a three-year grant from the Mathematical, Computational Sciences Division in the Office of Science at the U.S. Department of Energy. Net100 is a collaboration of the Pittsburgh Supercomputing Center (PSC), the National Center for Atmospheric Research (NCAR), Lawrence Berkeley National Laboratory (LBNL), and Oak Ridge National Laboratory (ORNL).

NLANR project

NLANR (National Laboratory for Applied Network Research) is the umbrella project funded by NSF to facilitate the individual research projects conducted on the high-performance networks. NCNE (National Center for Network Engineering) at PSC (Pittsburgh Supercomputer Center) subcontracts with NETS to help it provide engineering support. For more information on the NLANR project, see: http://www.nlanr.net/

NLANR Advanced Network Discovery (NANDISC) project

The NANDISC project seeks to create public-domain software that discovers network topologies, models the discovered information in open and standard ways, and displays the resulting models. Existing commercial solutions are proprietary and expensive and are often limited to a single vendor's products or a single layer of the OSI stack.

Access Grid

NETS participated in the network configuration and testing of the SCD Access Grid. NETS will continue to participate in the deployment of operational Access Grids.

Virtual Host Interface Project (VHIP)

A significant and long-standing problem with individual high-performance computing systems has been their inability to effectively and simultaneously attach to multiple networks. Simultaneous attachment to multiple networks is often referred to as "multi-homing" -- it is desirable both from the standpoint of increasing bandwidth and improved robustness in the face of potential network failure. However, limitations in the networking implementations of current operating systems prevent the additional interfaces from improving the reliability of the system. In many cases the multiple interfaces actually have a negative effect on the system as they introduce a complexity that induces additional modes of failure, several of which involve subtle interactions with the Domain Name System (DNS).

The VHIP project's purpose is to correct this situation by adding a new virtual host IP (VHIP) interface onto multi-homed servers, which will allow multi-homed servers to finally realize the full potential of their additional interfaces. The VHIP project will also explore other applications of the VHIP interface in the areas of redundant distributed services and client mobility. NETS submitted this as an NSF ITR proposal to perform the work that was not funded. NETS also submitted this as an NSF ITR proposal to perform the work that is pending approval.

Earth System Grid project

NETS provided network engineering support to the DOE Earth System Grid project. NETS will provide network engineering support to the DOE ESG II project in the coming year: http://www.earthsystemgrid.org/

Network technology tracking and transfer

SCD's Network Engineering and Telecommunications Section tracks networking technology via networking conferences, training classes, vendor meetings, beta tests, technology demonstration testing, email lists, networking journals, attending user conferences, and by meeting and exchanging information with universities and other laboratories. NETS staff continued to use all of these technology-tracking avenues during FY2001. New technologies are integrated into the production NCAR networks on an ongoing basis once a technology has met our capacity, performance, connectivity, reliability, usability, and other requirements.

Local Area Network projects

NETS supports both NCAR/UCAR network needs as well as the special networking needs of SCD itself. LAN projects are thus further subdivided as being either NCAR/UCAR network projects or SCD network projects.

NCAR/UCAR LAN projects

UCAR network infrastructure recabling projects

A large number of network infrastructure recabling projects were completed in FY2001. The common goal of all the recabling projects is to provide each workspace with a standard set of dedicated data communications links. The overall plan calls for each workspace to be provisioned with a standard Telecommunications Outlet (TO) that connects with four Category 6 (CAT6) twisted-pair cables and two pairs of multi-mode optical fiber. Additionally, intra-building (trunk) wiring must be installed to concentrate all workspace cables to intermediate and central locations.

Concurrent with recabling, each network device is delivered a dedicated 100 Mbps of bandwidth via a dedicated Ethernet packet-switch port. Such dedicated-port access offers substantial networking performance improvement over shared-media Ethernet access. By the end of FY2002, dedicated-port access will be available to all UCAR computers.

MINT-FL

MINT-FL is a project to completely rewire UCAR's FL1-FL3 facilities. This involves the installation of approximately 850 standard TOs, six supporting Telecommunication Closets (TCs), and new Ethernet switches throughout the FL1-FL3 buildings. Each computer at FL1-FL3 will be connected to its own dedicated 100-Mbps Ethernet packet-switch port to provide LAN access. MINT-FL began in February 2000 and was completed April 1, 2001.

For more information on MINT-FL:
http://www.scd.ucar.edu/nets/projects/NETSprojectplans/completeprojects/2001.complete.projects/mintfl/

Mesa Lab Advanced LAN Technology (MALT) project

MALT is a project to completely rewire UCAR's Tower B facilities, 2B, and classroom facilities. This involves the installation of standard TOs, supporting Telecommunication Closets (TCs), and new Ethernet switches throughout the MALT area. Each computer in MALT will be connected to its own dedicated 100-Mbps Ethernet packet-switch port to provide LAN access. MALT was completely designed and planned and began on June 1, 2001 and will be completed June 2002.

For more information on MALT: http://www.scd.ucar.edu/nets/projects/malt/

Pearl Street

UCAR acquired a new leased site at Pearl Street and Foothills Parkway. NETS worked extensively on the network and telecommunications design and installation, which included the installation of approximately 120 telecommunications outlets and one primary communications closet.

Miscellaneous LAN projects

NETS participated in the planning, design, and installation of the Corporate Technical Training Center (CTTC) networking, telecommunications, and physical security at Pearl Street.

NETS participated in the planning, design, and installation of the UNIDATA remodel networking, telecommunications, and physical security in FL4.

NETS participated in the planning, design, and installation of the MMM and COMET computer room remodel project networking, telecommunications, and physical security.

UCAR LAN engineering projects

Network Engineering Routing Design (NERD)

NERD was a project to redesign and deploy the UCARnet routing infrastructure. The goal of this project was to make the router structure simpler and more robust. Substantial re-engineering took place to better define the router boundary with the external world. Previously, internal and external routing functions were mixed together on a variety of routers. One of the goals of the NERD project was to end up with a subset of gateway routers that perform only external routing functions, and a subset that perform only internal routing functions. This separation simplifies a variety of configuration issues, including security filters. Another goal was to replace slower routers with new faster routers.

Ethernet packet switch re-engineering project

Several substantial improvements were made to the Ethernet packet-switching capabilities at NCAR. First, and probably most important, very large numbers of user-computers were converted from dedicated 10-Mbps Ethernet to dedicated 100-Mbps Ethernet. Also, 11 new Cisco 6509 Ethernet switches were installed in the network. These switches are faster and can also provide GigE host connectivity.

Network monitoring project

NETS continues to use HP Openview, flowscan, and Cricket as its principal monitoring tools. Information on HP Openview, Cricket, and other NETS tools can be found at: http://www.scd.ucar.edu/nets/tools/

NETS statistics can be viewed at: http://netserver.ucar.edu/nets/Statistics/

Performance monitors are also maintained at the request of two national network measuring organizations. NLANR OC3mon ATM OC3 monitors at NCAR and the FRGP and UCAID's Internet2, in conjunction with Advanced Network and Services, Inc., Surveyor network monitors at the FRGP. Information about the Surveyor project is available at: http://www.advanced.org/surveyor/
Information about MOAT and Coral are available at: http://moat.nlanr.net/

Local serial-access project

NETS supports several terminal servers for providing serial console access to various computers and networking equipment. Serial support is also provided for the very few serial terminals remaining at UCAR. The Annex terminal servers have been replaced.

NETS CSAC support project

The NCAR/UCAR Computer Security Advisory Committee (CSAC) is chartered by the SCD Director to assess the state of computer and network security at NCAR/UCAR, and to make recommendations to assist NCAR and UCAR management in setting policies related to the security of computers and other devices attached to the NCAR/UCAR network. Membership of CSAC is composed of technical representatives located throughout the various NCAR/UCAR organizations.

NETS has been involved with CSAC because almost all security plans use various types of network-connected devices located between the networks belonging to the external world and the UCAR networks that are being protected from the external world. These network-attached devices can operate as filters and/or authentication devices operating at one or more OSI (Open Systems Interconnection) layers, usually at the Network Layer (Layer 3) and higher. Based on CSAC recommendations, NETS continued to implement significant new gateway router filters to greatly improve network security for UCAR. Extensive testing and extensive coordination throughout UCAR is required to implement the recommended security filters.

Virtual Private Networks (VPNs)

A Virtual Private Network (VPN) is a technical solution that provides secure, private connections to network applications using a public or "unsecured" medium such as the Internet. With a VPN deployed across the Internet, virtual private connections can be established from almost anywhere in the world. NETS tested and deployed VPNs to simplify access to internal network services from outside our firewall for university users and internal UCAR users on travel. NETS has transferred this responsibility to DSG.

Multicast support activities project

Multicasting is a technology in which a single outbound stream of data can be made to arrive at multiple destinations. The data stream is multiplied in a tree-wise fashion using both software and hardware to effect the multiplication. Multicasting technology is particularly useful for videoconferencing and audioconferencing applications. NETS continues to support and enhance multicast services.

UPS project

NETS has continued installing UPS (Uninterruptable Power Supply) units into all new TCs so that all networking equipment will receive standby power in the event of a power failure. UPS units also help filter out damaging power spikes. Upgrading and maintaining these devices is an ongoing process.

Wireless

NETS has embarked on a program to evaluate, test, and deploy wireless Ethernet at UCAR. Initial testing is complete as is deployment to all public areas and conference rooms, and the deployment plan to all office space is in progress. For more information, see: http://www.scd.ucar.edu/nets/projects/wireless/

FDDI mitigation project

NETS has aggressively moved toward the removal of FDDI, an end-of-life technology, and is nearly done with this project.

Voice over IP

Data and voice communications have traditionally been managed separately at UCAR. However, since 1986, following industry trends, there has been a transition toward the co-mingling of digital voice and traditional data communications. As voice data transitioned from analog transmission to digital transmission, the difference between the data types became negligible from a transmission point of view -- it's all just ones and zeros. Networks can move digital data no matter what the source. Great economies of scale can and have been realized by combining voice and data types onto a single network. These cost savings have been what have largely driven the merging of voice and data.

A couple of new technologies are emerging that will even further blur the difference between voice and data networks. One technology is called Voice over IP (VoIP), and the second technology is Unified Messaging. By utilizing both of these advances, UCAR will realize advanced technical capabilities to enhance the electronic "office" of the future that combines mobility with integrated voice mail, electronic mail, fax, and video technologies.

VoIP in its full implementation is the complete merging of data, voice, video, and networks. Traditional separate voice systems such as PBXs, telephone cabling infrastructure, and proprietary telephone handsets go away. These components are replaced with computer-based call managers, VoIP telephones, or your computer and voicemail (Unified Message Server) devices that are Ethernet attached just like any other computer. With a VoIP system, the same cabling infrastructure that is used for traditional network devices can be used for VoIP devices as well. Such a combination simplifies the overall cabling infrastructure and lowers total costs of installation and maintenance. The same network monitoring system can be used to monitor both VoIP and data network devices.

It is the belief of NETS that these are the technologies of the near future. A joint VoIP and UMS testbed is in place to familiarize staff with this technology and assess its viability for larger scale deployment. The testbed will help determine whether UCAR should install this technology in new installations and/or transition existing facilities to this new technology.

SCD LAN projects

In addition to its overall NCAR/UCAR networking responsibilities, NETS has additional special support responsibilities for SCD. NETS handles or consults on most of the host-based SCD networking, including all supercomputing networking. NETS is also responsible within SCD for several other tasks including:

Ongoing SCD network support project

NETS is responsible for most aspects of daily operation of the SCD LANs and host-based networking. Among these responsibilities are monitoring, managing, tuning, troubleshooting, upgrading, reconfiguring, and expanding SCD LANs and host-based networking. NETS works closely with the system administrators of all SCD network-connected systems.

Supercomputing network support project

NETS supports almost all aspects of networking for all SCD supercomputers. This includes hardware, software, and routing configuration support for GigE, FDDI, and Ethernet interfaces. HiPPI (High-Performance Parallel Interface) IP routing configuration support is also provided for the supercomputer HiPPI connections.

ARCS RFP

SCD conducted an RFP for a new supercomputer. NETS participated in this process, contributing both networking requirements for the system along with a set of benchmarks designed to evaluate the networking performance of the candidate systems.

SCD Visualization Lab

NETS worked closely with VETS during the design phase of the new SCD Vislab to ensure that its networking needs are well met. The vislab required a higher density of network connections than most other SCD spaces. In addition, the lab will be reconfigurable between various seating arrangements. To meet these needs, NETS provided an increased number of wall plates, a flexible under-floor cable design allowing easy movement of network connections, and wireless connectivity. Construction of the NETS portion of this project is complete.

Metropolitan Area Network projects

Boulder Point-Of-Presence (BPOP)

The BPOP is a collaboration of entities within Boulder to achieve optimal intra-Boulder and wide area networking. NCAR aggregates traffic from the Department of Commerce Labs and the City of Boulder to the FRGP for Commodity Internet, Abilene, and intra-FRGP access. NOAA provides NCAR a third commodity Internet feed. Also, CU-Boulder and NCAR back up each other's FRGP links.

BRAN project

BRAN (Boulder Research and Administrative Network) is a strategic initiative in which UCAR has joined with CU-Boulder, NIST, NOAA, and the City of Boulder to construct and operate a private fiber network to interconnect the key facilities of these institutions and directly access certain common-carrier telecommunications facilities in Boulder.

All intra-Boulder MAN traffic now uses BRAN. UCAR has greatly reduced intra-Boulder circuit costs by utilizing BRAN. Active BRAN links include ML-FL, PS-FL, NOAA-ML, CU-ML. BRAN carries voice, MSS, and data traffic. NETS has continued to participate in the management committee and the technical committee. For more information, see: http://www.branfiber.net/

Voice and data network project

NETS continues to support communications systems from NCAR's ML site to NCAR's Jeffco and Marshall sites. Jeffco was upgraded to utilize Cell Relay Service (CRS) for data and a T1 connection for voice service.

The Marshall site was upgraded to a 100% wireless system this year for LAN and MAN voice, data, and security services. The T1 line to Marshall provides backup service for voice and data service between Marshall and the Mesa Lab.

The upgrades of these two systems allowed us to remove the nearly 15-year-old remnants of the Timeplex system. For a summary of the history of this system, see: http://www.scd.ucar.edu/news/01/fotoweek/1002.timeplex.html

Remote-working and home-access project

Remote access continues to be provided by digital T1 PRI access to a Cisco 5300 Remote Access Server (RAS). Each T1 PRI line provides twenty-three 56-Kbps channels that can support analog or ISDN dial-in access. Three such lines are in place. Long distance access via direct 1-800 lines is overlaid on one of the PRI lines. Telnet, PPP, and ARAP access are supported on the Cisco RAS devices. The robustness of this system was enhanced this year by implementing a spare test and substitute system.

Wide Area Network projects

Front Range GigaPOP (FRGP)

The Front Range GigaPOP (FRGP) is a consortium of universities, nonprofit corporations, and government agencies that are cooperating in a regional network aggregation point called the FRGP to share the costs of Wide Area Networking (WAN) services. The current FRGP partners are the Boulder Point of Presence (BPOP), Colorado State University (CSU), CU-Boulder, CU-Denver, CU-Health Science Center, CU-Colorado Springs, Colorado School of Mines, the University of Denver, the University of Wyoming, and Fort Lewis College. Additional partners, including the State of Colorado and the University of Northern Colorado, are likely to join soon. There are similar gigapops throughout the U.S. There are a number of advantages gained by sharing services through such a gigapop. Costs for WAN services are reduced for each partner, expertise among partners can be shared, a higher level of services can be purchased than individual institutions could afford, there is more buying power among a consortium, and there are great economies of scale.

Front Range Gigapop members
(Click image for detailed view.)

NCAR/UCAR has provided the engineering and Network Operations Center (NOC) support for the FRGP, with the service costs incurred by NCAR/UCAR being shared by all members. NETS believes that the greater service and bandwidth obtained through the FRGP are important enough for NCAR/UCAR to participate and provide the engineering and NOC services. FRGP has agreed that NETS has the most qualified engineering and NOC staff to provide the very best engineering and NOC services for the FRGP.

This is a critical service for the UCAR/NCAR staff as well as all the other partners and has proved to be an extremely successful technical project as well as an excellent collaboration with the Colorado research community. The FRGP provides NCAR/UCAR's primary WAN connectivity including Abilene connectivity. For more information, see: http://www.frgp.net/

Colorado State Network Access Point (CSNAP)

UCAR is investigating the possibility of aggregating local public entity IP traffic in Colorado and connecting it to the Front Range GigaPop (FRGP). The proposed point of aggregation for local public entities would be called the Colorado State Network Access Point (CSNAP), and this new entity would join the FRGP as a primary member. The FRGP is already the aggregation point for many of Colorado's higher education institutions. These institutions want to preserve the FRGP as the Colorado Multi-use NeTwork (MNT) is deployed, and forming the CSNAP is seen as one potential way to achieve this.

Initial discussions regarding this matter have taken place between representatives of the Colorado Commission on Higher Education (CCHE), Colorado Government Technology Services (CGTS), a division of the State of Colorado's General Support Services (GSS), NCAR, and various Colorado institutions of higher education. For more information, see: http://www.scd.ucar.edu/nets/projects/csnap/

The Quilt

The Quilt is a UCAID Project whose participants are non-profit advanced regional network organizations dedicated to advancing research and education in the United States by a) providing a broad range of advanced networking services to their constituents, including network engineering, management, and operation; regional connectivity and exchange; promotion and coordination of regional activities; and b) facilitating innovative and successful projects and productive working relationships. The Quilt's specific purposes and objectives are to a) provide advanced network services to the broadest possible research and educational community; b) promote end-to-end continuity, consistency, reliability, interoperability, efficiency and cost-effectiveness in the development and delivery of advanced network services by means which, at the same time, foster innovation and reflect the diversity of its members; and c) represent our common interests to backbone network service providers, industry, government, standard-setting organizations, and other organizations involved in or influencing the development and delivery of advanced network services.

The FRGP joined The Quilt in June 2001, and NCAR represents the FRGP on the Quilt Steering Committee. For more information The Quilt, see: http://www.thequilt.net/

TeraGrid

NCAR is actively pursuing a connection to the TeraGrid network. We are working with the TeraGrid participants as well as vendors to explore connectivity options, pricing levels, and possible funding opportunities.

vBNS+

The vBNS+ is a production ATM network provided and operated by MCI. The vBNS interconnects the NSF supercomputing centers, universities, and other customers. NETS has continued its support of the vBNS+, providing both network engineering and applications support and research. For more information, see: http://www.vbns.net/

Internet2/Abilene

Internet2 has launched the Abilene network, which is a national network for Internet2 university members constructed on fiber loaned by Qwest from its national fiber network. Abilene is constructed at 2.4-Gbps speeds and will be transitioned to 9.9-Gbps speeds as quickly as practical. As of October 2001, Abilene had 54 entities connected, many of them Gigapops, with over 202 institutions active, and in FY2000 UCAR joined UCAID and attached to Abilene via the FRGP. The Internet2 project website provides more information about Internet2, Abilene, and UCAID at: http://www.internet2.edu/

RAP WAN project

NETS continued to provide WAN networking service to RAP for their extensive field project requirements. This service was discontinued this year.

NREN WAN project

NETS continued to provide WAN networking service to NASA for their NREN connections in Boulder. This service was discontinued this year.

Other network projects

Projects listed in this section are ones that don't neatly fit into the Research, LAN, MAN, or WAN project classification scheme. These other projects are organized as NCAR/UCAR support projects and SCD support projects.

UCAR Database Integration Study (UDIS)

In September 2001, the ITC recommended that NETS lead a study on bringing together many databases and directory services throughout UCAR/NCAR/UOP. We have decided to call this effort the "UCAR Database Integration Study" (UDIS). Jeff Custard has been assigned to coordinate this effort for NETS and UCAR. This group will investigate ways in which the various databases around UCAR with information about "people" can be better coordinated. UDIS will also subsume the LDAP grassroots effort begun by Leonard Sitongia of CGD. LDAP is one thing that the UDIS should consider.

The "Lightweight Directory Access Protocol" (LDAP) is an increasingly popular standard for accessing directory information over the network. SCD has already formed a "Database Working Group" to examine these issues within SCD and develop solutions. That group will provide input into this UDIS process for SCD. Similar efforts may be happening (or planned) in other divisions around UCAR. If so, perhaps the UDIS can take input from those efforts as well and make coordination recommendations where appropriate. As they become available, we can link other web pages that may be available for these projects from this UDIS page. For more information, see: http://www.scd.ucar.edu/nets/projects/directories/index.html#whatis

Networking and telecommunications merge

Effective 11/26/2000, the SCD Networking Engineering and Telecommunications Section merged with the F&A Telecommunications and Physical Security Group to form the new Network Engineering and Telecommunications Section (NETS). This new section resides in SCD and is managed by Marla Meehl. This merge required the assimilation of six staff into SCD. The merge has proved to be quite successful.

Westnet

Westnet is an affinity group that grew out of the NSFnet regional network called Westnet. NETS' long-term participation continued with NETS continuing its leadership role. NETS is a member of the Westnet Steering Committee and leads the effort to plan and run bi-annual meetings that include technical presentations from members and vendors. Westnet provides powerful political and technical contacts with universities that are UCAR members and that share common concerns. The current Westnet members include CU-Boulder, CU-Denver , CSU, University of Wyoming, University of Utah, Utah State University, Arizona State University, University of Arizona, University of New Mexico, New Mexico State University, Idaho State University, Denver University, South Dakota School of Mines and Technology, New Mexico TechNet, Boise State University, and UCAR.

Project-tracking system

The Remedy-based project-tracking system continued in full production in FY2001. NETS work requests and projects are opened, tracked, and closed with this project-tracking system. The use of project-tracking tools is necessary because of the large number and variety of projects. It would be unwieldy to manually track just the personnel assignments for these hundreds of projects, much less track progress details of so many projects. In FY2001, NETS completed more than 1,698 work requests and over 30 projects. At the beginning of FY2001, 35 projects were in progress, and 402 work requests were queued.

NETS NCAB project

The Network Coordination and Advisory Board (NCAB) consists of appointed technical representatives from the NCAR divisions and UCAR. The purpose of NCAB is to advise NETS concerning network strategy, planning, policy, expansion, and management issues for all of NCAR and UCAR. The work of NCAB has been indispensable to the success of networking at UCAR.

Strategic plan

At the request of UCAR management, NETS formulated a comprehensive strategic plan outlining technical and budgetary requirements for UCAR networking for the next several years. This document is available at: http://www.scd.ucar.edu/nets/Documents/stratplan/strategy.html

This document is in the process of being revised and updated.

Conferences

NETS continues to provide networking support for classes, demonstrations, meetings, and conferences throughout NCAR/UCAR. This work involved the design, construction, configuration, and operation of the network components required by these activities.

NETS also sponsored and hosted two conferences in FY2001:

Outreach support

NETS personnel attended and presented at numerous conferences, meetings, and training sessions and prepared the numerous trip reports and presentations published at: http://www.scd.ucar.edu/nets /nar/reports/trips/ and http://www.scd.ucar.edu/nets/presentations/

Scot Colburn served on the Wheatridge High School computer and networking curriculum board.

Committee support

NETS representatives have attended and supported the following committees.

NCAR/UCAR committees:

SCD committees:

External committees:


Research data stewardship

The Data Support Section (DSS) maintains a large, organized archive of computer-accessible research data that is made available to scientists around the world. The archive represents an irreplaceable store of observed data and analyses and is used for major national and international atmospheric and oceanic research projects. DSS has much data, processing capability, and services that are not offered by other groups. The DSS group started working in 1965 and has been working on large projects and building the data archives ever since.

There are now over 550 distinct datasets in the DSS Research Data Archive on NCAR's Mass Storage System (MSS), ranging in size from less than 1 MB to over 1 TB. The total volume of data in the DSS archive was 2.4 TB in August 1990 and 17.5 TB in September 2001.

Data stored for Data Support, and total Mass Store

  Data Support Section

  Total NCAR Mass Store

Volume

Date Bit files Volume   Bit files Volume DSS/MSS
13 Aug 1990 61,335 2.437 TB   --- 14.430 TB 16.9%

4 Aug 1991

65,518

2.689 TB

 

715,000

19.400 TB

13.9%

3 Aug 1992

80,538

3.085 TB

 

1,060,000

27.270 TB

11.3%

Aug 1993

103,314

4.072 TB

 

1,351,271

36.280 TB

11.2%

15 Sep 1994

119,703

4.751 TB

 

1,849,466

47.423 TB

10.0%

14 Feb 1995

123,877

5.085 TB

 

1,966,990

52.456 TB

9.7%

24 Jan 1996

137,680

5.950 TB

 

2,486,471

67.590 TB

8.8%

28 Aug 1996

143,340

6.770 TB

 

2,888,639

78.964 TB

8.6%

28 Feb 1997

151,509

7.513 TB

 

3,289,224

91.399 TB

8.2%

17 Oct 1997

159,945

8.482 TB

 

4,046,678

110.359 TB

7.69%

2 Sep 1998

167,073

10.032 TB

 

5,038,611

147.439 TB

6.80%

7 Sep 1999

185,608

11.942 TB

 

6,737,448

206.885 TB

5.77%

25 Aug 2000

192,404

13.875 TB

 

8,187,688

267.796 TB

5.18%

6 Sep 2001

210,224

17.475 TB

 

10,781,364

370.706 TB

4.71%

  Note: The total data on the MSS passed 300 TB on January 2, 2001. Over the next 8.1 months, it added another 70.7 TB.  

DSS staff provides assistance and expertise in using the archive, and they help researchers locate data appropriate to their needs. Users may obtain copies of data by network access, on various tape media, or they may use data directly from the MSS. DSS staff also assist scientists by providing data access programs (to read and unpack data), other software for data manipulation, and dataset documentation. DSS has 10 staff members.

DSS main accomplishments during FY2001

Extend seven global 50-year datasets of observations

These daily observations are from about 1,500 global upper-air stations, and 7,500 land surface stations. There are also data from the ocean surface, from aircraft, and from satellites. Taken together these observations help to describe the temperature and motion of the earth's atmosphere several times each day. These data are needed for the several large projects that make new analyses of the atmosphere each 6 hours for 20 to 55 years. It is estimated that over 6,000 research projects have used the NCEP/NCAR reanalysis output. These observations are also needed for climate variability and climate trend studies.

Make more reanalysis output available

The NCEP-2 21-year reanalysis: Masao Kanamitsu, NCEP, ran an improved reanalysis at NERSC (in California) and produced the Reanalysis-2 collection of output products. We extracted these products from the NERSC Mass Store using FTP. The products are similar to those produced for the NCEP/NCAR Reanalysis-1 and are currently available for the period 1979 through May 2001. The total data volume is approximately 827GB.

NCEP continues to update the NCEP/NCAR 50-Year Reanalysis-1 products to near current. We add these updates to our archive and extract time series and monthly mean sets. There have also been improvements to periods in the archive where problems were found, and these improved products replace the previous run in the archive. The basic MSS archive and the CD-ROM products continue to be extremely popular.

Give good service to our 900 main users

DSS staff continues to handle requests from all of our main users. We fill data orders, help users access data into computers at NCAR, and send CD-ROMs. We give access to about 16 TB of data each year, and we serve many smaller users via hits on our web server.

Improve the data and layout on the web server

The web information interface for the SCD research data archives received a large-scale upgrade and improvement. The new interface covers all aspects of the DSS web presence and is not only superior for the data users, but also easy and efficient for the DSS staff to use and maintain. This information system has more than 2,500 html-formatted pages that are automatically updated. The web information continues to grow as data and metadata are routinely added by the DSS.

Increase the mesoscale model archives and give service

NCAR is the data center for certain NOAA/NASA mesoscale modeling projects. We routinely update the data from three main models (1) NCEP-Eta, (2) FSL maps, and (3) Canada Gem. By January 2000, we had 510 GB of data, which increased to 807 GB by February 2001. We also work with users who have received a lot of this data.

Document project: Assemble many documents and write more

Create better descriptions of our datasets, gather our older documents, and scan them for online access. This project started in March 1999; the production of scanned pages started March 2000. By August 2001, we completed the scanning of about 8,000 pages. We are approximately 70% done with the most critical work.

Make additions to COADS and other ocean datasets

In FY2001, a major milestone for COADS was achieved. New early data sources had been recovered through data archaeology efforts around the world. These sources and other new digital sources have been added to the collection for the period prior to 1950. This update and updates from previous years now forms the complete replacement and extension for Release 1 COADS (1985). The new archive now covers 1784-1997. The COADS project is a collaborative effort between NOAA/CDC, NOAA/NCDC, and NCAR/SCD.

Prepare more data for the high atmosphere archives (70 - 1000 km)

Many datasets were updated during the past year. A substantial effort has been preparing for the support of NASA's TIMED satellite, which will be launched in December 2001. It needs the CEDAR ground-based observations to use along with the satellite data. It needs the CEDAR data quickly, so more work has been done to enhance the automation of the ingest procedures.

Summary and outline of DSS projects

Summary and outline of DSS accomplishments and tasks, 1998 - 2001

Participate in producing data in large reanalysis projects

There are many aspects of world science that need a good day-by-day description of the whole atmosphere of the world (especially meteorology, oceanography, and climate science). In the past we had old hand-done analyses of some weather variables, then starting in 1962 we have archives of numeric analyses for the Northern Hemisphere, but the methods then were rather primitive compared with what can now be achieved. In the mid-1980s it appeared to a few people in the world that it would be beneficial to make a new analysis of old data (for perhaps 10 years) and that it even might be possible. NCAR Data Support participated in the effort to help start some actual projects. NCEP and NCAR linked up to start a very ambitious project to analyze 35 years. This evolved into a project to do 50 years. Now 53 years of global analyses are done (1948 - 2000). A colleague in DOE called this "the mother of all reanalyses."

Prepare and improve seven datasets of world observations

The seven main datasets of world observations were needed to make new analyses each 6 hours of the world's atmosphere for 1948 - 2000. The data were in many national archives and in a number of larger subsets. We needed to obtain the data (not easy), and do many diagnostic checks to find systematic errors and remove them. The work started in 1967 and became much more intense in 1991 when our work at NCAR DSS started to prepare for the NCEP/NCAR Reanalysis-1. The datasets of observations are now much better than anything that the world has ever had. There are still data (hard to get) that should be added, especially for earlier years.

This data preparation has been a huge amount of work, because each of the seven datasets has many component data sources, and we have been adding new components and refining and improving older components for improved coverage and accuracy. The seven main datasets are a significant benefit for world science. It is encouraging that it has worked out well. These observations and new reanalysis output will be key resources for world research during 2001 - 2010. These seven datasets describe the world weather history for 53 years, and they make it possible to create the new analyses.

Seven main sets of world observations

    Data
years

Number
of years

Work
started

Recent
work

Comments

a.

Rawinsondes
- upper air

1946-on

55

1967

1991

Some earlier data

b.

Pibals
- upper air

1942-on

59

1973

1991

Some earlier data

c.

Aircraft

1949-on

54

1973

1992

d.

Satellite cloud
winds

1967-on

34

1973

Cover better 1973-on

e.

Satellite soundings

1969-on

31

1973

1991

Better 1973-on

f.

Sfc 3-hr
synoptic

1948-on

53

1976

1992

Density incr 1967-on

g.

COADS ocean
surface

1854-on

145

1981

1988

Some earlier data

For each category of data, NCAR obtained and processed many separate data inputs. These are being used to prepare new global analysis at NCEP, ECMWF, and NASA.

Progress in preparing all of the observations

Version 1 of all of the observations was used for the NCEP/NCAR Reanalysis-1. Version 1 of observations for the last block of years (1948 - 1957) was ready by March 1998. There was still much work that could be done to add more data and decrease the errors in station location. Summary of progress:

Work on the DSS web server

Work started to extend the capabilities of the DSS web server in November 1999. More metadata were added, the layout was improved, and various new data inventories were added. One important milestone of improvements was reached in July 2000 and another in February 2001. Then this work had to slow down because other projects were being delayed too much.

Give access to much data; help many data users

Much of our time in DSS is needed to build the archive and extend it. We are also an operational unit to work with many users and provide much data and consulting help. Here are some statistics about our delivery of data.

The number of data requests and unique online data users

The table below shows a cross section of services and estimates the number of unique users who access the scientific data archives. The standard services account for more than 900 unique users, and self-service online large-volume users number another 600 or more per year. Many thousands of small-volume users access the online services for metadata and information and small data files.

Main data requests from users

Year

Custom data requests:
Tapes, FTP, CDROMS
NN Reanalysis data requests
(CDROMs sent)
NCAR MSS
Unique users
Data server
Large users
Total
requests

1995

376

11

391

 

778

1996

347

35

399

 

781

1997

329

146 (1150)

414

 

743

1998

331

164 (1893)

383

 

714

1999

308

141 (2401)

429

 

737

2000

362

124 (2524)

422

 

784

2001

426

44 (1170)

400

669

1495

Note: This table shows the number of custom data requests (data prepared for a single person or project), the number of requests for NCEP/NCAR Reanalysis CDROMs (numbers of CDROMS are shown in parentheses), the number of unique users who accessed the archives from the MSS, and the number of unique users who collected data files from the permanently online collections on the data server. Only users who have downloaded two or more data files with size greater than 50 KB (not metadata or web pages) from the data server are counted as large users. The combination of these values approximates the number of total users served annually by the research data collections.

Sending data on CD-ROMs

A CD-ROM has been made with data for each year of reanalysis (1953 - 1999, 47 years). During 1997 through 2001, DSS distributed 9,138 copies of these CD-ROMs (which held 5,939 GB of data). These have gone to 45 countries (46% to US, 10.5% to Japan, 4.2% to Canada, 3.3% to UK, 2.9% to India, etc.) Nearly all of these CD-ROMs have data from the NCEP/NCAR Reanalysis-1 project.

Volume of data being used

The total volume of DSS scientific data that was used from the MSS, distributed by custom data request (tapes, ftp, and CDROMS), and distributed on NCEP/NCAR Reanalysis CDROMS has increased from 5.6 TB in 1995 to 18.1 TB in 2001, as shown in the figure below. The numbers on the bar graphs give the volume of data in GB for each component.

Total use of DSS data

Data access by category from the MSS

Many users access research data directly from the NCAR MSS. In the figure below, we separate the access into categories to show how well these data serve the University and NCAR research communities. In 2001 the total access was over 12 TB with University users accounting for approximately 8 TB. The DSS staff also uses the MSS extensively to maintain, improve, develop new products, prepare custom data requests, and organize collections for large research projects. The DSS impact on the MSS is not shown here.

DSS data delivered to users from the MSS

Make additions to COADS and other ocean datasets

In 2001, a major milestone for COADS was achieved. New early data sources had been recovered through data archaeology efforts around the world. These sources and other new digital sources have been added to the collection for the period prior to 1950. This update and updates from previous years now forms the complete replacement and extension for Release 1 COADS (1985). The new archive now covers 1784-1997. The COADS project is a collaborative effort between NOAA/CDC, NOAA/NCDC, and NCAR/SCD.

Primary additions to COADS for the most recently completed update (1784-1949). The table includes the numbers of reports, period-of-record, and noteworthy comments.

Collection

Reports

Covering

Comments

UK Main Marine Data Bank 12.1 M 1854-1949 Also blended for 1980-97
US Merchant Marine Collection 3.5 M 1912-1946 US WW II records mostly discarded about 1974
US Maury Collection 1.3 M 1784-1863 Keyed by China; only available data for 1784-1803
Japanese Kobe Collection 1 M 1890-1932 Key additions: Pacific WW I: Manabe (1999)
World Ocean Database 1998 405 K 1874-1949 SST estimates from oceanographic profiles
Russian MARMET Data 268 K 1888-1949 Also blended through 1997
Norwegian Logbook Collection 201 K 1867-1889 About 200K reports remain undigitized
Arctic Drift Ships and Stations 16 K 1893-1925 Norwegian expedition ships Fram and Maud
Russian Makarov Collection 3.5 K 1804-1891 27 ships including the Vitiaz

Prepare more data for the high atmosphere archives (70 - 1000 km)

Many datasets were updated during the past year. A substantial effort has been preparing for the support of NASA's TIMED satellite, which will be launched in December 2001. It needs the CEDAR ground-based observations to use along with the satellite data. It needs the CEDAR data quickly, so more work has been done to enhance the automation of the ingest procedures.

The data delivery to users has been automated for some time using a few web-based interfaces.

The task to extend and update the CEDAR archive is a joint project between SCD and HAO that was started in 1984. NSF funds several large radars to sample the ionosphere. The world research community needed access to data from all of the radars, and from other types of instruments. This archive fills those needs.

The CEDAR (Coupling Energetics and Dynamics of Atmospheric Regions) program supports scientific investigation of the middle and upper atmosphere (about 70 - 1000 km) including measurement and modeling of the mesosphere, ionosphere, and thermosphere. Support is primarily through the NSF's Division of Atmospheric Sciences. The CEDAR database is an archive actively maintained with data from ground-based instruments including Incoherent Scatter and other Radars, spectrometers, interferometers, and LIDAR. As of May 2001, the database contains 16 GB representing 61 instruments.

DSS external linkages


Visualization and enabling technologies

The overall thrust of the Visualization and Enabling Technologies Section (VETS) is to provide the software, facilities, and human expertise to support a broad range of analysis and visualization capabilities while extending best practices, extending domain knowledge through basic research, and using results effectively for outreach.

VETS made a number of major advances during FY2001. We completed the creation of a next-generation Visualization Lab that included a sophisticated new physical space, integrated visual supercomputing, and collaboration technology in the form of an Access Grid node. NCL capabilities were dramatically expanded, new experimental software work was undertaken, and the Community Data Portal was populated successfully with a number of pilot projects. We succeeded in securing a leadership role in the DOE-sponsored Earth System Grid project, moved our NSF-funded VGEE project toward completion, and played a major role in the development of the Knowledge Environment for the Geosciences (KEG) concept and proposal to NSF's ITR program. We contributed to quite a number of other proposals as well.

We also began work on NCAR Strategic Initiatives in the area of web-based data provision and next-generation web environments. VETS staff engaged in a broad portfolio of collaborative interactions with people and projects at NCAR and other organizations. We also continued our tradition of maintaining a high level of visibility at conferences and in the public press and media. FY2001 will serve as a foundation for an exciting FY2002 program that will focus on advanced collaboration environments, sophisticated data and media portals, terascale analysis and visualization tools, and a new generation of web infrastructure and applications. This document briefly summarizes our progress for the year.

Facilities for analysis, visualization, and collaboration

SCD has operated its highly successful Visualization Lab for several years now, and it has proven to be a valuable tool for enabling new understanding of large-scale scientific data. It has proven equally useful as an important and popular window into our research activities at NCAR and UCAR. During FY2000, SCD developed plans for a next-generation facility aimed at collaborative group visualization and analysis with major enhancements in capability, capacity, and accommodation of larger groups. A new physical space was identified, pre-visualized, and designed. In FY2001 we completed the design specifications and proceeded to begin building the new facility. It was a large, complex project that took most of the year and had a number of phases and components. These included the remodeling of the physical space, moving and integration of primary computational systems, building the networking infrastructure (wired and wireless), managing a contract for control and room integration by an outside contractor, integrating a new generation of high-resolution 3D projection systems, and last but not least, developing an entirely new suite of software to drive the room's capabilities. We also upgraded the lab's storage array to a full RAID configuration and added some high-performance storage dedicated for our frequent demonstrations.

The new room sets a mark for scientific workspaces. It is a pleasant space, provisioned with comfortable seating, powered and networked modular furniture supplemented with wireless networking. We can comfortably seat about 35 people for a boardroom-style meeting and a few more for a theater-style presentation.

The new lab can flexibly be used in two modes: large-screen 3D visualization, and collaboration in the form of the AccessGrid. For visualization activities, visual supercomputers drive a tiled passive 3D projection system that illuminates the 24 x 9-foot silver screen. Using the room's sophisticated switching matrix, a different computational infrastructure can be selected, and the room becomes a human-scale interaction space and functions as an AccessGrid node.

  New Visualization Lab

The facility is a completely "fly-by-wire" system: the computers, lights, and audio systems are all operated by a dedicated touchpad display or a web interface via a browser.

Based on our early experiences, we have begun the process of making some incremental improvements to the new lab. In the upcoming year we'll be adding additional audio support for the AccessGrid, a new release of the electronic control system, and an expanded video-switching matrix that will allow more systems and better sharing of resources with offices.

Community software for data analysis and visualization

As part of SCD's FY2000 effort to develop roadmaps for our future, Open Source software was identified as a strategic element of the overall plan. At this writing, VETS manages approximately 1.5 million lines of analysis and visualization application code and conducts portability testing for most of the major Unix variants including Linux with Windows and Mac OSX on the horizon. Virtually all of the software has either been moved under a GPL source code license or a freely available binary license with the intent of having all applications available under a GPL license in FY2003.

Open Source is an important step for our research community because they have specialized software that is well adapted to our domain and that may be shared and extended for new purposes. It's an important step for us because we can extend our application software offerings with developments from the growing mass of other developers who have also adopted an Open Source path. Our software development and sharing efforts are detailed below.

NCAR Graphics

Used by thousands of researchers, NCAR Graphics is a mainstay in the geosciences community. The package provides software libraries that support a wide variety of graphing and visualization functions with extensive and robust mapping capabilities. NCAR Graphics delivers superb, publication-quality visuals that are as good or better than anything available from any toolset, including commercial tools. In FY2000, SCD released NCAR Graphics as an Open Source distribution under the GPL license. During FY2001 we saw nearly 9,000 downloads of the software, a remarkably large number for a "legacy" software package. It still appears to be popular and useful in the community. While we no longer actively develop or promote NCAR Graphics, we continue to supply a quality distribution, provide bug fixes, and add libraries as needed to support new functionality in NCL.

The NCAR Command Language (NCL)

The NCAR Command Language (NCL) is a scripting language for computation and visualization. While designed as a general-purpose language, SCD has worked extensively with CGD and others to equip and position NCL as the analysis environment of choice for the Community Climate System Model (CCSM). NCL handles netCDF, HDF, ASCII, binary, and GRiB files (including ECMWF coefficients) and can natively access CCM "history tapes." Layered interfaces tuned for climate analysis coupled with nearly 500 additional mathematical and graphical functions make NCL a formidable tool for the researcher.

SCD supplies NCL to the broad science and educational community and provides ongoing support and development for this 1.5-million-line software package. In FY2001, NCL was released for free to the public as a binary distribution along with new documentation, an FAQ, and new web-based support and distribution functions. We saw roughly 1,500 downloads of NCL over the web.

Over 60 new data-processing functions were added to NCL, including ones that do the following:

Several plotting functions and resources were updated to add more options for customizing graphics, including the generation of curly vectors, having more control over paneling multiple plots, and creating specialized XY plots. New functions were created for generating streamline and vector plots in pressure/height formats or over various map projections, and for creating histograms. Some utility functions were created for manipulating color maps and annotating plots.

Work began on integrating the RANGS/GSHHS database, a map database of very-high-resolution coastlines, and a new release including this important new functionality is planned for FY2002. Work also began on porting NCL to other environments, including Windows and Mac OSX.

Our FY2002 plans call for the provision of comprehensive support for all components of CCSM-2, much better capabilities for weather models (e.g. WRF, MM5), and a modest expansion of our staff resources for day-to-day porting, testing, and distribution. From a technical standpoint, this translates to completing interfaces for high-resolution map databases (particularly useful for oceanography), adding support for generalized 2D coordinate systems (e.g. curvilinear), and re-engineering higher-level software to expose all of this to users.

Several examples of NCL visualizations are shown here:
NCL output examples

NCAR version of Vis5D

Vis5D is a mainstay visualization tool in the scientific community and has proven to be a versatile application for the mesoscale community as well as for climate, turbulence, and ecosystems research. VETS has created significant extensions to the package, including provisions for use in a virtual environment, adaptations for visualizing very large datasets, a particle system feature, and interfaces for creating VRML (Virtual Reality Modeling Language) output. The enhanced application has been shared with the broad community and is used at universities, NOAA, NASA, other NSF centers, and the Naval Research Lab. Like the parent version of Vis5D, this software is extended and shared under the Gnu Public License (GPL). During FY2001, we integrated another software toolkit into Vis5D called "qslim," a very good surface decimation package. This allows the geometric complexity of objects to be optimized such that they're a tenth the size of the original, and correspondingly faster to render. This is one useful approach to dealing with the increase in dataset resolution in an era when high-end hardware advances have stagnated.

During FY2001, the Vis5D project became an Open Source community effort called Vis5D+, and we happily joined in, offering up our version of the software. We began the process of merging our technology with the public version, and we intend to continue and advance this process in FY2002.

Gvolsh

Gvolsh is a direct volume-rendering tool based on the shear-warp factorization approach that can effectively utilize multiple processors and deal with very large regularly gridded datasets. We continued to use gvolsh as an environment for experimenting with new rendering approaches, and we also developed a set of utilities that permit tightly coupled interaction between IDL and gvolsh. These utilities enable a user to seamlessly move back and forth between the IDL and gvolsh environments. A researcher may use gvolsh's powerful visual exploration capabilities to identify a feature of interest. Once a region of interest is identified, the new utilities enable a researcher to instantly apply IDL's data manipulation and analysis capabilities to the desired focus area.

xmovie

Our new Visualization Lab required new software with which to drive our tiled 3D display system. DOE has been developing a new animation tool called xmovie that is targeted at precisely this application. VETS collaborated with Lawrence Livermore National Lab (DOE) to add stereo capability and other scaling and placement functions to the tiled-display movie playback software, integrated it into our own production environment, and fed the improvements back into the core DOE distribution of the package.

Enterprise web services

Since 1993, SCD managed and maintained many individual websites for most of the divisions and programs at NCAR and UCAR using surplus equipment and an ad hoc arrangement for site administration. As the growth and use of the World Wide Web matured, access to data and information via web browsers increased dramatically. Web-based technologies began to form the basis for a large fraction of the interactive access to data and information throughout UCAR.

Working with members of the UCAR Information Technology Committee's (ITC) Web Advisory Group, SCD identified the major hardware and site-authoring tools required to meet all divisional and UCAR web services for the next three to five years. With funds from the ITC, a new system was purchased in late 1999 and was placed into service early the next year.

Since that time, SCD has concentrated on providing enterprise-wide web services to the organization. Toward this end, SCD installed and is now supporting a UCAR-wide search engine: the excellent Inktomi engine. Additionally, SCD is actively researching web technologies that can be used to support the many scientific efforts taking place at NCAR and UCAR. For example, SCD has researched the use of SourceForge for supporting distributed software development groups. SCD has also been actively developing ways of supporting centralized user authentication methods that will be necessary for web-based applications.

Over the past year, the usage of the NCAR/UCAR websites has accelerated at an incredible pace. The web servers currently sustain a load of almost 20 million hits per month, with mid-day peaks of over 30,000 hits per hour. To sustain this high load level with high reliability, the web architecture was completely redesigned during the FY2001. This was a major upgrade, and it comprised most of the work the Web Engineering Group accomplished over the last half of the reporting period.

The new architecture is cluster-based. It is centered on a centralized high-availability file server. Then there are two tiers of web server system employed to deliver the content to the user. The first tier is comprised of small, cheap systems. The intent is to be able to incrementally grow the capacity of the web cluster as needed for the non-dynamic content that is generated from the websites. This can easily be done with small, cheap systems. The second tier systems, currently under development, are intended to be the systems where more CPU-intensive and/or data-intensive web applications are run. The first-tier systems will be configured to pass requests for these more intensive applications on to the second-tier systems. This architecture allows the web servers to be incrementally "right-sized" for the types of loads that take place on the system. The following diagram illustrates the new architecture.

Clustered web server architecture diagram

We have also prepared a new program plan for this effort that includes additional software engineering support for the effort in FY2002. Plans call for much-improved access to web logs and statistics, web gateways to services such as mail, staff/user directory information services, security, streaming media, and search engine agents.

Research and special projects in data and visualization

The Earth System Grid I and II

During FY2001 we continued our efforts on the Earth System Grid project with our DOE collaborators. Mid-year we developed a Phase II proposal for the effort and were awarded The Earth System Grid II (ESG, http://www.earthsystemgrid.org/), a three-year DOE-sponsored collaboratory research project involving NCAR and several DOE labs (ORNL, ANL, LBNL, LLNL) and NCAR. The project is focused on building a DataGrid for climate research that facilitates management and access to climate model data across high-performance broadband networks. It builds on one of the more exciting recent developments in computational science, Grid Technologies and the Data Grid. The Data Grid is a next-generation framework for distributed data access in the high-performance computing arena, and it addresses security, transport, cataloging, replica management, and access to secondary storage. We successfully demonstrated the results of this work at the SC2000 conference and established NCAR as a node on the Data Grid. We intend for this important effort to have significant linkage with DLESE, DODS, NSDL/THREDDS, the Community Data Portal, and emerging international efforts in the DataGrid area. Securing the next-generation project is one of our more important accomplishments this year, as it positions us well to develop NCAR's computational environment as an important element of the international Grid.

The Visual Geophysical Exploration Environment (VGEE)

It is widely acknowledged that there are myriad opportunities to incorporate new information technologies into educational programs.

Building on its successful visualization program, SCD is a partner in the development of the Visual Geophysical Exploration Environment, an NSF-funded effort to incorporate advanced visualization technology into curricular material for undergraduate geosciences education. The aim is to develop new educational learning environments that blend web-based information-delivery environments with advanced media and a strong inquiry-based component, with exploratory visualization as a centerpiece.

  New Visualization Lab

This is a collaborative effort between SCD and UCAR's Digital Library for Earth System Education (DLESE), the University of Illinois, and the University of Georgia with SCD's role in the area of dataset preparation and visualization tool integration.

During the year we made major advances in the development of the actual visualization tool, developed an innovative new "smart probe" capability, and crafted a new and forward-looking graphical user interface (that is applicable to other higher-end environments). We also established linkage with the Unidata-led THREDDS project, with the intent of integrating the VGEE tools with the metadata discovery environment that is the focus of the project. We are in the final phases of VGEE, and in the upcoming year we will be completing the development of the visualization tool, deploying it in one or two classroom settings, producing a final report for the overall project, and exploring options for new funding sources.

The Community Data Portal (CDP)

Responding to a growing demand for flexible, web-based access to scientific data -- both by research investigators as well as the community of data consumers -- SCD launched a new project in FY2000 called the Community Data Portal (CDP). There are some nice new community technologies that serve as a substrate for this activity: DODS (the Distributed Oceanographic Data System), LAS (PMEL's Live Access Server), Ferret (PMEL's analysis system), and our own NCL. Building on these technologies in FY2001, SCD cultivated a number of collaborative pilot projects including ACACIA (CGD), Reanalysis (SCD/DSS), VEMAP (CGD), and space weather simulation (HAO), and moved this effort from the concept stage into an active development and release phase. We also began the process of analyzing system requirements in the context of simulated user loads, and subsequently specified and moved into acquisition phase for a new system, to be deployed in FY2002. Relationships were established between the CDP and the Earth System Grid project, the THREDDS project, and it was identified as an NCAR Strategic Initiative. FY2002 plans call for a dramatic scaling of the effort, support for one or more workshops, integration of university data holdings, and experimental work with very large geophysical turbulence datasets.

Chromium/WireGL

During FY2001, we started a small pilot project aimed at learning about and evaluating the potential of cluster-based visualization. Collaborating with Stanford University and Lawrence Livermore National Labs, VETS staff contributed to the design of the Chromium distributed-visualization framework. This technology appears to be useful and promising, and we plan to move forward by building a visualization cluster and deploying it in our lab for evaluation in FY2002.

A Knowledge environment for the geosciences

VETS worked with CSS in providing leadership to the development of an NSF ITR proposal for the Knowledge Environment for the Geosciences. The proposal was not funded but came close, and we continued to develop the idea during the year with the intention of moving it forward in FY2002.

Next-generation analysis and visualization software

Our current generation of analysis and visualization tools is approaching end-of-life relative to many of today's challenges and most of tomorrow's. Rather than continue to incrementally bandage existing software codes, a modern framework is required to facilitate the development of new applications that can cope with data volume (1-10 TB and more) and complexity, collaborative capabilities (e.g. AG), and efficient integration of the most promising new developments from computer science. During FY2001, we began the process of defining a new project that would address this important area. A proposal was prepared and submitted as an NCAR Strategic Initiative entitled "Frameworks and Applications for Terascale Data Analysis and Visualization," and VETS staff began meeting to discuss our future work in this area. This proposed new effort will build on our own long-term work in data analysis and visualization software (i.e. NCL, Vis5D, volume rendering, etc.), a variety of emerging community efforts by DOE, NSF, and university researchers, and our own collaborative research efforts. While the effort is not funded yet, we intend to pursue it aggressively in FY2002. In the meantime, we have been conducting exploratory research work in a variety of areas.

A white paper was authored on developing multi-resolution capabilities for large-scale volume visualization with an emphasis on turbulence simulation data. In the software realm, VETS staff experimented with Python, an object-oriented scripting language that has excellent capabilities for integrating legacy and extant software. Python's popularity is growing rapidly in the scientific community, and it's a strong candidate as an integrating layer for next-generation capabilities. We also began to explore the functionality and performance offered by Java, Java3D, and VisAD in the context of the Visual Geophysical Exploration Environment (discussed in Research).

Collaborative efforts

VETS collaborates with many modeling efforts, research projects, and working groups in the area of advancing data management, analysis, and visualization capabilities. We also work with other organizations to jointly develop new technology, and our move to an Open Source model will accelerate such activities in the future. Staff members work directly with researchers to identify and deploy analysis and visualization solutions on a project-by-project basis, a process that not only advances the research programs but also provides valuable insights into current and future requirements. The list below summarizes collaborative efforts engaged in by one or more VETS staff members:

Magneto-hydrodynamic simulations

Clyne collaborated with Dusan Odstrcil (NOAA) to develop new visualization techniques for understanding large-scale simulations of solar magnetic fields on a spherical grid.

  Solar magnetic field simulation

NCL and CSM

NCL, a general-purpose scripting language for geosciences data analysis and visualization, was described above. NCL continues to be a tight and productive collaboration among SCD and CGD staff.

Paleoclimate models

Tim Scheitlin worked with Christine Shields and Bette Otto-Bliesner (CGD) to visualize model data showing plausible El Nino events in a 150-year run of the Paleo CSM.

  Paleo El Nino

Circulation of the lower stratosphere

John Clyne collaborated with Bill Randel (ACD) on visualizing trajectory simulations of circulation in the lower stratosphere.

Supercell storm model with electrification

Tim Scheitlin collaborated with Ted Mansell of NOAA's Severe Storms Lab to visualize a fascinating new model that couples an electrification model with a more traditional storm model.

  Electrified supercell simulation

Decaying stratified turbulence

A collaboration of John Clyne (SCD), Jack Herring (NCAR/MMM), and Yoshi Kimura (Japan) to study decaying stratified turbulence. This is a demanding problem that combines a combination of flow visualization with scalar representation for datasets that range from 256-cubed to 512-cubed.

Jupiter jet simulations

John Clyne collaborated with Leo Rivier (NCAR and Princeton) to develop visualizations of a massively parallel model of jets in the Jovian atmosphere.

Weather Research Forecast (WRF) model

VETS staff served on the WRF Data Management and Analysis working group, providing technical guidance as well as NCL-based software tools that provided new analysis capabilities. A major FY2001 emphasis will be developing 2D and 3D analysis and visualization tools for the WRF effort.

The Community Data Portal (CDP)

VETS staff are collaborating with a number of projects and efforts including ACACIA, DSS/Reanalysis, THREDDS, VEMAP, and HAO's TIMEGCM effort.

The Visual Geophysical Exploration Environment (VGEE)

VETS staff collaborate with UCAR/DLESE, the University of Illinois, and the University of Georgia, as described above.

Chromium/WireGL

Collaboration with Stanford and DOE, as described above.

The Earth System Grid (ESG)

Collaboration with DOE centers and USC, as described above.

Common Model Infrastructure Working Group (CMIWG)
UCAR ITC Data Management Working Group (DMWG)
UCAR ITC Web Advisory Group (WAG)

VETS staff actively participates in these groups.

Wildland fire research

Don Middleton continued a collaborative effort with researchers Janice Coen and Terry Clark on the visualization of numerically simulated explosive wildfires and the analysis of observational field data.

Outreach and education

VETS continued its outreach program, albeit a slightly diminished one, due to the spinup of other new projects and building the new Visualization Lab. Early in the year we enhanced our mobile presentation capabilities by acquiring a deskside SGI Onyx2 and a pair of Barco Sim-6 stereo/3D projectors. We had appearances at SC2000, AGU2000, AMS2001, SIGGRAPH2001, the Envisioning Science conference at MIT, and at a public briefing in Washington DC on new computing technologies. We supported the normal heavy schedule of Visualization Lab presentations to visiting researchers, students, educators, and other visiting groups. A selection of the section's visualization work also appeared on a Weather Channel special segment called "Hot Planet." During the course of the year, VETS staff delivered 44 technical presentations and 11 educational ones.

Community service

Tim Scheitlin provided the keynote speech for the Colorado Computational Science Fair.

Ethan Alpert served as a technical advisor to the Weather Research Forecast (WRF) model effort, on the Data Management and Analysis subcommittee and as a member of the Common Model Infrastructure Working Group (CMIWG).

John Clyne served as the visualization program chair for the Cray User's Group (CUG), and as the technical program chair for DOECGF 2001.

Susan Cross served as a mentor for the SOARS program.

Fred Clare continued to serve on the NCAR Library Committee.

Don Middleton served as a co-PI on the NSF-funded Visual Geophysical Exploration Environment project, as a member of the Alliance Environmental Hydrology team, as an advisor to the NSF-funded Exploring Time project, as a member of Sun Microsystem's Visualization Advisory Council, as a program chair for the AMS Electronic Theater, as co-PI for the Earth System Grid and Earth System Grid II projects, and as co-PI for the Unidata-led Thematic Realtime Distributed Data Servers (THREDDS) project.

Technical presentations and educational outreach

(Note that many of the following are presentations in the Visualization Lab. Events in bold represent significant and/or large-audience efforts, as opposed to a quick meeting.)

Technical presentations

10/2000: Visualization Lab presentation to 64 StorageTek customers, Clyne presenting.
10/2000: For the UCAR Board of Trustees Meeting, Middleton prepared a composite PowerPoint presentation for Killeen and Kellie which included material on data access, NCAR Graphics and NCL open source futures, the new Vislab, the AccessGrid, forest fires, and CCM3 @ T170.
10/2000: Middleton provided a Vislab demonstration for IBM (Austin) visitors David Cohn and Gerry Hackett.
10/2000: Middleton presented two roadmaps at the SCD Retreat: Data Access, Analysis, and Visualization; Formal and Informal Science Education.
10/2000: Boote - Presented NCAR visualization work to a CU physics class.
11/7-11/2000: Alpert, Boote, Clyne, and Scheitlin conducted SC2000 demonstration Nov 7-11 in Dallas.
11/2000: Prepared visual materials to illustrate ARCAS and the Community Data Portal for EPRI Board Meeting and Chuck Hakkarinen of EPRI, David Brown
11/2000: Middleton - Presentation to the SCD Advisory Panel on the Community Data Portal.
12/15-19/2000: Alpert, Clyne, Cross, Scheitlin, and Middleton - "The Virtual Earth System," AGU2001 Conference, San Francisco, CA.
12/2000: Brown, Middleton - "The ARCAS Data System," presented at AGU2001, San Francisco, CA.
1/15-19/2001: Alpert, Boote, Clyne, Cross, Scheitlin - "The Virtual Earth System," AMS2001 Conference, Albuquerque, NM.
1/2000: Rodionov - Delivered a presentation about WEG projects to DSAC.
1/2000: Boote - presented an implementation plan for the WAG Strategic Plan to ITC on 1/22, and it was recommended for presentation to the Presidents Council.
1/2000: Scheitlin - Vislab presentation to Tim Killeen, a guest, and Bernard O'Lear (Scheitlin).
1/2000: Scheitlin - Vislab presentation to Dr. Oh (Director Meteorological Research Institute, Applied Meteorology Research Lab, Korea Meteorological Administration).
1/2000: Scheitlin - Vislab presentation to Lawrence Buja and NOAA colleague (Ken Buja).
1/2000: Middleton - "Visualizing Our Planet", invited presentation at the AMS2001 Electronic Theater, Albuquerque.
1/2000: Middleton - "The Community Data Portal", invited presentation for the UCAR Geosciences Education Workshop.
1/2000: Middleton - "Visualization for Education", invited presentation for the UCAR Geosciences Education Workshop.
2/2001: Middleton - "The Community Data Portal", presented to Unidata Program Committee
2/2001: Scheitlin - Vislab presentation to National Geographic film crew
2/2001: Scheitlin - Vislab presentation to Weather Channel interview with Wendy Abshire
2/2001: Scheitlin - Vislab presentations Denver AMS chapter (8)
2/2001: Scheitlin - Vislab presentations Bryan Hannegan, staff scientist, Senate Committee on Energy and Natural Resources.
3/2001: Clyne - Vislab presentation to a group of NSF visitors representing the National Optical Astronomy Observatory.
3/2001: Scheitlin - Vislab presentation to Harvey I. Leifert, Public Information Officer, AGU.
4/2001: Clyne - Guest lecture to CU undergraduate computer science class.
5/2001: Haley - Presentation to the SCD Advisory Panel on the results of the NCL survey.
5/2001: Clyne - "A Low-Cost Solution to Rendering Time-Varying Volume Data," presentation at DOE Computer Graphics Forum.
5/2001: Middleton - "NCAR's Visualization Program", Fraunhofer Institute for Computer Graphics, Frankfurt, Germany.
5/2001: Middleton - "Visualizing Our Planet", invited presentation at an SGI-sponsored summit on Weather and Climate, Silver Springs, Maryland.
6/2001: Scheitlin - Vislab presentations to Celia Chen (RAP) and 5 visitors from the Air Navigation and Weather Services, CAA Taiwan.
6/2001: Middleton - "Envisioning the Earth System", invited presentation at MIT Conference on Image and Meaning - Envisioning Science, Boston, MA.
6/2001: Middleton - "Emerging Data Access Systems", presentation to Ocean Observatories workshop and Vislab presentation, NCAR.
6/2001: Middleton - "Building the Virtual Commmunity", presentation to the new NCAR Advisory Panel, in new NCAR Vislab.
7/2001: Boote - Vislab presentation for David Radzanowski (OMB) and Jack Fellows.
7/2001: Scheitlin - Vislab presentation to Bo Connell, her student worker, and a new hire.
8/2001: Boote - Vislab presentation to a group of University of Arizona Graduate Students.
8/2001: Middleton - National Weather Service, "A Sampling of Enabling Technologies for Earth System Research", Boulder NOAA Site.
8/2001: Boote - Vislab presentations for Inspectors General from NSF and NOAA in both the old Vislab and the new Vislab.
8/2001: Clyne - SIGGRAPH2001, Presentation in Barco Exhibit in "Emerging Technologies" session entitled "From the Earth to the Sun: A Tour of NCAR Science".
8/2001: Middleton - "Building Cyberinfrastructure for the Earth System Sciences," invited presentation for the Workshop on Advanced Collaboration Environments, San Francisco, CA.

Education and outreach presentations

10/2000: Scheitlin - Worked with Michael Mejia (Producer for Integrated Science 7, a science television program for middle school students produced at the University of Alabama) and helped to develop digital media of NCAR Visualizations for use in his program.
11/2000: Scheitlin - Worked with Robert Pitschenede of the German FOCUS magazine and Jim Hack (CGD) to develop CCM3/T170 imagery for an article.
12/2000: Scheitlin presented the Vislab to Kevin Trenberth's DU science class (15).
3/2001: Scheitlin - Vislab presentation and demonstration for "60 Minutes Australia" interview of Tom Wigley.
3/2001: Boote - Provided Vislab presentations for a group of 60 students from Weber Elementary (Jeffco).
3/2001: Scheitlin - Vislab presentation to 18 CU meteorology students.
3/2000: Scheitlin - Vislab presentation to Harvey I. Leifert, Public Information Officer, AGU.
4/5/2001: Boote - Vislab presentation to a school group from Liberty School - Joes, Colorado.
4/2001: Scheitlin - Vislab presentation to six Laramie County Community College students.
5/2001: Scheitlin - Presentation on scientific visualization and the broader applications of computer graphics for the 2001 Colorado Computational Science Fair at the NCAR Mesa Lab.
5/2001: Scheitlin - Vislab presentation to 7 DU students and Penelope Canan.
5/2001: Scheitlin - Vislab presentation to 20 students and parents from Platte Valley 5th grade.

AccessGrid events

9/2001: Boote attended/operated the AccessGrid node for 3 AccessGrid training sessions: AG:Communicating Effectively Over the AG (9/17), AG:If I Had Only Known (9/18), and AG:Producing SCGlobal (9/18).
9/2001: Boote operated the AccessGrid node for a high-level meeting between Tim Killeen, Al Kellie, Don Middleton (NCAR), Rick Stevens (ANL), and Bob Wilhelmson (NCSA). This was the first real test of the AG node.

Visitors

1/2001: Visiting computer scientists and meteorologists from DWD in Germany. Middleton hosted.
2/2001: Dr. Ken Hay (University of Georgia) for 1.5 days in a "mini-workshop" to develop a new GUI for the VGEE project and to define a rough functional specification for the visualization tool. Scheitlin hosted.
3/2001: Jon Genetti, San Diego Supercomputing Center, Seminar entitled "Bringing Space Into Focus," Middleton hosting.
3/2001: Middleton hosted "Linux: It's Not Just for Servers Anymore," Daryll Strauss from VA Linux Systems Inc.
3/2001: Boote hosted a visit from Dr. Kim of The Korea Ocean R&D Institute.
8/2001: Dr. Chang Kim, director of an Ocean R&D Institute in South Korea, investigating technology we use in the Vislab in support of his own effort to build a visualization center. Boote hosted.

Publications

Pandya, R., D. Bramer, K. Hay, M. Marlino, D. Middleton, M. Ramamurthy, T. Scheitlin, and R. Wilhelmson, 2001: Using the virtual exploratorium to support inquiry-based learning in introductory geoscience courses: an ENSO example. Tenth Symposium on Education, American Meteorological Society, 2000 Albuquerque NM. January 2001.

Bramer, D., K. Hay, M. Marlino, D. Middleton, R. Pandya, M. Ramamurthy, T. Scheitlin, and R. Wilhelmson, 2001: The technology behind a virtual exploratorium: a resource for discovery-based learning in the geosciences. The Tenth Symposium on Education, American Meteorological Society, Albuquerque NM. January 2001.

Pandya, R, D. Bramer, K. Hay, M. Marlino, D. Middleton, M. Ramamurthy, T. Scheitlin and R. Wilhelmson, 2000: The Virtual Exploratorium: An inquiry-based learning environment for undergraduate geoscience education. Fall 2000 meeting of the American Geophysical Union, San Francisco CA. (EOS v. 18 #48, p. F300. Invited Talk.)


Assistance and support for NCAR's research community

Primary responsibility for helping researchers efficiently produce valid simulations on NCAR supercomputers is managed by the User Support Section (USS) to support part (e) of our mission. USS also supports researchers and SCD staff by providing supercomputer usage statistics and web publication services, and by supporting onsite servers, workstations, and application software at the NCAR Mesa Lab. In addition, USS provides management, office, and technical support for the UCAR Security Administrator, who maintains and enhances security for computers and networks inside the UCAR security perimeter.

Four groups and one Security Administrator handle this diverse range of services for researchers and NCAR staff:

Technical Consulting Group

In FY2001, the Technical Consulting Group's (TCG's) mission-based efforts have focused on the key elements of researcher support and documentation. These activities in FY2001 have been specifically focused on three mission-critical institution-wide projects: code conversion efforts surrounding the new IBM SP, network and computer security enhancements, and the ARCS procurement.

Consulting services

Providing software engineering and math libraries support for scientists using NCAR/SCD's high-performance scientific computing facilities is the core business and mission of TCG. This group is the primary point of contact for researchers with questions and concerns about their scientific computing efforts. TCG provides a single point of contact for SCD's community of researchers who need to resolve technical problems and obtain advice on optimal software design and implementation techniques. TCG leverages their contact with researchers by channeling customer needs into SCD's planning process. When the assistance of other SCD staff is required to resolve a problem, TCG coordinates all SCD efforts and manages the follow-through with customers. Collaborations with other SCD groups, researchers, vendors, and other high-performance computing centers are central to maintaining the expertise required to support this mission.

Production computing support

Supporting the production computing environment per TCG's core mission statement has continued to be critical to the success of the Scientific Computing Division. Historically, SCD's best asset in the eyes of the research community has been customer support. TCG maintains the highest standards in customer responsiveness, as well as diligence in system test and checkout to guarantee a stable and productive work environment for our customers. TCG recognizes the need to expand the quality of customer outreach, collaboration, and individualized service.

In addition to AIX operating system support on the IBM systems and Compaq's Tru64 UNIX, SCD provides support for platforms running Irix, UNICOS, and Solaris. TCG core focus has shifted somewhat this year. While TCG still dedicates a large portion of their effort to assisting researchers in their migration efforts to the new IBM, the user community has been requesting and receiving more support in the areas of software architecture and design, optimization, and debugging. This shift is indicative of NCAR's maturing relationship with the IBM SP.

Developing web content has been a complementary project to TCG's provision of code migration consulting and training classes. Over the last year, TCG worked closely with SCD's Digital Information Group (DIG) to modify a significant body of web-based documentation and examples covering the full range of introductory to advanced material and FAQs for users of our IBM systems. The content changes have been driven from several different directions, including changes due to system upgrades, instructional material, customer questions, problems encountered by TCG, frequently asked questions, and of course structured assistance to new customers. The web sites supplement and take advantage of vendor documentation where appropriate.

Training classes are brought in as customer interest and demands dictate needed topics. For instance, this year as the model-development efforts on the IBM matured, the research staff became interested in an advanced debugger, Totalview, from Etnus, LLC. Learning Totalview is a formidable task for a new user, and to ease the process, TCG decided a class on Totalview usage would significantly benefit all users. TCG identified and hired a third-party contractor to teach the class and provided him with a course syllabus designed to address our customer needs analysis. The class was delivered, and it received very positive feedback.

Finally, with the upcoming decommissioning of the Cray J90se computer systems, TCG has been assisting users with issues of migrating off the Cray systems. While most of these users are migrating to the IBM SP, a few are choosing to use other SCD platforms or moving their work to other institutions. No matter where they are migrating their jobs to, users often require assistance moving and using the data files created on the Cray systems. To assist users in this area, TCG staff wrote and maintains a library named NCARU which allows users of other computers to be able to read and write Cray-formatted data files. Over the course of FY2001, several modifications and enhancements were made to NCARU and its documentation. TCG assisted an MM5 researcher in developing software based on NCARU to convert an entire class of MM5 Cray output datasets for use on non-Cray Unix systems.

Security perimeter changes

FY2001 saw NCAR implement several computer and network security enhancements including a cleartext password ban that required significant changes in how remote users access the SCD computing facility. A project implemented across the entire institution, TCG's portion was to develop instructional documentation describing the changes and to provide examples and customer support for NCAR's community of off-site researchers. TCG's testing helped resolve problems on a wide range of services provided by SCD including the MIGS and IRJE systems, remote graphical displays, FTP, and even shell accounts. TCG worked with users, the UCAR Security Administrator, SCD Supercomputing Systems Group, and SCD Distributed Systems Group to develop effective solutions, including most notably the addition of X Window System proxying capability and SSH awareness to the firewall.

System testing

TCG is also responsible for testing the user environment in cooperation with HPS to ensure that operating system and programming environment software upgrades have minimal impact on productivity. To this end, TCG was involved in every operating system and compiler environment software installation. As problems were uncovered, TCG developed strategies and diagnostic code for testing and isolating each problem. TCG has been increasingly taking the lead in characterizing problems for the vendors and following the bug fixes through the pipeline. To streamline and structure these efforts, TCG has produced a customizable, extensible software framework for developing and running system tests. While few unit tests were initially added to the framework, TCG is now poised to begin building unit and regression tests in earnest.

In addition, TCG has begun using and maintaining the applications in the ARCS benchmark suite for ongoing system testing. This complex network of model software and control scripts amounts to over a million lines of code and requires over 30 GB of input data. The ARCS benchmarks have proven to be a valuable diagnostic tool for test and checkout of new hardware and operating system software upgrades. Ongoing development and refinement of the test has been a major effort.

ARCS procurement support

TCG staff provided the requirements for the programming environment sections of the ARCS RFP document as well as feedback on all other areas of the document. Once the vendor responses were returned, TCG staff participated in the analysis and scoring of the proposals through three rounds of best and final offers, and on to the selection of the winning proposal. After the winning proposal was selected, TCG began preparation for the acceptance test for the new hardware, in which the ARCS benchmark suite will play a significant role.

Digital Information Group

Providing online information services using web-based technologies

SCD's Digital Information Group (DIG) brings its expertise to bear in three principal areas for creating, delivering, and maintaining information for our community of earth sciences researchers:

DIG has overall responsibility for creating and maintaining web-accessible user information (including all documentation on divisional computing resources), SCD-specific information for staff and visitors, and materials documenting the scientific research supported by the division.

For the period October 1, 2000-September 30, 2001, the Digital Information Group had the following significant accomplishments:

SCD website design and architecture

DIG greatly expanded the 2000 redesign of the SCD website to further improve the site's navigation, appearance, usability, and accessibility. Improvements included:

DIG developed file-naming conventions and streamlined directory structures to standardize the SCD website's information architecture, permit easier site maintenance, and ensure compliance with UCAR and NCAR security requirements.

Web-based information and user documentation

To improve communications with our users, DIG developed SCD News (now a major component of SCD's website) into a robust, news-delivery service that provides constant updates on the SCD computing environment, including a "photo of the week" feature that has become very popular with users and staff.

DIG supplies "just in time" content for the SCD website by creating subsites and offering digital photo coverage for special events, advisory board meetings, conferences, and seminars hosted by the division. DIG publicizes important moments in the life of the division via clickable digital photo essays (for example, Delivery of NCAR's new Advanced Research Computing System). In addition, DIG now produces high-quality 8 x 10 color photographs of these events that serve as "take-home" mementos for participants.

During the review period, DIG staff continued to provide web-based information on all the computing systems maintained by the division -- especially the main compute engines that form the nucleus of SCD's Community Computing and Climate Simulation Laboratory research environments. Key activities in this area include:

DIG staff continued to take a lead role in assisting NCAR in its production of NCAR's web-based Annual Scientific Report, and DIG took lead responsibility for coordinating, editing, and publishing SCD's FY2000 Annual Scientific Report.

In 2001 DIG increased its production of hardcopy documentation for use in education and outreach activities. DIG staff also edited, compiled and produced a 220-page book for the NSF detailing results of the 2001 SCD User Survey. In addition, DIG inaugurated publication of a quarterly four-color brochure named "PowerCurve," detailing SCD efforts in support of science at NCAR.

Services provided to SCD

DIG staff continue to provide a high level of technical writing and editing skills to SCD management and staff whenever required. Our work with SCD staff often involves editing services for those who submit papers to conferences or research journals, while our work with the division director and his office ranges from drafting policy statements and guidelines to public relations and major documents such as:

DIG worked closely with SCD's Data Support Section (DSS) and the Visualization and Enabling Technologies Section (VETS) to create a website for the SCD Community Data Portal, an inter-organizational collaborative project that develops data exploration tools and provides real-time as well as retrospective data for UCAR/NCAR users around the world.

DIG also added new password-protected internal web pages to the SCD website to promote information exchange within the organization.

Website maintenance and development

SCD's website has grown to 5,130 digital pages containing 1,407 links, making it a complex website to administer. DIG staff work on the site on a daily basis to ensure the accuracy of online information and ensure link integrity, maintaining large portions of SCD's website by monitoring, verifying, and updating web pages. This "trench work" occupies a good deal of DIG staff time, but is essential to keeping SCD information accurate and reliable.

As more information is added (e.g., through new machine acquisitions, additional services offered, and internal "staff-only" materials), DIG seamlessly integrates new material into the site. In this regard, DIG frequently works with the Technical Consulting Group to produce new and updated information for users of SCD's computing resources.

DIG outreach, education, and training

DIG staff continues to provide a valuable resource for students participating in UCAR's Significant Opportunities in Atmospheric Research and Science (SOARS) program. This year marks the eighth consecutive year that we have provided writing and editing expertise to program students by serving as science writing mentors.

DIG also continues to teach classes and give presentations on website design and development and relevant topics that meet the evolving needs of UCAR staff as web technologies continue to evolve. Along those lines, DIG is pleased to continue our sponsorship of the Web Designers and Developers (WebD2) group that meets monthly at NCAR.

Finally, during the review period, DIG staff continued to refine and develop their own professional capabilities via training classes and conferences covering the newest in website authoring tools, such Adobe Photoshop, Macromedia Flash, Macromedia Dreamweaver, and QuarkXpress.

Distributed Systems Group

Overview of distributed services provided by DSG

The Distributed Systems Group (DSG) of SCD provides, maintains, and administers the necessary computer service infrastructure for the UCAR community to conduct daily computational activities. Examples of such service infrastructure are the NCAR electronic mail service, both Internet and Intranet World Wide Web servers, and Domain Name Service (DNS) for NCAR. Most of these services are provided transparently to NCAR users. In addition, DSG supports and administers interactive servers on which UCAR, NCAR, and SCD users conduct daily activities (e.g., email, text editing, MSS access), gateway servers that process 50% of supercomputer job submittals and MSS activity, network file service (NFS) for over 1,200 UCAR and NCAR users, security monitoring and filtering, library access and queries, SCD middleware servers (DCE), and computational service (for example, supporting a Sun Enterprise 6500 server for developing and running user modules and math software). Providing these services is an ongoing, year-to-year activity for DSG. The chart Office/Distributed servers and services shows current DSG services organized by server type.

DSG installs and administers personal computers (PCs) for SCD and the NCAR Director's office. The majority of PCs are Intel-based desktops and laptops running Microsoft Windows 98/NT/2000. DSG has installed a high-availability NT server configuration to provide services to Wintel-based clients comprised of both desktop systems and laptop systems. ("Wintel" refers to processors and support hardware running the Windows operating system. Intel hardware is also being used in SCD to run the Linux operating system. Further, Windows runs on other hardware such as AMD and Cyrix, but such systems are not deployed in SCD.) The high-availability NT server configuration is comprised of two servers -- a primary domain server and a backup domain server. This configuration and redundancy allows continued high availability of service to Wintel clients across various subnets. Furthermore, DSG administers a number of Sun workstations for SCD staff. In all, there are currently more than 300 desktops and laptops that DSG supports.

In addition to continuing to provide the above organization-wide services, DSG's focus in FY2001 was on the following primary areas:

Expansion and upgrade of servers and services

Current servers supported by DSG were designed and configured for a three-year life cycle. During the course of a fiscal year, minor hardware adjustments like adding more memory, expanding storage capacity, adding more or faster processors, or changing network interfaces are conducted on all the DSG servers. All these adjustments are done to keep a server in line with current and projected computational requirements. Usually by the end of a third year of service, a complete server replacement is required; the demands and specifications of current application releases outpace the processing power, size of memory, and disk capacity of a server that has been in production for three years. Additionally, a server in production for three years will experience an increase in hardware failures. The following illustrates how processing power, memory, and disk capacity expand rapidly over a three-year period for three servers performing similar activities:

DSG server configurations (mesa, mcphee, bison)
  1997
1998
2001
Processor (2): Ultra 167 MHz Ultra 250 MHz Ultra 750 MHz
Memory: 128 MB 256 MB 4 GB
Disks: 2 & 4 GB 4 & 9 GB 36 & 72 GB
Price: $18,000 $14,000 $18,000

Another factor that comes into play is the budget for server upgrades during a fiscal year. In any given fiscal year no predictable budget can be allocated and targeted specifically for servers. A number of servers supported by DSG including the web, electronic mail, security, and Unidata server are funded by non-SCD budgets. Some fiscal years' money is available for major upgrades and enhancements, other fiscal years' money is tight, and only critical hardware fixes can be made. The overall objective of a complete server replacement every three years has been met during the last 8 years.

During FY2001, a number of servers were upgraded to newer hardware models. Additionally, DSG has taken on a number of UCAR-wide services during FY2001 such as security and web server administration. The machine activity included:

New servers:

Upgrade of old servers to new ones:

Upgrade of processors, memory, and storage on existing servers:

During FY2001, a number of the older systems in SCD were phased out of production. Some of these workstations and servers are being stored for trade-ins for newer vendor equipment. Some of these servers include:

Clustered and fault-tolerant system configurations

DSG has installed high-availability configurations (hot standby) on a number of major systems over the last few years including:

DSG has been using high-availability servers for a long time, where one server acts as the primary service provider and the second is a hot standby. This traditionally involved two fairly expensive servers. Cluster server technology has advanced over the years and is rapidly replacing the use of a primary server and hot spare. In the clustered server approach, the servers are comprised of a rack of PC-based servers (that are inexpensive compared to traditional Unix vendor offerings) running Linux or OpenBSD as the operating system. In this type of configuration, there are usually three or more servers sharing the load. This allows one or more servers to fail and still have the service continue. It also allows additional servers to be added at a minimal cost to expand a service. Both the Linux and OpenBSD operating systems cost a minimal amount, and in some cases they are free if downloaded from a Linux vendor website. Clustering is a very cost-effective way of providing and protecting key services. The use of cluster server configurations for critical services will ensure continued and uninterrupted service to the UCAR user community. Critical services include such activities as DNS, user authentication, gateway, proxy, and web services. While clustering Intel-based PCs is highly effective for these types of service, it is not a suitable replacement for services that require a high level of I/O and for exporting file systems for a number of clients using network file system (NFS). These types of services will require a higher grade of server traditionally obtained through such companies as Sun or IBM who have their own clustering solution.

The clustering software allows adding or removing members without a major reconfiguration of the current system. Vendors such as Sun have recently entered the cluster arena by offering uniprocessor thin servers that fit into a hardware rack with their own propriety cluster software solution. Still, cost is the major difference between the Sun solution and one comprised of Intel-based servers running Linux. Clustering technology was used in FY2001 for the DNS server replacement system named "phoenix," the gateway server replacements, and new web proxy servers that will take over the proxy function from the main web back-end servers.

Expansion of Linux and other operating systems

The current DSG Unix environment, comprised of mostly Sun servers, is highly stable and reliable. There is no need for any radical changes in hardware platforms or operating systems. It is imperative that SCD, however, has exposure to the Linux operating system (e.g., supercomputers, and servers) because of the large amount of user interest and migration of major computer vendors (e.g., IBM, HP, Compaq, and SGI) to Linux. The general UCAR user interest with Linux is expected to increase with the release of the IA 64-bit processor due out in FY02. While Linux can be used on a limited basis on SCD servers, it is not quite ready for deployment on large-scale production servers. There are many reasons for this:

There is a large gray area in the niche where Linux servers are highly cost effective and have firmly entrenched themselves as major players. This gray area involves clustering servers to provide a service such as web proxy, DNS, or security. Since users do not have access to these servers, all their activities are transparent to the end users. The users then do not care how the services are carried out, only that they are performed efficiently and reliably. During FY2001, Linux was used for the replacement servers for DNS and authentication and proxy web service. The new gateway servers are in a cluster configuration running the OpenBSD operating system, not Linux.

SCD staff has shown a lot of interest in the new MacOS, version 10.1, which is a derivative of the UNIX operating system. Unlike UNIX, however, it allows the users to run Microsoft applications natively on their hardware. Currently there are a handful of MacOS systems that were obtained, but this number is expected to grow in FY2002.

Machine room reorganization of consoles and servers

At one time the DSG servers were physically located throughout the machine room. This made administration and monitoring the systems extremely difficult. After the creation of the computer room planning committee (CRPC), a section of the machine room was set aside to house all the DSG servers. Relocating all the DSG servers took several years.

The next phase of consolidation of DSG servers, which started in FY2001, is the elimination of the current monitor layout, where each DSG server has a monitor attached to it. DSG, in conjunction with the operations group, brought in different centralized monitoring boxes to test on DSG servers. This would help reduce the number of terminals in the Mesa Lab machine room to a few centralized centers where DSG and operations staff could perform administrative functions. Currently, a terminal is attached to each DSG server, and the instability of a monitor can result in crashing a system or periodic reboots. With centralization not only can redundant monitors be used, but multiple windows to critical services on a single console can be monitored by operations in their command center to detect hardware and software problems. This causes a horizontal layering of the machine room where monitors are at eye level and servers are located on the top or bottom racks. Using a terminal server, the system monitors can be eliminated, and all the servers can be run via a handful of monitors. When this is done, the servers can be placed in vertical configurations using racks to create additional space.

The majority of these activities took place during FY2001. A number of server vendors including Sun and IBM have been switching their servers to rack mounts as some of their low-end servers are only one inch high. It is important to take advantage of the newer server design features by planning the machine room layout to minimize computer space. During FY2001 all new servers acquired by DSG came with the rack mount option and were located in racks along with associated storage.

Desktop systems activity and accomplishments

DSG has two goals in the administration of Unix desktops -- to simplify system administration and to provide users with a comprehensive desktop work environment. During the last year, DSG started providing support of the Redhat Linux operating system on Intel-based desktops. Linux has grown in popularity within the Unix community and is the most popular Unix-like operating system in the world. Linux on Intel-based systems can be configured in either a standalone or dual-boot mode where the user could run Linux or Microsoft Windows separately. With DSG supporting Microsoft Windows, Sun Solaris, and Linux, SCD staff can choose from three of the most popular operating systems in the world.

Despite the rise of Linux in SCD, Sun is still the primary workstation used by the majority of SCD Unix users. With SCD users upgrading to Sun Ultra workstations, the complete phase-out of Sun's older Sparc line is nearly complete. Over the last year, the decommissioned Sparc 5s were traded in for Sun Ultra workstations or Intel-based systems running Windows 2000 or Redhat Linux. The new high-availability workstation configuration (flatiron) was set up to support the Ultra 10, 60, and Blade 100 desktop lines. The SCD Ultra workstations are booted and maintained from a centralized Sun server configuration running Solstice Autoclient software.

DSG upgraded Windows desktops and laptops to Windows 2000. This activity was actually started in mid-year 2000 after it was determined that Windows 2000 was stable and interacted with the existing Windows NT environment without any problems. During FY2001, new PCs were purchased with Windows 2000 installed as the default operating system. This saves system administration time by avoiding multiple operating system upgrades. With the constant turnover of laptops and desktops by PC users, a system lifetime of approximately three years, the Windows 2000 upgrade will only affect about two-thirds of the current Windows systems currently in production.

During FY2001, DSG reconfigured SCD desktop systems with the use of Dynamic Host Configuration Protocol (DHCP). DHCP allows a user to connect a workstation or laptop to an arbitrary port and have the IP address dynamically assigned to it. Sun Solaris workstations will be using DHCP after the Windows systems have been successfully implemented. Part of why DHCP will be successful is the flattening of the current subnets used by SCD into one large subnet. The use of one large subnet will make network administration and maintenance a lot easier. For instance, all the desktops will have the same interface in all the offices. Any machine can theoretically be moved to any office and work immediately without having to change the network connection.

During FY2001, DSG started setting up remote systems requiring access to SCD resources with a virtual private network (VPN). This lets the users conduct activities in a safe and secure environment.

Database Services Group

Managing computing accounts for researchers

During FY2001, SCD's Database Services Group (DBSG) managed 1,096 user accounts in 626 projects, not including users and projects that only had mass storage charges. Over 1,087 of the users had usage on the IBM, Cray, or SGI machines, and 871 used the Mass Storage System to read or write files during FY2001. By the end of FY2001, university researchers requesting 1,000 General Accounting Units or less could get an account within three days after materials were submitted.

During FY2001, four databases were administered, and reports of NCAR computer use were generated on a regular basis and upon special request. The four databases that use Oracle software include the computer accounting database to track computer resources used on NCAR computers, the Oracle database for the Remedy trouble ticket system, the database to log web accesses, and a test database for MSS file directory data. An upgrade was performed to bring the databases up to the current version of Oracle. Oracle software, including Oracle tools, was installed to ensure continued support from Oracle and to provide new features as well as bug fixes.

Monthly reports were generated to provide management and others with up-to-date information on how the NCAR computers were utilized. Reports included charts that were posted on the web. Reports may be accessed at http://www.scd.ucar.edu/dbsg/dbs/ Reports were generated for the online version of SCD's Advisory Panel book. Reports requested by management and others were also provided.

Work began on replacing Fortran programs that process daily machine use and currently run on the last remaining Cray J90. These Fortran programs are being replaced by Oracle tools. This will streamline daily accounting and make it easier to effect changes due to the addition or decommissioning of new computers and changes in the charging algorithm. This work will be completed in the second quarter of FY2002.

DBSG worked closely with TCG by helping users with requests for new accounts or passwords, questions about computer charges, and various other requests and questions.

Survey of researchers for NSF review

SCD conducted three surveys in the spring of 2001:

USS took the lead on the general survey of SCD computer users and on the survey of users of SCD-provided research data with assistance from many groups. VETS took the lead on the NCL survey.

For the general survey, SCD solicited responses from 287 users of SCD computers. This included 91 principal investigators, 193 lead users from projects using 100 General Accounting Units (GAUs) or more during 2000, plus 111 researcher who had used more than 500 GAUs during 2000. This resulted in 287 unique individuals. One hundred thirty-eight users responded, yielding a 48% response rate after one follow-up email to those who had not responded by the deadline.

For the NCL survey, SCD requested responses from 256 NCL or potential NCL users. This included people who had subscribed to the ncl-talk email list, people who had downloaded NCL from the web, people who had attended an NCL workshop, and people who had contacted SCD on NCL questions. Forty-nine NCL users responded for a 19% response rate with one follow-up email.

For the survey on SCD-provided research data, 191 were asked to respond to our survey on the data and services provided by SCD's Data Support Section (DSS). This list of researchers included those who ordered datasets from SCD and those who download data from the DSS data server. There was a 25% response rate from the one request to fill out the survey online.

Questionnaire data was summarized and results were reported using frequency charts. Bar charts were produced for many questions comparing the results from 2001 with the results from 1987, 1991, and 1995.

Full results are at http://www.scd.ucar.edu /nar/asr2001/scd2001surveys.htm

UCAR Security Administrator

SCD and UCAR continue to implement and enhance CSAC security recommendations and policies. This is a brief synopsis of FY2001 computer and network security accomplishments. The majority of the following security changes were evaluated and implemented by the UCAR Security Administrator in conjunction with the Computer Security Advisory Committee (CSAC). The UCAR Security Administrator advises UCAR system administrators in matters concerning security and possible violations. The goal is producing an environment that, as much as possible, retains our open network's advantages for our external and internal users, while also protecting their work from damage and disruption by external attackers.

Maintaining and enhancing UCAR's security perimeter

Enhancing UCAR's security incident detection and response

Establishing assured connection integrity and privacy


Enabling infrastructure

The Operations and Infrastructure Support Section's primary accomplishments were upgrading the computer room infrastructure to support the ARCS computer and successfully initiating the SCD portal project. OIS also produced the following results in FY2001.

Business continuity and disaster recovery plan

As part of a larger UCAR-wide effort, the Computer Production Group (CPG) was tasked with developing a comprehensive Business Continuity and Disaster Recovery (BCDR) plan for the Scientific Computing Division (SCD). The first part of this plan contained a business impact analysis exploring the potential effects of each type of disaster or event that SCD could encounter. Next, the plan determined risk levels associated with each particular type of disaster so that SCD can focus resources on the most probable disasters while having a cohesive plan in place in the event of less-probable disasters. Finally, the plan contains all business process information, (e.g. outside resources, vendors, scientific users, network-dependent users, and all universities) required to resume critical, essential, and necessary functions of SCD following a catastrophic event. After development of the BCDR, SCD must conduct an initial audit as well as subsequent audits at regular intervals.

With the BCDR plan in place, SCD will be prepared to implement a well-thought-out and highly orchestrated recovery process that should minimize loss of business while quickly restoring resources to the scientific community following a catastrophic disaster.

Network monitoring and NCAB monitoring host policy

Traditionally the Computer Production Group (CPG), in collaboration with the Network Engineering and Telecommunications Section (NETS), monitored the status of network-attached hosts throughout NCAR/UCAR even though there was no formal policy requiring them to provide this service. CPG plans to continue this service but needed to develop a formal policy limiting the number of hosts per division/program to equitably allocate space and support. CPG will provide a trained Junior System Administrator to help in many conversion tasks. This staff member will be enabled to make connectivity additions, changes, and second-level administration changes. The new policy dictates that in case of problems with monitored hosts, CPG operators are able to contact the appropriate system administrators.

Mass Storage System expansion

Data storage capacity for the Mass Storage System (MSS) is growing at a rate of 12 TB per month. To consistently and accurately manage this amount of data, CPG removed over 8,000 3490E and SD3 cartridges (800-MB and 50-GB capacity per tape respectively) and replaced them with 10,000 9840 and 9940 cartridges (40 GB - 60 GB per tape), and quickly initialized them to provide newer, more reliable technology and tremendous increases in space. Additionally, ten 9940 tape drives were installed. All told, SCD increased the MSS online and offline data storage capacity by a factor of three.

Application support services

IAG continued to provide application services to the organization through the design and development of four online surveys, upgrades to the UCAR conference Room Reservation System and the automation of daily Operations reports.

The completion of a major version upgrade and workflow redesign for Remedy (the division-wide trouble ticket system) improved staff productivity while providing additional functionality. This allowed staff to more efficiently track technical consulting and hardware and software issues to provide division management with accurate service-level and resolution-time statistics. IAG installed Migrator software on the Remedy systems to allow developers to seamlessly transfer data and workflow between development and production servers. This maximized productivity of existing equipment and eliminated user downtime for development and future upgrades.

The Computer Production Group (CPG) is responsible for the administration of all dial-up remote access system (RAS) user accounts. The Infrastructure Applications Group (IAG) along with CPG redesigned the online RAS account request form to add modification and deletion functionality and created a more intuitive and user-friendly interface. Additionally, SCD policy was revised, and it now specifies that all new accounts must gain management approval prior to being activated.

Research and new technology advances

IAG student assistants completed a major research project this year investigating Open Source database products (e.g. MySQL, PostgreSQL) for use in the division as alternatives to expensive commercial products. This study revealed the Open Source products to be robust and powerful enough to handle many applications such as online services, web server statistic processing, offline MSS data summarizing, and MSS data archiving processes. The next step is to work on deployment of these Open Source database packages over the next few years. This way the division will be able to greatly expand and improve its database usage with minimal licensing cost, leaving hardware costs as the only potential limiting factor.

IAG made a technological leap this year by moving from strictly Perl, cgi-bin, and GDBM database web-based products to including Java, JSP, Tomcat, Servlets, Jetspeed, GPDK, PHP, and other similar Jakarta Open Source software products. PostgreSQL and MySQL database programs were investigated and implemented to further establish our web-enabled services. The goal was twofold:

  1. To learn emerging technologies and eventually begin to move out ahead of the mainstream in the area of technology research
  2. To create more accessible, flexible and robust programs and services to UCAR staff and the scientific community. IAG strongly feels that we should not be simply following the curve, but we should be out in front, breaking new ground. In the future, IAG plans to present new information to the IT community via seminars and conferences.

Staffing changes

Our FY2001 staff changes reflected a desire to mentor and promote talented people and to follow our commitment for education and outreach to the local community. This philosophy was satisfied by moving a Computer Production Group (CPG) operator to the Infrastructure Applications Group (IAG) as a software engineer and hiring two part-time student assistants to replace contracted temporary staff in Operations. Additionally, IAG continues to utilize and mentor student assistants in software development and design.

Decommissioned machines

As in years past, technology continued to move at breakneck speeds. Reflecting this change, and making room for newer technology, two Cray J90s were decommissioned, and numerous smaller systems were removed from the computer room floor.


Community service activities

Ethan Alpert served as a technical advisor to the Weather Research Forecast (WRF) model effort, on the Data Management and Analysis subcommittee and as a member of the Common Model Infrastructure Working Group (CMIWG).

Brian Bevirt serves as a science writing mentor for the SOARS program.

Ginger Caldwell was the SC2001 Infrastructure Chair, a supercomputing conference sponsored by the IEEE Computer Society and ACM SIGARCH. She coordinates the annual Colorado Computational Science Fair held at NCAR for high school students.

Scot Colburn served on the State Commodity Internet RFP committee, the Wheatridge High School computer and networking curriculum board, and as a judge for the Boulder Valley Science Fair.

Jeff Custard serves as the chair of the Board of Directors of the Colorado Higher Education Computing Organization (CHECO).

Fred Clare continued to serve on the NCAR Library Committee.

John Clyne served as the visualization program chair for the Cray User's Group (CUG), and as the technical program chair for DOECGF 2001.

Susan Cross serves as a mentor for the SOARS program.

Rachelle Daily serves as Secretary to the Executive Board, Mass Storage Technical Committee, IEEE Computer Society. She attended and worked registration for the 18th IEEE Symposium on Mass Storage Systems in San Diego, CA. She maintained the membership list and alias for this board, and the database for the IEEE MSSTC Mailing List. She is a member of the logistics and planning team for the Computing in Atmospheric Research (CAS) meetings, sponsored by SCD and vendors. She coordinated meeting, catering and registration details for the international CAS 2001 Workshop, scheduled for October 2001 in Annecy, France.

George Fuentes is a member of IBM's SP-XXL Group. The SP-XXL group is comprised of IBM SP installations that are 128 nodes and greater. The SP-XXL group meets three times a year and provides technical input to IBM on new functionality that should be added to the AIX and PSSP operating system product toolset. He is a member of Compaq's AlphaServer User Group. The AlphaServer User Group meets three times a year and provides technical input to Compaq on new functionality that should be added to the Digital Tru64 and AlphaServer SC operating system product toolset.

Pam Gillman is a member of Compaq's AlphaServer User Group.

Roy Jenne negotiates data exchanges with Russia under the auspices of the U.S. State Department. Each year participants agree on a set of tasks and datasets, then prepare a document listing what has been accomplished with plans for the next year. Jenne has been leading this multi-agency effort for the U.S. since 1982. He also serves on the panel for the NCEP 20-year regional reanalysis project.

Steve Hammond is a member of the Technology Advisory Panel for the Artic Region Supercomputing Center located at the University of Alaska, Fairbanks. He was also a member of the organizing committee of the international workshop, Computers in Atmospheric Sciences 2001.

Gene Harano was on the Program Committee for the 18th IEEE Symposium on Mass Storage Systems in cooperation with the 9th NASA Goddard Conference on Mass Storage Systems and Techonologies. The conference was held in San Diego, CA, April 17-20, 2001. He is also a member of the Program Committee for the 10th NASA Goddard Conference on Mass Storage Systems and Technologies in cooperation with the 18th IEEE Symposium on Mass Storage Systems, which is scheduled for April 15-18, 2002 in College Park, MD. Program committee work began in FY2001 for this meeting. Gene is also a member of the Arctic Region Supercomputer Center (ARSC) Technology Panel. This technology panel assists the ARSC in assessing technology trends relevant to the ARSC.

Al Kellie participated in activities of the NCEP Advisory Panel Special Review Team. He is a member of the Unidata Policy Committee, the External Advisory Board to the IBM Deep Computing Institute, and the IBM eServer Advisory Council for UNIX. Al served as the NCAR contact for the Ocean Observatory Steering Committee held at NCAR this year. He co-chairs the Computing in Atmospheric Sciences (CAS) workshops, which are sponsored by SCD and vendors. (The next meeting is scheduled for October 2001 in Annecy, France).

Jeff Kuehn served on the steering committee for the Parallel Tools Consortium and helped to plan last year's annual meeting. Kuehn was the deputy chair of the Tutorials Program for the SC2001 Conference.

Lynda Lester sponsors the Boulder-based Web Designers and Developers group.

Marla Meehl serves on the Westnet Steering Committee, is chair of the Front Range GigaPop Management Committee (FMC), and is a member of the Quilt Steering Committee.

Don Middleton served as a co-PI on the NSF-funded Visual Geophysical Exploration Environment project, as a member of the Alliance Environmental Hydrology team, as an advisor to the NSF-funded Exploring Time project, as a member of Sun Microsystem's Visualization Advisory Council, as a program chair for the AMS Electronic Theater, as co-PI for the Earth System Grid and Earth System Grid II projects, and as co-PI for the Unidata-led Thematic Realtime Distributed Data Servers (THREDDS) project.

Bernard T. O'Lear is a member of the IEEE Computer Society Mass Storage Technical Committee Executive Committee. He co-chairs the Computing in Atmospheric (CAS) workshops, which are sponsored by SCD and vendors. (The next meeting is scheduled for October 2001 in Annecy, France).

Pete Peterson served as Chair, SOARS Steering Committee, and as a mentor for the SOARS program.

Tim Scheitlin provided the keynote speech about the broader applications of computer graphics for the Colorado Computational Science Fair.

Pete Siemsen served as the co-chair of the Front Range GigaPop Technical Committee (FTC).

Steve Thomas served as a reviewer for the following journals: Atmosphere-Ocean, Journal of Atmospheric and Oceanic Technology, Journal of Computational Physics, and Monthly Weather Review. He is also a member of the NCAR Geophysical Turbulence Program (GTP), the NCAR Scientist I Appointments committee, and a reviewer of NCAR ASP Post-doc Appointments. He was also a co-organizer of the workshop: Adaptive and high-order methods with applications in turbulence, NCAR Geophysical Turbulence Program, February 4-6, 2002.

VETS staff Don Middleton, Jeff Boote, John Clyne, and Tim Scheitlin provide an ongoing community service at conferences and for NCAR visitors by explaining and demonstrating state-of-the-art scientific visualization techniques and technology in the forms of Technical presentations and Education and outreach presentations. As the new Visualization Lab neared completion, they also began providing interactive virtual conferencing services to the scientific community via the Vislab's function as the NCAR AccessGrid node.


Educational activities

Colorado Computational Science Fair

For the eighth consecutive year, SCD co-hosted the Colorado Computational Science Fair (CCSF) with Colorado State University to encourage high school students to learn more about computational science. The 2001 CCSF was held on May 12. Sixty-nine students from Colorado entered 33 projects into the competition with almost all of the projects using internet connections provided by NCAR during the fair. Group and individual projects were submitted in the areas of Information Technology, Computational Science, and Robotics. Student projects in computational science were grouped for judging based on the highest level math class the student has completed. Ten researchers from industry, universities and NCAR judged the student projects and provided written feedback.

More detail is on the web at http://www.scd.ucar.edu/ois/ccsf/index.html

Computing grants to classrooms

SCD continues to provide access to its supercomputers for undergraduate and graduate university classes. Computing resources are provided for students engaged in modeling and simulations requiring high performance computers, and for classes studying recently introduced architectures. SCD also provides computing resources to graduate students in the atmospheric and related sciences for their thesis research.

SC2000 education program

Ginger Caldwell, USS, was the Education Program Publications Chair for the SC2000 Education Program, "A National Computational Science Leadership Program" held during the SC2000 conference in Dallas, Texas on November 4-9, 2000. After developing the proposal for this NSF-funded program in 2000, the focus in FY2001 was on the planning for the conference. Twenty-five teacher teams of four teachers each participated in this program which had significant representation from low-wealth schools. Teachers received 40 hours of training in computational biology, chemistry, and physics at the SC2000 conference. Over $1 million in grants have been awarded for this program so far with half from NSF.

SCD's participation in this national program has ensured that we are aware of the outcomes and strategies for enhancing computational science leadership among secondary teachers.


Papers and publications

Publications, refereed

Kistler, R., E. Kalnay, ---, R. Jenne, ---, 2001: The NCEP/NCAR 50-Year Reanalysis: Monthly Means CD-ROM and Documentation. Bull. Am. Meteor. Soc., 2, 247-267.

Rivier, L., R. Loft, and L. Polvani, 2001: An efficient spectral dynamical core for distributed memory computers. Mon. Wea. Rev., in press.

Swarztrauber, Paul N. and S.W. Hammond, 2001: "A Comparison of Optimal FFTs on Torus and Hypercube Connected Multicomputers," Par. Comp., 27, 847-859.

Thomas, S.J. and G.L. Browning, 2001: The accuracy and efficiency of semi-implicit time-stepping for mesoscale storm dynamics. J. Atmos. Sci., 20, 3053-3063.

Thomas, S.J. and R.D. Loft, 2000: Parallel semi-implicit spectral element methods for atmospheric general circulation models. J. Sci. Comp., 4, 499-518.

Papers and publications, non-refereed

Bramer, D., K. Hay, M. Marlino, D. Middleton, R. Pandya, M. Ramamurthy, T. Scheitlin, and R. Wilhelmson, 2001: The technology behind a virtual exploratorium: a resource for discovery-based learning in the geosciences. The Tenth Symposium on Education, American Meteorological Society, Albuquerque, New Mexico, January 2001.

Loft, R.D. and S.J. Thomas. Semi-implicit spectral element methods for atmospheric general circulation models. Proceedings of the Ninth ECMWF Workshop on High-Performance Computing in Meteorology, Reading, England, November 2000. Terascale Computing: The Use of Parallel Processors in Meteorology, World Scientific Publishers, Singapore.

Jenne, Roy. Observations for reanalysis. U.S.-Russia Bilateral Data Exchange Meeting, September 20, 2001, Obninsk, Russia.

Jenne, Roy. Reanalysis. Special Meeting of the Russian Weather Forecast Center, September 19, 2001, Moscow, Russia.

Jenne, Roy. Reanalysis and the Surface Cooperative Station Network. Meeting of U.S. State Climatologists, August 7, 2001, Omaha, Nebraska.

Pandya, R., D. Bramer, K. Hay, M. Marlino, D. Middleton, M. Ramamurthy, T. Scheitlin, and R. Wilhelmson, 2001: Using the virtual exploratorium to support inquiry-based learning in introductory geoscience courses: an ENSO example. Tenth Symposium on Education, American Meteorological Society, 2000 Albuquerque NM. January 2001.

Pandya, R, D. Bramer, K. Hay, M. Marlino, D. Middleton, M. Ramamurthy, T. Scheitlin and R. Wilhelmson, 2000: The Virtual Exploratorium: An inquiry-based learning environment for undergraduate geoscience education. Fall 2000 meeting of the American Geophysical Union, San Francisco CA. (EOS v. 18 #48, p. F300. Invited Talk.)

Thomas, S.J. and R.D. Loft. Parallel semi-implicit spectral element dynamical core for atmospheric general circulation models. Ninth annual conference of the Canadian CFD society, Waterloo, Canada, May 27-29, 2001.


Staff, visitors, and collaborators

SCD staff

This is the SCD staff list as of September 30, 2001.

Division office

Al Kellie, Director
Pete Peterson, Deputy Director
Bernie O'Lear, Associate Director
Tom Engel
Janice Kauvar
Rachelle Daily
Joan Fisher

Computational Science Section

Steve Hammond, Manager
Jennifer DeLaurant
Cecelia DeLuca
John Dennis
Rodney James
Rich Loft
David Neckels
Bill Spotz
Paul Swarztrauber
Steve Thomas

Data Support Section

Roy Jenne, Manager
Cecilia Banner
Roy Barnes
Joey Comeaux
Bob Dattore
Dennis Joseph
Chi-Fan Shih
Will Spangler
Gregg Walters
Steve Worley

Network Engineering and Technology Section

Marla Meehl, Manager
Bryan Anderson
Dolores Boyd
Blake Caldwell
Scott Colburn
Jeff Custard
Judy Green
Jennifer Griffin
Susan Guastella
Fabian Guerrero
Del Harris
Belinda Housewright
Basil Irwin
Jerome Martinez
Mike Martinez
David Mitchell
Long Moua
Richard Mumford
Peter ONeil
Teresa Shibao
Pete Siemsen
Ed Snyder
George Stringe
Jim VanDyke
Wes Wildcat

High Performance Systems Section

Gene Harano, Manager
Bill Anderson
Barb Bateman
Jeff Cowan
John Ellis
George Fuentes
Marc Genty
Pam Gillman
BJ Heller
Robert Kirby
Mark Love
John Merrill
Craig Ruff
Erich Thanhardt
George Williams

Operations and Infrastructure Support Section

Aaron Andersen, Manager
Susan Albertson
Melissa Breedlove
Dorothy Bustamante
Tim Clem
Gary Gard
Julie Harris
Erik Jeffries
Andy Kohler
Jirina Kokes
Linda LaBrie
Stan McLaughlin
Gary New
Katrina Smith
Lana Soller
Ingmar Thompson
Trevor Trumble
Louis Wynn

User Support Section

Ginger Caldwell, Manager
Dan Anderson
Ed Arnold
Brian Bevirt
Bob Campbell
Julie Chapin
Gaynez (Bo) Connell
John Fox
Jon Frudeger
Steve Gombosi
Scott Hays
Rich Johnson
Jeff Kuehn
Lynda Lester
Greg McArthur
Herb Poppe
Juli Rew
Michelle Smart
Mark Uris
Richard Valent
Greg Woods
Janie Young

Visualization and Enabling Technologies Section

Don Middleton, Manager
Ethan Alpert
Jeff Boote
Dave Brown
Fred Clare
John Clyne
Susan Cross
Mary Haley
Dave Kennison
Joseph Mendoza
Tamara Rittner
Andrei Rodionov
Tim Scheitlin

Visitors

6/2000-1/2002: Leonard Rivier, Columbia University, testing and evaluation work for FFTPACK and SPHEREPACK. Hosted by Loft and Thomas, CSS.

9/5/2000-8/15/2001: Jeffrey Weiss, University of Colorado, to research the visualization of large numerical simulations of turbulence and the application of scientific visualization to educational activities. Hosted by Middleton, VETS, and Tribbia, CGD/NCAR.

1/2001: Visiting computer scientists and meteorologists from DWD in Germany. Hosted by Middleton, VETS.

2/2001: Dr. Ken Hay, University of Georgia, for 1.5 days in a "mini-workshop" to develop a new GUI for the VGEE project and to define a rough functional specification for the visualization tool. Hosted by Scheitlin, VETS.

3/2001: Jon Genetti, San Diego Supercomputing Center, Seminar entitled "Bringing Space Into Focus," Hosted by Middleton, VETS.

3/2001: "Linux: It's Not Just for Servers Anymore," Daryll Strauss from VA Linux Systems Inc. Hosted by Middleton, VETS.

3/2001: Visit from Dr. Kim of The Korea Ocean R&D Institute. Hosted by Boote, VETS.

4/2001-5/2002: Maarten deKoenig, KEMA, The Netherlands, research on community data portal. Hosted by Middleton, VETS, and Wigley, CGD/NCAR.

5/2001-6/2001: Thomas Heinze, Technische Universitaet Muenchen, Zentrum Mathematik, to investigate and advance numerical methods that are efficient on parallel scalar systems. Hosted by Loft and Thomas, CSS, and Fournier, CGD/NCAR.

8/2001: Dr. Chang Kim, director of an Ocean R&D Institute in South Korea, investigating technology we use in the Vislab in support of his own effort to build a visualization center. Hosted by Boote, VETS.

2001: Christiane Jablonowski, University of Michigan Baroclinic Instability primitive equation test case intercomparisons. Hosted by Loft, CSS.

Collaborators

NCAR, NOAA/CDC, and NOAA/NCDC are collaborators on the Comprehensive Ocean-Atmosphere Data Set (COADS) Project. Steven Worley, DSS, is the co-PI from NCAR.

A collaboration, funded by NSF and managed by NCAR, was established with the All-Russian Research Institute for Hydrometeorogical Information to digitize and quality-control approximately 6,000,000 Russian R/V marine surface observations. This is a three-year project, and Steven Worley, DSS, is the co-PI from NCAR.