Significant accomplishment highlights

This page presents SCD's top two accomplishments for FY2002, then lists the top accomplishments of the management units defined by SCD's mission statement.

Distributed Shared Memory (DSM) systems grow in prominence and performance at NCAR

The production supercomputer environment managed by SCD for NCAR has evolved over the decades. During the last 18 years, SCD has brought NCAR's science into the world of multiprocessing supercomputers. Prior to the introduction of the four-CPU Cray X-MP in October 1986, all modeling was performed with serial codes. Since then, the focus has been on redeveloping codes to harness the power of multiple CPUs in a single system, and most recently, in multiple systems.

SCD's computer room is now dominated by Distributed Shared Memory (DSM) systems, most notably the IBM SP system blackforest, with preparations well underway to install the larger and more powerful IBM Cluster 1600 system bluesky.

Early in FY2002, phase I of the Advanced Research Computing System (ARCS) was delivered. This more than doubled the size of blackforest, to 1,308 processors. Shortly after this upgrade, a new IBM p690 system with 16 POWER4 processors was delivered. This machine was used for testing and system development in preparation for a large IBM Cluster 1600 system scheduled for delivery in October 2002. Also during FY2002, four systems were decommissioned and two new systems installed. In addition, there were major system software upgrades performed on all major supercomputers.

SCD Director Al Kellie said, "When we designed the ARCS RFP, we wanted a production-level, high-performance computing system that offered both capability and capacity computing. But more than that, we wanted to provide an upwardly compatible system architecture and user environment, and a stable environment for software engineering and development."

Thus, existing blackforest users found the same operating system, the same node configuration, the same batch system (LoadLeveler), and the same industry-standard debugger (TotalView). Usage of the new system continued to be split 50/50 between Community Computing and the Climate Simulation Laboratory (CSL), as it was before the upgrade. But whereas blackforest had become saturated in FY2001, the upgraded system offered plenty of capacity.

To prepare NCAR's computer room for the new ARCS equipment arriving in FY2002, SCD staff started work in early FY2001 to specify, procure, and install a new electrical power distribution system to support the next three years of the ARCS contract. This forms the cornerstone of a solid infrastructure for SCD to continue providing reliable, production-oriented services and equipment as tools for science.

The upgraded blackforest system went into production in December 2001, and it is accelerating research in global and regional climate change, droughts, short-and long-range weather prediction and warnings, wildland fires, turbulence, atmospheric chemistry, space weather, and other critical areas. The NSF purchased the machine for use at NCAR to advance a wide range of research topics in the agency's ten-year plan for the geosciences.

Progress in the multi-institutional collaboration on the Earth System Modeling Framework

The Earth System Modeling Framework (ESMF) is building software infrastructure for climate, weather, and data-assimilation applications. This effort is being conducted with an exceptionally broad group of collaborators: SCD, CGD, and MMM at NCAR; NOAA GFDL and NCEP; MIT; the University of Michigan; DOE ANL and LANL, and NASA-GSFC DAO and NSIPP. The project is organized around a series of 11 milestones, the first four of which were submitted during FY2002. Detailed information is provided in this report and at http://www.esmf.ucar.edu/

The Community Climate System Model-2 is one of 15 testbed applications planning to adopt the Earth System Modeling Framework. The plot above shows precipitable water at T170 resolution.

During FY2002, the Earth System Modeling Framework team accomplished the following:

• Negotiated a three-year, $9,800,000 contract with the NASA Earth Science Technology Office, $2,600,000 of which will go to NCAR SCD for implementing the core framework
• Established a collaborative development environment and communal repositories
• Collaboratively created an ESMF Software Developer's Guide and an exhaustive ESMF Requirements Document
• Organized a community meeting with 80+ attendees in Washington, DC, in May 2002 to review the Requirements Document and solicit additional input
• Assembled the ESMF Validation (EVA) Suite and performed performance baselines on application codes that will be adopting the framework
• Collaboratively developed an ESMF Architecture Document, an ESMF Implementation Report, and a Build and Test Plan
• Selected the ESMF Executive and Advisory Boards and hosted their first meeting at NCAR in September 2002
• Submitted the first Annual Report for the ESMF project

Highlights of SCD's High Performance Systems Section (HPS)

HPS provides two sections of this report: High performance computing and Data archiving and management.

1. Implementing MSS Class-of-Service:
   Class-of-Service (CoS) was implemented for MSS files in FY2002. The initial implementation provided two CoS attributes that relate to the total cost of storing data. A reliability attribute allows the owner of an MSS file to select file duplication. If selected, a second copy of the file is created and charged accordingly. As the number of MSS files written using this attribute increases, the total amount of data stored in the MSS will decrease proportionally. A usage attribute allows the owner of an MSS file to provide usage hints to the MSS. System backups can be identified with this attribute and will bypass the diskfarm cache. Those files will be written directly to tape media thus reducing the cost of having to later migrate those files from the diskfarm to tape.
2. Upgrading the Mass Storage Control Processor (MSCP):
   The MSCP runs the bulk of the MSS system software and must be sized to handle the workload. The MSCP capacity was increased by 80% in FY2002 in anticipation of the installation of the IBM POWER4 supercomputer acquired in the Advanced Research Computing System (ARCS) procurement. The POWER4 supercomputer is expected to be installed in early FY2003.
3. Upgrading blackforest to ARCS Phase 1 capability:
   FY2002 saw the start of the Advanced Research Computing System (ARCS) delivery. In early FY2002 the blackforest machine was more than doubled in size to 1,308 processors. Shortly after the blackforest upgrade, an IBM p690 Regatta with 16 processors was delivered. This machine was used for testing and system development in preparation for a large Regatta cluster that is scheduled for delivery in October 2002. Also during FY2002, four systems were decommissioned and two new systems installed. In addition, there were major system software upgrades performed on all major supercomputers.
4. Decommissioning the last Cray computer (chipeta): The end of an era:
   During the last 18 years, SCD has deployed a series of Parallel Vector Processor (PVP) systems ranging from a 2-CPU Cray Y-MP to a pair of 24-CPU Cray J90se systems. Massively Parallel Processing (MPP) systems included the Cray T3D with 128 processors and the Thinking Machines CM2 and CM5 systems. Most recently, Distributed Shared Memory (DSM) systems have been deployed; these include the Hewlett-Packard SPP-2000, SGI Origin2000, Compaq ES40 cluster, SGI Origin3800, and the IBM SPs. On September 3, 2002, chipeta, the last Cray system at NCAR, was decommissioned. The era of vector-processor Cray computers began on July 11, 1977, when CRAY-1A serial number 3 was delivered to NCAR. Since that day, SCD had continuously operated Cray supercomputers until the decommissioning of chipeta.

Highlights of SCD's Computational Science Section (CSS)

CSS provides the Computational science research and development section of this report.

1. Successful negotiation, staffing, and progress of the ESMF project:
   See above.

2. CSS initiative in making NCAR a Gelato Federation Member and in acquiring Itanium-2 technology:
   A significant CSS initiative in 2002 was the CSS-led effort that resulted in NCAR becoming a member of the Gelato Federation, an open software consortium focused on the Itanium architecture. This resulted in CSS and NETS Web100 funding, in the form of both salary for software development and equipment that amounted to approximately $360,000. This funding will both accelerate development of the Spectral Toolkit and provide NCAR with visibility within the Linux community.
3. Revitalizing the scientific track within SCD:
   CSS has created three new scientific positions: the scientific staff in CSS has grown from one to four in 2002. The CSS core mathematics and computer science research program has been expanded by the reclassification of one software engineer IV (Dr. Steve Thomas) to Scientist III, the addition of a Scientist I-level mathematician, an expert in conservative advection, and a joint appointment at the Scientist II level with the Computer Science Department at the University of Colorado.

4. Producing mathematical research in the area of iterative solvers:
   

   Spectral elements: Further progress was made in FY2002 on both 2-D shallow water and 3-D primitive equations SEAM models based on a spectral element horizontal discretization. Support for hybrid coordinates was added to the primitive equations dynamical core in anticipation of introducing physics into the model. Studies were done on new filtering techniques, based on the work of Fischer and Tufo, that do not require communications. (SEAM stands for "Spectral Element Atmosphere Model.")

   Numerical weather prediction models: In 2002, the collaborative effort between NCAR researchers Steven Thomas (SCD), Joshua Hacker (MMM), Piotr Smolarkiewicz (MMM), and Roland Stull (University of British Columbia) has led to a class of effective Krylov methods. They have developed a horizontal spectral preconditioner as an alternative to the more standard and much simpler line-Jacobi relaxation scheme. They evaluated the robustness of the proposed approach over a broad range of representative meteorological applications, and documented its superior performance in the context of a three-time-level semi-implicit semi-Lagrangian all-scale weather-prediction/research model.

Highlights of SCD's Data Support Section (DSS)

DSS provides the Research data stewardship section of this report.

1. Data development work:
   To upgrade reanalysis observations and to prepare for climate variability and other research, DSS has continued developing observational datasets. There are now seven major categories of observations, each with many component datasets. The NCEP/NCAR 50-year reanalysis (1948 - on) used Version 1 of the data. At end-FY2002, Version 3 was almost completed.
2. Updating research data products:
   DSS has always and continues to update numerous research data products on a regular basis. Data transfers using magnetic tape are becoming less frequent, and scheduled network transfers have replaced them. The advantage is that network transfers can be highly automated with new MSS files and online files becoming quickly available for users. The disadvantage is network transfers need to be carefully monitored to ensure dataset integrity. DSS continues to work on monitoring systems that include receipt reports and data delivery reconciliation.
3. Moving data processing tasks to UNIX server:
   DSS has transitioned all routine data processing off the Cray computer to a dedicated Sun server. In doing so, the capability to read and write legacy Cray data structures and the porting of many standard data-processing programs was accomplished. The new computing environment, with its direct linkage to the DSS information server, has improved our ability to serve data users.
4. Advancing the document project:
   Historically, DSS has written considerable hardcopy dataset documentation and collected many data reports. A project that will preserve these metadata is using scanning technology to create digital page images. Roy Jenne has been gathering many smaller documents into bundles of papers and writing more. The library now holds 235 documents and more than 15,000 image pages. This effort is ongoing. More scanning needs to be done, and overview guides that will aid the users need to be written.
5. Providing data services to users:
   Data services were provided to over 1,500 unique users during FY2002. About 900 of the users take data service directly from the MSS, obtain data prepared upon request, and receive data by receipt of CD-ROMs, ftp, and tapes. The remaining users are identified by unique IP address that access the online services and explicitly download files containing research data of substantial size. There are many more web hits for metadata, documents, etc.
6. Continuing reanalysis work:
   Involvement in Reanalysis with both data preparation and product distribution continues as a key activity for DSS. The Version 3 global surface and upper-air observations, and some early satellite data were provided to ECMWF for the ERA-40 project. Most of the observations are now Version 3, but not all. Some Version 3 surface data were also specially prepared during March - August 2002 for NCEP Regional Reanalysis (NRR), which is a 25-year reanalysis (1979 - on) covering North and Central America at high resolution. NCEP/NCAR Reanalysis (NNR) and NCEP DOE Reanalysis II are both run at NCEP in an operational mode now. Outputs from these reanalyses are downloaded frequently and provide users with one of the very best datasets for studying long-term climate trends.
7. Continuing marine data collaboration with Russia:
   The DSS marine data collaborative project with Russia (NSF funded) has proceeded on schedule. Marine surface observations from about 150 Russian research vessel cruises have been digitized, delivered to DSS, and verified by format translation into the I-COADS standard format. (International COADS, recently renamed, is a joint project between NCDC NOAA, CDC NOAA, and SCD NCAR). This project timetable has 1.5 years remaining and will result in 5-6 million additional records for I-COADS. We fulfilled our past obligation by sending the land surface GTS data for 1999 - 2000 to the Chinese National Oceanographic Data Center. We hope to establish another similar exchange with China whereby they will digitize the German Maury data collection (now available as scanned images) in return for additional GTS data.
8. Providing data stewardship:
   DSS staff are active and involved data stewards. For example, a collaboration was begun with Professor Yuk Ling Yung at California Institute of Technology where DSS has extracted radiative fields that define stratospheric forcing for 1979 - 06/2002 from the NNR. These data will be evaluated for the Northern Hemisphere Arctic Oscillation, a leading climate change indicator under scientific scrutiny now, and the resulting diagnostic time series will be returned to DSS for public distribution.

Highlights of the Network Engineering and Telecommunications Section (NETS)

NETS provides the Network engineering and telecommunications section of this report.

1. Participating in the BRAN Boulder Research and Administration Network (BRAN) expansion:
   BRAN is the Boulder Research and Administration Network, which is an eleven-mile fiber network in Boulder built and operated by the five BRAN partners to privately interconnect their Boulder-area facilities. UCAR has greatly reduced intra-Boulder circuit costs by utilizing BRAN fiber. In FY2002, the BRAN fiber network was extended into the ICG Boulder POP, thereby allowing direct access by BRAN members to the ICG network without the need to pay distance charges. Connecting BRAN to the ICG Boulder POP extends BRAN capabilities via leased and purchased ICG fiber. Two links have already been activated using this newly purchased dark fiber: the Mesa Lab to Center Green link and the Mesa Lab to FRGP link.

2. Implementing Front Range GigaPop (FRGP) upgrades:
   The Front Range GigaPOP (FRGP) is a consortium of universities, non-profit corporations, and government agencies that cooperate in an aggregation point called the FRGP to share Wide Area Networking (WAN) services connecting to the Commodity Internet, Abilene/Internet2, and to each other. UCAR operates the Front Range GigaPOP under contract to the other members. There are many cost and expertise advantages gained by sharing services through this GigaPOP. NCAR/UCAR has provided the engineering and Network Operations Center (NOC) support for the FRGP, with the service costs incurred by NCAR/UCAR being shared by all members. NETS believes that the greater service and bandwidth obtained through the FRGP are important enough for NCAR/UCAR to operate the FRGP by providing engineering and NOC services, and FRGP has agreed that NETS has the most qualified engineering and NOC staff to provide the very best engineering and NOC services for the FRGP.

   NETS made many significant improvements to the FRGP during FY2002.

3. Hosting the National Laboratory for Applied Network Research (NLANR)/ Internet2 (I2) Joint Techs Workshop and related meetings:
   The National Laboratory for Applied Network Research (NLANR) is funded by the National Science Foundation to provide technical, engineering, and traffic analysis support of NSF High Performance Connections sites and HPNSP (high-performance network service providers) such as the Abilene/Internet2 network. Utilizing the NOAA facilities in Boulder, Colorado, NCAR, NOAA, NIST, and NTIA very successfully co-hosted the Summer 2002 NLANR/I2 Joint Techs Meeting and IPv6 Workshop along with related meetings for Web100, Net100, ESCC, IPv6 Working Group, ATEAM, and JET (Joint Engineering Team).
4. Implementing the network configuration at UCAR's new Center Green campus:
   NETS evaluated the existing networking infrastructure in the new Center Green campus, reported on it to upper management, designed a temporary network, ordered equipment, materials, and supplies, and activated the network in less than 30 days to meet the aggressive schedule required to host the Board of Trustees meeting.
5. Participating in the Net100 project:
   The Net100 project is creating software that allows computer operating systems to adjust dynamically to network conditions. Net100 is funded by a three-year grant from the Mathematical, Information, and Computational Sciences (MICS) program in the Office of Science at the U.S. Department of Energy (DOE). Net100 is a collaboration of the Pittsburgh Supercomputing Center (PSC), the National Center for Atmospheric Research (NCAR), Lawrence Berkeley National Laboratory (LBNL), and Oak Ridge National Laboratory (ORNL).
6. Participating in the Web100 project:
   The Web100 project is funded by the National Science Foundation (NSF) as a collaborative project that will develop end-host TCP performance measurement and enhancement tools that will help end-hosts to automatically and transparently achieve high TCP data rates (100 Mbps) over the high-performance research networks. Initially the software and tools will be developed for the Linux operating system, but will be done in a standard, open manner so that they can easily be ported to other operating systems. The Web100 project achieved two of its key milestones during FY2002 by releasing the Web100 software to the general user community and submitting the Web100 TCP MIB to the Internet Engineering Task Force (IETF) for consideration as an Internet standard.

Highlights of SCD's Visualization and Enabling Technologies Section (VETS)

VETS provides the Visualization and enabling technologies section of this report.

1. Providing NCAR's new Visualization Lab:
   The ML Vislab and Access Grid became fully operational in FY2002, and they have become an indispensable resource for distance collaboration, distributed meetings, distributed presentations and training, and educational events. Over the last year we have supported and participated in several dozen events including testimony sessions for the NSF Blue Ribbon Panel on Cyberinfrastructure, weekly DOE/NSF Earth System Grid meetings, CCSM software engineering group meetings, Global Science and Technology Week events, multi-agency Grid Coordination meetings, SC Global, a Solar-Terrestrial Physics workshop, the NCAR Advisory Board, meetings with NSF, and the UCAR Board of Trustees event -- just to cite a few. This new resource has enabled us to ramp up our community interactions in a remarkable way, while at the same time reducing travel burdens in cost and time. There are intangible benefits as well: NCAR is an active and visible player "on the Grid" and we're positioned to proliferate the technology throughout our organization and into the community.
2. Publicizing strategic initiatives for data and web:
   The SCD Initiatives site was launched in September 2002 and showcases the progress of SCD across the four information technology initiatives in the NCAR Strategic Plan. In addition, the VETS internal and public websites, which are undergoing final testing, have been completely redesigned using best-practice approaches such as audience analysis, site strategy, wireframe interfaces, design comps, and template-based page development. All three websites sport a new visual look that raises the design standard for NCAR websites. WEG's beta implementation of a new streaming video server is a significant enhancement to our infrastructure, enabling users to start viewing multi-megabyte scientific animations within seconds instead of waiting for an entire movie to download. With these processes and infrastructure in place, we can now move our attention to the larger NCAR web presence.
3. Providing community data analysis and visualization software:
   SCD has continued to distribute NCAR Graphics as Open Source under the GNU Public License, and the NCAR Command Language (NCL) as "Open Binary" under a special license. Open Source is still planned for NCL in the near future. From the period October 1, 2001 to September 30, 2002, there were 1,533 accesses to the NCL download website, with an estimated 800 of these being unique downloads. For the same period, there were 7,795 accesses to the NCAR Graphics download site, with an estimated 1,800 of these being unique. NCL is now running on all major operating systems, with MS Windows (running Cygwin) and MacOSX being the latest operating systems added.

   We continue to work with the Climate and Global Dynamics division to add new functionality to NCL for the Community Climate System Model and for the scientific community. Since February 2001, CGD (with consulting help from SCD) has offered 12 hands-on workshops on NCL, with a total of 122 people attending.

   We worked indirectly with the Naval Research Lab to get their model and observational data plotted. This involved adding new functionality in NCL do curly vectors, high-resolution coastlines, and automatic generation of special map tickmarks on many different map projections. We have started to help the High Altitude Observatory division in laying the groundwork for using NCL as a post-processor for the Thermospheric General Circulation Model (TGCM). Other major new features in NCL include recognition for curvilinear grids and enhanced support for HDF-EOS (version 4) files and GRIB files.

   In determining the impact on users, it is helpful to consider the number of unique visits (not just number of hits) that the CSM Graphics Tutorial website (which is all based on NCL scripts) has received in the period February 2001 to September 2002. There were at least 13 foreign countries that visited the site 100 times or more. There were at least 19 North American universities that visited the site more than 100 times, with numbers going up into the 1,400s for some individual universities. The general trend shows an increasing number of visits every month with an average of 3,000 visits a month from May 2002 to September 2002. Locally, there have been about 2,000 visits to the site from February 2001 to September 2002.

Highlights of SCD's User Support Section (USS)

USS provides the Assistance and support for NCAR's research community section of this report.

1. Resolving user problems:
   The Technical Consulting Group (TCG) assists the scientific staff in all phases of program development and with all problems encountered with their software, system software, and system hardware. This year, TCG assisted users in migrating and optimizing the following major codes: CCSM 1.4 & 2.0 Klemp-Williamson Cloud Model, MOZART, WRF, TGCM, MHD3D, SEAM, MATCH, CGCM, AGCM/OGCM, POSEIDON, NEPTUNE, GSM, COAMPS, EULAG, CAM, POP, PCM, MM5, and NCOM.
2. Developing user documentation:
   TCG and the Digital Information Group collaborated on a fundamentally new suite of user documentation for the IBM SP systems and the new SGI Origin complex. The primary goals are to help users become self-sufficient in running codes as quickly as possible and in optimizing their codes for the architecture and environment of each computer system.
3. Developing system testing software:
   The primary software developed this year by TCG includes an automated, portable test harness. The test harness was completed in FY2002 and runs TCG-developed MSS tests, the UNICOMP Fortran90 Lite Testsuite, C/C++ tests, MPI tests, and the ARCS benchmark kernels (but not the full ARCS model codes) as well as documenting the outcome for analysis and archiving. It now stands at well over 100,000 lines of code.
4. Developing public relations documents:
   DIG has developed a series of pamphlets, entitled "PowerCurve" that highlight the central role SCD plays in furthering NCAR's scientific agenda. Each PowerCurve focuses on a specific area where SCD plays a crucial role in atmospheric research, e.g., computing, visualization, networking and data storage technologies. In addition DIG developed a "History of Supercomputing at NCAR" poster that has received wide acclaim. DIG also completely redesigned SCD's Functional Diagram for the SCD website.
5. Streamlining the daily accounting process:
   The Database Services Group (DBSG) streamlined the daily accounting update process by replacing Fortran code with easier-to-maintain UNIX scripts and Oracle PL/SQL code. DBSG tracked computer resource use for new machines: blackforest, chinook, and dave; tracked a new class of service for MSS files, and made appropriate changes to implement revalued GAUs.
6. Implementing stronger security measures:
   The Distributed Systems Group (DSG) implemented a number of security measures to provide a more secure user work environment including virus and spam scanning of UCAR electronic mail.

Highlights of SCD's Operations and Infrastructure Support Section (OIS)