|
|
|
A number of this year's activities serve as steps to realizing our strategic plans, and highlights include new efforts in web-based data access, the development of a next-generation Visualization Lab, a move to Open Source application software, and participation in some exciting new research projects. The overall thrust of the program is to provide the software, facilities, and human expertise to support a broad range of analysis and visualization capabilities while extending best practices, extending domain knowledge through basic research, and using results effectively for outreach.
As part of SCD's FY2000 effort to develop roadmaps for our future, Open Source software was identified as a strategic element of the overall plan. At this writing, VETS manages approximately 1.2 million lines of analysis and visualization application code and conducts portability testing for most of the major Unix variants including Linux. Virtually all of the software has either been moved under a GPL source code license or a freely available binary license with the intent of having all applications available under a GPL license by early FY2003. NCAR Graphics and NCL are still available as a licensed package for those who require enhanced levels of installation and usage support, and this is supported by the revenue from the packaged software.Open Source is an important step for our research community because they have specialized software that is well adapted to our domain and may be shared and extended for new purposes. It's an important step for us because we can extend our application software offerings with developments from the growing mass of other developers who have also adopted an Open Source path.
Used by thousands of researchers, NCAR Graphics is a mainstay in the geosciences community. The package provides software libraries that support a wide variety of graphing and visualization functions with extensive and robust mapping capabilities. NCAR Graphics delivers superb, publication-quality visuals that are as good or better than anything available from any toolset, even commercial ones. Mid-year, NCAR Graphics was made generally available under a Gnu Public License (GPL), a step warmly received by the community.
NCAR Graphics is a mature package, and while most of the development time was spent in fixing bugs and enhancing portability, several new features were added. These included new mapping functions, new political boundaries, a special map database for NCEP climate divisions, enhanced maps for Canada, Mexico, and U.S. counties, a new resampling library (ngmath), and various other enhancements in support of NCL.
The NCAR Command Language (NCL) is a scripting language for computation and visualization. While designed as a general-purpose language, SCD has worked extensively with CGD to equip and position NCL as the analysis environment of choice for the Climate System Model (CSM). NCL handles NetCDF, HDF, ASCII, binary, and GRiB files (including ECMWF coefficients) and can natively access CCM "history tapes." Layered interfaces tuned for climate analysis coupled with over 400 additional mathematical and graphical functions make NCL a formidable tool for the climate researcher. As part of our movement toward Open Source software, NCL binaries have been made freely available. This creates enhanced opportunities for global scientific collaboration, allowing distributed research teams to share tools and scripts without having to agree on purchasing various commercial tools.Most of the enhancements made to NCAR Graphics were in support of NCL functionality, so all of the new capabilities noted above apply directly to NCL. Additional improvements included new functions for reading in multiple data files in one action, attribute querying, enhanced weather maps, and the addition of several numeric input types. As a fairly mature environment, NCL is showing a lot of promise as an infrastructure layer for many other endeavors.
For example, working with the Weather Research Forecast (WRF) model team, a Python wrapper for NCL was developed along with support for the native WRF terrain-following coordinate system. CGD has begun development of several web-based data analysis tools that are based on NCL. RAP's award-winning realtime weather website uses NCL to produce its weather map, which is served from UCAR's homepage. We have also experimentally integrated NCL as a graphics and processing engine for the Live Access Server (LAS) web-based data provision environment (discussed under A new community data portal) and added the ability to directly access DODS servers.
Community acceptance of NCL appears to be growing nicely and is often observed to be accounting for a substantial portion of the cycles expended on dataproc. It is now a fairly mature tool, but in FY2001 we expect to continue to extend domain-specific functionality in support of both CSM and WRF, including new support for HDF-EOS datasets. Opportunities related to additional integration of NCL capabilities with the increasingly popular Python scripting language will also be explored.
Vis5D is a popular free 3D visualization package widely used in the geosciences, especially by the mesoscale community. SCD has developed a customized version of this software which provides support for very large datasets (greater than 2 GB), delivers an interactive stereo/3D virtual exploration capability, and supports output of Web-3D objects in the form of VRML (the Virtual Reality Modeling Language). SCD continued to support and extend this GPL software for the broad community, and our version is in use at universities and corporations as well as at other NSF, DOE, and DOD research labs.
Work began in FY2000 to cleanly integrate all of our enhancements into the new 5.3 version of Vis5D with an expected release in FY2001. We also experimented with SGI's Volumizer API for volume rendering and with geometry compression for improved rendering performance.
Direct volume rendering is a valuable approach for scientific visualization, especially in the area of turbulence. SCD has developed the volsh package, which provides high-performance, high-quality parallelized volume rendering for most Unix-based platforms. While volsh is a fairly mature tool, the addition of a rudimentary capability for tying the volume renderer to more traditional analysis tools (e.g. IDL) constituted an important enhancement to the overall capability.
Research in the earth system sciences is extraordinarily demanding from the data analysis perspective. Researchers need to access, explore, and analyze collections of observational and simulation data that are complex, diverse, and extremely large. Current simulation efforts in climate, weather, and turbulence produce terabyte datasets, and these must often be compared and contrasted with one another and to observational data. VETS manages a fairly comprehensive portfolio of dedicated facility assets aimed at various aspects of this area: web-based data access, production systems for "classical analysis" (e.g. dataproc), and visual supercomputing facilities for next-generation virtual exploration.
Responding to a growing demand for flexible, web-based access to scientific data -- both by research investigators as well as the community of data consumers -- SCD put a new data-provision system into production: motherlode. Motherlode is slated to support both an enhanced mode of operation for Unidata's Internet Data Distribution System (IDD) as well as web-based provision of important retrospective and simulation datasets. At the time of this writing, the top-level IDD service is in full production with an important new capability: a 30-day backing store of data products. This allows data consumers to repopulate data archives in the event of a service outage, data loss, or similar requirement.SCD launched a number of pilot data service projects including data provision for CGD's ACACIA project and experimental work in Reanalysis-II data. There are some nice new community technologies that serve as a substrate for this activity: DODS (the Distributed Oceanographic Data System), LAS (PMEL's Live Access Server), Ferret (PMEL's analysis system), and our own NCL. SCD has integrated these components onto motherlode and begun the process of populating the systems with datasets.
Samples of the ACACIA and Reanalysis-II data are available on the web. In FY2001 several of these pilot projects will be released to the general community and monitored so that we can develop a better understanding of the community demands, requirements, and future opportunities. The overall plan is to provide data provision systems along with a solid integrated layer of infrastructure such that local and university research programs can instantiate their own web-based data access services with a very low cost of entry.
SCD has operated its highly successful Visualization Lab for several years now. It has proven to be a valuable tool for enabling new understanding of large-scale scientific data, and it has also served as an important and popular window into our research activities at NCAR and UCAR.During FY2000, SCD developed plans for a next-generation facility aimed at collaborative group visualization and analysis with major enhancements in capability, capacity, and accommodation of larger groups. A new physical space was identified, pre-visualized, and designed with the latter phase of the project brought to completion late in this reporting period. Located in the Mesa Lab near the computer room, the new space will feature major improvements in its computational infrastructure, a next-generation stereo-3D display system, and an AccessGrid environment for collaboration.
The new lab is designed such that it is relatively easy to reconfigure for various modes of operation. In theater mode, the new room will be able to accommodate about 50 people, a major improvement in our ability to host large groups, an activity that is becoming more common. Equipped with modular furniture and extensive networking (wired and wireless), the space will lend itself beautifully to small group data exploration and collaboration as well as larger group sessions such as presentations, AccessGrid sessions, workshops, and symposia that require access to sophisticated visual technologies.
During FY2000, most of the enhancements to the computational infrastructure were initiated and several were completed. We retired a 7-year-old SGI Challenge system, replacing it with a 2-processor SGI Origin200 service system and an 8-processor SGI Origin2000 computation platform. We upgraded our existing SGI Onyx-2, adding an additional InfiniteReality-2 graphics pipe (for a total of 2) and additional physical memory for a total of 7 GB. A Lightwave KVM (Keyboard/Video/Mouse) switch was installed to support the operation of the new lab and to enhance VETS staff access to lab systems. The Lightwave switch flexibly routes a visual supercomputer station to an office or lab location via a fiber-optic connection controlled over the network. We also addressed major intra-lab data-sharing problems by installing a fiber-optic-based Storage Area Network (SAN) and expanding overall storage to approximately 0.5 TB.
The general architecture of the evolving facility is shown here:
The new Visualization Lab facility is slated to open in early summer 2001. It will be a model environment for next-generation data exploration and analysis for the geosciences.
VETS collaborates with several modeling efforts and working groups in the area of advancing data management, analysis, and visualization capabilities. We also work with other organizations to jointly develop new technology, and our move to an Open Source model will accelerate such activities in the future.Staff members work directly with researchers to identify and deploy analysis and visualization solutions on a project-by-project basis, a process that not only advances the research programs but also provides valuable insights into current and future requirements.
A full summary of VETS collaborations appears in the Collaborations and outreach section of this report.
SCD ASR - Table of contents
