Visualization and enabling technologies

Overview

SCD's Visualization and Enabling Technologies Section (VETS) has a primary focus of advancing the knowledge development process. Our activities span the development and delivery of software tools for analysis and visualization, the provisioning of advancing visualization and collaboration environments, web engineering for all of UCAR, R&D endeavors in collaboratories, the development of a new generation of data management and access, Grid R&D, novel visualization capabilities, and a sizable outreach effort.

VETS grew to 22 staff members in FY2003, which includes several student contributors and new positions coming from external funds. We were awarded continued NCAR funding for a Cyberinfrastructure Strategic Initiative, and this covers two additional staff positions. We were successful in our bid to the NSF ITR Program and received an award to join with U.C. Davis in developing new visualization technology. We also received a three-year grant from NASA to develop a Grid environment for biogeochemical modeling and analysis, an effort that will have substantial synergy with our Community Data Portal (CDP) and Earth System Grid (ESG) projects. We complete another year as a strong contributor to the Unidata-led THREDDS effort. VETS contributed as an unfunded partner to the NSF Alliance Expedition, Scientific Workspaces of the Future, which is moving forward the tool agenda in the context of the Access Grid. We submitted a proposal to join NSF's Extensible Terascale Facility (ETF) and were competitive, but did not receive an award.

Our NCAR Command Language (NCL) software continues to grow in popularity in atmospheric science research, but it is also being adopted by other agencies, the military, the Earth Simulator Center in Japan, and even in classrooms. We added some important new capabilities this year, in particular the ability to effectively visualize data with curvilinear coordinate systems, which is important for CCSM and other research efforts. This year we launched an aggressive effort to refactor our core software structure in support of developing a new language layer, Python, on top of NCL's formidable capabilities. This is an important first step toward a larger, next-generation framework. We also deployed SGI's Visual Area Networking as a pilot project aimed at evaluating one approach to delivering powerful 3D visualization onto a wide variety of user desktops.

The Web Outreach and Redesign (WORD) effort is one of the primary components funded out of our NCAR strategic initiative, and it is aimed at fundamentally elevating our ability to use the web to communicate, share, and work with our stakeholders and the general public. WORD spans the UCAR, NCAR, and UOP web presences, and despite its scope, we made excellent progress this year on developing consensus on direction and overall design. Our development of web applications for the UCAR community continued to accelerate and we completed most of the development work for a new "people directory" service as well as a new web-based mail service that ties in all of the many web servers across UCAR. We have two efforts in the area of web/Grid-based data access and management -- the Community Data Portal (CDP, NCAR/UCAR) and the Earth System Grid (ESG, DOE/NSF). We made excellent progress in both of these, particularly in the area of metadata development and authoring and the design and engineering of the associated web portals. We expect both of these efforts to go into full production in FY2004, when hundreds, perhaps even thousands, of datasets will be searchable and accessible by a broad science and education community.

VETS continued a very strong outreach program, providing dozens of presentations in our Visualization Lab. We also spun up a new program where UCAR's Public Visitors Program (PVP) prepares and delivers highly visual presentations to visiting educational groups, and the results thus far have been very positive. Through teamwork, we are able to accommodate a much greater number of visitors with only a modest impact on SCD technical staff. We had a strong presence at the SC2002 conference and showed off a new design scheme in our exhibit. SC2002 was our only formal exhibit in FY2003, which reflected our strategic plan to reduce our exhibit participation in conferences in favor of more technical R&D and a growing, stronger presence presenting and publishing papers. This new direction recouped a substantial amount of high-level staff time.

Responding to recent NSF reviews of SCD, NCAR, and UCAR, we have begun an aggressive effort to define and gather metrics of performance. Our initial thrust areas in this regard are web services, visualization services, and visualization and analysis software. In FY2004 we will be working to instrument our Community Data Portal and Earth System Grid environments such that these can be monitored for usage levels as well.

FY2004 promises to be an exciting year where we see a number of our emerging efforts (e.g. CDP, ESG, WORD) reach a critical level of maturity -- one that will afford us the opportunity to begin some important new strategic integrative efforts. A primary thrust will be beginning planning, requirements, and some development of a larger-scale framework that encompasses data management, access, analysis, visualization, and collaboration. Building on our successful delivery of NCL to the community, our Visualization Lab, our dataportal activities, substantial advances in metadata definition and development, and our work with the AccessGrid, we will be driving a new effort to construct coupled analysis and visualization systems that build on all of these strengths. This work will draw on NSF core funding and our ongoing funded relationships with DOE, NASA, and other strategic partners. In a complementary fashion, we have begun developing strategic plans for a next-generation knowledge development environment.

Cyberinfrastructure Strategic Initiative projects

WORD project

A key component of the World Class Web initiative, the WORD group is building an umbrella website that encompasses the current NCAR, UCAR, and UOP top-level sites. This new integrated site will make our web presence a more engaging, educational, and entertaining experience while maintaining an appropriate tone for our scientific and academic audiences. The site will serve the unique needs of six audiences: the scientifically literate public, students and educators, the research community, journalists, members, affiliates, and staff. Feature stories that rotate monthly will highlight our compelling research and activities. The WORD group is composed of over 20 individuals from across the three organizations, and it plans to launch in the spring of 2004.

Community Data Portal (CDP)

The NCAR Community Data Portal (CDP) is an NCAR Strategic Initiative aimed at developing a central institutional gateway to the large and diversified data holdings of UCAR, NCAR, and UOP. The ultimate goal is to provide a state-of-the-art data portal with a broad spectrum of functionality ranging from data search and discovery to catalogs and metadata browsing, and from high performance and reliable data download to analysis and visualization. At the current level of funding, the CDP still needs to be considered a prototype project aimed at assessing the viability of a larger institutional effort.

During 2003, extremely good progress was made in developing, instrumenting, and populating the CDP web portal (see http://dataportal.ucar.edu/), specifically:

• The underlying software architecture of the web portal was changed from pure Java Servlets running in a Tomcat container to Struts Actions, thus realizing a clear separation between the three software layers of view, control, and business logic. At the same time, a completely new user interface (the "view") was designed and implemented (see picture below).
• A new "Google-like" data search and discovery system was developed to allow locating datasets through a query resolved against the dataset's metadata. The system (based on the Apache Lucene software) supports a rich, powerful query syntax and is integrated with the metadata viewing and catalogs browsing applications (see later). It is expected that in the next year the free text search will be complemented by other kinds of advanced, structured queries yet to be developed.
• The THREDDS schema (see http://www.unidata.ucar.edu/projects/THREDDS) was adopted for the purposes of data cataloguing and hierarchical organization, while Dublin Core is currently recommended as the preferred choice for encoding high level search and discovery metadata (although the CDP may ingest and use any kind of metadata). Applications were written for browsing and viewing all these kinds of metadata through customized stylesheets.
• The CDP capabilities were increased by allowing web-based download of data from the NCAR Mass Storage System.
• The CDP data holdings were augmented with several sample datasets provided by various divisions and projects across UCAR and NCAR, including:
• ACD (MOZART model and TRACE-P model evaluation data)
• CGD/CAS Satellite Data
• CGD CCSM model data
• SCD/DSS full data holdings (!)
• SCD/VETS visualization gallery
• Unidata IDD model data
• WACCM model data
• Several software packages were installed or upgraded on the host machine and currently provide a broad range of server-side functionality, including:
• LAS (Live Access Server) for data visualization, aggregation, and download
• OPeNDAP (Open Network Data Access Protocol), for remote data access and sub-setting
• OPeNDAP Aggregation Server, for remote data access and aggregation
• ArcIMS (Arc Internet Map Server), for interoperability with GIS applications

The CDP web portal was presented and promoted both within UCAR and NCAR (through the sponsorship of the NCAR Data Management Working Group) and in a national and international context (through collaborations with the Earth System Grid and the Earth Science Portals projects). It is expected that in the next year the level of collaboration with other Web and Data Portals (ESG, GCMD, DLESE, BDAC, NOAA/CDC, GFDL) will significantly increase, spreading the first seeds of a real, effective federation of Earth Sciences sites.

Mountain Region TeraGrid

SCD’s VETS and NETS sections led the effort to develop a proposal to join NSF’s Extensible Terascale Facility (ETF). Our proposal, entitled the Mountain Region TeraGrid (MRT), proposed to link data and visualization resources at NCAR and the University of Utah’s Scientific Computing Institute (SCI) via a 10Gb/second network interface through the National LightRail initiative. Our bid focused upon the provision of Grid-based delivery of data and visualization resources along with many scientific endeavors that could take advantage of this next generation distributed system. Our proposal was reviewed favorably but was not selected for funding this year. We intend to bid again in FY2004, as this is a potentially important component of our efforts in the area of Cyberinfrastructure.

Visualization lab and services

SCD’s Visualization Lab, now in full operation for two years, qualifies as a resounding success. We designed it to serve as a next-generation collaborative scientific environment and meeting space and it has dramatically exceeded expectations (which were already fairly high). It sees constant use for technical and scientific endeavors along with a steady stream of outreach activities. In FY2003 we added two new dedicated support staff to support the operation and maintenance of the facility, and this has greatly enhanced our ability to support the heavy load of events that we support via the Vislab.

Computational facilities

VETS operates a state of the art visual supercomputing facility, offering powerful interactive visual computing platforms, batch platforms for data post processing tasks, and a shared, high performance Storage Area Network (SAN) for containing large working sets of data. Following the previous year’s grand opening of the new SCD Visualization Lab, FY03 was relatively quiet with equipment upgrades and acquisitions intentionally minimized. Nevertheless, VETS augmented its SAN with an additional 4.5 TBs of RAID disk, bringing our shared storage total capacity to ten TBs, three of which are presently reserved for use by NCAR’s Geophysical Turbulence Program. VETS also deployed two new systems: a 10-processor, 4-pipe SGI Onyx3800, dedicated to VizServer usage (see below); and a dual processor SGI Octane2. The year also saw two rounds of major OS upgrades on all the lab’s SGI systems.

Experimental visualization cluster / Commodity graphics systems

The performance of commodity graphics chips has increased at a rate exceeding Moore’s law for the past several years! Inexpensive gaming chips now offer performance surpassing by many metrics traditional high-end graphics systems. Furthermore, with the advent of programmable Graphics Processing Units (GPUs) the capabilities offered by these graphics platforms is unprecedented. VETS has begun to explore the utility of these new chips in a number of areas including:

• Mentoring a SOARS student in the development and evaluation of an entirely GPU based version of 2D Image Based Flow Visualization. The exploratory work yielded a 2x increase in performance while idling the CPU.
• Developing benchmark codes that take advantage of the vertex array extensions needed to achieve high geometry rates for dynamic scenes.
• Developing a hardware texture based volume rendering engine for VETS multiresolution data browser, mdb.
• Evaluating performance of 64-bit computing platforms with commodity graphics including HP’s image-based remote rendering technology, Golden Gate.

Access Grid  

The SCD Visualization Lab’s Access Grid (AG) continues to grow in use and popularity, and it supports weekly DOE-funded Earth System Grid meeting, the DOE SciDAC Climate modeling effort as well as the CCSM (Community Climate System Model) Software Engineering Group. More recently we have begun to use the AG in support of another collaborative scientific effort, the NSF-funded Center for Integrated Space Weather Modeling (CISM).  Beyond these regularly scheduled meetings, the use of the Access Grid doubled this year with over 40 events.  VETS staff continue to champion the AG technology, and have organized and presented an AG seminar to introduce NCAR staff to the Access Grid video conferencing system.  Overall, the AG is becoming a core fabric for the conduct of several large-scale, high-profile scientific research efforts and technology-in-the-service-of-science efforts. Consequently, the SCD Access Grid’s functionality and usefulness has been recognized across the organization, prompting the MultiMedia services group to acquire and support several new portable systems for other divisions.

Several upgrades were added to the SCD system including the integration of a video mixer to provide overlays and captions to outbound AG video streams, system upgrades for the display and video machines, and the installation and testing of the latest AG software (v2.0).  AG operators are also continually updating online documentation regarding the support and operation of the SCD system.   VETS staff successfully implemented a portable AccessGrid system from InSORS that can be set up in almost any meeting room and supports 4 or 5 local participants.  This system has provided essential backup during schedule conflicts in the vislab.

Beyond the routine use of the SCD Access Grid for meetings and conferences, the AG provides a platform for research in collaborative video conferencing.   The SCD AG was used to remotely participate in a Scientific Workspaces of the Future (SWOF) demonstration at the University of Illinois where new collaborative technologies were being shown, and also for a demonstration of collaborative visualization using Vis5d with ANL researchers Mickelson, Jones, and Papka.

VETS staff also continue to provide consultation to outside organizations that are interested in installing their own Access Grid nodes, and they regularly collaborate with other sites to provide testing.  Sites that VETS has assisted include the University of Hawaii, Dartmouth, Manchester University, the Desert Research Institute, Indiana University, and the University of Colorado.

VizServer

For many years VETS has had much success in the production of many compelling and meaningful scientific visualizations, appearing in publications and presentations, worldwide. In all but a few cases expert VETS visualization staff, working closely with NCAR scientists, produced these results. In FY03 we removed one of the final barriers to greater visualization usage among scientists and researchers themselves by enabling users to harness the power of VETS visual supercomputing resources from the convenience of their own offices using only a commodity desktop or laptop computer. SCD’s VizServer pilot project, kicked off in FY03, employs a proprietary, image-based remote display technology to enable virtual visual supercomputing by exploiting ubiquitous high-speed networking found throughout ML and FL campus offices. The technology permits researchers to access otherwise highly distance limited visual computing resources, maintained in the SCD Visualization Lab, from virtually anywhere with reasonable network connectivity. Though much of FY03 was spent testing the system and shaking out the bugs, scientists at the Foothills Lab, Mesa Lab, and most recently, NOAA, have begun to take advantage of this powerful new service. One researcher uses the service on an almost daily basis, while several others have employed the service for specific projects. In FY04 we plan to expand our user base further through outreach efforts directed at scientists with highly demanding 3D visualization needs. We project that the current system is capable of accommodating three or four regular (daily) users. This pilot project is aimed at evaluating the relative effectiveness of providing a shared high-end resource vs. dedicated high-end desktop systems. One of the primary reasons that VizServer (and similar approaches) is very interesting to us and worth of such attention is that it mitigates the need for moving terascale data on a regular basis.

Metrics

In the last quarter of FY03 VETS instituted systems accounting to measure utilization of key Vis Lab resources, including usage of the Visualization Lab room itself. The tables below provide a sampling of these metrics for FY03Q4.

Howard University Access Grid    

SCD was given charge to provide funding and support to bring an Access Grid collaboration system online at Howard University.  Howard is building a new lab for their atmospheric science program and this space has been identified as the initial location of the AccessGrid.

VETS staff established a schedule for the design, acquisition, and installation of the system and collaborated with Howard University Professor Dr. Everette Joseph to review room layout options and specify requirements for the project. VETS staff also contacted InSORS, an access grid vendor, to acquire quotes and recommendations for the installation.

At the end of FY2003, the design was finished and system recommendation had been made.  It is expected that installation will begin in the first quarter of FY2004.

Data analysis and visualization research and development

NCL and NCAR Graphics (and metrics)   

The NCAR Command Language (NCL) is a scripting language that provides robust file I/O and data analysis in addition to scientific visualization based on the publication quality plots produced by NCAR Graphics.  NCAR Graphics is a software package that provides C and Fortran libraries for calling a wide variety of graphing and visualization functions, including extensive contouring and mapping capabilities. NCL has been offered for free in binary format since late 2000, and has grown rapidly in the scientific community as evidenced by the increase in the number of software downloads, the number of people visiting the various NCL websites, and the growing interest in NCL workshops. NCAR Graphics has been open source since the year 2000.

A recent survey of NCL users showed that in addition to the atmospheric and oceanographic sciences, NCL is used by graduate students, professors, and researchers in areas of mechanical engineering, geology, geophysics, electrical engineering, earth sciences, biogeochemistry, and theoretical physics. In these fields, NCL is used for data analysis and visualization of a wide variety of model runs, in teaching graduate-level atmospheric-related courses, for automatic generation of web pages, for use in real-time web sites, for converting from one data file format to another, and for publication graphics in journals, presentations, and dissertations.

These users hail from all areas of the globe including Australia, Argentina, Canada, China, Denmark, England, France, Germany, Greece, Italy, Japan, Korea, Norway, South Africa, Switzerland, Taiwan, United States, Vietnam, and Yugoslavia.

In FY2003, support for plotting unique 2-dimensional grids was added. These types of grids include the Parallel Ocean Program (POP) grid (which was developed at Los Alamos National Laboratory and incorporated in the Community Climate System Model), spherical geodesic grids, and the ORCA, ISSCP, and SEAM grids.

In FY2003, significant performance enhancements were made and over twenty new functions added to NCL. There were also several user-requested features added, including direct PDF output (for producing high-quality graphics for publications), a more robust GRIB file reader, a mini reference document available in hard-copy, a means for creating user-defined dash patterns and markers (a long-requested feature), and a robust print statement for writing out columnar data.  Works in progress include incorporation of the special grids mentioned above into the contouring and vector capabilities of NCL, a Python interface to NCL's graphical and file handling capabilities, a Vis5D data file writer, and ports to new 64-bit systems.

We started gathering metrics on NCL usage, and in FY2003 there was a 34% increase in the number of accesses to the download web page for NCL over FY2002, and a 30% increase in the number of postings to the NCL email list. The number of hits and visits to the NCL/NCAR Graphics documentation web site saw an increase from 19,076 visits and 288,310 hits in November 2002 to 26,530 visits and 4,640,999 hits in September 2003. Note: these statistics show unusual spikes in the last three months and will be further monitored in the upcoming months. On the CGD web site, which hosts its own NCL documentation, the number of hits went from 60,000 in October 2002 to 90,000 in August 2003, roughly a 50% increase in usage. These increases may partly be due to the number of NCL workshops offered in FY2003, of which there were 2 off-site workshops and 5 local workshops for a total of almost 100 attendees. Many of these attendees went back to their sites and taught NCL to their colleagues. Several upcoming NCL workshops are planned for naval oceanographic researchers and local scientists, as well as university users in Australia, France, and Purdue.

Python-based data analysis and visualization

VETS continued to expand its knowledge and usage of the Python scripting language as a tool for data analysis and visualization. 

Python was used as the language for producing two animations - the Sea Ice and CO2 Levels animation using data from the CCSM2, and the Water Vapor and Precipitable Water animation (documented in the Collaborative Efforts section of this document).  The Python language reduced the development time for these animations primarily due to its flexibility and the availability of several pre-existing extension modules, such as array and image processing capabilities.

Much progress was made toward providing a Python interface for the NCAR Command Language (NCL) graphics functions.  This interface will make those functions available to a much wider audience, increase the opportunities for collaboration, and in general be a significant contribution to the community.  An alpha release of the Python interface to NCL (and the base level C functions) is scheduled for late calendar year 2003.

NSF ITR award

The study of the dynamical aspects of physical phenomenon or chemical processes is paramount to the advances of many sciences. Great investments have been made to advance numerical models and the computing platforms upon which these codes are run. Essential to the understanding of the resulting data is the ability to interactively visualize and explore complex, dynamic phenomena. However, the state of the art of existing visualization techniques with respect to their ability to handle temporal dimensions is poor. As a result, increasingly the challenge for scientists, often more daunting than performing numerical simulations themselves, is effectively exploring and understanding immense, time-varying volumes data sets generated by these codes.

Building on our previous efforts, VETS, in collaboration with U.C. Davis, Ohio State University, JPL, and the University of Tokyo, authored a NSF ITR proposal to pursue a five-year investigation into new methods for the visualization of time varying data. “Gleaning Insight in Large Time-Varying Scientific & Engineering Data” has been awarded and work on this effort will begin in FY04. VETS contribution to this effort will focus on extending our progressive data access techniques discussed above to better handle time varying data and irregular data. We will also use the bulk of awarded funds to support a software engineer to help develop Open Source user tools based on the most promising technologies explored under this award.

Visual Geophysical Education Environment (VGEE)

The VGEE is a curriculum and interactive tool-set designed to help students learn science concepts by constructing scientific visualizations from real data.   The VGEE was developed by a collaboration of atmospheric and computer scientists, educational theorists, undergraduate teachers, and students.  The VGEE team included Atmospheric Scientists from the University of Illinois and West Chester University, education researchers from the University of Georgia, developers of the Digital Library for Earth Science Education, and VETS staff members. The entire VGEE project is web-based and catalogued in the Digital Library for Earth System Education (DLESE). 

The VGEE software design continued to be re-architected in FY2003 by the Unidata Metapps developers to use the feature-rich IDV toolset, and VETS continued their involvement by providing test data including a new VGEE ocean data set developed for a demo at AGU. The project term for VGEE is now at an end, so progress in FY2004 will be focused on authoring the final report.

Multi-resolution toolkit (mtk)

The challenges associated with high resolution numerical modeling commonplace at NCAR and elsewhere do not end once the simulation completes. The torrents of data produced by these codes stress our abilities to analyze and manipulate these model outputs in our effort to extract meaningful information. A promising new approach that VETS began exploring in FY03 to address this problem is hierarchical data representation combined with progressive data access. The premise behind this tactic is that many scientific inquires may be conducted using less computationally demanding, coarsened approximations of the data. In some instances the coarsened representations may be adequate to completely satisfy the inquiry, in others they may help to greatly reduce the temporal and/or spatial domain upon which further full resolution inquires may than be conducted.

Our progressive data access scheme is based upon the hierarchical representation properties of wavelet transformations. We have integrated our novel data scheme into a number of visual as well as non-visual analysis tools including, Kitware’s VTK, RSI’s IDL, and our own interactive direct volume renderer, mdb. We have begun to experiment with this scheme in collaboration with scientists from NCAR’s Geophysical Turbulence Program (GTP), exploring the results of several large turbulence simulations, of which one has a spatial resolution of 504x504x2048 and contains hundreds of time steps, representing terabytes of data. Our progressive access scheme has permitted essential interactive interaction with this vast data, which otherwise would have not been possible, while using only a lowly desktop PC.

Next-generation analysis and visualization software

Building upon the data management, analysis, visualization, and middleware activities described in this report, we are poised to launch an integrative next-generation effort that fuses many of these new capabilities. This will be a primary focus in FY2004, and it will involve collaboration with a number of strategic partners and funding entities, including NASA, DOE, and NSF. Preliminary discussions, meetings, and workshops with DOE took place in FY2003.

Enterprise web services

The Web Engineering group is advised by the UCAR/ITC Web Advisory Group (WAG). The WAG has made good progress this year toward the completion of a WAG Implementation Plan which will turn the existing WAG Strategic Plan into concrete project recommendations for the Web Engineering Group (WEG) and other web development groups at UCAR/NCAR/UOP. The WAG voted to become a representational body and has recruited individuals from nearly all the divisions and programs. In addition, the group has completed a survey of 80 internal web staff to determine priorities across the organization.

Web facilities

The UCAR Web Engineering Group (WEG) is currently in the process of a dramatic architecture shift that will take further advantage of the Sun clustered environment we moved to in 2002 and 2003. Using new features of Sun clustering software, we are integrating several separate clusters for file and applications services into one master back-end cluster. This will provide for increased flexibility and more efficient use of drive storage resources which we doubled in 2003 to meet increased demand. In FY2004, we will also be adding a development/test cluster for the purpose of testing code before moving it into production. The implementation of this popular request from our customers will mean higher quality launches of new websites and smoother software upgrades. The new architecture is pictured below with planned changes depicted in color.

Tools and services

In 2003, the WEG continued to address the WAG strategic plan with its own prioritized list of projects while at the same time contributing to the development of the formal WAG implementation plan.

The Tomcat Java application server launched in 2002 has proven quite popular and now hosts 5 applications. WebMail, which launched in 2002 now boasts over 300 registered users. The newest addition is People Search, a WEG-developed application which is a replacement for the NCAR phone directory and EQC directory change request utility. This web application brings together directory information from around the organization into a central location and enables searching of both staff and university members.

WEG formalized streaming media services through the deployment of the Real Helix Server and worked with various groups, in particular UCAR Multimedia, to support the organizational use of this service.

The WEG now hosts a SOAP server to provide for centralized hosting of XML-based web services, a web standard that is gaining in popularity among developers. The WEG also developered the first UCAR web service which enables any UCAR website to access gatekeeper authentication via SOAP.

We performed software evaluations of several product categories including content management systems, collaborative software development, and portal frameworks in preparation for offering these as new WEG services.

Web outreach, advocacy, and community building

The UCAR web community has seen some important shifts in 2003. The WAG is now a representational body and the WORD group is bringing important integration to our overall web presence. UCAR Communications will play a central role in management of the new WORD umbrella site and the WEG is providing the web engineering and hosting resources for the project. Both WAG and WORD have done extensive outreach and community building in 2003, including a WAG open house, WAG survey, outreach to all divisions and programs to invite new members, and numerous presentations to a variety of groups on the WORD project.

In FY2004, we plan to complete the WAG Implementation Plan, and focus more on outreach to divisions and programs to promote our core hosting services with internal marketing efforts, "quick start" documentation, training, and initial consultations.

WORD project

One of our primary web efforts and one that WEG will be deeply involved in is the Web Outreach and Redesign (WORD) effort. This project is a component of our Cyberinfrastructure Initiative and is described in the early paragraphs in this section.

Web metrics

The WEG web cluster served over 105 million web pages in FY2003. On a monthly basis, this translates to 8.8 million pages served to over 500,000 unique visitors totalling 180 GB of content served. Peak loads have reached over 800,000 pages per day. We are now reporting a metrics summary on a monthly basis to management.

Our plans for enhancing metrics in FY2004 include moving three new applications into production: Cacti which is a server performance monitoring tool, Sawmill which is a more advanced and configurable web traffic reporting tool, and Tomcat web application performance monitoring.

Data and knowledge systems R&D

Over the past two years, VETS has established a very active program in data management, access, and related Grid technologies. We briefly described our various synergistic efforts below.

The Earth System Grid

• Design of a new web portal, with some commonality with CDP.
• The development of a new software architecture built upon a model-viewer-controller paradigm and the Struts framework.
• Developed a new metadata schema for climate research data;
• Co-developed the NcML (netCDF Markup Language) in collaboration with Unidata;
• Authored high-level discovery metadata for roughly one hundred extant simulations;
• Deployed a new Replica Location Service (RLS) on the ESG infrastructure;
• Released an alpha version of OpenDAP-g distributed data protocol server based on Grid technology;
• Published a paper on ontologies as they relate to the ESG;
• Established strong relationships with the Earth Science Portal (ESP) effort, NOMADS (NOAA), the Unidata THREDDS project, and the U.K. e-Science program;
• Executed a successful in-progress project peer review with some very complimentary review feedback. Primary results from this review indicated that the project was effectively managed, we were aware of and leveraging other related efforts, and had good technical direction. Primary recommendations were to accelerate deployment for users and to simplify the overall system in favor of rapid adoption.

Our primary goal in FY2004 is to facilitate the management and access to IPCC (Intergovernmental Panel on Climate Change) data that will begin to arrive in CY2004. We expect to have a production release of the ESG portal in early 2004.

Community Data Portal (CDP)

Our Community Data Portal (CDP) effort is a cornerstone of our activities in Cyberinfrastructure and was detailed earlier in this report.

GridBGC

NCAR and CU were awarded a three-year grant through NASA's Advanced Information Science and Technology (AIST) program one of 11 awards made nationally. The effort is aimed at developing a new modeling system that encompasses the remote execution of a biogeochemical model (a future component of CCSM), the Grid-mediated movement of initialization and output data, and associated web portals and analysis tools. This is a close collaboration among CGD (PI and scientist), SCD CSS and VETS, and CU’s Computer Science Department. Recruiting took place in FY2003 and formal project work will begin in FY2004.

KEG   

VETS led the effort to develop another submission of the Knowledge Environment for the Geosciences (KEG) proposal to NSF’s Large ITR solicitation. This second submission was also unsuccessful, although it did receive some favorable reviews. During the time that we have been working on this very-broad proposal, we have managed to initiate work and gain some funding in most of the areas articulated in the proposal. Thus, the overall concept appears to be sound but is perhaps too broad for a single project. KEG, to some degree, is serving as a strategic roadmap for many of our emerging efforts and strategic plans.

Collaborative science and technology projects

VETS continues to collaborate on many different modeling efforts and research projects across the organization.  Staff members work directly with NCAR researchers and scientists outside the organization to jointly develop new technologies and visualization solutions to their problems.  The examples below summarize the collaborative visualization efforts engaged in by VETS staff in FY2003.

CCSM sea ice

Clare and Scheitlin worked with Marika Holland (CGD) to produce several animations showing a 150-year change of polar sea ice thickness in response to increasing atmospheric CO2 levels.

Hazardous gas dispersion

Scheitlin worked with RAP researchers to produce a visual simulation of the 3D dispersion of a hazardous gas released over Salt Lake City.

Big Elk forest fire

Scheitlin collaborated with Janice Coen (MMM) to produce a weather and fire behavior simulation of the Big Elk Fire (July 17, 2002 - Pinewood Springs, CO).

TIEGCM, TIDI, and GSWM

Clyne continued collaborations with HAO’s Qian Wu and Tim Killeen on the production of numerous visualizations of TIMED Doppler Interferometer (TIDI), Global Scale Wave Model (GSWM), and Thermosphere Ionosphere Electrodynamic General Circulation Model (TIEGCM) data.

CCM3-T170 special request

In collaboration with Barry Baker of The Nature Conservancy Fred Clare produced an animation depicting the evolution of total water vapor and precipitable water during a one-year period over Southeast Asia.  This animation was based on a previous animation of Don Middleton using data from the T170 version of the NCAR Community Climate Model with a one hour time step.

Geophysical turbulence program

Clyne continued collaborations with science staff from NCAR’s Geophysical Turbulence Program. Clyne was a contributor to an NSF CMG proposal submitted by GTP and awarded for funding starting in FY04. Clyne led an SCD Opportunity Fund proposal in collaboration with GTP to develop domain specific analysis packages for CFD data. Lastly, Clyne and Mendozoa worked closely with HAO’s Mark Rast on the visual analysis of numerous solar turbulence simulations such as the 3D solar plume simulation depicted below.

Outreach activities  

VETS continued is outreach program with visualization demonstrations and conference presentations at SC2002 and CAS2002, along with many demos provided in the SCD visualization lab to visiting students, researchers, Congressional Staffers, and others. 

Also, the collaboration with the Public Visitor expanded in FY2003 with the PVP staff giving regular demonstrations in the vislab and providing school children with exposure to high-end graphics and scientific visualization techniques. 

There were 24 total Education and Outreach presentations in the vislab in FY2003 including:

10/16 Foothills Academy High School

10/25 Metro State University

10/28 Local 4^th Grade School Group

11/15 Boy Scout Troup

12/4 COMET Class Participants

1/16 Storage Tek visitors

2/7 Colorado College Students

2/21 Louisville Middle School 6^th Graders

3/18 CU Graphics Students

5/28 3^rd Grade Visitor with PVP staff

6/9 SOARS Students

6/16 John’s Restaurant Staff

6/16 Middle/High School Teachers affiliated with R. Johnson conference

6/19 Undergraduate Leadership Workshop Participants

7/15 Upward Bound Visitors

7/21 Climate and Global Change Workshop for Geoscience Education Participants

7/22 Colorado School of Mines Students

7/22 SOARS Students

7/25 Xian University Professors

8/4 Walter Orr Robert’s Family visitors

9/16 Colorado College Undergraduates

9/25 Greeley Mountain View School Group

VETS technical presentations and Access Grid events

Technical presentations

There were 53 total technical presentations including:

Cinquini gave invited presentations on the Community Data Portal and Earth System Grid at the Earth Science Portal Workshop in Daresbury, U.K.

Middleton, Cinquini, Stobbs, and Brown gave a technical presentation to a joint assembly of the UCAR ITC Data Management Working Group (DMWG) and Web Advisory Group (WAG) on the Community Data Portal and Earth System Grid projects.

Clyne gave a talk in the SGI exhibit booth at SC2002 entitled Numerical Simulation in the Earth Sciences

Clyne gave an HAO Science Discovery talk on the Multiresolution Toolkit (MTK) and SCDs VizServer pilot

Clyne presented his paper, The Multiresolution Toolkit: Progressive Access for Regular Gridded Data, at the IASTED Visualization, Image, and Image Processing (VIIP) conference.

Clyne was an invited speaker at an ARSC workshop on advanced display devices.

Middleton was an invited speaker at the DOE Townhall Meeting on Scientific Data Management in response to the Japanese ESS: “Data Management and Access for Climate Simulation Research”.

Middleton was an invited speaker and session chair at ECMWF Workshop, “Towards Terascale Computing” and presented “The Earth System Grid”.

Middleton gave a presentation on data and collaboratory efforts and relationships for the UCAR Cooperative Agreement Panel Meeting.

Middleton gave an invited presentation on The Earth System Grid at the Earth Simulator Center in Yokohama, Japan.

Middleton was an invited speaker at the Computing in Atmospheric Sciences (CAS) conference in Annecy, France, presenting “NCAR Data and Grid Efforts”.

Middleton gave an invited talk on the Earth System Grid at the DOE SciDAC Principle Investigator meeting in Napa, Ca. (March 2003).

Middleton gave a presentation on data analysis and visualization futures and strategies to the CCSM Scientific Steering Committee.

Middleton gave an invited presentation at the Colorado Grid workshop at CSU in Fort Collins, Co.

Middleton gave a presentation on visualization activities at ESRI headquarters in Redlands, Ca.

Middleton gave an invited presentation at the DOE Visualization Frameworks workshop entitled “Visualizing the Multiscale Earth System”.

Middleton gave an invited presentation at the DOE Ultrascale workshop entitled “Ultrascale Analysis and Visualization: A Geosciences Perspective”.

Middleton gave an invited presentation entitled “Visualizing the Earth System” and the SIGGRAPH/Eurographics Meeting: Visualization Meets Hollywood at Snowbird, Utah.

Middleton gave invited presentations at the bi-annual ECMWF workshop on computing, the UCAR Cooperative Agreement Panel Meeting, the DOE SciDAC PI Meeting, the Earth Simulator Center in Japan, DOE Visualization Frameworks meeting, DOE Ultrascale Workshop, the CAS conference, and the IEEE SIGGRAPH/Eurographics Visualization Meets Hollywood Campfire workshop.

Many other technical presentations featuring demonstrations and presentations of NCAR science were presented in the vislab to the following visitors and guests:

10/1 NASA Goddard TV Producer

10/8 UCAR Members

10/9 UCLA Representatives

10/29 RAP Project Sponsors

11/1 DGRP Program Manager

11/18 House Appropriations Committee Staff

12/9 UCAR Auditors

12/12 Watarau Visit

1/28 David Himes (NOAA)

1/31 GIS Initiative

2/12 National Association of Science Writers

2/14 UCAR Finance

2/18 USDA Cartographer

2/20 DKRZ Computer Center Director

3/14 ESRI Visitors

3/24 Audio Tour Company

4/16 NOAA Visitors

5/13 NOAA Arctic Research Meeting Participants

5/23 JOSS

5/30 State Department Visitors

6/20 CIRES Participants

6/23 Visitors From NSF Inspector General's Office

6/23 British Consulate Visitor

6/24 Cliff Jacob’s Intern Demo

6/30 Petersen Visitor Demo

7/1 R.Y. Deshpande (Lester)

7/9 NCAR E&O Staff

7/9 NCAR Contracts Staff

7/18 NCAR HR Staff

8/6 Congressional Staff

8/11 National Association of State Universities and Land-Grant Colleges Visitors

8/20 Allen Cutler Congressional Staff Visitors

8/27 NSF Examiner David Trinkle

9/19 Chinese Meteorological Center Delegation

Access Grid events 

There were 91 total Access Grid Meetings including:

Weekly ESG Meetings

Monthly SciDAC meetings

Ongoing 10 Scientific Workplaces of the Future (SWOF) meetings

11/6 LEAD Planning

11/21 I2 Virtual Briefing

12/6 “Engaging the Audience in Interactive Digital Media Arts Installations”

12/17 NERSC User Services Lecture

1/17 CISM Modelers

1/27 Demo to CU

2/4 – 2/5 CCSM Software Engineering Working Group

2/11 Climate Grid Activities in Europe

2/14 CISM Magnetosphere/Ionosphere Modelers

2/18 Net100

2/21 Demo to Brian Jackson

3/4 – 3/5 CISM Magnetosphere/Ionosphere Modelers

3/6 – 3/7 ACE-Asia Data Intercomparison/Harmony

3/14 ACE CISM Magnetosphere/Ionosphere Modelers

4/18 CISM Magnetosphere/Ionosphere Modelers

4/30 – 5/1 NPACI

5/2 Teragrid Discussion

5/9 BADC

5/12 CCSM Oversight Committee

5/13 Power4 Advisory Panel

5/14 Teragrid Discussion

5/20 U of Mississippi AG Demo

5/29 NUG (Warren Washington Presentation)

6/5 “Art on the Grid”

6/10 Net100

6/11 University of Arkansas Medical Sciences

7/23 Northeastern University

9/11 AG 2.x Town Hall

9/18 AG 2.x Town Hall

9/23 Demo to NCAR’s MMM Division

9/24 Howard University AG Discussion

Publications in press

Middleton, D., Wilhelmson, R., Scheitlin, T., 2003: Weather and Climate Visualization in The Visualization Handbook, editors C. Johnson and C. Hansen, Academic Press, 2003, in press.

Add ESP Workshop, ESG Workshop, metadata workshop