Computational science research and development

CSS's mission is to help realize the end-to-end scientific simulation environment envisioned by the NCAR Strategic Plan. The mission of CSS is to develop much of the critical software and intellectual infrastructure needed to achieve the plan's ambitious goals. To this end CSS's role is to track computer technology, learn to extract performance from it, pioneer new and efficient numerical methods, create software frameworks to facilitate scientific advancement, particularly interdisciplinary geoscience collaborations, and share the resultant software and findings with the community through open source software, publications, talks, and websites.

CSS has been productive in all of these areas. Research and development activities funded in FY2002 are up and running. The section had nine papers published or accepted in peer-reviewed journals in the last year, and has four more currently submitted for publication. CSS also generated five posters or papers published in conference proceedings. One patent application was initiated. CSS staff initiated two grants in FY2003, and participated in four new proposals, one of which, an internal Opportunity Fund proposal, was partially funded. Two external NSF proposals were unsuccessful and one, an internal Strategic Initiative proposal, is currently outstanding. A critical DOE cooperative agreement research grant (CCPP) was renewed for another year at full funding.

CSS has continued to grow steadily in the past year. Two new staff members have been added: one project scientist, Dr. Amik St. Cyr has been added to support newly funded research in adaptive mesh refinement with non-conforming spectral elements. A Scientist-1 mathematician, Natasha Flyer, has also been added to CSS. Both have already made significant contributions that are noted below.

Applied computer science activities

In 2003, CSS applied computer science efforts have centered on four activities: participating in the Gelato Federation community to advance Linux-Itanium systems; evaluating emerging computational technologies and parallel computing systems; participating in a team to benchmark, evaluate and performance-tune the Community Atmospheric Model (CAM) component of the Community Climate System Model (CCSM); and perform applied research to develop computationally efficient space-filling curve and GMRES algorithms. This latter computer science research is a good example of research activities merging interdisciplinary elements of both applied mathematics and computer science to solve practical problems more efficiently. These activities, particularly the increasingly fruitful partnership of CSS with the University of Colorado department of computer science, represent significant advances in the long-term process of building a native computer science program within SCD.

Gelato membership

The Itanium very long instruction word (VLIW) architecture represents an important departure from the traditional superscalar RISC microprocessor and CISC-like Pentium architectures used in the geosciences departments at most universities today. Since the VLIW relies on the compiler rather than on-chip circuitry to extract parallelism from the instruction stream, developing robust optimizing compilers for Itanium is critical. As Itanium microprocessors become plentiful in the geoscience community, access to reliable compilers, ported modeling applications, and open-source high-performance mathematical libraries optimized for this architecture enable scientific progress on Itanium Linux systems.

In the past year, CSS's role as a member of the Gelato Federation, an organization devoted to the advancement of the Linux-Itanium technical solution, has been to "beta test" the Intel Fortran and C++ compilers on the Intel Itanium and Itanium-2 processors by porting and tuning a variety of applications, such as CAM2 and MM5 to this platform. In addition to the Spectral Toolkit mathematics library development work described below, SCD networking researchers in NETS have ported the Web100 software to the Linux Itanium platform.

Benchmarking and performance modeling activities

CSS has also been extensively involved in evaluating and benchmarking emerging parallel computing technologies, such as new IBM POWER4 systems and the new IBM Federation interconnect; commodity IA-32 Linux clusters and system interconnects; NEC and Cray vector supercomputers; the Itanium-based SGI Altix parallel computer; and the IBM BlueGene/L massively parallel supercomputer. In particular, Dr. Henry Tufo in CSS has led an extensive evaluation of Linux clusters. Dr. Tufo, who is also a computer science professor at the University of Colorado, has acquired a 128-processor Linux system with Pentium-4 processors. In a superb example of a synergistic NCAR-university collaboration, Dr. Tufo has used this platform both as a teaching vehicle for university students and as a benchmarking platform for NCAR applications such as CAM and CCSM. John Dennis, a PhD student at CU and software engineer in CSS, has developed a detailed BlueGene/L performance model of the performance of a space-filling curve version of the HOMME spectral element dynamical core. This model has shown that a 65,536-processor BlueGene/L system could sustain approximately 25 Tflops on a 10 km global model. Rory Kelly of CSS has worked closely with application support staff from Silicon Graphics to port applications to the SGI Altix, an Itanium Linux distributed shared memory system. CSS has also worked with technologists from Clear Speed, Inc, a supercomputer startup in the signal processing market, to evaluate the suitability of their technology for atmospheric modeling.

Tuning the Community Atmospheric Model (CAM)

Running CAM at T85 for IPCC makes CAM the rate-limiting step in the coupled system integration. Since August of 2003, CSS has participated in a "Tiger Team" composed of staff from SCD, CGD, IBM, and the DOE. The goal of the team has been to make a performance tuning push for T85 on the IBM platform. CSS has been instrumental in collecting performance data to determine optimal run configurations and to prioritize candidates for optimization. The optimization effort has resulted in performance gains of approximately 30%.

Space-filling curves for spectral elements

The NCAR spectral element atmospheric model employs a gnomonic projection of a cube onto the surface of the sphere. The six cube faces are each subdivided into an array of quadrilateral spectral elements. When the cubed-sphere is partitioned using the METIS system, both computational load imbalance and communication requirements can lead to sub-optimal performance. John Dennis has investigated using Hilbert, Peano, and nested Hilbert m-Peano space-filling curves as the basis of an inverse space-filling partitioning algorithm. The resulting partitions from Dennis's implementation allow a maximum 14% increase in the sustained floating-point execution rate versus METIS on O(1000) processors, when running a relatively high-resolution climate simulation.

Block GMRES algorithm performance

John Dennis has also collaborated in an effort to develop an alternative to the standard restarted GMRES algorithm for solving a single right-hand side linear system Ax=b based on solving the block linear system AX=B. Additional starting vectors and right-hand sides are chosen to accelerate convergence. Algorithm performance, i.e. time to solution, is improved by using the matrix A in operations on groups of vectors, or "multivectors," thereby reducing the movement of A through memory. The efficient implementation of the method depends on a fast matrix-multivector multiply routine. Numerical results that show that the time to solution of the new method is up to 2.5 times faster than that of restarted GMRES on preconditioned problems.

Applied mathematical research activities

The applied mathematics research in CSS is diverse, but in the past year it has become increasingly focused on problems relevant to NCAR's core geoscience mission. In particular, the development of new numerical cores for atmospheric modeling is a central focus of the section's research. Since this research now extends well beyond spectral elements, to discontinuous Galerkin methods and adaptive mesh refinement, the element-based research core has been renamed from the Spectral Element Atmospheric Model (SEAM) to the High-Order Multiscale Modeling Environment (HOMME). This name both more accurately describes the research thrust of the section and the current capabilities of the dynamical core; it also eliminates name confusion with other applications.

High-Order Multi-Scale Modeling Environment (HOMME)

To avoid naming conflicts with other, pre-existing models, and to recognize the expansion of the scope of work beyond spectral element methods, the Computational Science Section's Spectral Element Atmospheric Model (SEAM) has been renamed the High-Order Multiscale Modeling Environment (HOMME). In 2003, Dr. Steve Thomas, Dr. Rich Loft, and John Dennis in CSS have continued to develop the High-Order Multiscale Modeling Environment (HOMME) spectral element dynamical core. In the past year they performed extensive testing of the spectral element dynamical core in collaboration with Prof. Lorenzo Polvani of Columbia University. A new test problem was developed, based on a baroclinic shear instability located in the northern hemisphere. Balanced initial conditions are derived using both analytic expressions and by computing a meridional integral to machine precision using Gaussian quadrature. The resulting initial boundary value problem is integrated for 12 days with numerical dissipation applied using time splitting. The purpose of the test is to determine if different dynamical cores can obtain numerically converged solutions. We have compared results produced by the spectral element model with the GFDL spectral transform FMS model and found them to be in close agreement. These studies have resulted in a joint paper submitted to Monthly Weather Review.

A semi-implicit formulation of the shallow water equations permits a much larger time step than the explicit model, potentially more than doubling the integration rate, as measured in simulated days per day. The semi-implicit primitive equation solver is based on vertical eigenmode decomposition and an iterative conjugate-gradient elliptic solver. Our success last year with the spectral element "weak" formulation of the semi-implicit shallow water equations on a staggered Gauss (pressure) Gauss-Lobatto (velocity) grid led us to assume that the primitive equations would be amenable to a similar treatment. This proved not to be the case. After discovering that our previous theoretical concerns about spurious pressure wave solutions were unfounded, non-staggered explicit and semi-implicit variants of the shallow water equations were recently implemented.

Currently, work continues on the non-staggered semi-implicit Eulerian formulation of the 3D primitive equations. Further development of a semi-implicit semi-Lagrangian formulation of the spectral element dynamical core will permit a large time step beyond the advective CFL limit. Semi-Lagrangian time stepping was successfully implemented in the shallow water equations with Amik St. Cyr, using the Operator-Integration Factor Splitting (OIFS) scheme of Maday et al (1990), relying on Eulerian sub-cycling of the homogeneous advection equation. We plan to include a conservative discontinuous Galerkin (DG) scheme for moisture transport. Traditional semi-Lagrangian schemes permit a longer time step than Eulerian schemes but can introduce numerical dissipation due to interpolation. In recent years, there has been considerable progress in developing conservative variants for global models. The OIFS scheme is formally equivalent to a semi-Lagrangian scheme, without interpolation.

As noted in the applied computer science section, John Dennis has continued to study and refine the space-filling-curve-based partitioning method that was introduced into HOMME as a result of his research. Space-filling curves allow the mapping of a 2D grid of elements on the cube sphere onto a 1D line of elements. In particular, this past year Mr. Dennis has investigated nesting Hilbert and Peano space-filling curves for element counts along a cube edge that factorize into powers of 2 and 3.

Mesh refinement: Non-conforming spectral elements

In the past year, in an effort funded by a grant from the NSF's Mathematics Directorate, Dr. Amik St-Cyr, John Dennis, and Dr. Steve Thomas of CSS have successfully implemented element-based mesh h-refinement in HOMME using the geometrically non-conforming spectral element method of Fischer, Kruse, and Loth (2002). The method is interpolation based, using trace matching between non-conforming interfaces to satisfy the required inter-element continuity. Locally refined meshes on the sphere have been generated and the results successfully validated using the test cases described in the Williamson et al (1992) paper "A standard test set for numerical approximations to the shallow water equations in spherical geometry," J. Comp. Phys., 102, 211-224. Convergence and performance studies of these simple two-dimensional problems are currently being performed. The immediate goal is to determine whether the same error level obtained in a uniform resolution simulation can be maintained at less cost using geometrically non-conforming elements in combination with semi-implicit, semi-Lagrangian time stepping, when compared to more conventional unrefined Eulerian models.

Although not explicitly tested yet, these researchers expect the 3D hydrostatic primitive equations to validate in the case of non-conforming spectral elements as well. An attractive initial test case for the primitive equations, for both static and dynamic mesh refinement, is the baroclinic instability test problem recently developed by Polvani et al (2003).

Discontinuous Galerkin methods and conservative advection schemes

The Discontinuous Galerkin (DG) method provides a class of high-order accurate conservative algorithms for solving nonlinear hyperbolic systems based on the exchange of fluxes at element interfaces instead of enforcing C0 continuity. By design, DG schemes are nonlinearly stable, highly parallelizable, and have the ability to capture discontinuities of the exact solution without producing spurious oscillations. These mathematical and computational properties make DG a very attractive method for atmospheric numerical modeling. Applying the Discontinuous Galerkin method to such models has been one of the objectives of Dr. Nair's research over the past year.

As a first step, Dr. Nair has developed one-dimensional versions of both the Discontinuous Galerkin (DG) and Spectral Volume (SV) methods. After comparing these, the DG scheme was subsequently extended to 2D Cartesian geometry and monotonic limiter options were added. Next Dr. Nair implemented this DG advection scheme on the sphere, using the same cubed-sphere geometry employed by the HOMME spectral element model. As a final step in FY2003, Dr. Nair developed a complete DG flux form shallow water equation model based on cubed-sphere geometry with an explicit third-order Runge-Kutta time integration scheme. The preliminary test case results for this DG-shallow water model are very encouraging, and a poster entitled "Development of a discontinuous Galerkin atmospheric model on the cubed-sphere" was presented by Dr. Nair at the ADAPT '03 conference, held Oct 10-11, 2003, at RPI in Troy, New York.

Currently, Dr. Nair is working on further developments that will enable the creation of high-order accurate, inherently conservative, scalable integration schemes. In particular, he is developing a semi-implicit time integration scheme for the DG-shallow water equation model and implementing a robust WENO (Weighed Essentially Non-Oscillatory) monotonic scheme. He is also working on a semi-Lagrangian (SL) conservative transport scheme that is simultaneously conservative and capable of taking large time steps.

If Dr. Nair is successful in creating an efficient mass conserving DG scheme, it will be integrated into the full set of governing equations for the Community Atmospheric Model (CAM). In such a model, the cell fluxes across element boundaries will be the same as those computed for the dynamical fields, leading to a more physical representation of global transport processes when compared to traditional Eulerian and semi-Lagrangian approaches. Working with Ram Nair, CSS staff and university collaborators will then combine the new scheme with the OIFS time-stepping algorithms; an approach neatly compatible with adaptive-mesh refinement.

Elliptic solvers for semi-implicit non-hydrostatic NWP models

The collaborative effort between NCAR researchers Dr. Steve Thomas (SCD), and Dr. Piotr Smolarkiewicz (MMM) shifted in 2003 to the development of a spectral preconditioner for the EULAG anelastic global model. Initial results for an isotropic turbulence problem are very encouraging as the new preconditioner drastically reduces the model computation time. An MPI parallel implementation is being developed, and the method will be applied to atmospheric flow problems with realistic topography.

Dr. Thomas has also collaborated with Jim Purser at NCEP to develop a spectral preconditioner for the semi-implicit semi-Lagrangian dynamical core version of the Weather Research Forecast (WRF) model.

Shallow water flows develop singularities on the sphere

In the past year, a series of computational experiments have been conducted using the vector harmonic method to compute the solutions of nonlinear time-dependent shallow water equations on the sphere. The vector harmonic method is very stable without implicit or explicit smoothing either in space or time. The two-thirds truncation rule is not required, and the solutions converge quadratically corresponding to the temporal truncation error. Nevertheless, apparent singularities in the solutions form for certain test cases at finite time for the stationary sphere. No singularities were observed in the limited set of tests performed on the rotating sphere.

Several computational experiments were performed in this study. The initial condition of the first experiment consisted of a Gaussian geopotential dome of fluid on a non-rotating sphere. The dome collapses to produce a wave that travels from the pole and reconstructs at its antipole. The reconstruction of the dome at the antipole is not identical to the initial Gaussian shape because of the nonlinear nature of the flow. This oscillation then continues to between the poles and remains smooth until day 7, when it develops a sharp spicule at the original pole. Sharp corners in the velocity field appear to develop at earlier times. The second experiment is initialized with a Gaussian vortex dome on a steady sphere. The vortex dome does not collapse but rather is sustained by a geopotential low that develops early in the flow. The initial conditions of the third experiment are identical to the first, but occur on a rotating sphere. The resulting flows are considerably different and lack of apparent singularities.

Spherical harmonic dynamical research core (BOB)

During the past year, the spectral dynamical core BOB (Built-on-Beowulf) has matured substantially as a research tool. The build environment has been improved, and support for IBM SP clusters has been added. A website for the model has been created (http://www.scd.ucar.edu/css/dynamical_cores/) with downloadable documentation and tarfiles. Email model support is also provided. Technical documentation for BOB will also be released in the form of an NCAR technical note. Dr. Richard Scott and Lorenzo Polvani at Columbia University have recently used the primitive equations version of the model to study polar vortex dynamics.

Designs for a new version of the spectral element core have been started. The new design will be based on Fortran90, and will share many modules with our spectral element research code. It will leverage the STK library calls, and it represents an overall reduction in the amount of software maintained by CSS to support research activities in numerical methods/dynamical cores.

Coronal Mass Ejections

Dr. B.C. Low of NCAR has proposed a theory for the formation of the solar magnetic "bubbles" named Coronal Mass Ejections (CMEs), based on the forced (gravitation and pressure) and force-free elliptic equation for the solar magnetic stream function. Natasha Flyer, Bengt Fomberg of the University of Colorado, and Steve Thomas applied arc-length continuation and Newton's method to solve this highly nonlinear elliptic problem. The sought-after magnetic bubble solutions for the force-free equation have now been found in the case of high powers of the stream function. These solutions exceed the critical Ally energy, thus providing strong evidence that the numerical solutions are physical. A paper describing the work has been submitted.

Finally, Dr. Natasha Flyer, a recent Scientist-1 hire in CSS, has started additional important CME research projects. In particular she is studying, with Dr. Mei Zhang, solution branches in 2D parameter space for CMEs and the effects of nonlinearity on the limiting values of the magnetic energy and flux in a force-free field.

Initial Boundary Value Problems (IBVPs)

Since starting in CSS three months ago, Dr. Flyer has continued her long-standing research interest in IBVPs by working on the development of numerical methodology to restore the spectral accuracy of the solution of Burger's equation near boundaries at initial times for high Reynolds numbers (>100). This problem has applications both to geophysical turbulence and shocks in supersonic flows. The second equation Dr. Flyer is currently interested in is the Kuramoto-Shivashinsky equation, which has both solitary and shock-like solutions. It is used to model long waves on the interface of two fluids as well as drift waves in plasmas. Regardless of the partial differential equation being addressed, the importance of this topic lies in the fact that boundary and initial data arise from different circumstances causing a singularity to arise in the solution or a derivative of the solution in the corners of the time-space domain. This wreaks havoc on spectral codes and motivates the development of numerical strategies that restore spectral accuracy.

Development activities

Earth System Modeling Framework (ESMF)

The Earth System Modeling Framework (ESMF) is building software infrastructure for climate, weather, and data assimilation applications. Collaborators include NCAR SCD, CGD and MMM, NOAA GFDL, NOAA NCEP, MIT, the University of Michigan, DOE ANL, DOE LANL, and NASA/GSFC GMAO. The project is organized around a series of 11 milestones, the first four of which were submitted during FY2002.

The fifth ESMF milestone, submitted during FY2003, marked the first release of ESMF software and the second ESMF Community Meeting. The second Community Meeting was held at GFDL in Princeton, New Jersey on May 15, 2003. Over 120 scientists, program managers, software developers, vendors, and others involved in Earth system modeling attended. The ESMF team outlined how applications can be structured using ESMF gridded components and couplers and described how model developers can use ESMF data structures and utilities to develop and streamline codes. A point emphasized throughout the meeting was that ESMF can be adopted incrementally, in either a top-down or bottom-up manner. The ESMF team also presented a prototype version of the framework software, ESMF Version 1.0, and demonstrated its use in a coupled fluid flow simulation. ESMF Version 1.0 can be downloaded from the ESMF website, http://www.esmf.ucar.edu/

Since the ESMF 1.0 release, the ESMF project has been issuing monthly internal software releases. The most recent of these included the ability to compute interpolation weights in parallel and perform basic regridding operations.

ESMF recently created an ESMF Partners program to engage groups who are committed to or interested in becoming early adopters of the framework. This currently includes groups at the Goddard Institute for Space Studies, UCLA, and the Center for Ocean-Land-Atmosphere Studies, as well as the NASA GSFC Land Information Systems project. ESMF partners receive notification of all internal releases and interface reviews, and are encouraged to provide feedback.

Spectral Toolkit mathematical library development

The Spectral Toolkit is a research and development effort in CSS to modernize legacy numerical software libraries that originated at NCAR, while integrating recent work in algorithm development and high-performance software implementation along the way. The scope of the current project is spectral transforms (FFT, Legendre, Spherical Harmonic) relevant to atmospheric data analysis and numerical simulation. It is the intent of the project to target microprocessor-based desktop and server systems for maximal performance. The project is implemented in ISO standard C++ using generic object-oriented programming methodologies to ensure the broadest portability and software lifetime currently possible while providing extensible, flexible, and efficient code.

Activities during the past fiscal year include a complete rewrite of the real and complex FFT code, and completion of the final working prototype of the spherical harmonic transform. High-performance 2D and 3D multithreaded real and complex FFTs have been released: software and documentation may be downloaded from the CSS website http://www.scd.ucar.edu/css/software/stk/

These multidimensional FFTs are supported on Windows, Linux, and a variety of proprietary Unix OSes. The new FFT code was a rewrite in C++ of an earlier C version that resulted in a source code reduction of 30%, while also obtaining a 10% increase in performance. The new code runs as high as 40% of peak floating point efficiency, and 25% to 700% faster than other popular open source FFT packages for sequence lengths of interest in atmospheric sciences.

The spherical harmonic transforms are composed of FFTs, transpositions, and Legendre Transforms (LTs). The multiple instance LTs in STK are formulated as the multiplication of a complex matrix of spectral coefficients by a real matrix of Associated Legendre Polynomials (ALPs). This matrix-matrix multiplication has been cache blocked with adjustable blocking factors to provide high performance on a variety of microprocessor architectures, including Pentium and Itanium. To save memory, the ALPs are computed as needed using a recursion relation, rather than storing all O(N^3) entries. This allows large transforms to be calculated on small computer systems: for example, a T341 spherical harmonic implementation of the shallow water equations can run on a laptop computer with as little as 256 MB of main memory.

The new spherical harmonic transform code is highly accurate: it includes new Gaussian quadrature algorithms developed by Dr. Paul Swarztrauber capable of stable generation of thousand-point Gaussian weights sequences at quad-precision accuracy (10^-31), as well as a blocked multiple instance analysis and synthesis that achieves up to 90%+ of peak floating point efficiency on a wide variety of platforms. For example, a multiple instance T682 sustains 95% of peak (5.7 Gflops) on the Itanium2 processor. The entire package, containing multithreaded FFT and Spherical Harmonic transforms, with FORTRAN interface is currently scheduled for final release in December 2003.

These transforms will significantly augment the capabilities of the Spectral Toolkit and bring us closer to the goals of incorporating the essential functionality of SPHEREPACK into STK and constructing a spectral dynamical core based on it.

Patent applications

A second patent application was filed by UCAR on behalf of Dr. Paul Swarztrauber for the "Communication Machine."

FY2003 grants and proposals

NSF, "An Adaptive Mesh, Spectral Element Formulation of the Well-Posed Primitive Equations for Climate and Weather Simulations," $501,006, funded, period of performance, 10/1/02 - 9/30/05.

NASA, "Implementing an efficient supercomputer-based Grid Compute Engine for a high-resolution, high-volume terrestrial carbon cycle model," $1,272,116, period of performance 8/1/03 - 7/31/06.

NCAR Opportunity Fund (internal): "Targeting Vector and Microprocessor Systems: Automatic Code Generation Tools for NCAR High Performance Simulation Components," 10/1/03 - 9/30/05. $199,810. Partially funded, under negotiation.

DOE/SNL, "LDRD: Massively Parallel Scalable Atmosphere Model," $650,000, 10/1/03 - 9/30/06. Parallel NCAR Director's Stragtegic Iniative proposal, Transport and Diffusion Modeling for Emergency Response, 1/01/04 - 1/01/07, $1,375,000. status: Strategic Initiative proposal pending.

NSF, "Algorithms, Applications, and Environments for Emerging Petascale Architectures," $3,671,466, 9/1/03 - 8/31/07, declined.

NSF, "Terrascale Extensions: Mountain Region TeraGrid (MRT)," 10/1/03 - 9/30/06, $3,068,031, declined.

Conference papers and posters

Nair, R.D., S.J. Thomas, and R.D. Loft, "Development of a discontinuous Galerkin atmospheric model on the cubed-sphere," ADAPT '03 conference, RPI, Troy NY.

St.-Cyr, A., and S.J. Thomas, "Non-conforming spectral element atmospheric model," ADAPT '03 conference, RPI, Troy NY.

Dennis, J.M., "Partitioning with Space-Filling Curves on the Cubed-Sphere," conference paper in Proceedings of Workshop on Massively Parallel Processing at IPDPS '03, Nice, France, April 2003.

Dennis, J.M., Tufo, H.M. and Loft, R.D., "Partitioning the Cubed-Sphere for BlueGene/L," poster at the Workshop on BlueGene/L: Applications, Architecture and Software, Reno, Nevada, October 13, 2003.

Loft, R.D., "Supercomputing Challenges for Geoscience Applications," invited paper at the National Research Council/The National Academies, Computer Science and Telecommunications Board's Committee on the Future of Supercomputing Workshop held in Santa Fe, New Mexico on 24-25 September 2003.

Talks

Thomas, S.J., Workshop on current developments in shallow water models on the sphere, Technical University of Munich, Garching, Germany, March 10-14, 2003: Validation experiments with the BOB and SEAM dynamical cores.

Thomas, S.J., Annual SIAM meeting, Montreal June 16-20, 2003: "An initial value problem for testing the dynamical core of atmospheric general circulation models."

Thomas, S.J., AFOSR workshop on time-stepping methods, Brown University, Providence RI, Aug 18-22, 2003: "The NCAR spectral element climate dynamical core: Semi-implicit, semi-Lagrangian formulation." Invited talk.

Dennis, J.M., Workshop on Massively Parallel Processing at IPDPS'03, Nice, France, April 26, 2003: "Partitioning with Space-Filling Curves on the Cubed-Sphere."

Nair, R., 2003: "Efficient conservative semi-Lagrangian schemes over the sphere." SIAM conference in Computational Sciences. 2003, Feb. 10-13, San Diego, CA.

Tufo, H., "Terascale Spectral Element Algorithms and Implementations," presented at the University of New Mexico, Albuquerque, NM, November 7, 2002.

Tufo, H., "Terascale Spectral Element Algorithms and Implementations," presented at CERFACS, Toulouse, France, December 12, 2002.

Tufo, H., "Terascale Spectral Element Algorithms and Implementations," SCI Distinguished Lecture Series, Univ. of Utah, Salt Lake City, Utah, February 21, 2003.

Tufo, H., "Spectral Element Methods for the Shallow Water Equations," SIAM Conference on Computational Science and Engineering (CSE03), San Diego, CA, February 10, 2003.