|
|
|
CSS conducts research in areas such as computational science, numerical analysis, computational fluid dynamics, parallel communication algorithms for massively parallel architectures, and numerical solutions to partial differential equations.
We develop software packages and numerical libraries to make use of our research results and our numerical and computational expertise. We are in the process of developing highly efficient numerical routines for use by the atmospheric and related sciences community. These codes are optimized for use on RISC-based microprocessor systems.
A new area that we are starting to investigate is software frameworks and object-oriented techniques for applications in Earth system simulation.
CSS is involved in technology tracking (performance monitoring and benchmarking studies) of both hardware and software. This work evaluates and ensures the efficient use of future computing resources and is critical in selecting the most appropriate computers for the future production computing needs of NCAR and the university community. On the software side, staff play an active role evaluating programming languages, programming environments, and paradigms.
The section also engages in collaborative research and development projects with groups within NCAR and from other institutions and agencies. These projects benefit the broader NCAR community. Currently, CSS is involved in projects in conjunction with the Climate and Global Dynamics (CGD) Division of NCAR and with support from the Department of Energy's (DOE) Climate Change and Prediction Program (CCPP), to develop a state-of-the-art climate model coupler that will run efficiently and effectively on distributed-memory parallel computers.
Finally, CSS staff are active in the areas of education and outreach. We organize workshops, provide guest lectures at universities, host postdocs and summer visitors, and give seminars and talks at conferences. Cecelia DeLuca and Steve Hammond organized seminars and workshops on software engineering, and on software framework design and use. During FY2000, CSS staff gave 20 scientific and technical presentations.
These are the FY2000 highlights of CSS computational science research.
An efficient spectral transform method for solving the shallow water equations on the sphere
Bill Spotz and Paul Swarztrauber are developing a new, faster, more memory-efficient spectral model for the shallow water equations on the sphere. The model is based on a true double Fourier expansion of variables, meaning the transformation between spectral and physical space can be accomplished with fast Fourier transforms rather than the slower associated Legendre transforms. It is well-known that this model, by itself, is unstable due to non-isotropic representation of waves near the poles. We solve this problem by projecting the prognostic variables onto the space of spherical harmonics at the end of every time step. For the shallow water equations, this reduces the number of associated Legendre transforms per time step from nine to six, and concentrates the transforms into a projection operator that can be further optimized. Work has concentrated on improving the efficiency of this projection (sometimes called a spherical filter in the literature) and implementing it in a double Fourier model.The projection has been improved in two significant ways. First, the memory requirement has been improved by a factor of N (where Nis the number of latitude points) by introducing and using a complementary space of basis functions orthogonal to the traditional associated Legendre functions. Second, the operation count for the projection operator has been cut roughly in half by using an orthogonal complement representation of the projection matrix, which leads to a faster algorithm for half the zonal wave numbers. The projection is still an O(N3) algorithm, but is twice as fast as doing a traditional forward transform followed by a backward transform.
The double Fourier model that makes use of the projection is remarkably like the traditional spectral transform method. It solves the vorticity-divergence form of the shallow water equations by advancing spectral coefficients forward in time. All of its communication on parallel computers is concentrated in nine transposes per time step of quantities between spectral coefficients decomposed by zonal wave number and physical grid quantities decomposed by latitude. Differences between the double Fourier and traditional spectral model include an equally spaced latitude grid with points at the poles; fast Fourier transforms in the meridional direction; simple tridiagonal solvers required for both time-stepping and the solution of elliptic equations required for velocity calculations and semi-implicit time-stepping; and the projection at the end of the time-step. We anticipate a significant savings in the memory used and time spent in Legendre transforms when all the components of the model have been completed, while maintaining the accuracy, stability, and parallel efficiency of the traditional spectral transform model.
References
Paul N. Swarztrauber and William F. Spotz, "Generalized discrete spherical harmonic transforms," J. Comp. Phys. vol. 159, no. 2, 10 April 2000, pp. 213-230.William F. Spotz and Paul N. Swarztrauber, "A Performance Comparison of Associated Legendre Projections," submitted to the J. Comp. Phys.
Work with MMM on time-stepping schemes and solvers
Steve Thomas continues his collaboration with Joe Klemp and Bill Skamarock in MMM to develop fully 3D semi-implicit schemes and associated elliptic solvers for the dynamical core of the Weather Research and Forecast (WRF) mesoscale model (Skamarock et al 1997).A hydrostatic pressure or mass coordinate due to Rene Laprise (1992) has been implemented, and we are developing a novel semi-implicit scheme with Joe Klemp and Bill Skamarock. The Laprise coordinate is now used in the Canadian (GEM) and French (Aladin) models, and it is being considered by other modeling groups around the world such as HIRLAM, DWD, and Australia.
In addition, modelers in the cloud physics group (Piotr Smolarkiewicz, Terry Clark) are investigating efficient elliptic solvers for anelastic and compressible models. We are working on a spectral type preconditioner for the GCR(k) solver developed by Piotr Smolarkiewicz for use in the global EULAG model. Terry Clark is moving to a semi-implicit model for very high resolution fire modeling studies where high temperatures affect compressibility.
References
W. C. Skamarock, P. K. Smolarkiewicz and J. B. Klemp. "Preconditioned conjugate-residual solvers for Helmholtz equations in nonhydrostatic models." Mon. Wea. Rev., vol. 125, 1997, pp. 587-599.R. Laprise. "The Euler equations of motion with hydrostatic pressure as an independent variable." Mon. Wea. Rev., vol. 120, 1992, pp. 197-207.
Spectral-element methods in geophysical fluid dynamics
Steve Thomas and Richard Loft are investigating the use of spectral element techniques for geophysical fluid dynamics problems. Spectral elements combine the accuracy and exponential convergence of conventional spectral methods with the geometric flexibility of finite elements. Additionally, there are several apparent computational advantages to using spectral element methods on RISC microprocessors. In particular, the computations are naturally cache-blocked, and derivatives may be computed using nearest-neighbor communications. The explicit spectral element atmospheric model of Taylor et al (1997) has demonstrated close to linear scaling on a variety of parallel computers including the NCAR IBM SP and Linux clusters. Explicit formulations of PDEs arising in geophysical fluid dynamics are time-step limited by the phase speed of gravity waves. Semi-implicit time integration schemes remove the stability restriction but require the solution of an elliptic BVP. An efficient semi-implicit scheme and solver for the spectral element model have been lacking until now (Loft and Thomas, 2000). By employing a weak formulation of the governing equations, it is possible to obtain a symmetric Helmholtz operator and then apply an efficient preconditioned conjugate gradient solver. This approach accelerates the simulation rate of the semi-implicit relative to the explicit model for practical climate resolutions.These techniques can be applied to geophysical turbulence problems, and Annik Pouquet (GTP senior scientist) plans to collaborate with CSS to apply spectral elements and adaptive mesh refinement. Direct numerical simulations (DNS) of turbulent flows based on the Navier-Stokes equations are the ultimate goal of this work.
References
Taylor, M., J. Tribbia, and M. Iskandarani, 1997a: "The spectral element method for the shallow water equations on the sphere." J. Comp. Phys., 130, 92-108.Loft, R. D., and Thomas, S. J., 2000: "Semi-implicit methods for spectral-element general circulation models," Proceeding of the ECMWF workshop of the use of parallel processing in meteorology. November 2000.
Coronal Mass Ejections (CMEs) with HAO
Steve Thomas and Paul Swarztrauber are collaborating with HAO scientist B. C. Low to solve a highly nonlinear elliptic problem for the sun's magnetic field. Solutions of this problem will in turn provide initial conditions for time-dependent MHD simulations of coronal mass ejections. Our research is part of a much larger effort funded by NASA to simulate and understand space weather phenomena which could possible threaten satellites, the international space station, and national power grids.
Computing the points and weights for Gauss-Legendre quadrature
Paul Swarztrauber is investigating efficient methods for computing the points and weights for Gauss-Legendre quadrature. The eigensystem method has been used for computing the points and weights required by Gauss quadrature. The points are given as the eigenvalues of a symmetric tridiagonal matrix obtained from the three-term recurrence relation that is satisfied by the Legendre polynomials. For an n-point quadrature, the error growth is O(n) for the points and O(n2) in the weights. This is somewhat intuitive because the points cluster quadratically near the end points and therefore coalesce, to machine accuracy, at the same rate. Here we compute the points, which are nearly equally spaced. This, together with extrapolation, provides an accurate initial approximation for Newton's method. The resulting algorithm is robust as well as superior in both accuracy and speed. Error growth in the points is bounded near machine accuracy. The error growth in the weights is O(n), which is determined by the same asymptotic error growth in the trigonometric functions. Relative accuracy is retained near the poles.
References
Paul N. Swarztrauber. "Computing the points and weights for Gauss-Legendre quadrature," submitted to SIAM J. Sci. Comput. October 2000.
Linux PC clusters
Linux clusters are systems constructed entirely of commodity hardware components. They are continuing to prove themselves as stable and cost-effective compute platforms. Numerous government and university labs have substantial Linux clusters installed, and the Forecast System Laboratory (FSL) has selected a Linux cluster as its main compute platform. John Dennis and Rich Loft of CSS have an ongoing project to track the development of Linux clusters. This has included the installation and maintenance of two 16-processor systems at NCAR, as well as assisting Columbia University with their installation of a similar system at the Lamont-Doherty Earth Observatory.The primary NCAR system consists of eight 2-processor 450-MHz Pentium II nodes. Each node contains 512MB of memory, and is connected with a 1-Gbps Myrinet interconnect. The Columbia University system is identical except the nodes consist of 2-processor 800-MHz Pentium IIIs. Preliminary results on the Columbia Linux cluster system include 1,001 MFLOPS for CCM3.2, which represents a price-performance of $55 per MFLOPS. John Dennis, Rich Loft, and Leo Rivier (visiting CSS from Columbia Univ.) have collaborated with Lorenzo Polvani of Columbia University and CGD scientist R. Saravanan to develop a scalable high-resolution spectral shallow water model (BOB). This model, running at T170L2, has achieved 1,312 MFLOPS on 16 processors and represents a price-performance of $42 per MFLOPS.
A comparison of BOB's performance between the NCAR 450-Mhz system using Myrinet, the Columbia 800-Mhz system using Myrinet, and the IBM SP system with WinterHawk 2 nodes can be seen in the figure below. It shows that the Columbia cluster runs at 64% the performance of the IBM SP system on 16 processors. These results are very encouraging, especially considering the cost differential between the two systems.
Performance comparison of Linux clusters
In addition to benchmarking these systems, we have an ongoing effort to develop expertise with the software components, system administration practices, and tools required to deploy Beowulf systems.
In response to the recently released NASA Cooperative Agreement Notice (CAN) titled, "Increasing Interoperability and Performance of Grand Challenge Applications in the Earth and Space Sciences," CSS has led an effort to develop an Earth System Modeling Framework (ESMF). The participants include people from GFDL, NCAR, NASA/GSFC-DAO, NASA/GSFC-NSIPP, ANL, NCEP, LANL, MIT, and U. Mich. The ESMF will enable the composition of high-performance, extensible and interoperable codes for weather prediction, climate simulations, and data assimilation. The multi-institutional and cross-disciplinary interactions among the broad community involved in the proposed work, together with the greater ease in composing robust and exchangeable components that the framework will provide, will result in a new generation of diverse and performance-portable earth science applications.The framework will consist of component coupling services, a set of data constructs that support operations on a variety of data grids and decompositions, and a portable, optimized set of low-level utilities. The data constructs and low-level utilities will be used by the coupling services and may also be used separately to compose scientific applications. Thus our work will promote software reusability as well as interoperability. Work on a set of prototype low-level utilities for the ESMF began during summer 2000. We are currently incorporating these utilities into the next release of CCSM.
From this coordinated community effort, three interdependent proposals are being developed. The first is to develop an ESMF that covers the core framework development effort. The other two focus on framework utilization for modeling and data assimilation applications.
SCD ASR - Table of contents