Plan to expand the NCAR Supercomputing Facility (NSC) with Wyoming partners




		This is a visualization of a coronal mass ejection impacting the Earth's magnetosphere. These phenomena can have significant impacts on satellites, astronauts, and systems ranging from GPS to power grids. Significant challenges in this field that will benefit from petascale computing include first-principles modeling of solar convection and its contribution to the 22-year solar cycle, and crucially, modeling the emergence of magnetic flux from the solar convection zone and the conditions that lead to solar flares and coronal mass ejections. (Image courtesy of Michael Wiltberger, NCAR HAO.)


		Hypoxia conditions off the Southern New Jersey Coast are set up by a combination of topography and upwelling factors. This process illustrates the intimate connection between biological, chemical, and physical processes in the ocean. (Image courtesy of Dale Haidvogel, Rutgers University.) The coastal oceans are also challenging to understand, being both complicated and complex not only physically, but also chemically and biologically. An important computing goal for the future will be to simulate regional oceanic ecosystem dynamics within the context of a unified Earth system climate model. Future progress will depend on petascale computing and associated technologies in the area of coastal ocean modeling and prediction.


		(a) This 500,000-cell grid was derived from a fully heterogenous geological model containing 2,388,015 hexahedral cells. It was developed for studying permeability heterogeneity in sedimentary aquifers. (b) A region of the grid reveals details near two passive wells. High-fidelity geological reservoir simulation tools improve our ability to investigate longstanding problems in soil permeability. This model can be applied to studies of enhanced oil recovery techniques and CO₂ injection. Underground CO₂ sequestration based on recaptured utility emissions is increasingly being advocated as a viable approach for mitigating global climate change. This work indicates the cross-disciplinary synergy that can arise within a petascale computing facility dedicated to Earth system science problems. (Image courtesy of Ye Zhang, University of Wyoming.)

In partnership with the University of Wyoming, the State of Wyoming, and the NSF, NCAR proposes to develop an expanded NCAR Supercomputing Center (NSC) in Cheyenne, Wyoming. The shared vision for this project is summarized by the following points:

The primary purpose of the NSC is to enable Earth System science discoveries.
The facility should be dedicated to Earth system science problems.
Because environmental problems know no boundaries, the NSC should serve to broaden participation in this geoscientific enterprise.
The facility should be world class and built to last.
The NSC facility should be efficient and as green to build and operate as practicable.
Time is of the essence: many questions have huge societal impacts and won't wait.

This vision for the NSC project is well aligned with the NSF strategic plan and the NSF vision for cyberinfrastructure. The project is driven by science and is being proposed in direct response to the exploding demand for both capability and capacity HPC resources needed by Earth System science researchers. Whether because of a need for greater model resolution, increased model complexity, better statistics, more predictive power, longer simulation times, or a combination of these factors, geosciences investigators are clamoring for petascale computing, data analysis, and visualization resources and exascale data management capabilities.

Technology trends dictate that such capabilities require the availability of a large-scale computing center capable of handling the multi-megawatt heat loads of future systems. The size and infrastructure requirements for the NSC have been determined with these technology trends in mind, and the conceptual design for the facility will support the supercomputing needs of Earth System science researchers for the next two to three decades.

The NSC is designed to fit in with NSF's larger CI vision and will help foster the creation of a national petascale cyberinfrastructure. The proposed facility is sized to be a peer with other NSF Track-2 facilities and will thus be well positioned to serve as a "geosciences stepping stone" to the NSF's multi-disciplinary, one-petaflop-sustained Track-1 facility.

NSC partner selection background

During FY2006 and early FY2007, NCAR and UCAR personnel undertook an in-depth analysis of all possible options for securing a replacement facility for the aging NCAR ML machine room. This rigorous review process involved discussions with members of the community, various institutional governance and advisory bodies, NSF personnel, and subject matter experts. It was determined that pursuing new facility construction with a partner is the most cost-effective solution for meeting the ongoing high-performance computing needs of the atmospheric and related sciences.

Correspondingly, NCAR and UCAR initiated an effort to identify and explore possible facility construction partners, a process that involved conducting each of the following:

Initial discussions with potential partners to outline the vision of the project and to determine initial interest in pursuing a partnership arrangement
Technical feasibility studies of each potential partner institution to determine if a proposed partnership arrangement would meet the project technical requirements outlined in the facility conceptual design plan
Financial feasibility studies of the offering made by each proposed facility partner that met the identified technical requirements

The initial list of possible facility partners included:

Colorado School of Mines (CSM; Golden, Colorado)
Colorado State University (CSU; Fort Collins, Colorado)
IBM (Boulder, Colorado campus)
Sun Microsystems/Storage Tek (Louisville, Colorado campus)
University of Colorado at Boulder (CU; Boulder, Colorado)

During the initial discussions phase, most of these institutions expressed significant interest in at least hosting a facility. Some also expressed interest in forming full partnerships with NCAR, involving not only intellectual collaborations but also providing financial backing to the supercomputing facility construction effort. Sun Microsystems/Storage Tek declined to participate in further discussions, and IBM was eventually removed from consideration when the company elected not to respond to the technical feasibility questionnaire. While initial discussions were taking place with these five potential partners, a delegation from the State of Wyoming contacted NCAR and UCAR representatives and expressed strong interest in the proposed supercomputing facility construction project. After an initial meeting with NCAR and UCAR representatives to discuss the project, the Wyoming team was added to the list of potential project partners.

Detailed FY2007 accomplishments

During FY2007, the Colorado School of Mines, Colorado State University, the University of Colorado at Boulder, and the University of Wyoming and partners in the State of Wyoming all provided plans for a strategic partnership. All four institutions presented proposals with compelling intellectual and research agendas. After reviewing the technical and financial feasibility responses received from CSM, CSU, CU, and Wyoming, NCAR and UCAR determined that CU and the University of Wyoming were offering facility partnership proposals with the greatest potential benefit to NCAR, UCAR, NSF, and the broader community. Over the course of a roughly two-to-three-month negotiating process, NCAR and UCAR staff reached agreement with both CU and the Wyoming team regarding the financial and technical terms of the facility partnership offers being made. Regular briefings with NSF/ATM management and the UCAR Board of Trustees were conducted during this same time to ensure that all entities remained fully apprised. In January 2007, NCAR formally announced the selection of the University of Wyoming and the State of Wyoming as partners for construction of the NSC. The Wyoming proposal was deemed to provide the best path forward for the community to gain access to the greatest amount of computing at the earliest possible time. The proposed site for the facility is in the Front Range Business Park on the west side of Cheyenne, Wyoming.

Throughout the remainder of FY2007, NCAR and Wyoming personnel held a series of formal and informal meetings to further discuss and develop ideas regarding the partnership plan. In April 2007, a retreat was held involving NCAR and University of Wyoming personnel to engage in dialogue regarding the intellectual components of the partnership. NCAR and Wyoming personnel also met in Cheyenne to begin discussing some of the technical aspects of the NSC construction process. During FY2007, NCAR project personnel worked on developing construction cost estimates and analyzing impacts on the construction budget that may be caused by ongoing volatility in materials and labor costs for commercial construction projects. In addition, the NCAR project team worked with the University of Wyoming to assemble a comprehensive project proposal detailing the scientific and technical drivers for the project, the partner selection process, and a comprehensive project management plan and budget for the NSC construction effort. NCAR personnel are also developing requirements and a draft statement of work (SOW) to be released as part of the RFP for NSC architecture and engineering services.

Impacts

The impacts of this project have broad significance. As envisioned, the NSC will be a key component of provisioning HPC resources for Earth System sciences research well into the future. This large, complex project has involved five years of effort on the part of NCAR personnel to arrive at this point and will require nearly four more years of work before NSC supercomputing resources will be available for community use. NCAR plans to fully integrate the NSC into the NSF TeraGrid, and the project will be carried out within the guidelines of the NSF Strategic Plan and the NSF cyberinfrastructure vision. The challenges presented to researchers to effectively utilize petascale systems are immense, and the availabity of large-scale parallel systems to be provided by the NSC—coupled with access to software engineering and parallel algorithm research expertise—are needed to ensure that investigators can prepare their codes and simulations to run effectively on NSF Track-1 computing resources.

This work is supported by NSF Core funding.