[Previous] [Table of contents] [Next]

NCAR's Mass Storage System

The NCAR Mass Storage System (MSS) is a central, large-scale data archive that stores data used and generated by climate models and other programs executed on NCAR's supercomputers and compute servers. At the end of FY1999, NCAR MSS managed more than 6.8 million files containing a total of over 212 terabytes (TB) of stored data, and the net growth rate of data in the MSS was approximately 5 TB per month. On average, 74,000 cartridges are being mounted per month, approximately one half of these by operators and the other half in the StorageTek Powderhorn Automated Cartridge Subsystems (ACS). The StorageTek Powderhorn ACSs use robotics to mount and dismount cartridges. On a daily basis, the MSS handles approximately 15,000 requests resulting in the movement of over 680 GB of data. During FY1999, data transfers to and from the MSS exceeded 240 TB.

While some of the data stored on the NCAR MSS originate from field experiments and observations, the bulk of the data is generated by global climate-simulation models and other earth-science models that run on supercomputers, and SCD faces an increasing demand to archive data from ever-faster supercomputers. Essentially, the faster the supercomputer, the more data there are to be archived. Ever greater demands for archiving data will result from the growing use of coupled atmospheric/oceanic simulation models.

MSS history

The NCAR Mass Storage System has evolved over the last fifteen years. Prior to late 1989, the Mass Storage was comprised strictly of off-line, manual-mount media. In November 1989, the first STK Powderhorn Automated Cartridge System (a.k.a. "silo") was acquired, commencing a new era of mass storage at NCAR. The following figure shows the various technologies that have been utilized to provide for storage of critical datasets throughout NCAR's history.

During FY1999, the NCAR Mass Storage System grew from 5,135,472 files, with a total of 151.57 TB to 6,850,168 files, with a total of 212.04 TB. This was an average net growth rate of 5.04 TB per month during FY1999.

MSS today

MSS access methods

The NCAR MSS provides direct storage-device access via a High-Performance Data Fabric (HPDF). The data fabric consists of host computer High Performance Parallel Interface (HiPPI) channel interfaces, non-blocking HiPPI switches capable of supporting multiple bi-directional 100 MB/sec data transfers, and protocol converters that connect the HiPPI data fabric to the IBM-style device control units. The data fabric provides data paths directly between the MSS storage devices and the client compute servers. To utilize this data fabric, SCD has written a file-transport type of interface to enable users to copy files between their host systems and the MSS. The data fabric can support 14 independent file-transfer operations between the storage devices and the compute servers, with 4 transfers sustaining 3 MB/sec each and 10 transfers sustaining 10 MB/sec each, for an aggregate total of 112 MB/sec.

HiPPI technology continues to be deployed only in a niche market. It has not shown signs of spreading into the commodity marketplace, and as a result the cost of HiPPI technology has remained high and the number of HiPPI vendors is dwindling. The lack of availability and support of HiPPI technology is becoming a critical issue to the continued operation of the MSS. Replacement technologies are on the horizon, but they are not yet widely available nor are they functional enough to immediately replace HiPPI. Promising replacement technologies are Fibre Channel and Network Attached Storage Devices. Fibre-Channel-attached RAID units are available today at extremely attractive costs. Over the next few years, the number and types of available Fibre-Channel-attached devices are expected to grow and include tape storage. Once tape devices can be Fibre Channel attached, SCD intends to evaluate the replacement of our HiPPI fabric with Fibre Channel.

Network Attached Storage Devices (NASD) is another emerging technology that is being closely tracked by SCD. Today a handful of vendors supply Network File System (NFS)-based NASD devices. Some vendors are developing "local-disk"-attached NASD products using Fibre Channel and HiPPI connections. SCD's current strategy is to deploy a Fibre Channel infrastructure and add NASD to it at a later time. The end result will be the decommissioning of our HiPPI fabric and ESCON and BMX storage devices, and the wholesale replacement of those older technologies with new, vendor-supported (and hopefully standards-based) technologies.

MSS storage hierarchy

The NCAR MSS currently uses two levels of storage: online and offline. The most frequently accessed data are kept on the fastest storage media, which is the online storage devices: 180 GB of IBM 3390 Model 3 disks and two StorageTek Powderhorn ACSs. The Powderhorn ACSs use 3490E-based technology as well as StorageTek SD-3 (Redwood) technology. Currently, the NCAR MSS has two Powderhorn ACSs, one using 3490E tapes and the other using SD-3 tapes, providing a total online capacity of approximately 300 TB. Lower in the storage hierarchy is a 3490E offline cartridge tape library holding 166,700 cartridges that can be staged with one of the 16 external IBM 3490E manually mounted cartridge drives. StorageTek SD-3 drives have been added to the offline storage level for providing secondary copies of the Powderhorn-resident SD-3 files.

"Migration" refers to the massive task of transferring tens of terabytes of data from old media to modern media before the equipment that uses the old media becomes obsolete. This task by itself is straightforward; however, this data migration must be handled as a background task while the processing and storage components of the system remain fully dedicated to supplying prompt, 24-hour-per-day service to users. When the migration is complete, the total capacity of the offline archive (assuming no reduction in the offline archive's available floor space in the SCD machine room) will exceed 1 petabyte. A data migration to StorageTek 9840 technology is scheduled for spring 2000. With additional silos and 9840 drives, the data migration can be accomplished in one year.

Expansion of the MSS storage hierarchy is planned over the next five years with the introduction of new tape technologies, new ACSs, and with the integration of a front-end file server having its own HSM to offload active and temporary data. The MSS archive will become a back-end store for the file server accessed only by the front-end HSM. A single global name space will be provided for all data managed by SCD. Evaluation of HSM solutions will continue in FY2000.

MSS import/export capability

Another important capability of the NCAR MSS is the ability to import and export data to and from external portable media. Importing data involves copying data from portable media to the MSS data archive, while exporting data involves copying data from the MSS data archive to portable media. Import/export allow users to bring data to NCAR with them, as well as take data away. Import also allows data from field experiments to be copied to the NCAR MSS archive.

Options to exchange data with smaller satellite storage systems are being investigated. Using this technique, data generated at NCAR could be transferred to remote sites for further analysis. The NCAR SCD storage model would thus be geographically distributed, rather than centrally located and administered.

In addition to 3480 and 3490E cartridge tapes and 9-track round tapes, the NCAR MSS also offers import/export to single and double-density Exabyte cartridge tapes. The deployment of an MSS-IV Import/Export server in FY2000 will provide the ability to support many more device types, such as CD-ROM, DAT, and newer Exabyte media, to name a few.

MSS enhancements and upgrades

In FY1999, SCD made the following improvements to the NCAR MSS:

Y2K compliance: Modifications and tests were made to the MSS to ensure Y2K compliance.

MSCP upgrade: An additional CPU and 256 MB of memory were added to the Mass Storage Control Processor (MSCP). The operating system was also upgraded to OS390 V1.2.

HPDF upgrades: Serial High Performance Parallel Interfaces (HiPPI) deployment continued with the replacement of 4 S-BUS parallel HiPPI interfaces. The remaining S-BUS parallel HiPPI interfaces will be converted to serial HiPPI in FY2000. Two new HiPPI switches were integrated into the HPDF which will replace two older switches that are no longer supported by the vendor.

MSS hosts: The IBM SP HiPPI interface was debugged and integrated into the MSS. Additionally, five new MSS hosts were added, and three were decommissioned.

DCS project: The DCS server was upgraded and is now configured with a back/test server. Also in FY1999, DCS was ported to the Linux platform.

New tape technology: The StorageTek 9840 tape drives were beta tested and integrated into the MSS. The 9840 technology will replace the existing StorageTek SD-3 (redwood), StorageTek 4490, and IBM 3490E drives. A three-month friendly-user 9840 trial period was successfully completed in FY1999. Production deployment of the 9840 drives is scheduled for October 1999. Migration of the existing archive to the 9840 technology is scheduled to begin in spring 2000.

MSS-IV: An additional Data Migration Server was deployed on the SGI Origin2000 platform, bringing the total of Data Migration Servers to three. An EXAByte Import/Export server was deployed and was in friendly user testing mode at the end of FY1999.

MSS growth

NCAR Mass Storage System growth during FY1999 continued at projected rates with the introduction of a 160-node IBM SP into the SCD computing environment. The addition of this new supercomputing power effectively doubled the average number of Gflops being delivered to user applications. Projecting this growth into the future, it is not difficult to realize that new storage paradigms and user education will be required, since without this the growth in just three to five years will be untenable.

The following table compares year-end statistics for FY1996, FY1997, FY1998, FY1999, and projected statistics for year-end FY2005. The FY2005 estimates assume a flat budget for supercomputing, historical data storage trends at NCAR, and Moore's Law growth in computer performance per unit cost. Even with the most optimistic vendor projections for storage densities and costs, these estimates indicate that the NCAR MSS would require between one and two dozen ACSs and the annual MSS budget will exceed that for supercomputers.

MSS growth statistics and expectations

MSS roadmap

A description of the four-year MSS roadmap is available on the web at http://www.scd.ucar.edu/hps/PAPERS/MSS_Roadmap991111/

Key issues addressed over the next four years are managing data growth and integrating new storage technologies to keep pace with the projected growth in computing cycles.

---

[Previous] [Table of contents] [Next]