"Offloading MSS files" refers to the process of moving files from the NCAR Mass Storage System (MSS) to a computer system or a medium external to the MSS, for the purpose of removing these files from the MSS. This activity is generally associated with closure of an CISL computing account when a project ends and allocated MSS resources are no longer designated to maintain the data files.
The customer's point-to-point bandwidth and host storage capacity are limiting factors for all methods involving transfer over networks including the Internet. For remote customers at this time, a bandwidth of 5 MB per second is considered excellent, transferring 1 GB in less than five minutes.
NCAR MSS customers may request assistance in evaluating offloading methods and in estimating the time required to transfer a specific list of MSS files to a system external to CISL. The best way to request assistance or information is via CISL Customer Support. When the data amount is several terabytes, offloading may not be practical. In this case, CISL recommends finding a sponsor (an alternate designated project and account) willing to pay fees for the MSS archival. CISL will provide advice on finding a sponsor.
The customer must provide the exact list of files to be offloaded as an itemized list of individual files with byte count. If there are more than 2,000 files, CISL requests that the customer ask CISL to prepare the list.
Here are some common questions and answers about the methods people use to offload files from the MSS.
What methods does CISL recommend to offload MSS files?
Customers can transfer MSS files from the CISL supercomputers via the scp or sftp commands. This method is not recommended since it utilizes supercomputer resources to transfer MSS files. Customers who require this method need to follow the document File transfer to the supercomputers from outside the security perimeter.
A second transfer method is the msrcp command. This method is limited because it requires the customer's local system to reside within the NCAR security perimeter. Additionally, msrcp requires special software installed on the local server by the local system administrator. As of June 2005, CISL web documentation for msrcp installation is under development. Interested customers may request the status of the documentation from CISL.
A third transfer method is MSS FTP. This method also requires the customer to install special software on their local system. Please see the document Using FTP to read/write MSS files for more information.
A fourth method uses the services of the CISL Data Support Section (DSS). In this case, a fully qualified MSS file list is required, and requests are handled in the order received and as time permits. The process includes coordination with DSS staff to arrange the process where MSS files are moved to an FTP-accessible server, typically in segments of less than 200 GB. Repeated segment moves and downloads are required to completely transfer large archives. The customer is responsible to initiate, monitor, and verify that the files are accurately downloaded. Confirmation of successful download is required and implicitly approves the removal of the files from the DSS server. To request this service, please email the list of MSS files to datahelp@ucar.edu.
Finally, customers may request that MSS files be placed on external media on a cost-of-reproduction basis. In this way, MSS data may be offloaded to a select set of media including tape, CDs, DVDs, or RSB2 devices. This set will vary in time as certain media become obsolete. CISL recommends this method when the customer's file transfer rate is too slow to transfer files over the Internet. Please email requests for media information, cost estimates, size limitations, and a delivery time estimate to datahelp@ucar.edu.
What are the file transfer rates for these methods?
Transfer rates depend on the point-to-point hardware and software setup between the user's local system and the NCAR retrieval system, including the MSS, the network, the user's local system, and related CISL systems. On request, CISL will advise the customer in determining an average rate. CISL media preparation and delivery is highly dependent on the current workload. Customers should request the work as far in advance as possible.
Do the transfers involve security firewalls other than the NCAR/CISL security perimeter?
It is important to know the answer at the outset, because firewalls increase the setup complexity for file transfers. Customers are expected to work with their system administrators to determine the transfer setup through firewalls on their side.
Customers may request assistance from CISL for offloading MSS files. CISL will assist with advice on preparing the itemized list of files and individual byte counts (when necessary), with information on the transfer methods, and with advice on file transfer rates. An CISL staff member will be the customer point of contact during the offloading project if the customer requests this.
The customer is expected to contact CISL when there are problems to be resolved or information to be shared. For example, the CISL Allocations Specialist will need to be informed if the customer runs out of GAU resources during the offloading operation. And the MSS system administrators need to know, at the outset, the following information: the name of the requester or person offloading the files, the start of the project, and the estimated end. Upon completion of offloading, the customer's files will be permanently removed from the MSS.
If you have questions about this document, please contact us by any of the methods listed on this web page (please notice the location of office visits, and also our office hours): CISL Customer Support.
Last updated: 05/16/2007
© Copyright 1992-2006. University Corporation for Atmospheric Research (UCAR). All Rights Reserved.
Address of this page: http://www.scd.ucar.edu/docs/mss/offload.html