Overview of computing at NCAR -
Transferring files

Last update: 03/28/2008

File transfers to and from remote systems are allowed only while you are logged on to an NCAR supercomputer. File transfers between remote systems and other systems inside the UCAR security perimeter are allowed only when you are logged on to a system inside the UCAR security perimeter. Transfers between systems inside and outside any NCAR firewall must be made with a secure method that uses encryption such as the secure copy command (scp) and secure ftp (sftp).

Using sftp (secure file transfer protocol) at NCAR

sftp is an interactive file transfer program similar to FTP. sftp performs all operations over an encrypted SSH transport. It may also use many features of SSH, such as public key authentication and compression. sftp connects and logs into the specified host, then enters an interactive command mode.

From within sftp, all of the normal ftp commands are present. sftp does not support anonymous logins. A user account is required on the remote end.

For usage information, see the man page for sftp on the computer from which you are issuing the command.

Copying files using secure copy (scp)

The secure copy command (scp) works with secure shell (SSH). You must use scp to transfer files into and out of the UCAR security perimeter. You also use scp to transfer files within the security perimeter.

For usage information, see the man page for scp on the computer from which you are issuing the command.

Using the FTP proxy from an NCAR supercomputer

The following procedure shows how to use FTP when you are logged on to an NCAR supercomputing system: you FTP to the roy-s system and log in using your user_id and the domain name for the remote host. Then if your account on the remote host requires a password, you supply that password.

Interactive use

When you are logged on to an NCAR supercomputer, and you want to FTP to a remote host, type:
   ftp roy-s
The gatekeeper system will respond like this:
   Connected to roy-s.ucar.edu.
   220 abby.ucar.edu FTP proxy (Version V1.3) ready.
   Name (roy-s:user_id):
At the Name prompt, type:
   user_id@remote_host
This connects you to your desired site. If you are prompted for a password, use your password for the remote host. Examples:

Name (roy-s.ucar.edu:user_id): anonymous@ftp.ncep.noaa.gov

Name (roy-s.ucar.edu:user_id): john_doe@cs.umich.edu
Password: ********

You can then perform all your normal FTP operations (get, put, ls, etc.), and the NCAR security filters will not affect your file transfers.

Batch use

It is preferable to store output from your jobs on the Mass Storage System. We do not generally recommend using FTP from within batch jobs because it can hang. It also risks exposing login and password information to other users and is therefore also ill-advised from a security standpoint in production batch jobs. However, if you need to FTP a file to your home computer, you can use the FTP proxy. For example, you would use the following method in your supercomputer shell script:
ftp -n -v roy-s << 'EOF'
    user jsmith@rapunzel.noaa.gov rapunzel_password_here
    cd some_directory

    put ${pgmname}.lst
    put ${pgmname}.out
    quit
'EOF'

Transferring files to and from the Mass Storage System (MSS)

Data moves to and from the MSS using the msrcp command. This command is only available on CISL servers (bluevista, bluefire, frost, the Data Analysis and Visualization cluster, and some others that are not available to users). These are the only systems that can move data directly to and from the MSS.

Note: The deprecated msread and mswrite commands still exist on some servers and in some API libraries. Older codes that use the MSS library may still function with these older API libraries. However, msread and mswrite are scheduled to be removed, so you should use msrcp for new codes and convert old codes to msrcp.

You cannot transfer data directly between your home computer and the MSS. You must use an intermediary machine, preferably the data analysis and visualization cluster.

The following example shows the use of bluevista as an intermediary machine to move data from the MSS to any system (inside or outside the UCAR security perimeter) that is not a CISL server:

Connect to bluevista via roy.ucar.edu using SSH. Use msrcp to read your MSS data to your /ptmp/logon_name directory. (If you do not have a /ptmp/logon_name directory, you are strongly encouraged to make one and use it for your temporary files.)

Use scp to read your data to your computer.

Remove your data from the CISL system.

Notes: SSH is required for you to use scp. You also use the /ptmp directory on CISL systems to store your data.

To move data from any non-CISL server to the MSS:

Connect to the CISL system using SSH (via roy for bluevista or directly for bluefire). Use scp to write your data to the /ptmp directory.

Use msrcp from the CISL system to write your data to the MSS. Remove your data from the CISL system.

For more information about msrcp, see Reading or writing MSS files with msrcp or the msrcp man page on any of the CISL servers.

File transfer to the supercomputers from outside the security perimeter

For information on how to transfer files to bluevista in this regard, please see File transfer to the supercomputers from outside the security perimeter.

For information on how to transfer files to bluefire in this regard, please see the bluefire quickstart guide.

Access for unattended file transfers

To perform unattended file transfers, see Access for unattended file transfers.

Overview of computing at NCAR - Table of contents

If you have questions about this document, please contact us via any of the methods (phone, email, ticket, or in person) described here: CISL Customer Support.
© Copyright 2003-2008. University Corporation for Atmospheric Research (UCAR). All Rights Reserved.

Address of this page: Address of this page: http://www.cisl.ucar.edu/docs/access/transfer.html

Overview of computing at NCAR - Transferring files