SCD News > Tips and techniques

Transferring files between SCD computers and other computers

Six ways to ship 'em in and move 'em out

SCD technical consultant
Juli Rew
by Juli Rew
Note: This article is obsolete; for current recommended file transfer methods, please see the section on transferring files in the document, Overview of Computing at NCAR.
Suppose you want to send a file from an SCD computer to a different computer (or vice-versa) either interactively or in a batch job. What are your choices?
We were able to come up with no less than six different methods! Here is a summary of these methods, along with their pros and cons, some examples, a warning, and some final recommendations.

The six methods

cp (copy)

Pros: Easy and fast.
Cons: Works to other computers only for cross-mounted file systems -- these are normally files that begin with /fs.
Example:

cp from.file /fs/cgd/home0/joe/to.file

rcp (remote copy)

Pros: Easy.
Cons: Requires setup of .rhosts files. May fail if host is down. May prevent a checkpointed batch job from restarting. Any I/O in .cshrc can cause problems.
Notes: Prohibited from external hosts to SCD computers as of September 20, 2000. OK if invoked from an SCD computer. For more information, see http://www.scd.ucar.edu/tcg/consweb/Info/Timely/Gotcha_2000_07.html.
Example:

rcp from.file my.other.computer:to.file

scp (secure copy)

Pros: Same as rcp, but with better security.
Notes: Uses ssh for data transfer and uses the same authentication mechanism. Available on most SCD platforms.
Example:

scp from.file my.other.computer:to.file

ftp (File Transfer Protocol)

Pros: Well-known syntax.
Cons: Somewhat tricky in a UNIX script (requires "here" document). Can sometimes cause batch job to hang.
Notes: Must go through the FTP proxy on the NCAR computer gate.ucar.edu, or use passive mode FTP, which makes the "here" script trickier.
Examples:
Interactive use
When you are logged on to an NCAR supercomputer or meeker, and you want to FTP to a remote host, type:
ftp gate.ucar.edu

The gatekeeper system will respond like this:
Connected to gate.ucar.edu. 220 gate1 FTP proxy (Version V2.0) ready. Name (gate.ucar.edu:user_id):

At the Name prompt, type:
user_id@remote_host

This connects you to your desired site. If you are prompted for a password, use your password for the remote host. Examples:

Name (gate.ucar.edu:user_id): anonymous@ftp.ncep.noaa.gov
Name (gate.ucar.edu:user_id): john_doe@cs.umich.edu
Password: ********

You can then perform all your normal FTP operations (get, put, ls, etc.), and the NCAR security filters will not affect your file transfers.
A news file describing this procedure appears on all NCAR supercomputers (type "news ftp | more"). If you have any questions or problems, please contact the SCD Consulting Office by sending email to consult1@ucar.edu or by calling 303-497-1278.
Batch use
It is preferable to store output from your jobs on the Mass Storage System (MSS). We do not generally recommend using FTP from within batch jobs because it can hang. However, if you need to FTP a file to your home computer, you can use the FTP proxy on the gateway computer. For example, you would use the following method in your script:
ftp -n -v << 'EOF' open gate.ucar.edu user jsmith@rapunzel.noaa.gov rapunzel_password_here cd some_directory put ${pgmname}.lst put ${pgmname}.out quit 'EOF'
Note: The "quote site batch" method of job submission is no longer available on SCD production computers. An alternative method using passive mode is as follows:

ftp -n -v << 'EOF' open rapunzel.noaa.gov user jsmith rapunzel_password_here quote pasv cd some_directory put ${pgname}.lst put ${pgname}.out quit 'EOF'

The remote computer's FTP must support passive mode for this to work.

rft (remote file transfer)

Pros: Supported, reliable, robust.
Cons: Complicated syntax. Need Network Queueing Environment (NQE) on both computers.
Example:

rft -host meeker.ucar.edu -nopassword -function \ put from.file to.file

netng (MIGS)/netug (IRJE)/netme (meeker)

Pros: Easy to use. With netng and netug, the file will be automatically forwarded to your home computer if invoked from a job submitted via IRJE or MIGS. (To use the netug and netng commands directly, you will need to specify the "host=" parameter. See the man pages.)
Cons: Can send only to MIGS, IRJE, or meeker. IRJE has a 300-MB file size limit; MIGS limits are system-dependent (see your system administrator).
Example:

netng FLNM=from.file flnm=to.file

Availability

From
Method Crays data-
proc ute black-
forest meeker Non-SCD
computers

cp¹ y y y y y y

rcp y y y y y y
scp - y y y y ?

ftp y y y y y y

rft² y - y - - ?

netng³ y y y - - -

netug⁴ y y y - - -

netme⁵ y - y - - -

¹ Only if file system is cross-mounted (or local).
² Both computers must be running NQE.
³ Only for MIGS users.
⁴ Only for IRJE users.
⁵ Can only send to meeker.ucar.edu.

Warning
None of the six methods described above is 100% foolproof. That is, all are subject to certain conditions that might prevent a successful transfer of files. If you are running interactively, you can probably detect if a problem occurs, and then can try again or take some other action. However, if you are running in a batch job, your job might hang or might skip the transfer. It is important to note that netxx and rft are a lot more reliable than rcp/scp/ftp in batch jobs.
We recommend that you always take the precaution in a batch job of first writing your data to the MSS. Writing to the MSS is very reliable (although it too is not 100% reliable!). This way, if your transfer method fails, you will still be able to later retrieve the MSS copy. This is especially important for critical data or data that was expensive to generate.
For example, a simple:

msrcp my.important.file mss:my.important.file

will write the file to the MSS and keep it for 32 days. You will get a notice via email that the file will be purged at that time unless you take steps to keep the file.
Note: All systems may not have msrcp. Alternatively, you can use:

mswrite my.important.file

Recommendations
The most widely available commands are cp, rcp, and ftp. Because cp only works for a few cross-mounted systems, it may not be feasible for many users. FTP can be too cumbersome. We recommend using scp (or ssh). Rcp, in spite of the initial setup (.rhosts) and potential interference from .cshrc files, also works well, but may be phased out soon in favor of scp.
Remember, for safety in a batch job, be sure to write the file to the MSS first.

For more information
For more information on any of the commands mentioned, please see their man pages. In addition, SCD offers a document, Getting Started with SSH at NCAR, that provides information about using SSH.
Thanks to Tom Parker, who supplied the idea and much information for this article.
Back to contents