msrcp — Copy files to or from the MSS.
If you are using asynchronous msrcp commands in your batch job, you must ensure that you wait for all of the msrcp commands to complete before your batch job exits. Otherwise the batch queue manager will kill any remaining asynchronous msrcp processes that are running in the background, resulting in failures to write your data to the MSS.
There are several strategies you might use to ensure all of your data gets written to the MSS.
Do not use the -async option, but wait for each msrcp
command to finish before proceeding with the next step in your job.
If you do use the -async option, record the DCS
job id written to the msrcp command's standard output (stdout)
stream, then later pass that job id to the dcswait command to
verify that all of the msrcp commands have completed before
exiting your batch job.
Use a separate "file harvester" script (either as a separate batch
job or interactively after the completion of your batch job) to
archive the contents of your batch job output directories. The
-overwrite and -srcdelete options
can be useful to make sure your MSS destination directories have been
synchronized with your job's output file directories.
Which ever solution you use, it is important to examine the exit status codes of the msrcp command(s) for a success code of zero, or you will need to rerun the appropirate command(s).
The msrcp command is used to transfer files between the local host
and the MSS. msrcp copies the files named in the
source_path list to the destination named in
target. Conceptually, this command is similar to the UNIX
rcp command, however this command can only transfer files between the
local host and the MSS. Indirect, or third party, transfers are not
supported.
Either target, or every argument of the
source_path list must be preceded with the
prefix mss:. This indicates which of
the arguments refer to the MSS and specifies the direction (read or write)
of the request. If none of the command line arguments include this prefix,
msrcp exits with an error message.
Metacharacters in MSS paths are supported, however the entire argument
referring to the MSS path must be escaped to prevent the local host shell from
attempting to expand the metacharacters. See the examples below.
Each source_path argument is a pathname residing either
on the local host, or, if preceded with mss:, the MSS.
If a source_path
argument does not begin with with a / character,
the path is assumed to be relative to the current working directory on
either the local host or the MSS. If the -R option
is present, the source_path arguments
may contain directory and/or file pathnames and target
must be a directory. Otherwise each argument in the source_path
list is expected to be a regular file.
The target argument is the destination pathname.
If target is prefixed with mss:,
then target refers to a MSS pathname.
If the target argument does not begin with a leading
/, it is assumed relative to the current working
directory on either the local host or the MSS. The target
argument can refer to a directory. In this case, a final destination pathname is constructed
using the source_path argument. Since MSS directories are
automatically created as specified in MSS pathnames, it is possible to
specify a non-existent MSS directory as the target.
See Example 35, “Target path construction.” for an examples of how msrcp
constructs the destination path names.
If the command line arguments refer to more than one file, msrcp will attempt to transfer the files in parallel. The amount of parallelism is controlled by the DCS server, and the specific amount of parallelism is beyond the control of the user. Thus, you may reduce wall clock time by specifying more than one file on the msrcp command line. In this case, the files are not guaranteed to arrive in any specific order. If there is an error during a multi-file request, msrcp will complete as many of the file transfers as possible. Any error conditions will be reported to standard error (stderr). Note that some error conditions will result in the request being prematurely terminated.
The msrcp command has the ability to transfer files asynchronously (See
the -async option below). If the asynchronous option is specified,
msrcp submits the request and waits for the status of the submittal.
If the submittal is successful, msrcp then prints the DCS job identifier
to standard out (stdout) and immediately exits. This job identifier may
be then used as an argument to either dcsq(1NCAR) or dcswait(1NCAR).
If the asynchronous request fails,
an email message containing the cause of failure is sent to you.
proj
Supply proj as the project number for
access charging.
Service the request in asynchronous mode. If specified, msrcp will print out the job identifier to standard out (stdout) and immediately exit with a 0 (zero) status. The transfer will occur at some later time. Users are responsible for verifying the transfer succeeded, which can be done using the dcswait command.
cos
Provide cos as the class of service
information for files written to the MSS. See the msclass(1NCAR)
man page for details.
commentSpecify the comment. The comment must be between 1 and 80 characters, and must be appropriately quoted if it contains whitespace or other characters that will be interpreted by the local shell.
nThis option is accepted for compatibility purposes with the previous version of DCS, and is ignored.
Synonym for the -overwrite never option.
condition
Controls the overwrite behavior of msrcp. The values for
condition are one of always,
never or sizemismatch.
The default if this option is not specified is always, which causes
msrcp to always replace the destination file if it exists.
If never is specified, msrcp will never replace the
destination file. If sizemismatch is specified, msrcp
will not replace a file of the same size.
A diagnostic containing the MSS pathname and the
local pathname will be output at command termination for all files
not transferred. If some files are not transferred, a zero exit status code
will be returned if there were no other errors.
Include the parent directory names (if any) of the
source file paths in the constructed destination path under target.
The default, if -parent is not specified, is to only use
the final component of the source path. See Example 35, “Target path construction.”
for details.
n
The retention period in days. This option sets the MSS retention
period for files being created on the MSS. If not specified, the
default is 32. The system silently enforces a maximum limit of
n to 4096.
Do not issue error messages about missing source files.
If no other errors detected, causes the exit status
to indicate success (zero). This option is meant to make
retrying writes to the MSS convenient when using the
-srcdelete.
Recursively copy all files and directories represented by
source_path... to target,
which must be a directory. If target is
on the MSS and it does not exist, it will be used as the name of
a new directory to automatically create.
Note that when copying directory trees in this manner, the MSS pathname
limit of 128 characters is enforced. Should msrcp build a MSS
pathname greater than 128 characters, the transfer will fail.
See Example 35, “Target path construction.” for details.
Delete the source file (on the local system) if the transfer to the MSS is successful or the destination MSS file was not replaced. Can only be used when copying files to the MSS.
password
Supply the read password for the following MSS files. If another
read password was specified prior to this point in this command line,
and the following MSS files do no have a read password, a null
password must be given: -rpwd ''. If the MSS
files do not have a read password, this option must not be used.
password
Supply the write password for the following MSS files. If
another write password was specified prior to this point in this command
line, and the following MSS files do no have a write password, a
null password must be given: -wpwd ''. If the MSS
files do not have a write password, this option must not be used.
source_path
A pathname of a file (or a directory if the -R
option was given) to be copied to the destination.
target
A pathname of a file or a directory. If the -R
option was given, target must be
a directory.
Example 29. Write a file to the MSS.
This example writes the file myfile from the local system
to the MSS with the name /USER/mssfile, sets the write
password to secret and sets the retention period
to 180 days.
msrcp -wpwd secret -period 180 myfile mss:/USER/mssfile
Example 30. Read a single file from the MSS to the local host.
In this example, msrcp constructs a MSS pathname from the current MSS
working directory and datafile and transfers the file to
the current working directory on the local host with a name of
new_data. By default, the MSS current
working directory is your MSS root directory, thus the MSS pathname
might be something like /USER/datafile.
msrcp mss:datafile new_data
Example 31. Copy files selected by shell patterns.
This example uses metacharacters to copy all MSS files in the MSS
directory /USER/data that have a leading character
of either D or E,
followed by output, and have any number
of trailing characters to the current directory:
msrcp 'mss:/USER/data/[DE]output*' .
Note the source argument is enclosed in quotes (to prevent local
shell expansion) and the use of dot (.)
as a reference to the current local directory.
Example 32. Recursive copy.
This example uses recursion to copy all the files in the local
directory subdir as well as descend into any
sub-directories subdir may have and copy those
files as well. Assume that in this particular example, the MSS
target directory, /USER/foo does not already exist
on the MSS.
msrcp -R subdir mss:/USER/foo
All the files (and directories) in the local directory tree beginning at
subdir will be copied to the MSS to the newly created
MSS directory /USER/foo/subdir.
Example 33. Move several file to the MSS.
This example effectively moves the local files with names starting with file
from the local system to the MSS directory /USER/mssdir. Destination
MSS files will not be overwritten if they exist. The source files will be removed if they have been
successfully written to the MSS or if the destination MSS files were already present.
The write password for files that are transferred to the MSS will be set to
secret and the retention period for files that are transferred
to the MSS will be set to 180 days.
msrcp -wpwd secret -period 180 -srcdelete -overwrite never \
file* mss:/USER/mssdir
Example 34. Synchronize a local directory with an MSS directory.
This example reads files from the MSS into a local directory if they are not already present in the local directory.
msrcp -R -overwrite never mss:/USER/mssdir localdir
Note that this will place the files under localdir/mssdir.
To place mssdir in the current directory, use
. as the destination directory name.
Example 35. Target path construction.
This example shows the default behavior that msrcp uses when constructing
target path names. Assume that in this particular example, the MSS target
/USER/dest does not already exist on the MSS, and
that a/b is a local file.
msrcp a/b mss:/USER/dest
The file a/b is written to the MSS as
/USER/dest.
This example shows the default behavior that msrcp uses when constructing
target path names. Assume that in this particular example, the MSS target
/USER/dest is a pre-existing directory, and
that a/b is a local file.
msrcp a/b mss:/USER/dest
The file a/b is written to the MSS as
/USER/dest/b.
This example shows the behavior that msrcp uses when constructing
target path names if the -parent option is specified.
Assume that in this particular example, the MSS target
/USER/dest either is not present or is a
pre-existing directory, and that a/b is a local file.
msrcp -parent a/b mss:/USER/dest
The file a/b will be written to the MSS
as the file /USER/dest/a/b.
This example shows the behavior that a recursive msrcp uses when constructing
target path names if the -parent option is specified.
Assume that in this particular example, the MSS target
directory, /USER/dest either is not present or is a
pre-existing directory, and that a/b is a local directory
tree and c is a local file.
msrcp -parent -R a/b c mss:/USER/dest
All the files (and directories) in the a/b tree
will be written to the MSS under /USER/dest/a/b.
The file c will be copied to the MSS to /USER/dest/c.
When used for reading files from the MSS, and providing there are no errors,
msrcp will always transfer the files, regardless of whether the local file exists
(unless the -overwrite never or -overwrite sizemismatch
option is given). This behavior is
distinctly different from the (non-DCS) msread command.
The msrcp command cannot be used to copy a MSS file to another MSS file.
In other words, specifying both source_path
and target with the MSS prefix
(mss:) is illegal.
When used for reading multiple MSS files, the passwords specified must match all the files in the request, otherwise, msrcp will terminate in error.
When writing files to the MSS using the -srcdelete
option, if any of the source arguments are aliases of a file (either links or symlinks),
only the links named by the source arguments will be deleted. The file itself will
not be deleted unless it was also named as one of the source arguments to be written
to the MSS.
-overwrite never or -overwrite sizemismatch
option can potentially cause differences in the metadata attributes of a group of files on the MSS.
This will happen if a subset of the source files to be written already exist on the MSS with
one set of metadata attributes and a different set are specified by msrcp options.
The attributes that can be affected are the class of service, comment, project number,
retention period and the read and write passwords.
-async option merely instructs the msrcp
command to submit the transfer request and immediately exit after
starting a background msrcp process. Unlike the
-nowait option on the (non-DCS) mswrite
command, no special processing of the request occurs. All MSS
attributes of the request must be correct, and the files must exist,
unmodified, at the time of the actual transfer. The background
msrcp process must not be killed or the transfer will fail.
The msrcp command will print any error messages to stderr. In general, msrcp will attempt to print the affected file name and associated message, for every error encountered. In some cases msrcp will print out both the local file name and the MSS file name. There are numerous possible error messages msrcp can generate, however the more common usage type error messages are described below.
The command line implied a transfer to/from a remote host and the MSS. The msrcp command can only transfer files between the invoking host and the MSS.
Only a single path was specified on the command line.
An attempt to specify the MSS as both a source and a target was detected.
The MSS prefix was not specified in either the source paths or the target path.
A source file or directory did not exist at the time of the invocation.
A source was not a plain file and the -R
option was not specified.
The target points to a plain file and multiple MSS files are implied.
A source argument implied a directory while the MSS target is a plain file.
The transfer implied multiple MSS files and the target directory does not exist.
An attempt was made to send an empty file to the MSS. This file was ignored and msrcp will attempt to transfer the remaining files to the MSS.
See Environment Variables section of the Command Behavior and Envrionment chapter for details.
See Exit Status Codes section of the Command Behavior and Environment chapter for details.