The MSS is a collection of hardware and software dedicated to providing archival storage services adequate for handling vast quantities of data. The major components are shown in this table:

The basic unit of exchange between the MSS and its clients is the MSS file, which is a named collection of one or more bytes (however, see Chapter 6, Usage Guidelines for cautions about storing small files). The maximum size of an MSS file increases from time to time, and the current maximum size is 12 billion (12,000,000,000) bytes. The MSS does not interpret the contents of MSS files, nor does it provide any data format conversion services. When an MSS file is transferred between the MSS and the user's system, the entire contents is moved in one operation. The MSS is not a file server; it is not possible to request the transfer of only a subset of an MSS file.

MSS files are stored for a limited amount of time, known as the retention period. If an MSS file has not been referenced for a period of days equal to its retention period, it is automatically purged from the MSS. After an MSS file has been purged, it is recoverable for a short period of time. See the section called “Retention Period” for more details.

The MSS supports directories that hold one or more MSS files and/or subdirectories. This structures the name space into a tree. Each authorized MSS user has a top-level directory. Directories are not persistent, they are created as needed and are automatically removed when empty. See the section called “Directory Behavior” for more details.

The DCS interface to the MSS is modeled, where possible, on the equivalent POSIX/UNIX style commands. The DCS suite consists of a single file movement command, several metadata commands, and several request management and status commands. The following commands are suitable for a command-line interface based on the current MSS:

The commands, documented later, are intended to be as consistent as possible with their POSIX counterparts, and the reader is referred to the POSIX standard (IEEE Std 1003.2-1992, also known as POSIX.2), except where prohibited by the special constraints of the MSS (for example msfind, msmv). The MSS is an archive service and all aspects of its operation cannot be hidden from the user, hence the following commands are also necessary:

The following POSIX commands can not be provided with the current MSS, as the MSS lacks the required functionality:

The DCS commands are also callable from C, C++ and Fortran programs. This can be performed via the system(3) C Library routine, by directly managing the fork(2)/exec(2) process or by using the one of the DCS libraries:

This section gives a brief summary of the changes from the current DCS version 3.3 software.

Each MSS file has a name (also known as the pathname) of 1 to 128 characters associated with it. The name is similar to, but does not have exactly the same semantics as, a POSIX (Unix) pathname. The name has the following components:

The following restrictions apply:

Here are some example MSS file names:

When MSS files have either been removed or purged, they become trash MSS files. In this case, their names are prefixed with /trash. The trash prefix does not count towards the name length. See Chapter 5, MSS File Life Cycles for more details.

MSS directories are not persistent. This means there are no empty directories, and they will be automatically created as needed and removed when empty. For example, if an MSS file with the name /SIMPSONS/Bart/secretstuff exists, then the directories /SIMPSONS and /SIMPSONS/Bart also exist. If /SIMPSONS/Bart/secretstuff is removed, and no other MSS files exist with /SIMPSONS/Bart as a path prefix, then the directory /SIMPSONS/Bart will be automatically removed. At that point, if no other MSS files exist with /SIMPSONS as a path prefix, then the directory /SIMPSONS will also be automatically removed.

The MSS provides no low level commands to directly manipulate directories. All of the commands described in this document that accept a directory name as an argument may recursively descend the directory tree (some require the -R option to be specified) to issue the appropriate command on all MSS files contained within that tree. For example, the command:

would cause thousands of individual move operations to take place if dir1 contains thousands of MSS files. For convenience from this point on, we will pretend that directories behave as in the POSIX standard unless specifically noted.

Unless otherwise noted, the DCS commands that accept MSS file pathnames as input operands will accept POSIX-style shell patterns for expansion (also known as shell wildcards, as described briefly below and in POSIX section 3.13).

A pattern consists of normal characters, which match themselves, and metacharacters. The metacharacters are !, *, ?, and [. These characters lose their special meanings if they are quoted by preceeding them with a back slash (\). An asterisk (*) matches any string of characters. A question mark (?) matches any single character. A left bracket ([) introduces a character class. The end of the character class is indicated by a right bracket (]); if the ] is missing then the [ matches the [ character itself, rather than introducing a character class. A character class matches any of the characters between the square brackets. A range of characters may be specified using a minus sign (-). The character class may be complemented by making an exclamation point (!) the first character of the character class. To include a ] in a character class, make it the first character listed (after the !, if any). To include a minus sign, make it the first or last character listed.

Csh-style brace patterns like {pat1,pat2,...} will also be expanded. Csh-style tilde expansion (~user or ~/) is also done.

In the following table, each type of wildcard is marked with a notation about its relative expansion speed. Those wildcards marked as "Slow" will have to read the entire directory contents to determine all matching file names. This could be slow if the directory has more than a few hundred files, or very slow for directories with more than a thousand files. See the section called “Metadata Guidelines” for more wildcard usage guidelines.

Each MSS file has an associated Class of Service (COS) that provides user specified hints or other instructions to the MSS on how to manage the file. The class of service is set by msrcp(1NCAR) when the MSS file is first written, and can be changed by the msclass(1NCAR) command. See the msclass(1NCAR) documentation for details of the supported classes of service.

Each MSS file can have a comment of up to 80 characters associated with it. The comment can contain any printable characters the user desires, and is meant for documentation purposes. When you provide a comment, you must quote it if it contains spaces or shell filename pattern characters. The comment can be set by msrcp(1NCAR) and set or changed by mscomment(1NCAR).

Each MSS file is owned by a single user (indicated by the MSS file's scientist number) and is charged to a project number. In addition, the user name (UCAS account) of the user who writes the MSS file is kept. The user name may not match the scientist number if the MSS file ownership has been changed. MSS file ownership may be changed by sending a request to CISL Customer Support (see CISL Customer Support web site for contact methods).

Files are owned by the user who wrote them. Ownership of a file is not based on the directory in which the file resides. In other words, user JOE can write a file with the pathname /LEE/DATA, but JOE still owns the file. Such files will appear in listings of /LEE, but charges will be made to the project set by the creator, JOE. Both JOE and LEE will receive purge notices (see Chapter 5, MSS File Life Cycles for more information).

The MSS does not implement MSS file access permissions. However, read and write passwords of 1 to 8 characters may be set by msrcp(1NCAR) when the file is created or later set or changed by mspasswd(1NCAR). If a read password is present, it must be supplied to msrcp before it will be copied from the MSS to the user's system. If a write password is present for an existing MSS file, it must be supplied to msrcp before an MSS file will be copied from the user's system to overwrite the existing MSS file.

The read and/or write passwords, if any have been set, must be given to any command that modifies MSS file metadata. No password is required to touch (update the last reference time) an MSS file, nor to recover an MSS file.

Each MSS file has seven reference times associated with it. They are the times of creation, last read, last write, and the last four references. The reference times are updated when a files is read, written or has its metadata changed. These reference times, along with the last read time, can be zero, which indicates the file has never been read nor has had its metadata modified.

Each MSS file also has read and write counters, which indicate how many times the file has been read or written.

Each MSS file has a retention period that specifies how many days it should be kept after the time the file is last referenced (by a read, write or metadata update). The retention period ranges from 1 to 32,767 days, and the default is 32 days. If the MSS file is not referenced within its retention period, it is purged from the MSS (see Chapter 5, MSS File Life Cycles for further details). The retention period may be set by msrcp(1NCAR) when writing the file or changed by msretention(1NCAR) later.

Access to DCS services is restricted to those systems located behind the UCAR network security perimeter. Systems with (UCAR interior) firewalls located between them and the DCS Servers must have the appropriate firewall entries made to permit certain TCP traffic to pass between the DCS Client system and the DCS Servers. For systems that do not meet these requirements, MSS file access via the MSS FTP Server may be an alternative.

DCS is supported for a variety of Unix systems. See the DCS 4.0 Support Policy for more details. See the Installation Instructions for details on installation and configuration of the DCS commands including on how to configure them for use from behind an UCAR internal firewall.

The normal cd command is somewhat unique in that it is directly performed by your shell. The shell sets its own (internal) working directory, which child processes (commands) inherit. Thus, to get similar behavior, the mscd command must also be performed by your shell in some fashion. To accomplish this, the mscd command must be a shell function (for Bourne or ksh) or an alias (csh or tcsh) which calls another command. This auxiliary command, mscdsetup, is used both to initialize the shell and to perform the actual directory change operation.

Initially, the user must cause the mscdsetup(1NCAR) command to be run before any use is made of mscd. This can be done explicitly or implicitly via a shell startup file (note that the Bourne and ksh shells only execute $HOME/.profile at login time, not when subshells are started). When mscdsetup is run, it will initialize the NCAR_MSSPWD environment variable (if not already set) and initialize the mscd alias or function for use.

Because the DCS commands are a mixture of POSIX and non-POSIX, the command line options accepted are a mix between POSIX style (single character) and MSS-specific (multiple character) options. The MSS-specific options can sometimes be abbreviated to the shortest unique prefix. In the man pages for each command, the MSS-specific options will shown in the following form:

Where -pre is the minimum required prefix and the suffix fix may be optionally given.

The DCS Client has some configurable parameters that control the retry of job submission and how long the DCS Client will wait for several phases of the job's processing to complete. These parameters may be tuned, if desired. Note that it is possible for the system administrator for the system upon which the DCS Client runs to have modified the defaults shown below.

These options are set via the --config string command line argument. The value for string is of the form:

where namen specifies the option name and the corresponding value is provided by valuen. The following table describes the option names and values. Time limits are given in seconds, and may be optionally suffixed with m to specify minutes or h to specify hours:

For example, to set the job start time limit to 2 hours, use:

The DCS commands look for a project number to use in this order:

Currently, charging is only done for storage and file transfer. Metadata and DCS request management operations are not charged. This may change in the future, and so all commands accept the -project option.

MSS file read and write passwords are difficult to fit into a POSIX-style interface. However, since the MSS currently implements neither permission nor access control lists (ACLs), we must deal with read and write passwords.

Since many of the commands accept multiple MSS file names, and the individual files could have separate read or write passwords, you may give the password options multiple times on a single command line. The given password option applies to every MSS file from that point on (reading left to right), until changed by another option. In this fashion, a group of MSS files with the same password or different passwords may be manipulated. However, this could quickly become confusing.

The following two options apply to existing MSS files that have a read or write password set currently, or to a new file being created with msrcp:

You may give a null password (i.e. -rpwd '' with no space between the quotes) to indicate subsequent MSS files do not (or should not) have a password. For example, if the MSS file abc has a read password of secret, and the MSS file def has no read password, you could set the comment for both MSS files with either of the following commands:

or:

The following options may be used to set a new read or write password with some of the metadata commands:

You may give OFF as the password (i.e. -newr OFF) to indicate subsequent MSS files should have the current password removed. For example, if the MSS files abc and def have a read password of secret, and you wish to set a new read password of private for abc and remove the password for def, you could use the following command:

Perhaps the worst simple example is moving an existing MSS file file1 with read abc and write def passwords to replace an existing MSS file file2 with read ghi and write jkl passwords and specifying new read mno and write pqr passwords:

The msls command displays simulated permissions for MSS files and directories. For directories, the owner and other permissions are always displayed as rwx, and the group permissions are always displayed as ---. For files, the owner permissions are always displayed as rw-, group permissions are always displayed as ---, and the other permissions indicate whether the MSS file actually has a read and/or write password. The other permissions display an r if the file is readable by everyone (no read password), and display a w if the file is writable by anyone (no write password). For example:

The file named one is read/write for others (it has no read password and no write password). File two is readable by others, but not writable by others (no read password, but it does have a write password). File three is neither readable nor writable by others (it has both read and write passwords). Of course, if you know the read/write password, then you can read/write the file.

While shell patterns (wildcards), msfind and recursive operations allow you to conveniently manipulate multiple MSS files with a single command, they may not be flexible enough for every situation. If a list of the desired MSS files can be generated, you can use the xargs(1) program to drive the appropriate DCS command with the contents of the list. The xargs command breaks up the list of files into manageable chunks so that it does not exceed your shell's command line limit. Hence the desired DCS command may actually be run several times, each with a different portion of the filename list. The xargs command has several options, including the -t option that allows you to see the full command it's executing.

Here are some points to be aware of when using xargs:

The DCS commands will use the following environment variables, if they are set, to provide defaults for various parameters and/or options.

A non-zero exit status code indicates that the job encountered one or more problems while trying to execute the request. Error messages are sent to stderr.

The DCS commands are passed to remote servers for execution. This means that your system must have network connectivity for the entire time your DCS command is running. If, for any reason, when the DCS server starts to process your request, it can not contact the DCS Client on your system or forward progress is delayed for too long, the request will be automatically terminated. The reason this is done is because the MSS is a shared resource, it is not fair to other users to tie up DCS Server resources on commands that may never complete.

The MSS is limited in the number of operations it can perform at any one time. For example, during periods when few requests are active, your metadata requests may process at the rate of 40 MSS files or directories per second. Thus, it may take a long time to process MSS directory trees with many thousands of files, especially during the busy times of the day. You can use the dcsq command to see if your request is active, or is queued. Some MSS enhancements are planned for the near future to improve the performance of metadata commands.

Currently, the purge processing is performed once a week early on Sunday mornings. The following actions are taken by the purge program:

Before an MSS file is moved into the trash by the purge processing, up to two email notifications will be sent to the owner of the file. If the owner of the top-level directory under which the MSS file resides is different from the owner of the MSS file, the top-level directory owner will also receive the email notifications.

Figure 5.1. The life cycle of a usage=normal MSS file.

Figure 5.2. The life cycle of a usage=backup MSS file.

Figure 5.3. The life cycle of a temporary MSS file.

An MSS file is removed when a user runs the msrm command successfully. When this happens, the MSS file becomes a trash MSS file, and the trash expiration date is 30 days in the future. If the MSS file is not recovered from the trash before those 30 days are up, it is permanently deleted.

Figure 5.4. The life cycle of a removed MSS file.

When MSS files are removed or are purged, they are placed into a special area of the MSS called the trash. The name of the MSS file is prefixed with /trash. While in the trash, the only metadata operations that can be performed are msls and msrecover. MSS files may not be read or written while they are in the trash. While MSS files are in the trash, but before they have been permanently deleted, they can be restored with the msrecover command.

You can report problems to CISL Customer Support (see CISL Customer Support web site for contact methods) or by calling 303-497-1278. Using the web site will make it easier to get the information accurately to the MSS administrators.

Please be prepared to provide as much of the following information as possible: