Copyright © 2008 University Corporation for Atmospheric Research
Feb 14, 2008
Table of Contents
This document describes the File Transfer Protocol (FTP) Service of the National Center for Atmospheric Research (NCAR) Mass Storage Service (MSS).
The MSS is a collection of hardware and software dedicated to providing archival storage services adequate for handling vast quantities of data. Documentation describing the MSS fully can be found at NCAR Mass Storage Service User Reference, which describes the MSS DCS software.
The MSS FTP Service provides access to MSS files for those systems that can not run the MSS DCS software. The MSS FTP Service provides only a way to retrieve MSS directory listings and to read and write MSS files. It does not provide MSS file management commands. It implements a large subset of the File Transfer Protocol as described by a number of Internet Engineering Task Force (IETF) standards and requests for comments (RFCs), along with MSS specific extensions.
In this documentation, there will be references to commands sent
on your behalf by your FTP Client to the MSS FTP Service, as defined
by the IETF standards and RFCs.
These commands are either three or four characters long and will be shown
in bold upper case characters, like this: ALLO.
Commands you issue to your FTP Client will be shown in bold lower case
characters, like this: put file. If, for some
reason, you must have your FTP Client issue a command directly to the
MSS FTP Service, you probably can use the FTP Client quote
command. MSS FTP Service commands are not case sensitive, but any MSS related
parameters such as UCAS account name, passwords, pathnames and comments
are case sensitive.
mssftp.ucar.edu.
Table of Contents
This chapter describes the requirements and restrictions for use of the MSS FTP Service.
ALLO command prior to the
STOR command if the file is larger
than 500,000,000 (500 million) bytes.
See the section called “Writing a File to the MSS.” for how to accomplish this.
The following requirements and restrictions are common to all FTP Client programs. See Chapter 7, Compatible TLS-enabled FTP Clients for information about some compatible FTP Client programs.
SITE
commands to the MSS FTP Service if you need to use non-default
MSS specific file attributes. See the section called “MSS Specific File Attributes.”
for a description of how to possibly accomplish this.
TYPE I) mode.
STRU F), which
is usually the default for FTP Clients
MODE S), which
is usually the default for FTP Clients
Access from outside the UCAR security perimeter has additional restrictions. Please read this section carefully.
AUTH TLS or
AUTH SSL command before it sends
the user name (USER) and password
(PASS).
PROT C). FTP Clients that insist
on securing both the control and data connections
will not work.
Table of Contents
This chapter describes how to use the MSS FTP Service. The following examples assume a command line FTP Client similar to that commonly available on Unix systems.
Use your FTP Client to
connect to host mssftp.ucar.edu.
(Normally your FTP Client will connect to port 21 automatically,
and the method to specify a port number to use is FTP Client
specific.)
Enter your UCAS name and password when prompted. At this point you will
be placed into your MSS top level directory (with a pathname of a
slash / character followed by the upper case
version of your UCAS account name). For example:
% ftp mssftp.ucar.edu
Trying 128.117.8.214...
Connected to mssftp.ucar.edu (128.117.8.214).
220 MSS FTP service ready for new user.
Name (mssftp.ucar.edu:auser):
331 User name okay, need UCAS password
Password:
230-This service is available for authorized use only. Use of this service
for unauthorized purposes is a violation of federal law. All activity
will be logged and monitored, and by using this service, you consent to
such logging and monitoring.
NOTE: Put (STOR) commands will limit files written to the MSS to
500,000,000 bytes unless an ALLO command giving the size of the
file to be stored is sent first. Some FTP clients may do this
for you automatically.
230 User logged in, proceed.
Remote system type is UNIX.
Your FTP Client undoubtedly has a way to display help information.
A command line client may accept a help command,
possibly with optional arguments.
Your FTP Client should also provide a way for you to display help information returned from the MSS FTP Service. This help information shows what the MSS FTP Service expects to receive from your FTP Client when you issue commands. It also provides a URL that points to this documentation. For example:
ftp> rhelp 214-The following commands are recognized: ABOR ALLO AUTH CCC CDUP CLNT CWD EPRT EPSV FEAT HELP LIST MODE MDTM MLSD MLST NLST NOOP OPTS PASS PASV PBSZ PORT PROT PWD QUIT REST RETR SITE SIZE STAT STOR STRU SYST TYPE USER Send 'HELP <command>' for command specifics. 214 Documentation available at http://www.cisl.ucar.edu/mss/ftp/
Documentation on a specific MSS FTP Service command can also be obtained. For example:
ftp> rhelp allo 214 Syntax: ALLO <sp> <bytes>
Many systems these days are located behind network firewalls that prevent inbound TCP connections. Newer FTP Clients are cognizant of this situation and automatically try to use the "passive data connection mode", which is usually firewall compatible. However, if you are using an older FTP Client, you may need to tell it to use passive mode. For example:
ftp> passive Passive mode on.
In addition, your local firewall may restrict the ports to which your computer may attempt to establish outbound connections. The MSS FTP Service is configured to permit passive connections from FTP Clients on TCP ports from 49152 to 49183, inclusive. The response from the MSS FTP Service to the PASV command from the FTP Client states which passive port will be used for that specific data transfer request.
Use your FTP Client's specific method to retrieve directory and file listings. Some GUI based FTP Clients may do this automatically after you login or change directories. For example:
ftp> dir 200 PORT command ok. 125 Data connection open; transfer starting. total 99811 drwx---rwx 2 AUSER 01234567 0 Oct 10 2001 NCAR -rw------- 2 AUSER 01234567 5695506 Aug 7 12:51 ftptest -rw----rw- 3 AUSER 01234567 359984 Aug 7 12:55 ftptest2 -rw----rw- 3 AUSER 01234567 215921 Aug 7 13:15 ftptest3 226 ASCII Transfer complete.
You should be able to have your FTP Client change to a different working directory. This allows you to use relative pathnames instead of absolute pathnames. For example:
ftp> cd somedir 200 CWD command ok.
All transfers of MSS files must be done in binary (or image)
mode to prevent corruption of data. You may be able to use the
binary and/or image
command to your FTP Client. For example:
ftp> binary 200 TYPE command ok.
To read a file from the MSS, issue your FTP Client's normal command to retrieve a file from an FTP server. If the file is not in the cache, it will notify you that it is being fetched from the MSS. You can expect a delay while this happens, and the data transfer to your FTP Client will not start until the entire file is in the cache. For example:
ftp> get ftptest2 local: ftptest2 remote: ftptest2 200 PORT command ok. 125-Fetching file from MSS. 125 Data connection open; transfer starting. 226 Binary Transfer complete. 359984 bytes received in 0.00293 secs (1.2e+05 Kbytes/sec)
Sometimes an FTP session is broken before you can completely retrieve a file from the MSS. If your client supports it, you can restart a file retrieval at a specific byte offset and just fetch the missing portion. The commands to do this are FTP Client specific, but one client uses this sequence:
ftp> restart 50000 restarting at 50000. ftp> get ftptest2 local: ftptest2 remote: ftptest2 200 PORT command ok. 350-Restarting at 50000. 350 Send RETR to initiate transfer. 125 Data connection open; transfer starting. 226 Binary Transfer complete. 309984 bytes received in 0.00193 secs (1.6e+05 Kbytes/sec)
Some FTP Clients may be able to do this automatically for you. You may also be able to simulate this manually by writing the missing portion to a second file then manually joining the pieces together. For example:
ftp> quote rest 75000 350-Restarting at 75000. 350 Send RETR to initiate transfer. ftp> get ftptest2 localfile.part2
To write a file to the MSS, issue your FTP Client's normal command used to send a file to an FTP server. The file will be placed into the cache first, then written to the MSS.
ftp> put testfile9 local: testfile9 remote: testfile9 200 PORT command ok. 125 Data connection open; transfer starting. 226 Binary Transfer complete. 5696894 bytes sent in 0.155 secs (3.6e+04 Kbytes/sec)
To efficiently write files to the MSS, the MSS FTP Service restricts
the size of the file that can be written if the size is not provided
ahead of time. See the section called “Displaying the MSS Specific Parameters.” on how to see what the current limit is
for your FTP session. To write files that are larger than this
limit, you (or your FTP Client) must tell the MSS FTP Service how large
the incoming file is. You do this by issuing the ALLO
command before sending the file. Some FTP Clients may do this
for you automatically. If your FTP Client does not, you can do it
manually: For example:
ftp> quote allo 123456789 200 ALLO command ok. ftp> put bigfile
The MSS FTP Service supports setting an MSS file's class of service, comment, project number, retention period and the read and write passwords. All of these parameters will be used when writing files to the MSS. If you do not set them, the normal defaults (as provided by the DCS msrcp command) will apply. For reads of files from the MSS, only the read password will apply. Once set, the values will be remembered and applied to each subsequent read or write as appropriate. See the NCAR Mass Storage Service User Reference, which describes these attributes in detail.
The FTP standard defines a way to pass site specific commands to
an FTP server from a client. This is knows as a SITE
command. They way you issue it is FTP Client specific, but you may be able
to issue a site or quote site command
to your FTP Client.
To set a specific class of service value, issue the SITE CLAS
command. The argument is the same class of service as that provided
to the DCS msclass command. For example:
ftp> site clas reliability=economy 200 CLAS command ok.
To reset the class of service to the default value, issue the
SITE CLAS command with no argument. For example:
ftp> site clas 200 Command ok, class of service defaulted.
To set a comment, issue the SITE CMNT
command. The argument is the MSS file comment. For example:
ftp> site cmnt This is a comment. 200 CMNT command ok.
To reset the comment to an empty value, issue the SITE CMNT
command with no argument. For example:
ftp> site cmnt 200 Command ok, comment turned off.
To set the project number to be used for file transfers,
as well as the number to assign when writing a file to the MSS,
issue the SITE PROJ
command. The argument is the project number. For example:
ftp> site proj 01234567 200 PROJ command ok.
To display the current project number, issue the
SITE PROJ command with no argument. For example:
ftp> site proj 211 Current project is 01234567
To set a retention period, issue the SITE PER
command. The argument is MSS file comment. For example:
ftp> site per 180 200 PER command ok.
To reset the retention period to the default value, issue the
SITE PER command with no argument. For example:
ftp> site per 200 Command ok, retention period defaulted.
To set a read password, issue the SITE RPWD
command. The argument is the read password. For example:
ftp> site rpwd abc123 200 RPWD command ok.
To reset the read password to an empty value, either issue the
SITE RPWD command with no argument
or the exact string OFF (case sensitive).
For example:
ftp> site rpwd 200 Command okay. Read password turned off.
To set a write password, issue the SITE WPWD
command. The argument is the write password. For example:
ftp> site wpwd abc123 200 WPWD command ok.
To reset the write password to an empty value, either issue the
SITE WPWD command with no argument
or the exact string OFF (case sensitive).
For example:
ftp> site wpwd 200 Command okay. Write password turned off.
To display the MSS specific parameters, along with other
parameters related to your FTP session, issue the
STAT command. Your FTP Client may
call it something like rstat or you
may need to issue it with a quote stat
client command. For example:
ftp> rstat 211-FTP session status: User: AUSER @ (::ffff:127.0.0.1@45310) Scientist: 0000 Project: 01234567 Class of service: usage=economy Comment: This is a comment. Retention period: default Read password: is off Write password: is set Session idle timeout: 600 (seconds) Number of transfers: 0 Bytes transferred: 0 Allocation size (ALLO parameter): 0 Restart location (REST parameter): 0 Maximum file size: 12,000,000,000 (bytes) Maximum file size for STOR when ALLO is not used: 500,000,000 (bytes) 211 End of status.
Documentation about the MSS specific SITE
commands can be also displayed. For example:
ftp> site help 214-The following SITE commands are recognized: HELP CLAS CMNT PER PROJ RPWD WPWD Send 'SITE HELP <command>' for command specifics. 214 Documentation available at http://www.cisl.ucar.edu/mss/ftp/ ftp> site help clas 214 Syntax: SITE CLAS [<sp> <class-of-service>]
Table of Contents
This chapter provides some guidelines about how to efficiently use the MSS.
Think about what data you need to store on the MSS. Be selective, don't store data indiscriminately just because the MSS is there. In particular, temporary data (i.e. files with a lifetime of less than one month) should probably never be placed on the MSS.
Store fewer large files instead of numerous small files. Files smaller than 64 KB are very inefficient due to overhead, and a one byte file requires 112,640 bytes on the storage medium due to the blocking and checksum overheads, and you will be charged for all 112,640 bytes. Using the tar or cpio command to containerize existing sets of small files before transferring them to the MSS will also reduce the latency to read or write the small files.
Make use of single copies via Class of Service
(SITE CLAS) when the data doesn't require the
highest level of reliability (see the documentation for the DCS msclass
command for details). Remember that the
default reliability class of service generates dual copies, and that you will
therefore be charged double as well. See
the section called “Setting the Class of Service” for details.
Avoid repetitious reads if it all possible. For data that you will access frequently, cache it elsewhere.
Set a reasonable retention period when the MSS file is first created by using
the SITE PER FTP command.
Try to estimate the useful lifetime of your data.
For example, some data may only be useful while a specific type of computer is
in use. In this case, a 5 year retention period (or the expected life of that
computer) might be best. You can always change the retention period later
if circumstances warrant it. See the section called “Setting the Retention Period”
for details.
The MSS is not a file server, and clients such as middleware must not use it as such. Middleware must use disk cache management to locally buffer file traffic to and from the MSS. Repeated reads of data should be satisfied from the middleware's disk cache as much as possible. The disk cache should be sized appropriately. Do not write temporary data to the MSS.
Data providers wanting to provide web access to NCAR data assets that are archived on the MSS should do everything they can to hide the underlying MSS file and directory pathname structure from end clients. Since many middleware web applications tend to abstract from files, this should not pose any heavy additional burden on the data provider.
Since the computer systems that host middleware servers often also use the MSS to back up their filesystems, data providers must be careful to specifically exclude backing up those cache areas which contain copies of data already archived on the MSS. Failure to do so can (and has) result(ed) in significant redundant backing up of MSS data, and its associated charges.
The creation of more than several hundred files or more than 10 GB via MSS FTP Service in a day can be problematic if the network between your system and UCAR is slow. Users requiring this rate of data storage are encouraged to submit a request to CISL Customer Support (see CISL Customer Support web site for contact methods) to obtain consultation.
User options and guidelines for offloading data are discussed at the Offloading MSS Files web site.
Users wanting to offload more than 10 GB of data from the MSS via MSS FTP Service should submit a request to CISL Customer Support to obtain consultation. Please include in your request a list of the file names (or parent directories) that you will be reading. CISL will return to you a list with the file sizes that is ordered to enhance optimal retrieval from tape media. You can feed the resulting list to the mss_mirror_prep Perl script to generate input scripts for the lftp FTP client program. To obtain the documentation for using mss_mirror_prep, you can use one of the Perl pod documentation utilities similar to the following commands:
pod2man mss_mirror_prep > mss_mirror_prep.man groff -Tascii -man mss_mirror_prep.man
Users are advised that typical transaction rates during busy periods for metadata
operations (retrieving file and directory listings or expanding wildcards) can
be as low as 10 per second. Avoid trying to retrieve listings for directories
with large numbers of entries (such as /DSS), as this
could take several hours before any output will be returned. The MSS Group
is working on resolving this issue, but the projected deployment of the
solution is currently sometime in 2007.
When feasible, generate lists of MSS file names programmatically instead of using wildcard patterns. Capturing the output msls for a directory then using the grep on your local system may be faster than trying to use slow wildcard patterns in a directory containing many files.
Table of Contents
This chapter describes some common problems and how to prevent or correct them.
ftp mssftp.ucar.edu Trying 128.117.8.214... Connected to mssftp.ucar.edu (128.117.8.214). 220 MSS FTP service ready for new user. Name (mssftp.ucar.edu:auser): 534 Security is mandatory. Login failed.
See the section called “Changing the Working Directory.” for a description of how to change the working directory. If you encounter an error, one of the following reasons may be the cause.
If an MSS directory does not have anything under it, it will be removed automatically. This is true for an MSS user's top level directory also. For example, you may see this response:
ftp> cd /BUSER 550-Could not set working directory. 550 No such file or directory
If the top level directory is missing, but refers to a valid MSS user, tell your FTP Client to store the file with an absolute MSS pathname. For example:
ftp> put ftptest2 /BUSER/file local: ftptest2 remote: /BUSER/file
If you want to create a new subdirectory, tell your FTP Client to store the file with the appropriate MSS pathname. For example:
ftp> put ftptest2 newdir/file local: ftptest2 remote: newdir/file
ftp> cd / 550-Could not set working directory. 550 Root directory access denied
ftp> get ftptest2 local: ftptest2 remote: ftptest2 200 PORT command ok. 504 TYPE A not supported for RETR. ftp> bin 200 TYPE command ok. ftp> get ftptest2 local: ftptest2 remote: ftptest2 200 PORT command ok. 125 Data connection open; transfer starting. 226 Binary Transfer complete. 359984 bytes received in 0.00298 secs (1.2e+05 Kbytes/sec)
ftp> put testfile9 local: testfile9 remote: testfile9 200 PORT command ok. 550 REST not supported for STOR. ftp> restart 0 restarting at 0. ftp> put testfile9
ftp> put testfile9 local: testfile9 remote: testfile9 200 PORT command ok. 550 write password mismatch ftp> site wpwd 200 Command okay. Write password turned off. ftp> put testfile9
PROT C).
See Chapter 7, Compatible TLS-enabled FTP Clients for some compatible FTP Clients.
The FTP standard is not very clear on if pathname wild card patterns are to be supported by the FTP server. It strongly implies that it is the responsibility of the FTP Client to perform wild card pattern matching. If your FTP Client supports this, refer to its documentation for help.
However, historically, many FTP servers have attempted to perform wild
card pattern matching where appropriate and when it doesn't conflict
too much with the FTP standard. The MSS FTP Service will attempt
to do this for the LIST (used for directory
listings) and NLST (typically used for multiple
get) commands. The LIST
argument may be a full DCS pattern. The NLST
argument only implements a subset as defined by the POSIX
fnmatch function: the *,
? and [] patterns
(i.e. no csh-style extensions).
The MSS FTP Service does implement the recent MLSD
and MSLT commands, but as per their definitions,
do not perform wild card pattern matching on their arguments.
These commands may be used by FTP Clients to generate the
input list for wild card pattern matching of their own.
If you issue a "get *" command to your FTP Client, it may have passed something different to the MSS FTP Service that did not match anything in your current working directory. Get the files manually or see if your FTP Client has a way to change its behavior.
Table of Contents
This chapter provides some guidelines about how to provide effective problem reports. The more information you can provide initially, the easier it will be to find and fix any problems.
You can report problems to CISL Customer Support (see CISL Customer Support web site for contact methods). Using the web site will make it easier to get the information accurately to the MSS administrators.
Please be prepared to provide as much of the following information as possible:
Your contact information, including your name, phone number and email address.
Your UCAR Central Authentication Services (UCAS) account name and the project number under which you expected your data to be stored.
The host name of the computer on which you were running the FTP Client.
If this computer is not directly managed by CISL, provide the full
domain name. Also please provide the operating system name and revision,
(which can be done by providing the output of the
uname -a command) and the name
and version of the FTP Client you were using.
The dates and times you were accessing the MSS FTP Service.
A problem description. Please include the actual output from the FTP Client if possible.
Table of Contents
This chapter provides some information about FTP client software that supports TLS secured control connections and has been tested and found to be compatible with the MSS FTP Service. The information in this chapter should not be considered as an endorsement for any specific commercial FTP Client package.
CISL does not normally supply precompiled versions of FTP clients due to the varying security policies at remote sites. Please ask your local system administrator for assistance in installing a compatible FTP client. In particular, you are responsible for complying with any licensing and use restrictions. Additional FTP Client software packages that might work are listed at http://www.ford-hutchinson.com/~fh-1-pfh/ftps-ext.html#client.
The lftp program has superior error recovery and scriptability.
| Homepage | http://lftp.yar.ru/ |
| Tested Version | 3.5.14 |
| Build Notes | Enable SSL support if you are external to the UCAR security perimeter. |
| Usage Notes |
Set the following parameters either in your
$HOME/.lftprc file or while running
lftp:
set net:timeout 1800For external users, also set the following parameters: set ftp:ssl-force yes set ssl:verify-certificate no |
| Homepage | http://www.columbia.edu/kermit/ck80.html |
| Tested Version | 8.0.212 Dev.26 |
| Build Notes |
8.0.212 is the first version to support large files (> 2 GB).
For Mac OS X, you may need to add |
| Usage Notes |
set auth tls verify off set ftp authtype tls set ftp data-protection-level clear ftp open mssftp.ucar.edu |
| Homepage | http://www.ford-hutchinson.com/~fh-1-pfh/ftp-tls-20051021.tar.gz |
| Tested Version | 20051021 |
| Build Notes |
Compile with -D_FILE_OFFSET_BITS=64
to enable large file (> 2 GB) support.
|
| Usage Notes |
ftp -z certsok mssftp.ucar.edu WarningNewer versions of ftp-tls have a "progress meter". This must be disabled or the data transfers will fail. |
The lftp program has superior error recovery and scriptability.
| Homepage | http://lftp.yar.ru/ |
| Tested Version | 3.5.14 |
| Build Notes | Enable SSL support if you are external to the UCAR security perimeter. |
| Usage Notes |
Set the following parameters either in your
$HOME/.lftprc file or while running
lftp:
set net:timeout 1800For external users, also set the following parameters: set ftp:ssl-force yes set ssl:verify-certificate no |
| Homepage | http://www.columbia.edu/kermit/ck80.html |
| Tested Version | 8.0.212 Dev.26 |
| Build Notes |
8.0.212 is the first version to support large files (> 2 GB).
For Mac OS X, you may need to add |
| Usage Notes |
set auth tls verify off set ftp authtype tls set ftp data-protection-level clear ftp open mssftp.ucar.edu |
| Homepage | http://www.coreftp.com |
| Tested Version | 1.3c (build 1447.6) |
| Settings Notes | Bring up the Site Manager dialog from the Tools menu. Select the "AUTH TLS" and "OpenSSL" options. Bring up the Advanced Site Settings dialog and select the Connections page. Deselect "Keep Alive". Bring up the Transfers page and select "Set default transfer mode to BINARY". |
| Homepage | http://www.inicom.net/pages/en.ffxp-home.php |
| Tested Version | 3.4.0 (build 1145) |
| Usage Notes | Bring up the Site Manager dialog. On the Options tab deselect "Send anti-idle keep alives", "Send noop during transfer" and "Show hidden files (LIST -al)". On the SSL tab, check the "Auth TLS" button. |
| Homepage | http://www.columbia.edu/kermit/k95.html |
| Tested Version | 2.1.3 |
| Usage Notes |
set auth tls verify off set ftp authtype tls set ftp data-protection-level clear ftp open mssftp.ucar.edu |
| Homepage | http://www.smartftp.com/download/ |
| Tested Version | 2.0.997 |
| Settings Notes | Bring up the Settings dialog from the Tools menu. Select the Connection "Keep Alive" subpage and deselect the "Enable Keep Alive" option. Select the Connection SSL subpage and set "AUTH Mode" to "TLS", "Control Connection Mode" to "Private (Secure)", "File Transfer" to "Clear" and "List Transfer" to "Clear". Select the Transfer ASCII/Binary page and set "Default Transfer Type" to "Binary Transfer Type". |
| Connection Properties Notes | When you create a new "Remote Browser" for the MSS FTP Service, set "Protocol" to "FTP over SSL Explicit". |
Table of Contents
This chapter gives links to references you may find useful.
The CISL Customer Support web site lists several ways you can use to contact CISL Customer Support. You may also call 303-497-1278.
Documentation that fully describes the MSS is available at NCAR Mass Storage Service User Reference.
These are the relevant standards and RFCs used by the MSS FTP Service.