This Month's GOTCHA

Internet Remote Job Entry Limits and the Mass Storage System

Split Up Command Files to the MSS

by Greg Woods

Last month, SCD began enforcing a limit on the size of IRJE .msc command files. In the past, some users were submitting command files to update hundreds or thousands of MSS files (with "touch" commands) all at once. There are two problems with this; since each touch request spawns another job and waits for that job to complete, these jobs can tie up an active slot for a very long time. If we receive several of these at once, which is quite common on Mondays after everybody receives their MSS purge notices, the entire system can be brought to a near standstill for other jobs. A second problem is that if anything should happen to the system during the execution of one of these jobs, the ENTIRE JOB has to be started over again.

Thus, we have had to limit the number of files that can be touched with a single job. People who want to touch large numbers of MSS files must split their command file up into several requests. While this may be somewhat inconvenient, it allows the job scheduler to move all users' jobs through the system much more smoothly. It is also easier to fairly schedule 20 requests that touch 50 files each than it is to schedule one giant request to touch 1000 files.

All .msc command files longer than 100 lines (not counting comments) will be rejected and an explanatory message sent to the submitting user.

Splitting is Easy

It is easy to split a file into smaller chunks using the split command. Once you have a file containing the MSS filenames you want acted upon, type:

where msslist is the file containing the list of filenames, to split it into files of 90 names each. By default, the files created will be named xaa, xab, xac, etc., or you can give them a new name. See the split man page for more information.

IRJE File Size Limits

You may have noted in the IRJE documentation that the maximum size file that may be sent from IRJE to the MSS is 200 MB. Since the MSS can accept files as large as 400 million bytes, why is this limit imposed?

The IRJE system is currently set to 200 MB because for the vast majority of sites, the Internet link is too slow to reliably transmit files any larger than that, and even if it could, it would take too long. There is limited temporary space available (which is shared by all IRJE and MIGS users), and a 400-million-byte file would take up a significant portion of that space.

It should also be noted that there is still a limit of a little under 200 MB on the size of a file that can be transmitted over MASnet, an internal NCAR network. This would affect the maximum size of a file that could be submitted to the Text and Graphics System (TAGS) or submitted to or returned from a Cray, regardless of whatever limit is set by IRJE.

Previous GOTCHAs!

Maintained by: consult1@ncar.ucar.edu
Comments & suggestions welcomed.