Last month, SCD began enforcing a limit on the size of IRJE .msc command files. In the past, some users were submitting command files to update hundreds or thousands of MSS files (with "touch" commands) all at once. There are two problems with this; since each touch request spawns another job and waits for that job to complete, these jobs can tie up an active slot for a very long time. If we receive several of these at once, which is quite common on Mondays after everybody receives their MSS purge notices, the entire system can be brought to a near standstill for other jobs. A second problem is that if anything should happen to the system during the execution of one of these jobs, the ENTIRE JOB has to be started over again.
Thus, we have had to limit the number of files that can be touched with a single job. People who want to touch large numbers of MSS files must split their command file up into several requests. While this may be somewhat inconvenient, it allows the job scheduler to move all users' jobs through the system much more smoothly. It is also easier to fairly schedule 20 requests that touch 50 files each than it is to schedule one giant request to touch 1000 files.
All .msc command files longer than 100 lines (not counting comments) will be rejected and an explanatory message sent to the submitting user.
Splitting is Easy
It is easy to split a file into smaller chunks using the split command. Once you have a file containing the MSS filenames you want acted upon, type:
The IRJE system is currently set to 200 MB because for the vast majority of sites, the Internet link is too slow to reliably transmit files any larger than that, and even if it could, it would take too long. There is limited temporary space available (which is shared by all IRJE and MIGS users), and a 400-million-byte file would take up a significant portion of that space.
It should also be noted that there is still a limit of a little under 200 MB on the size of a file that can be transmitted over MASnet, an internal NCAR network. This would affect the maximum size of a file that could be submitted to the Text and Graphics System (TAGS) or submitted to or returned from a Cray, regardless of whatever limit is set by IRJE.