|Knowledge Center Contents Previous Next Index|
Job Email and Job File Spooling
Mail Notification When a Job Starts
When a batch job completes or exits, LSF by default sends a job report by electronic mail to the submitting user account. The report includes the following information:
- Standard output (stdout) of the job
- Standard error (stderr) of the job
- LSF job information such as CPU, process and memory usage
The output from stdout and stderr are merged together in the order printed, as if the job was run interactively. The default standard input (stdin) file is the null device. The null device on UNIX is /dev/null.
bsub mail options
Sends email to the job submitter when the job is dispatched and begins running. The default destination for email is defined by LSB_MAILTO in lsf.conf.
If you want mail sent to another user, use the -u user_name option to the bsub command. Mail associated with the job will be sent to the named user instead of to the submitting user account.
If you want to separate the job report information from the job output, use the -N option to specify that the job report information should be sent by email.
Output and error file options (-o output_file, -e error_file, -oo output_file, and -eo error_file)
The output file created by the -o and -oo options to the bsub command normally contains job report information as well as the job output. This information includes the submitting user and host, the execution host, the CPU time (user plus system time) used by the job, and the exit status.
If you specify a -o output_file or -oo output_file option and do not specify a
-e error_file or -eo error_file option, the standard output and standard error are merged and stored in output_file. You can also specify the standard input file if the job needs to read input from stdin.
note:The file path can contain up to 4094 characters for UNIX and Linux, or up to 255 characters for Windows, including the directory, file name, and expanded values for %J (job_ID) and %I (index_ID).
The output files specified by the output and error file options are created on the execution host.
See Remote File Access for an example of copying the output file back to the submission host if the job executes on a file system that is not shared between the submission and execution hosts.
Disabling job email
If you do not want job output to be sent by mail, specify stdout and stderr as the files for the output and error options (-o, -oo, -e, and -eo). For example, the following command directs stderr and stdout to file named /tmp/job_out, and no email is sent.bsub -o /tmp/job_out sleep 5
On UNIX, If you want no job output or email at all, specify /dev/null as the output file:bsub -o /dev/null sleep 5
The following example submits myjob to the night queue:
bsub -q night -i job_in -o job_out -e job_err myjob
The job reads its input from file job_in. Standard output is stored in file job_out, and standard error is stored in file job_err.
Size of job email
Some batch jobs can create large amounts of output. To prevent large job output files from interfering with your mail system, you can use the LSB_MAILSIZE_LIMIT parameter in lsf.conf to limit the size of the email containing the job output information.
By default, LSB_MAILSIZE_LIMIT is not enabled-no limit is set on size of batch job output email.
If the size of the job output email exceeds LSB_MAILSIZE_LIMIT, the output is saved to a file under JOB_SPOOL_DIR, or the default job output directory if JOB_SPOOL_DIR is undefined. The email informs users where the job output is located.
If the -o or -oo option of bsub is used, the size of the job output is not checked against LSB_MAILSIZE_LIMIT.
LSB_MAILSIZE environment variable
LSF sets LSB_MAILSIZE to the approximate size in KB of the email containing job output information, allowing a custom mail program to intercept output that is larger than desired. If you use the LSB_MAILPROG parameter to specify the custom mail program that can make use of the LSB_MAILSIZE environment variable, it is not necessary to configure LSB_MAILSIZE_LIMIT.
LSB_MAILSIZE is not recognized by the LSF default mail program. To prevent large job output files from interfering with your mail system, use LSB_MAILSIZE_LIMIT to explicitly set the maximum size in KB of the email containing the job information.
The LSB_MAILSIZE environment variable can take the following values:
- A positive integer: if the output is being sent by email, LSB_MAILSIZE is set to the estimated mail size in KB.
- -1 :if the output fails or cannot be read, LSB_MAILSIZE is set to -1 and the output is sent by email using LSB_MAILPROG if specified in lsf.conf.
- Undefined: If you use the output or error options (-o, -oo, -e, or -eo) of bsub, the output is redirected to an output file. Because the output is not sent by email in this case, LSB_MAILSIZE is not used and LSB_MAILPROG is not called.
If the -N option is used with the output or error options of bsub, LSB_MAILSIZE is not set.
Directory for job output
The output and error options (-o, -oo, -e, and -eo) of the bsub and bmod commands can accept a file name or directory path. LSF creates the standard output and standard error files in this directory. If you specify only a directory path, job output and error files are created with unique names based on the job ID so that you can use a single directory for all job output, rather than having to create separate output directories for each job.
note:The directory path can contain up to 4094 characters for UNIX and Linux, or up to 255 characters for Windows.
Specifying a directory for job output
Make the final character in the path a slash (/) on UNIX, or a double backslash (\\) on Windows. If you omit the trailing slash or backslash characters, LSF treats the specification as a file name.
If the specified directory does not exist, LSF creates it on the execution host when it creates the standard error and standard output files.
By default, the output files have the following format:
The following command creates the directory /usr/share/lsf_out if it does not exist, and creates the standard output file job_ID.out in this directory when the job completes:bsub -o /usr/share/lsf_out/ myjob
The following command creates the directory C:\lsf\work\lsf_err if it does not exist, and creates the standard error file job_ID.err in this directory when the job completes:bsub -e C:\lsf\work\lsf_err\\ myjob
For more information
See the Platform LSF Configuration Reference for information about the LSB_MAILSIZE environment variable and the LSB_MAILTO, LSB_MAILSIZE_LIMIT parameters in lsf.conf, and JOB_SPOOL_DIR in lsb.params.
File Spooling for Job Input, Output, and Command Files
About job file spooling
LSF enables spooling of job input, output, and command files by creating directories and files for buffering input and output for a job. LSF removes these files when the job completes.
You can make use of file spooling when submitting jobs with the -is and -Zs options to bsub. Use similar options in bmod to modify or cancel the spool file specification for the job. Use the file spooling options if you need to modify or remove the original job input or command files before the job completes. Removing or modifying the original input file does not affect the submitted job.
note:The file path for spooling job input, output, and command files can contain up to 4094 characters for UNIX and Linux, or up to 255 characters for Windows, including the directory, file name, and expanded values for %J (job_ID) and %I (index_ID).
File spooling is not supported across MultiClusters.
Specifying job input files
Use the bsub -i input_file and bsub -is input_file commands to get the standard input for the job from the file path name specified by input_file. The path can be an absolute path or a relative path to the current working directory. The input file can be any type of file, though it is typically a shell script text file.
LSF first checks the execution host to see if the input file exists. If the file exists on the execution host, LSF uses this file as the input file for the job.
If the file does not exist on the execution host, LSF attempts to copy the file from the submission host to the execution host. For the file copy to be successful, you must allow remote copy (rcp) access, or you must submit the job from a server host where RES is running. The file is copied from the submission host to a temporary file in the directory specified by the JOB_SPOOL_DIR parameter in lsb.params, or your $HOME/.lsbatch directory on the execution host. LSF removes this file when the job completes.
The -is option of bsub spools the input file to the directory specified by the JOB_SPOOL_DIR parameter in lsb.params, and uses the spooled file as the input file for the job.
Use the bsub -is command if you need to change the original input file before the job completes. Removing or modifying the original input file does not affect the submitted job.
Unless you use -is, you can use the special characters %J and %I in the name of the input file. %J is replaced by the job ID. %I is replaced by the index of the job in the array, if the job is a member of an array, otherwise by 0 (zero). The special characters %J and %I are not valid with the -is option.
Specifying a job command file (bsub -Zs)
Use the bsub -Zs command to spool a job command file to the directory specified by the JOB_SPOOL_DIR parameter in lsb.params. LSF uses the spooled file as the command file for the job.
Use the bmod -Zs command if you need to change the command file after the job has been submitted. Changing the original input file does not affect the submitted job. Use bmod -Zsn to cancel the last spooled command file and use the original spooled file.
The bsub -Zs option is not supported for embedded job commands because LSF is unable to determine the first command to be spooled in an embedded job command.
About the job spooling directory (JOB_SPOOL_DIR)
If JOB_SPOOL_DIR is specified in lsb.params:
- The job input file for bsub -is is spooled to JOB_SPOOL_DIR/lsf_indir. If the lsf_indir directory does not exist, LSF creates it before spooling the file. LSF removes the spooled file when the job completes.
- The job command file for bsub -Zs is spooled to JOB_SPOOL_DIR/lsf_cmddir. If the lsf_cmddir directory does not exist, LSF creates it before spooling the file. LSF removes the spooled file when the job completes.
The JOB_SPOOL_DIR directory should be a shared directory accessible from the master host and the submission host. The directory must be readable and writable by the job submission users.
JOB_SPOOL_DIR can be any valid path up to a maximum length up to 4094 characters on UNIX and Linux or up to 255 characters for Windows.
JOB_SPOOL_DIR must be readable and writable by the job submission user, and it must be shared by the master host and the submission host. If the specified directory is not accessible or does not exist, bsub -is cannot write to the default directory LSB_SHAREDIR/cluster_name/lsf_cmddir and the job will fail.
Except for bsub -is and bsub -Zs, if JOB_SPOOL_DIR is not accessible or does not exist, output is spooled to the default job output directory .lsbatch.
For bsub -is and bsub -Zs, JOB_SPOOL_DIR must be readable and writable by the job submission user. If the specified directory is not accessible or does not exist, bsub -is and bsub -Zs cannot write to the default directory and the job will fail.
If JOB_SPOOL_DIR is not specified in lsb.params:
- The job input file for bsub -is is spooled to LSB_SHAREDIR/cluster_name/lsf_indir. If the lsf_indir directory does not exist, LSF creates it before spooling the file. LSF removes the spooled file when the job completes.
- The job command file for bsub -Zs is spooled to LSB_SHAREDIR/cluster_name/lsf_cmddir. If the lsf_cmddir directory does not exist, LSF creates it before spooling the file. LSF removes the spooled file when the job completes.
If you want to use job file spooling, but do not specify JOB_SPOOL_DIR, the LSB_SHAREDIR/cluster_name directory must be readable and writable by all the job submission users. If your site does not permit this, you must manually create lsf_indir and lsf_cmddir directories under LSB_SHAREDIR/cluster_name that are readable and writable by all job submission users.
Modifying the job input file
Use the -i and -is options of bmod to specify a new job input file. The -in and -isn options cancel the last job input file modification made with either -i or -is.
Modifying the job command file
Use the -Z and -Zs options of bmod to modify the job command file specification. -Z modifies a command submitted without spooling, and Zs modifies a spooled command file. The -Zsn option of bmod cancels the last job command file modification made with -Zs and uses the original spooled command.
For more information
See the Platform LSF Command Reference for more information about the bsub and bmod commands.
See the Platform LSF Configuration Reference for more information about the JOB_SPOOL_DIR parameter in lsb.params, and the LSF_TMPDIR environment variable.
Platform Computing Inc.
|Knowledge Center Contents Previous Next Index|