bjobs

displays information about LSF jobs

Synopsis

bjobs [-A] [-a] [-W] [-w | -l] [-X] [-x] 
[-app application_profile_name] [-g job_group_name] 
[-sla service_class_name] [-J job_name] [-Lp ls_project_name] 
[-m host_name | -m host_group | -m cluster_name] 
[-N host_name | -N host_model | -N cpu_factor] 
[-P project_name] [-q queue_name] 
[-u user_name | -u user_group | -u all | -G user_group] 
job_ID | "job_ID[index_list]" ... 
bjobs [-A] [-d] [-p] [-r] [-s] [-W] [-w | -l] [-X] [-x] 
[-app application_profile_name] [-g job_group_name] 
[-sla service_class_name] [-J job_name] [-Lp ls_project_name] 
[-m host_name | -m host_group | -m cluster_name] 
[-N host_name | -N host_model | -N cpu_factor] 
[-P project_name] [-q queue_name] 
[-u user_name | -u user_group | -u all | -G user_group] 
job_ID | "job_ID[index_list]" ... 
bjobs [-w | -l | -aps] [-A] [-a] [-d] [-p] [-s] [-r] [-X] [-x] 
[-m host_name] [-q queue_name] 
[-u user_name | -u user_group | -u all | -G user_group] 
[-g job_group] [-sla service_class] [-P project_name] 
[-N host_spec] [-Lp license_project] [-app application_profile] 
[-J name_spec] [job_ID | "job_ID[index_list]" ...] 
bjobs [-h | -V] 

Description

By default, displays information about your own pending, running and suspended jobs.

bjobs displays output for condensed host groups. These host groups are defined by CONDENSE in the HostGroup section of lsb.hosts. These host groups are displayed as a single entry with the name as defined by GROUP_NAME in the HostGroup section of lsb.hosts. The -l and -X options display uncondensed output.

If you defined LSB_SHORT_HOSTLIST=1 in lsf.conf, parallel jobs running in the same condensed host group are displayed as an abbreviated list.

To display older historical information, use bhist.

Options

-A

Displays summarized information about job arrays. If you specify job arrays with the job array ID, and also specify -A, do not include the index list with the job array ID.

You can use -w to show the full array specification, if necessary.

-a

Displays information about jobs in all states, including finished jobs that finished recently, within an interval specified by CLEAN_PERIOD in lsb.params (the default period is 1 hour).

Use -a with -x option to display all jobs that have triggered a job exception (overrun, underrun, idle).

-aps

Displays absolute priority scheduling (APS) information for pending jobs in a queue with APS_PRIORITY enabled. The APS value is calculated based on the current scheduling cycle, so jobs are not guaranteed to be dispatched in this order.

Pending jobs are ordered by APS value. Jobs with system APS values are listed first, from highest to lowest APS value. Jobs with calculated APS values are listed next ordered from high to low value. Finally, jobs not in an APS queue are listed. Jobs with equal APS values are listed in order of submission time. APS values of jobs not in an APS queue are shown with a dash (-).

If queues are configured with the same priority, bjobs -aps may not show jobs in the correct expected dispatch order. Jobs may be dispatched in the order the queues are configured in lsb.queues. You should avoid configuring queues with the same priority.

-d

Displays information about jobs that finished recently, within an interval specified by CLEAN_PERIOD in lsb.params (the default period is 1 hour).

-l

Long format. Displays detailed information for each job in a multiline format.

The -l option displays the following additional information: project name, job command, current working directory on the submission host, initial checkpoint period, checkpoint directory, migration threshold, pending and suspending reasons, job status, resource usage, resource usage limits information, runtime resource usage information on the execution hosts.

Use bjobs -A -l to display detailed information for job arrays including job array job limit (%job_limit) if set.

If JOB_IDLE is configured in the queue, use bjobs -l to display job idle exception information.

If you submitted your job with the -U option to use advance reservations created with the brsvadd command, bjobs -l shows the reservation ID used by the job.

If LSF_HPC_EXTENSIONS="SHORT_PIDLIST" is specified in lsf.conf, the output from bjobs is shortened to display only the first PID and a count of the process group IDs (PGIDs) and process IDs for the job. Without SHORT_PIDLIST, all of the process IDs (PIDs) for a job are displayed.

If you submitted a job with multiple resource requirement strings using the bsub -R option for the order, same, rusage, and select sections, bjobs -l displays a single, merged resource requirement string for those sections, as if they were submitted using a single -R.

If you submitted a job using the OR (||) expression to specify alternative resources, this option displays the Execution rusage string with which the job runs.

For jobs submitted to an absolute priority scheduling (APS) queue, -l shows the ADMIN factor value and the system APS value if they have been set by the administrator for the job:

-p

Displays pending jobs, together with the pending reasons that caused each job not to be dispatched during the last dispatch turn. The pending reason shows the number of hosts for that reason, or names the hosts if -l is also specified.

With MultiCluster, -l shows the names of hosts in the local cluster.

Each pending reason is associated with one or more hosts and it states the cause why these hosts are not allocated to run the job. In situations where the job requests specific hosts (using bsub -m), users may see reasons for unrelated hosts also being displayed, together with the reasons associated with the requested hosts.

The life cycle of a pending reason ends after the time indicated by PEND_REASON_UPDATE_INTERVAL in lsb.params.

When the job slot limit is reached for a job array (bsub -J "jobArray[indexList]%job_slot_limit") the following message is displayed:

The job array has reached its job slot limit. 

-r

Displays running jobs.

-s

Displays suspended jobs, together with the suspending reason that caused each job to become suspended.

The suspending reason may not remain the same while the job stays suspended. For example, a job may have been suspended due to the paging rate, but after the paging rate dropped another load index could prevent the job from being resumed. The suspending reason is updated according to the load index. The reasons could be as old as the time interval specified by SBD_SLEEP_TIME in lsb.params. So the reasons shown may not reflect the current load situation.

-W

Provides resource usage information for: PROJ_NAME, CPU_USED, MEM, SWAP, PIDS, START_TIME, FINISH_TIME.

-w

Wide format. Displays job information without truncating fields.

-X

Displays uncondensed output for host groups.

-x

Displays unfinished jobs that have triggered a job exception (overrun, underrun, idle). Use with the -l option to show the actual exception status. Use with -a to display all jobs that have triggered a job exception.

-app application_profile_name

Displays information about jobs submitted to the specified application profile. You must specify an existing application profile.

-G user_group

Only displays jobs associated with a user group submitted with bsub -G for the specified user group. The -G option does not display jobs from subgroups within the specified user group.

The -G option cannot be used together with the -u option. You can only specify a user group name. The keyword all is not supported for -G.

-g job_group_name

Displays information about jobs attached to the job group specified by job_group_name. For example:

bjobs -g /risk_group
JOBID   USER    STAT  QUEUE      FROM_HOST   EXEC_HOST   JOB_NAME   SUBMIT_TIME
113     user1   PEND  normal     hostA                   myjob     Jun 17 16:15
111     user2   RUN   normal     hostA       hostA       myjob     Jun 14 15:13
110     user1   RUN   normal     hostB       hostA       myjob     Jun 12 05:03
104     user3   RUN   normal     hostA       hostC       myjob     Jun 11 13:18 

Use -g with -sla to display job groups attached to a service class. Once a job group is attached to a service class, all jobs submitted to that group are subject to the SLA.

bjobs -l with -g displays the full path to the group to which a job is attached. For example:

bjobs -l -g /risk_group

Job <101>, User <user1>, Project <default>, Job Group </risk_group>, Status <RUN>, Queue 
<normal>, Command <myjob>
Tue Jun 17 16:21:49: Submitted from host <hostA>, CWD </home/user1;
Tue Jun 17 16:22:01: Started on <hostA>;
... 

-J job_name

Displays information about the specified jobs or job arrays. Only displays jobs that were submitted by the user running this command.

The job name can be up to 4094 characters long for UNIX and Linux or up to 255 characters for Windows.

-Lp ls_project_name

Displays jobs that belong to the specified LSF License Scheduler project.

-m host_name ... | -m host_group ... | -m cluster_name ...

Only displays jobs dispatched to the specified hosts. To see the available hosts, use bhosts.

If a host group is specified, displays jobs dispatched to all hosts in the group. To determine the available host groups, use bmgroup.

With MultiCluster, displays jobs in the specified cluster. If a remote cluster name is specified, you see the remote job ID, even if the execution host belongs to the local cluster. To determine the available clusters, use bclusters.

-N host_name | -N host_model | -N cpu_factor

Displays the normalized CPU time consumed by the job. Normalizes using the CPU factor specified, or the CPU factor of the host or host model specified.

-P project_name

Only displays jobs that belong to the specified project.

-q queue_name

Only displays jobs in the specified queue.

The command bqueues returns a list of queues configured in the system, and information about the configurations of these queues.

In MultiCluster, you cannot specify remote queues.

-sla service_class_name

Displays jobs belonging to the specified service class.

bjobs also displays information about jobs assigned to a default SLA configured with ENABLE_DEFAULT_EGO_SLA in lsb.params.

Use -sla with -g to display job groups attached to a service class. Once a job group is attached to a service class, all jobs submitted to that group are subject to the SLA.

Use bsla to display the configuration properties of service classes configured in lsb.serviceclasses, the default SLA configured in lsb.params, and dynamic information about the state of each service class.

-u user_name... | -u user_group... | -u all

Only displays jobs that have been submitted by the specified users or user groups. The keyword all specifies all users. To specify a Windows user account, include the domain name in uppercase letters and use a single backslash (DOMAIN_NAME\user_name) in a Windows command line or a double backslash (DOMAIN_NAME\\user_name) in a UNIX command line.

The -u option cannot be used with the -G option.

job_ID | "job_ID[index]"

Displays information about the specified jobs or job arrays.

If you use -A, specify job array IDs without the index list.

-h

Prints command usage to stderr and exits.

-V

Prints LSF release version to stderr and exits.

Output

Pending jobs are displayed in the order in which they are considered for dispatch. Jobs in higher priority queues are displayed before those in lower priority queues. Pending jobs in the same priority queues are displayed in the order in which they were submitted but this order can be changed by using the commands btop or bbot. If more than one job is dispatched to a host, the jobs on that host are listed in the order in which they are considered for scheduling on this host by their queue priorities and dispatch times. Finished jobs are displayed in the order in which they were completed.

Default Display

A listing of jobs is displayed with the following fields:

JOBID

The job ID that LSF assigned to the job.

USER

The user who submitted the job.

STAT

The current status of the job (see JOB STATUS below).

QUEUE

The name of the job queue to which the job belongs. If the queue to which the job belongs has been removed from the configuration, the queue name is displayed as lost_and_found. Use bhist to get the original queue name. Jobs in the lost_and_found queue remain pending until they are switched with the bswitch command into another queue.

In a MultiCluster resource leasing environment, jobs scheduled by the consumer cluster display the remote queue name in the format queue_name@cluster_name. By default, this field truncates at 10 characters, so you might not see the cluster name unless you use -w or -l.

FROM_HOST

The name of the host from which the job was submitted.

With MultiCluster, if the host is in a remote cluster, the cluster name and remote job ID are appended to the host name, in the format host_name@cluster_name:job_ID. By default, this field truncates at 11 characters; you might not see the cluster name and job ID unless you use -w or -l.

EXEC_HOST

The name of one or more hosts on which the job is executing (this field is empty if the job has not been dispatched). If the host on which the job is running has been removed from the configuration, the host name is displayed as lost_and_found. Use bhist to get the original host name.

If the host is part of a condensed host group, the host name is displayed as the name of the condensed host group.

If you configure a host to belong to more than one condensed host groups using wildcards, bjobs can display any of the host groups as execution host name.

JOB_NAME

The job name assigned by the user, or the command string assigned by default at job submission with bsub. If the job name is too long to fit in this field, then only the latter part of the job name is displayed.

The displayed job name or job command can contain up to 4094 characters for UNIX, or up to 255 characters for Windows.

SUBMIT_TIME

The submission time of the job.

-l output

The -l option displays a long format listing with the following additional fields:

Project

The project the job was submitted from.

Application Profile

The application profile the job was submitted to.

Command

The job command.

CWD

The current working directory on the submission host.

Initial checkpoint period

The initial checkpoint period specified at the job level, by bsub -k, or in an application profile with CHKPNT_INITPERIOD.

Checkpoint period

The checkpoint period specified at the job level, by bsub -k, in the queue with CHKPNT, or in an application profile with CHKPNT_PERIOD.

Checkpoint directory

The checkpoint directory specified at the job level, by bsub -k, in the queue with CHKPNT, or in an application profile with CHKPNT_DIR.

Migration threshold

The migration threshold specified at the job level, by bsub -mig.

Post-execute Command

The post-execution command specified at the job-level, by bsub -Ep.

PENDING REASONS

The reason the job is in the PEND or PSUSP state. The names of the hosts associated with each reason are displayed when both -p and -l options are specified.

SUSPENDING REASONS

The reason the job is in the USUSP or SSUSP state.

loadSched

The load scheduling thresholds for the job.

loadStop

The load suspending thresholds for the job.

JOB STATUS

Possible values for the status of a job include:

PEND

The job is pending, that is, it has not yet been started.

PSUSP

The job has been suspended, either by its owner or the LSF administrator, while pending.

RUN

The job is currently running.

USUSP

The job has been suspended, either by its owner or the LSF administrator, while running.

SSUSP

The job has been suspended by LSF. The job has been suspended by LSF due to either of the following two causes:

• The load conditions on the execution host or hosts have exceeded a threshold according to the loadStop vector defined for the host or queue.
• The run window of the job's queue is closed. See bqueues(1), bhosts(1), and lsb.queues(5).

DONE

The job has terminated with status of 0.

EXIT

The job has terminated with a non-zero status - it may have been aborted due to an error in its execution, or killed by its owner or the LSF administrator.

For example, exit code 131 means that the job exceeded a configured resource usage limit and LSF killed the job.

UNKWN

• mbatchd has lost contact with the sbatchd on the host on which the job runs.

WAIT

For jobs submitted to a chunk job queue, members of a chunk job that are waiting to run.

ZOMBI

A job becomes ZOMBI if:

• A non-rerunnable job is killed by bkill while the sbatchd on the execution host is unreachable and the job is shown as UNKWN.
• The host on which a rerunnable job is running is unavailable and the job has been requeued by LSF with a new job ID, as if the job were submitted as a new job.
• After the execution host becomes available, LSF tries to kill the ZOMBI job. Upon successful termination of the ZOMBI job, the job's status is changed to EXIT.
• With MultiCluster, when a job running on a remote execution cluster becomes a ZOMBI job, the execution cluster treats the job the same way as local ZOMBI jobs. In addition, it notifies the submission cluster that the job is in ZOMBI state and the submission cluster requeues the job.

RUNTIME

Estimated run time for the job, specified by bsub -We or bmod -We.

RESOURCE USAGE

For the MultiCluster job forwarding model, this information is not shown if MultiCluster resource usage updating is disabled.

The values for the current usage of a job include:

CPU time

Cumulative total CPU time in seconds of all processes in a job.

IDLE_FACTOR

Job idle information (CPU time/runtime) if JOB_IDLE is configured in the queue, and the job has triggered an idle exception.

MEM

Total resident memory usage of all processes in a job. By default, memory usage is shown in MB. Use LSF_UNIT_FOR_LIMITS in lsf.conf to specify a larger unit for display (MB, GB, TB, PB, or EB).

SWAP

Total virtual memory usage of all processes in a job. By default, swap space is shown in MB. Use LSF_UNIT_FOR_LIMITS in lsf.conf to specify a larger unit for display (MB, GB, TB, PB, or EB).

NTHREAD

Number of currently active threads of a job.

PGID

Currently active process group ID in a job.

PIDs

Currently active processes in a job.

RESOURCE LIMITS

The hard resource usage limits that are imposed on the jobs in the queue (see getrlimit(2) and lsb.queues(5)). These limits are imposed on a per-job and a per-process basis.

The possible per-job resource usage limits are:

• CPULIMIT
• PROCLIMIT
• MEMLIMIT
• SWAPLIMIT
• PROCESSLIMIT
• THREADLIMIT
• OPENFILELIMIT

The possible UNIX per-process resource usage limits are:

• RUNLIMIT
• FILELIMIT
• DATALIMIT
• STACKLIMIT
• CORELIMIT

If a job submitted to the queue has any of these limits specified (see bsub(1)), then the lower of the corresponding job limits and queue limits are used for the job.

If no resource limit is specified, the resource is assumed to be unlimited. User shell limits that are unlimited are not displayed.

EXCEPTION STATUS

Possible values for the exception status of a job include:

idle

The job is consuming less CPU time than expected. The job idle factor (CPU time/runtime) is less than the configured JOB_IDLE threshold for the queue and a job exception has been triggered.

overrun

The job is running longer than the number of minutes specified by the JOB_OVERRUN threshold for the queue and a job exception has been triggered.

underrun

The job finished sooner than the number of minutes specified by the JOB_UNDERRUN threshold for the queue and a job exception has been triggered.

Job Array Summary Information

If you use -A, displays summary information about job arrays. The following fields are displayed:

JOBID

Job ID of the job array.

ARRAY_SPEC

Array specification in the format of name[index]. The array specification may be truncated, use -w option together with -A to show the full array specification.

OWNER

Owner of the job array.

NJOBS

Number of jobs in the job array.

PEND

Number of pending jobs of the job array.

RUN

Number of running jobs of the job array.

DONE

Number of successfully completed jobs of the job array.

EXIT

Number of unsuccessfully completed jobs of the job array.

SSUSP

Number of LSF system suspended jobs of the job array.

USUSP

Number of user suspended jobs of the job array.

PSUSP

Number of held jobs of the job array.

Examples

bjobs -pl 

Displays detailed information about all pending jobs of the invoker.

bjobs -ps 

Display only pending and suspended jobs.

bjobs -u all -a 

Displays all jobs of all users.

bjobs -d -q short -m hostA -u user1 

Displays all the recently finished jobs submitted by user1 to the queue short, and executed on the host hostA.

bjobs 101 102 203 509 

Display jobs with job_ID 101, 102, 203, and 509.

bjobs -X 101 102 203 509 

Display jobs with job ID 101, 102, 203, and 509 as uncondensed output even if these jobs belong to hosts in condensed host groups.

bjobs -sla Uclulet 

Displays all jobs belonging to the service class Uclulet.

bjobs -app fluent 

Displays all jobs belonging to the application profile fluent.

See also

bsub(1), bkill(1), bhosts(1), bmgroup(1), bclusters(1), bqueues(1), bhist(1), bresume(1), bsla(1), bstop(1), lsb.params(5), lsb.erviceclasses(5), mbatchd(8)