Preemptive scheduling

The preemptive scheduling feature allows a pending high-priority job to preempt a running job of lower priority. The lower-priority job is suspended and is resumed as soon as possible. Use preemptive scheduling if you have long-running, low-priority jobs causing high-priority jobs to wait an unacceptably long time.

Contents

  • About preemptive scheduling

  • Scope

  • Configuration to enable preemptive scheduling

  • Preemptive scheduling behavior

  • Configuration to modify preemptive scheduling behavior

  • Preemptive scheduling commands

About preemptive scheduling

Preemptive scheduling takes effect when two jobs compete for the same job slots. If a high-priority job is pending, LSF can suspend a lower-priority job that is running, and then start the high-priority job instead. For this to happen, the high-priority job must be pending in a preemptive queue (a queue that can preempt other queues), or the low-priority job must belong to a preemptable queue (a queue that can be preempted by other queues).

If multiple slots are required, LSF can preempt multiple jobs until sufficient slots are available. For example, one or more jobs can be preempted for a job that needs multiple job slots.

A preempted job is resumed as soon as more job slots become available; it does not necessarily have to wait for the preempting job to finish.

Preemptive queue

Jobs in a preemptive queue can preempt jobs in any queue of lower priority, even if the lower-priority queues are not specified as preemptable.

Preemptive queues are more aggressive at scheduling jobs because a slot that is not available to a low-priority queue may be available by preemption to a high-priority queue.

Preemptable queue

Jobs in a preemptable queue can be preempted by jobs from any queue of a higher priority, even if the higher-priority queues are not specified as preemptive.

When multiple preemptable jobs exist (low-priority jobs holding the required slots), and preemption occurs, LSF preempts a job from the least-loaded host.

Scope

Preemptive scheduling does not apply to jobs that have been forced to run or backfill and exclusive jobs.

Default behavior (preemptive scheduling not enabled)

With preemptive scheduling enabled (preemptive queue)

With preemptive scheduling enabled (preemptable queue)

Configuration to enable preemptive scheduling

The preemptive scheduling feature is enabled by defining at least one queue as preemptive or preemptable, using the PREEMPTION parameter in the lsb.queues file. Preemption does not actually occur until at least one queue is assigned a higher relative priority than another queue, using the PRIORITY parameter, which is also set in the lsb.queues file.

Both PREEMPTION and PRIORITY are used to determine which queues can preempt other queues, either by establishing relative priority of queues or by specifically defining preemptive properties for a queue.


Configuration file

Parameter and syntax

Default behavior

lsb.queues

PREEMPTION=PREEMPTIVE

  • Enables preemptive scheduling

  • Jobs in this queue can preempt jobs in any queue of lower priority, even if the lower-priority queue is not specified as preemptable

PREEMPTION=PREEMPTABLE

  • Enables preemptive scheduling

  • Jobs in this queue can be preempted by jobs from any queue of higher priority, even if the higher-priority queue is not specified as preemptive

PRIORITY=integer

  • Sets the priority for this queue relative to all other queues

  • The larger the number, the higher the priority—a queue with PRIORITY=99 has a higher priority than a queue with PRIORITY=1


Preemptive scheduling behavior

Preemptive scheduling is based primarily on parameters specified at the queue level: some queues are eligible for preemption, others are not. Once a hierarchy of queues has been established, other factors determine which jobs from a queue should be preempted.

There are three ways to establish which queues should be preempted:

  • Based on queue priority—the PREEMPTION parameter defines a queue as preemptive or preemptable and preemption is based on queue priority, where jobs from higher-priority queues can preempt jobs from lower-priority queues

  • Based on a preferred order—the PREEMPTION parameter defines queues that can preempt other queues, in a preferred order

  • Explicitly, by specific queues—the PREEMPTION parameter defines queues that can be preempted, and by which queues


When …

The behavior is …

Preemption is not enabled—no queue is defined as preemptable, and no queue is defined as preemptive

  • High-priority jobs do not preempt jobs that are already running

A queue is defined as preemptable, but no specific queues are listed that can preempt it

  • Jobs from this queue can be preempted by jobs from any queue with a higher value for priority

A queue is defined as preemptable, and one or more queues are specified that can preempt it

  • Jobs from this queue can be preempted only by jobs from the specified queues

A queue is defined as preemptive, but no specific queues are listed that it can preempt

  • Jobs from this queue preempt jobs from all queues with a lower value for priority

  • Jobs are preempted from the least-loaded host

A queue is defined as preemptive, and one or more specific queues are listed that it can preempt, but no queue preference is specified

  • Jobs from this queue preempt jobs from any queue in the specified list

  • Jobs are preempted on the least-loaded host first

A queue is defined as preemptive, and one or more queues have a preference number specified, indicating a preferred order of preemption

  • Queues with a preference number are preferred for preemption over queues without a preference number

  • Queues with a higher preference number are preferred for preemption over queues with a lower preference number

  • For queues that have the same preference number, the queue with lowest priority is preferred for preemption over queues with higher priority

  • For queues without a preference number, the queue with lower priority is preferred for preemption over the queue with higher priority

A queue is defined as preemptive, or a queue is defined as preemptable, and preemption of jobs with the shortest run time is configured

  • A queue from which to preempt a job is determined based on other parameters as shown above

  • The job that has been running for the shortest period of time is preempted

A queue is defined as preemptive, or a queue is defined as preemptable, and preemption of jobs that will finish within a certain time period is prevented

  • A queue from which to preempt a job is determined based on other parameters as shown above

  • A job that has a run limit or a run time specified and that will not finish within the specified time period is preempted

A queue is defined as preemptive, or a queue is defined as preemptable, and preemption of jobs with the specified run time is prevented

  • A queue from which to preempt a job is determined based on other parameters as shown above

  • The job that has been running for less than the specified period of time is preempted


Case study: Three queues with varying priority

Consider the case where three queues are defined as follows:
  • Queue A has the highest relative priority, with a value of 99
  • Queue B is both preemptive and preemptable, and has a relative priority of 10
  • Queue C has the lowest relative priority, with the default value of 1

The queues can preempt as follows:

  • A can preempt B because B is preemptable and B has a lower priority than A

  • B can preempt C because B is preemptive and C has a lower priority than B

  • A cannot preempt C, even though A has a higher priority than C, because A is not preemptive, nor is C preemptable

Calculation of job slots in use

The number of job slots in use determines whether preemptive jobs can start. The method in which the number of job slots in use is calculated can be configured to ensure that a preemptive job can start. When a job is preempted, it is suspended. If the suspended job still counts towards the total number of jobs allowed in the system, based on the limits imposed in the lsb.resources file, suspending the job may not be enough to allow the preemptive job to run.

The PREEMPT_FOR parameter is used to change the calculation of job slot usage, ignoring suspended jobs in the calculation. This ensures that if a limit is met, the preempting job can actually run.


When …

The effect on the calculation of job slots used is …

Preemption is not enabled

  • Job slot limits are enforced based on the number of job slots taken by both running and suspended jobs.

  • Job slot limits specified at the queue level are enforced for both running and suspended jobs.

Preemption is enabled

  • The total number of jobs at both the host and individual user level is not limited by the number of suspended jobs—only running jobs are considered.

  • The number of running jobs never exceeds the job slot limits. If starting a preemptive job violates a job slot limit, a lower-priority job is suspended to run the preemptive job. If, however, a job slot limit is still violated (i.e. the suspended job still counts in the calculation of job slots in use), the preemptive job still cannot start.

  • Job slot limits specified at the queue level are always enforced for both running and suspended jobs.

  • When preemptive scheduling is enabled, suspended jobs never count against the total job slot limit for individual users.

Preemption is enabled, and PREEMPT_FOR=GROUP_JLP

  • Only running jobs are counted when calculating the per-processor job slots in use for a user group, and comparing the result with the limit specified at the user level.

Preemption is enabled, and PREEMPT_FOR=GROUP_MAX

  • Only running jobs are counted when calculating the job slots in use for this user group, and comparing the result with the limit specified at the user level.

Preemption is enabled, and PREEMPT_FOR=HOST_JLU

  • Only running jobs are counted when calculating the total job slots in use for a user group, and comparing the result with the limit specified at the host level. Suspended jobs do not count against the limit for individual users.

Preemption is enabled, and PREEMPT_FOR=USER_JLP

  • Only running jobs are counted when calculating the per-processor job slots in use for an individual user, and comparing the result with the limit specified at the user level.


Preemption of backfill jobs

With preemption of backfill jobs enabled (PREEMPT_JOBTYPE=BACKFILL in lsb.params), LSF maintains the priority of jobs with resource or slot reservations by preventing lower-priority jobs that preempt backfill jobs from "stealing" resources from jobs with reservations. Only jobs from queues with a higher priority than queues that define resource or slot reservations can preempt backfill jobs. For example,

If …

Is configured …

And a priority of …

The behavior is …

queueR

With a resource or slot reservation

80

Jobs in this queue reserve resources. If backfill scheduling is enabled, backfill jobs with a defined run limit can use the resources.

queueB

As a preemptable backfill queue

50

Jobs in queueB with a defined run limit use job slots reserved by jobs in queueR.

queueP

As a preemptive queue

75

Jobs in this queue do not necessarily have a run limit. LSF prevents jobs from this queue from preempting backfill jobs because queueP has a lower priority than queue R.


To guarantee a minimum run time for interruptible backfill jobs, LSF suspends them upon preemption. To change this behavior so that LSF terminates interruptible backfill jobs upon preemption, you must define the parameter TERMINATE_WHEN=PREEMPT in lsb.queues.

Configuration to modify preemptive scheduling behavior

There are configuration parameters that modify various aspects of preemptive scheduling behavior, by

  • Modifying the selection of the queue to preempt jobs from

  • Modifying the selection of the job to preempt

  • Modifying preemption of backfill and exclusive jobs

  • Modifying the way job slot limits are calculated

  • Modifying the number of jobs to preempt for a parallel job

  • Modifying the control action applied to preempted jobs

  • Control how many times a job can be preempted

Configuration to modify selection of queue to preempt


File

Parameter

Syntax and description

lsb.queues

PREEMPTION

PREEMPTION=PREEMPTIVE[low_queue+pref …]
  • Jobs in this queue can preempt running jobs from the specified queues, starting with jobs in the queue with the highest value set for preference

PREEMPTION=PREEMPTABLE[hi_queue …]
  • Jobs in this queue can be preempted by jobs from the specified queues

PRIORITY=integer

  • Sets the priority for this queue relative to all other queues

  • The higher the priority value, the more likely it is that jobs from this queue may preempt jobs from other queues, and the less likely it is for jobs from this queue to be preempted by jobs from other queues


Configuration to modify selection of job to preempt


Files

Parameter

Syntax and description

lsb.params

lsb.applications

PREEMPT_FOR

PREEMPT_FOR=LEAST_RUN_TIME

  • Preempts the job that has been running for the shortest time

NO_PREEMPT_RUN_TIME

NO_PREEMPT_RUN_TIME=%
  • Prevents preemption of jobs that have been running for the specified percentage of minutes, or longer

  • If NO_PREEMPT_RUN_TIME is specified as a percentage, the job cannot be preempted after running the percentage of the job duration. For example, if the job run limit is 60 minutes and NO_PREEMPT_RUN_TIME=50%, the job cannot be preempted after it running 30 minutes or longer.

  • Requires a run time (bsub -We or RUNTIME in lsb.applications), or run limit to be specified for the job (bsub -W, or RUNLIMIT in lsb.queues, or RUNLIMIT in lsb.applications)

NO_PREEMPT_FINISH_TIME

NO_PREEMPT_FINISH_TIME=%
  • Prevents preemption of jobs that will finish within the specified percentage of minutes

  • If NO_PREEMPT_FINISH_TIME is specified as a percentage, the job cannot be preempted if the job finishes within the percentage of the job duration. For example, if the job run limit is 60 minutes and NO_PREEMPT_FINISH_TIME=10%, the job cannot be preempted after it running 54 minutes or longer.

  • Requires a run time (bsub -We or RUNTIME in lsb.applications), or run limit to be specified for the job (bsub -W, or RUNLIMIT in lsb.queues, or RUNLIMIT in lsb.applications)


Configuration to modify preemption of backfill and exclusive jobs


File

Parameter

Syntax and description

lsb.params

PREEMPT_JOBTYPE

PREEMPT_JOBTYPE=BACKFILL

  • Enables preemption of backfill jobs.

  • Requires the line PREEMPTION=PREEMPTABLE in the queue definition.

  • Only jobs from queues with a higher priority than queues that define resource or slot reservations can preempt jobs from backfill queues.

PREEMPT_JOBTYPE=EXCLUSIVE

  • Enables preemption of and preemption by exclusive jobs.

  • Requires the line PREEMPTION=PREEMPTABLE or PREEMPTION=PREEMPTIVE in the queue definition.

  • Requires the definition of LSB_DISABLE_LIMLOCK_EXCL in lsf.conf.

PREEMPT_JOBTYPE=EXCLUSIVE BACKFILL

  • Enables preemption of exclusive jobs, backfill jobs, or both.

lsf.conf

LSB_DISABLE_LIMLOCK_EXCL

LSB_DISABLE_LIMLOCK_EXCL=y
  • Enables preemption of exclusive jobs.

  • For a host running an exclusive job:
    • lsload displays the host status ok.

    • bhosts displays the host status closed.

    • Users can run tasks on the host using lsrun or lsgrun. To prevent users from running tasks during execution of an exclusive job, the parameter LSF_DISABLE_LSRUN=y must be defined in lsf.conf.

  • Changing this parameter requires a restart of all sbatchds in the cluster (badmin hrestart). Do not change this parameter while exclusive jobs are running.


Configuration to modify how job slot usage is calculated


File

Parameter

Syntax and description

lsb.params

PREEMPT_FOR

PREEMPT_FOR=GROUP_JLP

  • Counts only running jobs when evaluating if a user group is approaching its per-processor job slot limit (SLOTS_PER_PROCESSOR, USERS, and PER_HOST=all in the lsb.resources file), ignoring suspended jobs

PREEMPT_FOR=GROUP_MAX

  • Counts only running jobs when evaluating if a user group is approaching its total job slot limit (SLOTS, PER_USER=all, and HOSTS in the lsb.resources file), ignoring suspended jobs

PREEMPT_FOR=HOST_JLU

  • Counts only running jobs when evaluating if a user or user group is approaching its per-host job slot limit (SLOTS, PER_USER=all, and HOSTS in the lsb.resources file), ignoring suspended jobs

PREEMPT_FOR=USER_JLP

  • Counts only running jobs when evaluating if a user is approaching their per-processor job slot limit (SLOTS_PER_PROCESSOR, USERS, and PER_HOST=all in the lsb.resources file)

  • Ignores suspended jobs when calculating the per-processor job slot limit for individual users


Configuration to modify preemption of parallel jobs


File

Parameter

Syntax and description

lsb.params

PREEMPT_FOR

PREEMPT_FOR=MINI_JOB

  • Optimizes preemption of parallel jobs by preempting only enough low-priority parallel jobs to start the high-priority parallel job


Configuration to modify the control action applied to preempted jobs


File

Parameter

Syntax and description

lsb.queues

TERMINATE_WHEN

TERMINATE_WHEN=PREEMPT

  • Changes the default control action of SUSPEND to TERMINATE so that LSF terminates preempted jobs


Configuration to control how many times a job can be preempted

By default, if preemption is enabled, there is actually no guarantee that a job will ever actually complete. A lower priority job could be preempted again and again, and ultimately end up being killed due to a run limit.

Limiting the number of times a job can be preempted is configured cluster-wide (lsb.params), at the queue level (lsb.queues), and at the application level (lsb.applications). MAX_JOB_PREEMPT in lsb.applications overrides lsb.queues, and lsb.queues overrides lsb.params configuration.


Files

Parameter

Syntax and description

lsb.params

lsb.queues

lsb.applications

MAX_JOB_PREEMPT

MAX_JOB_PREEMPT=integer
  • Specifies the maximum number of times a job can be preempted.

  • Specify a value within the following ranges:

    0 < MAX_JOB_PREEMPT < INFINIT_INT

    INFINIT_INT is defined in lsf.h

  • By default, the number of preemption times is unlimited.


When MAX_JOB_ PREEMPT is set, and a job is preempted by higher priority job, the number of job preemption times is set to 1. When the number of preemption times exceeds MAX_JOB_ PREEMPT, the job will run to completion and cannot be preempted again.

The job preemption limit times is recovered when LSF is restarted or reconfigured.

Preemptive scheduling commands

Commands for submission


Command

Description

bsub -q queue_name

  • Submits the job to the specified queue, which may have a run limit associated with it

bsub -W minutes

  • Submits the job with the specified run limit, in minutes

bsub -app application_profile_name

  • Submits the job to the specified application profile, which may have a run limit associated with it


Commands to monitor


Command

Description

bjobs -s

  • Displays suspended jobs, together with the reason the job was suspended


Commands to control


Command

Description

brun

  • Forces a pending job to run immediately on specified hosts. For an exclusive job, when LSB_DISABLE_LIMLOCK_EXCL=y , LSF allows other jobs already running on the host to finish but does not dispatch any additional jobs to that host until the exclusive job finishes.


Commands to display configuration


Command

Description

bqueues

  • Displays the priority (PRIO) and run limit (RUNLIMIT) for the queue, and whether the queue is configured to be preemptive, preemptable, or both

bhosts

  • Displays the number of job slots per user for a host

  • Displays the number of job slots available

bparams

  • Displays the value of parameters defined in lsb.params.

badmin showconf

  • Displays all configured parameters and their values set in lsf.conf or ego.conf that affect mbatchd and sbatchd.

    Use a text editor to view other parameters in the lsf.conf or ego.conf configuration files.

  • In a MultiCluster environment, badmin showconf only displays the parameters of daemons on the local cluster.