lsb.modules

The lsb.modules file contains configuration information for LSF scheduler and resource broker modules. The file contains only one section, named PluginModule.

This file is optional. If no scheduler or resource broker modules are configured, LSF uses the default scheduler plugin modules named schmod_default and schmod_fcfs.

The lsb.modules file is stored in the directory LSB_CONFDIR/cluster_name/configdir, where LSB_CONFDIR is defined in lsf.conf.

Changing lsb.modules configuration

After making any changes to lsb.modules, run badmin reconfig to reconfigure mbatchd.

PluginModule section

Description

Defines the plugin modules for the LSF scheduler and LSF resource broker. If this section is not configured, LSF uses the default scheduler plugin modules named schmod_default and schmod_fcfs, which enable the LSF default scheduling features.

Example PluginModule section

The following PluginModule section enables all scheduling policies provided by LSF:
Begin PluginModule 
SCH_PLUGIN              RB_PLUGIN           SCH_DISABLE_PHASES schmod_default               ()                      () 
schmod_fairshare             ()                      () 
schmod_fcfs                  ()                      () 
schmod_limit                 ()                      () 
schmod_parallel              ()                      () 
schmod_reserve               ()                      () 
schmod_preemption            ()                      ()
schmod_advrsv                ()                      () 
schmod_mc                    ()                      () 
schmod_jobweight             ()                      () 
schmod_cpuset                ()                      () 
schmod_pset                  ()                      ()
schmod_ps                    ()                      () 
schmod_aps                   ()                      () 
End PluginModule

PluginModule section structure

The first line consists of the following keywords:

  • SCH_PLUGIN

  • RB_PLUGIN

  • SCH_DISABLE_PHASES

They identify the scheduler plugins, resource broker plugins, and the scheduler phase to be disabled for the plugins that you wish to configure.

Each subsequent line describes the configuration information for one scheduler plugin module, resource broker plugin module, and scheduler phase, if any, to be disabled for the plugin. Each line must contain one entry for each keyword. Use empty parentheses ( ) or a dash (-) to specify the default value for an entry.

SCH_PLUGIN

Description

Required. The SCH_PLUGIN column specifies the shared module name for the LSF scheduler plugin. Each plugin requires a corresponding license. Scheduler plugins are called in the order they are listed in the PluginModule section.

By default, all shared modules for scheduler plugins are located in LSF_LIBDIR. On UNIX, you can also specify a full path to the name of the scheduler plugin.

The following modules are supplied with LSF:

schmod_default

Enables the default LSF scheduler features.

Licensed by: LSF_Manager

schmod_fcfs

Enables the first-come, first-served (FCFS) scheduler features. schmod_fcfs can appear anywhere in the SCH_PLUGIN list. By default, if schmod_fcfs is not configured in lsb.modules, it is loaded automatically along with schmod_default.

Source code (sch.mod.fcfs.c) for the schmod_fcfs scheduler plugin module is installed in the directory
LSF_TOP/7.0/misc/examples/external_plugin/

Use the LSF scheduler plugin SDK to modify the FCFS scheduler module code to suit the job scheduling requirements of your site.

See Platform LSF Programmer’s Guide for more detailed information about writing, building, and configuring your own custom scheduler plugins.

schmod_fairshare

Enables the LSF fairshare scheduling features.

schmod_limit

Enables the LSF resource allocation limit features.

Licensed by: LSF_Manager

schmod_parallel

Enables scheduling of parallel jobs submitted with bsub -n.

schmod_reserve

Enables the LSF resource reservation features.

To enable processor reservation, backfill, and memory reservation for parallel jobs, you must configure both schmod_parallel and schmod_reserve in lsb.modules. If only schmod_reserve is configured, backfill and memory reservation are enabled only for sequential jobs, and processor reservation is not enabled.

schmod_preemption

Enables the LSF preemption scheduler features.

schmod_advrsv

Handles jobs that use advance reservations (brsvadd, brsvs, brsvdel, bsub -U)

schmod_cpuset

Handles jobs that use IRIX cpusets (bsub ‑ext;sched] "CPUSET[cpuset_options]")

The schmod_cpuset plugin name must be configured after the standard LSF plugin names in the PluginModule list.

schmod_mc

Enables MultiCluster job forwarding

Licensed by: LSF_MultiCluster

schmod_ps

Enables resource ownership functionality of EGO-enabled SLA scheduling policies

schmod_pset

Enables scheduling policies required for jobs that use HP-UX processor sets (pset) allocations (bsub ‑ext[sched] "PSET[topology]")

The schmod_pset plugin name must be configured after the standard LSF plugin names in the PluginModule list.

schmod_aps

Enables absolute priority scheduling (APS) policies configured by APS_PRIORITY in lsb.queues.

The schmod_aps plugin name must be configured after the schmod_fairshare plugin name in the PluginModule list, so that the APS value can override the fairshare job ordering decision.

Licensed by: LSF_HPC

schmod_jobweight

An optional scheduler plugin module to enable Cross-Queue Job Weight scheduling policies. The schmod_jobweight plugin must be listed before schmod_cpuset and schmod_rms, and after all other scheduler plugin modules.

You should not use job weight scheduling together with fairshare scheduling or job preemption. To avoid scheduling conflicts, you should comment out schmod_fairshare and schmod_preemption in lsb.modules.

Scheduler plugin SDK

Use the LSF scheduler plugin SDK to write customized scheduler modules that give you more flexibility and control over job scheduling. Enable your custom scheduling policies by configuring your modules under SCH_PLUGIN in the PluginModules section of lsb.modules.

The directory
LSF_TOP/7.0/misc/examples/external_plugin/

contains sample plugin code. See Platform LSF Programmer’s Guide for more detailed information about writing, building, and configuring your own custom scheduler plugins.

RB_PLUGIN

Description

RB_PLUGIN specifies the shared module name for resource broker plugins. Resource broker plugins collect and update job resource accounting information, and provide it to the scheduler.

Normally, for each scheduler plugin module, there is a corresponding resource broker plugin module to support it. However, the resource broker also supports multiple plugin modules for one scheduler plugin module.

For example, a fairshare policy may need more than one resource broker plugin module to support it if the policy has multiple configurations.

A scheduler plugin can have one, multiple, or none RB plugins corresponding to it.

Example

NAME                 RB_PLUGIN 
schmod_default      () 
schmod_fairshare    (rb_fairshare) 

Default

Undefined

SCH_DISABLE_PHASES

Description

SCH_DISABLE_PHASES specifies which scheduler phases, if any, to be disabled for the plugin. LSF scheduling has four phases:
  1. Preprocessing — the scheduler checks the readiness of the job for scheduling and prepares a list of ready resource seekers. It also checks the start time of a job, and evaluates any job dependencies.

  2. Match/limit — the scheduler evaluates the job resource requirements and prepares candidate hosts for jobs by matching jobs with resources. It also applies resource allocation limits. Jobs with all required resources matched go on to order/allocation phase. Not all jobs are mapped to all potential available resources. Jobs without any matching resources will not go through the Order/Allocation Phase but can go through the Post-processing phase, where preemption may be applied to get resources the job needs to run.

  3. Order/allocation — the scheduler sorts jobs with matched resources and allocates resources for each job, assigning job slot, memory, and other resources to the job. It also checks if the allocation satisfies all constraints defined in configuration, such as queue slot limit, deadline for the job, etc.
    1. In the order phase, the scheduler applies policies such as FCFS, Fairshare and Host-partition and consider job priorities within user groups and share groups. By default, job priority within a pool of jobs from the same user is based on how long the job has been pending.

    2. For resource intensive jobs (jobs requiring a lot of CPUs or a large amount of memory), resource reservation is performed so that these jobs are not starved.

    3. When all the currently available resources are allocated, jobs go on to post-processing.

  4. Post-processing — the scheduler prepares jobs from the order/allocation phase for dispatch and applies preemption or backfill policies to obtain resources for the jobs that have completed pre-processing or match/limit phases, but did not have resources available to enter the next scheduling phase.

Each scheduler plugin module invokes one or more scheduler phase. The processing for a give phase can be disabled or skipped if:

The plugin module does not need to do any processing for that phase or the processing has already been done by a previous plugin module in the list.

The scheduler will not invoke phases marked by SCH_DISABLE_PHASES when scheduling jobs.

None of the plugins provided by LSF should require phases to be disabled, but your own custom plugin modules using the scheduler SDK may need to disable one or more scheduler phases.

Example

In the following configuration, the schmod_custom plugin module disables the order allocation (3) and post-processing (4) phases:
NAME             SCH_DISABLE_PHASES 
schmod_default   ()                
schmod_custom    (3,4)              

Default

Undefined