Last update: 03/03/2008

Queues and charging for resource usage on lightning

This section provides information about lightning queues and charging. Charges depend on the queue in which your job runs. To select a queue, it is important to consider the charging formula.

To make informed decisions about queues and charging, you need a basic understanding of lightning's system architecture. It is worthwhile to review the lightning main page if you are not already familiar with the system.

Charging

Charges for jobs run on lightning are assessed in General Accounting Units (GAUs).

Charging formula

This formula specifes how your computing account is charged for running jobs on lightning.

GAUs charged = wallclock hours used * number of nodes used * number of processors in that node * computer factor * queue charging factor

The "computer factor" is a multiplier that equalizes the way GAUs are consumed on different computing platforms. Faster computers have higher computer factors. The computer factor for lightning is 0.34.

The "queue charging factor" is a multiplier that reflects the priority given to jobs in a queue: higher-priority jobs are charged more.

Exceeding allocation threshold limits*

Jobs from NCAR divisions or CSL proposal groups that have exceeded either the 30-day or 90-day usage limits* will be placed in the hold queue and run at a priority below jobs in the economy queue. Affected jobs will be charged at 1/3 the rate they would have been charged if they had been run in a regular queue ("rg").

Jobs from NCAR divisions or CSL proposal groups that have exceeded both the 30-day and 90-day usage limits* will be rejected, and users will receive an email suggesting that they submit their jobs to a standby queue. Note that standby queue time limits are six hours, so users may need to change their job's time limit before resubmitting to a standby queue.

* The IBM SP-cluster systems reference manual provides details about how jobs are scheduled for execution when an allocation threshold limit has been exceeded. See the section Allocation thresholds for projects influence job scheduling.

Queue names and uses

The queue names and uses are:

• The special queue is used for large, long-running jobs that must be approved by the SCD Division Director. To request use of this queue, send email to cal@ucar.edu
• The premium queue is for jobs that need to run before jobs in the lower queues.
• The regular queue is for medium-priority jobs.
• The economy queue is for low-priority jobs.
• The standby queue is for jobs that can be run only when idle nodes are available. This is a very low-priority queue that is designed to make otherwise unused cycles available to users. Therefore, turnaround time in the standby queue may be extremely long. Caution: If use of the standby queue increases to the point where it interferes with normal work, the standby jobs will be suspended and will remain in a "hold" state until unused cycles become available.
• The share queue runs on the two interactive nodes only. Examples of jobs appropriate to run in the share queue include:
• Short interactive jobs
• Pre-staging data to the MSS before a simulation run
• Post-staging data from the MSS after a simulation run
• General script or application development or checkout
• General simulation runs that do not require large amounts of memory, I/O, or CPU time.
• The hold queue is an automated queue for jobs from divisions or groups that have exceeded their allocation threshold limit. Jobs in the hold queue have priority only over jobs in the standby queue.

• The debug queue is a daytime queue dedicated to debugging applications that will be run on lightning. This queue is for jobs being converted to run on lightning and jobs being developed to run on lightning.

Queues for batch jobs on lightning

The queue structure for lightning is:

Queue	CPUs	Maximum wallclock hours	Memory	Queue charging factor	Availability
special	256	Unlimited	4 GB	1.0	By special permission
premium	256	6 hrs.	4 GB	1.5	Any time
regular	256	6 hrs.	4 GB	1.0	Any time
economy	256	6 hrs.	4 GB	0.5	Any time
standby	256	6 hrs.	4 GB	0.1	Any time
share*	4	6 hrs.	8 GB	1.0	Any time
hold**	256	6 hrs.	4 GB	0.33	Automated
debug	16	0.5 hrs.	4 GB	1.0	10am - 6pm daily
* The share queue is available on the login nodes only.
** Jobs in the hold queue are automatically moved to lower priority when allocation threshold limits are exceeded.

Full-capacity issues

Be aware that usage issues arise when nearly all of lightning's nodes are being used. Users running jobs that require the entire system are also affected.

To provide the debug queue in the daytime, 8 compute nodes (16 processors) are removed from the special, premium, regular, economy, and standby queues, and only 120 nodes (240 processors) are available to these other queues.

So jobs that need more than 240 processors cannot run until after 6:00 pm. And when the system is saturated with jobs, note that more processors are available in the evenings.

For more detailed system information

To obtain queue information, type:
bqueues -l
while logged on to lightning.

To check the status of all users' jobs on lightning, type:
bjobs -u all
while logged on to lightning, or to see a summary of all running jobs, type:
lsfq

To see your GPFS disk quota, type:
/usr/lpp/mmfs/bin/mmlsquota
while logged on to lightning. Note that disk space is oversubscribed to maximize the amount that is used. Because of this, everyone cannot use all of their disk space simultaneously.

lightning main page

If you have questions about this document, please contact SCD Customer Support. You can also reach us by telephone 24 hours a day, seven days a week at 303-497-1278. Additional contact methods: consult1@ucar.edu and during business hours in NCAR Mesa Lab Suite 39.

© Copyright 2005-2008. University Corporation for Atmospheric Research (UCAR). All Rights Reserved.

Address of this page: http://www.scd.ucar.edu/computers/lightning/queue.charge.html