Last update: 01/30/2007

Queues and charging for resource usage on blueice

This section provides information about blueice queues and charging. Charges depend on the queue in which your job runs. To select a queue, it is important to consider the charging formula.

To make informed decisions about queues and charging, you need a basic understanding of blueice's system architecture. It is worthwhile to review the blueice main page if you are not already familiar with the system.

Charging

Charges for jobs run on blueice are assessed in General Accounting Units (GAUs).

Charging formula

This formula specifies how your computing account is charged for running jobs on blueice:

GAUs charged = wallclock hours used * number of nodes used * number of processors in that node * computer factor * queue charging factor

The "computer factor" is a multiplier that equalizes the way GAUs are consumed on different computing platforms. Faster computers have higher computer factors. The computer factor for blueice is 0.87.

The "queue charging factor" is a multiplier that reflects the priority given to jobs in a queue: higher-priority jobs are charged more.

Exceeding allocation threshold limits*

Jobs from NCAR divisions or CSL proposal groups that have exceeded either the 30-day or 90-day usage limits* will be placed in the hold queue and run at a priority below jobs in the economy queue. Affected jobs will be charged at 1/3 the rate they would have been charged if they had been run in a regular queue ("rg").

Jobs from NCAR divisions or CSL proposal groups that have exceeded both the 30-day and 90-day usage limits* will be rejected, and users will receive an email suggesting that they submit their jobs to a standby queue. Note that standby queue time limits are three hours, so users may need to change their job's time limit before resubmitting to a standby queue.

* The blueice user guide provides details about how jobs are scheduled for execution when an allocation threshold limit has been exceeded. See the section Allocation thresholds for projects influence job scheduling.

Queue names and uses

The queue names and uses are:

• The special queue is used for large, long-running jobs that must be approved by the SCD Division Director. To request use of this queue, send email to cal@ucar.edu
• The premium queue is for jobs that need to run before jobs in the lower queues.

• The bts_{prm,reg,sby,lrg} queues are for use by a small group of users who have been invited by NSF for use on Break-Through Science projects. These queues will run from Feb 1 through April 30 of 2007.
• The regular queue is for medium-priority jobs.
• The economy queue is for low-priority jobs.
• The standby queue is for jobs that can be run only when idle nodes are available. This is a very low-priority queue that is designed to make otherwise unused cycles available to users. Therefore, turnaround time in the standby queue may be extremely long. Caution: If use of the standby queue increases to the point where it interferes with normal work, the standby jobs will be suspended and will remain in a "hold" state until unused cycles become available.
• The share queue runs on two nodes The share queue is for jobs that are not compute-intensive or intensive on other resources such as memory. This queue runs on two nodes only, the same ones as the debug queue (see below). It has a three-hour time limit, with a maximum of 32 processors (2 nodes). If you know your jobs will keep the share queue fairly busy, we recommend that you run them outside the peak-demand hours of 08:00-17:00 MT weekdays. At all times, the number of jobs submitted to the share queue by the same user should be limited to 25. Examples of jobs appropriate to run in the share queue include:
  • Short interactive or debugging jobs;
  • Pre-staging data to(from) the MSS before(after) a simulation run;
  • Post-staging data from the MSS after a simulation run;
  • General script or application development or checkout;
  • Jobs requiring only one processor; and
  • General simulation runs that do not require large amounts of memory, I/O, or CPU time.

• The debug queue is provided for interactive debugging (for example TotalView). It runs on the same two nodes as the share queue. It has a 30-minute time limit, with a maximum of 32 processors (2 nodes).
• The hold queue is an automated queue for jobs from divisions or groups that have exceeded their allocation threshold limit. Jobs in the hold queue have priority only over jobs in the standby queue.

Queues for batch jobs on blueice

The bts_{prm,reg,sby,lrg} queues are for a small number of special capability computing campaigns that are approved through NCAR management.

Queue	Maximum wallclock hours	Memory	Queue charging factor	Availability	Node limit per job
special	Unlimited	32 GB	1.0	By special permission	100
premium	6 hrs.	32 GB	1.5	Any time	25
regular	6 hrs.	32 GB	1.0	Any time	25
lrg_mem*	6 hrs.	64 GB	1.0	Any time	25
economy	6 hrs.	32 GB	0.5	Any time	25
standby	2 hrs.	32 GB	0.1	Any time	25
share**	3 hrs.	32 GB	1.0	Any time	2
debug**	0.5 hrs.	32 GB	1.0	Any time	2
hold***	6 hrs.	32 GB	0.33	Automated	25
bts_prm	6 hrs.	32 GB	1.0	By special permission	100
bts_reg	6 hrs.	32 GB	1.0	By special permission	75
bts_sby	6 hrs.	32 GB	1.0	By special permission	75
bts_lrg	6 hrs.	64 GB	1.0	By special permission	25
* The lrg_mem queue is specifically for jobs with large memory requirements that cannot run in the other queues.
** The share and debug queues are on two nodes distinct from the login nodes.
*** Jobs in the hold queue are automatically moved to lower priority when allocation threshold limits are exceeded.

Queue-capacity issues

Jobs with large node counts relative to the queue node limit, or with wallclock specification close to the queue wallclock limit, may remain pending in the queue a long time (days) before starting.

For more detailed system information

To obtain queue information, type:
bqueues -l
while logged on to blueice.

To check the status of all users' jobs on blueice, type:
bjobs -u all
while logged on to blueice, or to see a summary of all running jobs, type:
lsfq

To see your GPFS disk quota, type:
/usr/local/bin/spquota or
/usr/local/bin/spinfo
while logged on to blueice. Note that disk space is oversubscribed to maximize the amount that is used. Because of this, everyone cannot use all of their disk space simultaneously.

blueice main page

If you have questions about this document, please contact SCD Customer Support. You can also reach us by telephone 24 hours a day, seven days a week at 303-497-1278. Additional contact methods: consult1@ucar.edu and during business hours in NCAR Mesa Lab Suite 39.

© Copyright 2006. University Corporation for Atmospheric Research (UCAR). All Rights Reserved.

Address of this page: http://www.scd.ucar.edu/computers/blueice/queue.charge.html