IBM-SP user tips: Job requirements on blackforest vs. bluesky and babyblue

Bluesky

This article concerns differences in PWR3 and PWR4 architectures among bluesky, blackforest, and babyblue. These tips are to help you get your jobs through the system with fewer problems and less intervention from the SCD Consulting Office due to dropped jobs.

Job specification

SCD software engineers have observed a number of unrunnable jobs on babyblue and blackforest due to incorrect job specification. These jobs probably ran on bluesky's 8- and 32-way nodes, but are incorrect for babyblue and blackforest 4-way nodes.

Jobs requesting more tasks per node than processors cannot be scheduled, and they are automatically dropped from the queue. The SCD consultants then contact the job's owner to tell them the bad news.

To spare yourself this annoyance and loss of your research time, please check your job's LoadLeveler scripts for task and node correctness before submitting them. In particular, check these LoadLeveler keyword combinations:

#@tasks_per_node

#@total_tasks

#@task_geometry

#@node

For example, an incorrect task-node combination for babyblue and blackforest 4-way nodes is

#@ total_tasks = 8

#@ node =1

A correct combination is

#@ total_tasks=8

#@ node=2

Job memory

Another area of concern is job memory. Users can run jobs with larger memory on bluesky, because each bluesky processor has 2 GB memory whereas each blackforest memory has 512 KBs.

Total memory per blackforest and bluesky node follows:

Blackforest

• Normal (4 CPU) — Winterhawk: 2GB
• Special (16 CPU)— Nighthawk: 8GB (assuming 512KB per CPU)

Bluesky

• 8 CPU LPAR: 16GB
• 32-CPU LPAR: 64GB

Jobs run on the IBMs with excessive memory requirements for the job nodes will cause the job to swap in and out of memory, with miserable performance. (This is true on bluesky as well as for blackforest and babyblue.) These jobs are usually dropped manually by SCD software engineers when swapping is detected, so we recommend that you check your job's memory requirement when you move from bluesky to blackforest.

However, you normally cannot determine the job's memory requirement by inspecting the LoadLeveler script; you need to get at the job's memory information by looking at the job's documentation or by running experiments.

If you need to run a job which may cause swapping on the IBM-SPs, please contact the SCD Consulting Office before doing so. Phone: 303-497-1278 weekdays 8–5 Mountain Time; email: consult1@ucar.edu