Last update: 08/02/2006

lightning main page

Overview

Lightning is a clustered Symmetric MultiProcessing (SMP) system. Lightning provides compute nodes that are equally divided between Community Computing users and Climate Simulation Lab users.

Lightning has 128 batch computing nodes and two interactive (login) nodes. Each compute node has two 2.2 GHz processors and 4 GB of shared memory. Both interactive nodes have two 2.2 GHz processors and 8 GB of shared memory. The batch job submittal system is LSF (Load Sharing Facility). Lightning resides inside the SCD security perimeter.

Hardware

Processors: 256 Opteron processors with a 2.2-GHz clock cycle
Compute nodes: 128 for batch computing
Interactive nodes: 2 for logins
Main memory: 4 GB of ECC RAM on a node is shared between processors on that node and is accessible through a dedicated memory controller on each processor, delivering a peak bandwidth of 5.3 GB/s per processor. Interactive nodes have 8 GB memory shared between processors on that node.
Cache memory:

Level 1 cache: Two-way set-associative, 64-byte lines, 64 KB of data and 64 KB of instruction cache in each processor
Level 2 cache: 16-way set associative, 64-byte lines, 1 MB of L2 cache in each processor

Disk storage: Each user application can access 20 GB of temporary space in /ptmp. Each user account has access to 3 GB of storage space.
Interconnect fabric ("the switch"): A 128-port Myrinet switch interconnects all the batch nodes through a single-port Myrinet PCI adapter for message passing. The switch has a ~6.3 microsecond latency and 248 MB/s each way point-to-point asymptotic bandwidth with full duplex link. Note that the interactive nodes are not connected to the Myrinet switch.
Gigabit Ethernet network hosts the TCP/IP traffic including GPFS, NFS, MSS, etc.

Software

Operating System: SUSE Linux Enterprise Server 8
Compilers: PGI Server suite: pgf90, pgf77, pgcc, pgCC, pghpf
Profiler: pgprof
Debugger: pgdbg
Utilities: gmake, gcc, contact the SCD consultants for information.
Software libraries:

• ACML

• netcdf-3.5.1

• NCAR Graphics

• LAPACK

• SPHEREPACK

• HDF
Batch subsystem: LSF
File System: General Parallel File System (GPFS), a UNIX-style file system that allows applications on multiple nodes to share file data. GPFS supports very large file systems and stripes data across multiple disks for higher performance.

More information about lightning's software environment appears in the seminar materials for lightning compilers (PowerPoint slides).

Who can use this system

Lightning is equally shared by Community Computing users and Climate Simulation Lab users. The number of nodes available to each group is flexible to maximize the system's productivity -- e.g., if one group is not using all of its available compute nodes, then those nodes may be used by members of the other group.

Recommendations for usage

Lightning is a compute server; it does not support data analysis work.

There are three ways to use lightning:

To use more than one processor on a node, use OpenMP threading directives on the node, or use MPI processes on the node, or use a mixture of both.
To pass information between nodes, you must use MPI.
To take full advantage of parallelism: use OpenMP threads, MPI, or a mixture of both on the node plus use MPI between nodes.

Note that lightning has "thin" nodes with only two processors and 4 GB of shared memory per node. Because of this, your use of OpenMP may not be as productive as it is on nodes with more processors.

How to get an account

CSL users may receive a lightning login as part of their CSL allocation. See the CSL announcement of opportunity for details on submitting proposals.

Community Computing users who have General Accounting Unit (GAU) allocations are eligible to apply for an account on lightning. Community users may request a lightning login by contacting SCD Customer Support. Please include the following information with your login request:

Your login name
Your project number
A short description of how you intend to use lightning

System security and access method

Lightning is inside the UCAR supercomputer security perimeter, so access from outside must be via the gatekeeper system roy.ucar.edu using a CRYPTOCard. If you do not have an SCD-issued CRYPTOCard, you can request one by submitting a Extraview work request or calling 303-497-1200. Instructions for navigating the UCAR security perimeter are provided in the document "Overview of computing at NCAR" and the relevant section is Accessing computing services.

You must use ssh, scp, and sftp on lightning; there are no rsh, rcp, and ftp commands.

You will need Secure SHell (SSH) on the workstation you use to access IBM SP-cluster systems. Information about using SSH for computing at NCAR is described in the document "Overview of computing at NCAR" and the relevant section is SSH - Secure SHell.

Once you have authenticated your access at roy.ucar.edu using your Common Authentication System (CAS) password or your public key and one-time password from your CRYPTOCard, then you may access lightning from your proxy prompt by keying in your CAS password (or public key if you have installed key-based authentication at roy).

Queues and charging

The class (queue) structure for lightning is described in the document Queues and charging for resource usage on lightning.

Documentation

Lightning user guide

If you have questions about this document, please contact SCD Customer Support. You can also reach us by telephone 24 hours a day, seven days a week at 303-497-1278. Additional contact methods: consult1@ucar.edu and during business hours in NCAR Mesa Lab Suite 39.