lightning main page
Lightning is a clustered Symmetric MultiProcessing (SMP) system.
Lightning provides compute nodes that are equally divided between
Community Computing users and Climate Simulation Lab users.
Lightning has 128 batch computing nodes and two interactive
(login) nodes. Each compute node has two 2.2 GHz processors and
4 GB of shared memory. Both interactive nodes have two 2.2 GHz
processors and 8 GB of shared memory. The batch job submittal
system is LSF (Load Sharing Facility).
- Processors: 256 Opteron processors with a 2.2-GHz clock cycle
- Compute nodes: 128 for batch computing
- Interactive nodes: 2 for logins
- Main memory: 4 GB of ECC RAM on a node is shared between processors
on that node and is accessible through a dedicated memory controller on
each processor, delivering a peak bandwidth of 5.3 GB/s per processor.
Interactive nodes have 8 GB memory shared between processors on that
node.
- Cache memory:
- Level 1 cache: Two-way set-associative, 64-byte lines, 64 KB
of data and 64 KB of instruction cache in each processor
- Level 2 cache: 16-way set associative, 64-byte lines, 1 MB of
L2 cache in each processor
- Disk storage: Each user application can access 20 GB of temporary
space in /ptmp. Each user account has access to 3 GB of storage space.
- Interconnect fabric ("the switch"): A 128-port Myrinet switch
interconnects all the batch nodes through a single-port Myrinet PCI
adapter for message passing. The switch has a ~6.3 microsecond
latency and 248 MB/s each way point-to-point asymptotic bandwidth
with full duplex link. Note that the interactive nodes are not
connected to the Myrinet switch.
- Gigabit Ethernet network hosts the TCP/IP traffic including
GPFS, NFS, MSS, etc.
- Operating System: SUSE Linux Enterprise Server 8
- Compilers: PGI Server suite: pgf90, pgf77, pgcc, pgCC, pghpf
- Profiler: pgprof
- Debugger: pgdbg
- Utilities: gmake, gcc, contact the
CISL consultants for information.
- Software libraries:
- • ACML
- • netcdf-3.5.1
- • NCAR Graphics
- • LAPACK
- • SPHEREPACK
- • HDF
- Batch subsystem:
LSF
- File System: General Parallel File System (GPFS), a UNIX-style
file system that allows applications on multiple nodes to share file
data. GPFS supports very large file systems and stripes data across
multiple disks for higher performance.
More information about lightning's software environment appears in
the seminar materials for
lightning compilers
(PowerPoint slides).
Lightning is equally shared by Community Computing users and
Climate Simulation Lab users. The number of nodes available to each
group is flexible to maximize the system's productivity -- e.g.,
if one group is not using all of its available compute nodes, then
those nodes may be used by members of the other group.
Lightning is a compute server; it does not support data analysis
work.
There are three ways to use lightning:
- To use more than one processor on a node, use OpenMP threading
directives on the node, or use MPI processes on the node, or use a
mixture of both.
- To pass information between nodes, you must use MPI.
- To take full advantage of parallelism: use OpenMP threads, MPI,
or a mixture of both on the node plus use MPI between nodes.
Note that lightning has "thin" nodes with only two processors and
4 GB of shared memory per node. Because of this, your use of OpenMP
may not be as productive as it is on nodes with more processors.
CSL users may receive a lightning login as part of their CSL
allocation. See the
CSL announcement of opportunity
for details on submitting proposals.
Community Computing users who have General Accounting Unit (GAU)
allocations are eligible to apply for an account on lightning.
Community users may request a lightning login by contacting
CISL Customer
Support. Please include the following information with your
login request:
- Your login name
- Your project number
- A short description of how you intend to use lightning
Lightning access requires using a CRYPTOCard. If you do not have
an CISL-issued CRYPTOCard, you can request one by submitting an
Extraview work request or calling 303-497-1200.
You must use ssh, scp, and sftp on lightning; there are no rsh,
rcp, and ftp commands.
You will need Secure SHell (SSH) on the workstation you use to
access IBM SP-cluster systems. Information about using SSH for
computing at NCAR is described in the document "Overview of
computing at NCAR" and the relevant section is
SSH - Secure SHell.
The class (queue) structure for lightning is described in the document
Queues and charging for
resource usage on lightning.
Lightning user guide
If you have questions about this document, please contact us by any of
the methods shown here:
CISL Customer Support.
Copyright © 2004-2006. University Corporation for Atmospheric
Research. All Rights Reserved.
|
