Overview of parallel programming for CISL computers

Last updated: 09/21/2009

CISL's production computing platforms are clusters of symmetric multiprocessor (SMP) computers. While you can run single-processor jobs on these cluster systems, compute-intensive tasks require you to use multiple processors. The cluster systems at NCAR have many nodes with 32 processors per node.

To exploit the power of cluster systems, programmers combine many processors and may also combine multiple compute nodes.

This is called parallel programming.

Parallel programmming has two aspects:

• Using parallel threads on an SMP compute node
• Using message passing between compute nodes. Message passing is used to exchange data between nodes.

Parallel programs are called "hybrid" when they use both threads and message passing.

Threads run multiple processors within a node using OpenMP directives. The Supercomputing98 (SC98) conference featured an excellent OpenMP tutorial.

Note: Many vendors supply their own proprietary thread package that is often integrated with the compiler (such as Cray's !MIC$ directives for autotasking). However, CISL recommends that you use OpenMP since it is the POSIX thread standard and is available on all systems.

Message passing exchanges data between multiple nodes using MPI, the Message Passing Interface. Argonne National Laboratory provides a good selection of MPI tutorials.

Note: Many vendors supply their own proprietary message passing library that is often tuned to their architecture. However, CISL advises you to use the POSIX-standard message-passing library -- MPI -- for which efficient implementations are provided on each vendor's platform.

Book introduces parallel programming in detail

If you are not familiar with parallel programming, Ian Foster's book Designing and Building Parallel Programs provides an excellent introduction.

CISL documentation