Getting started on bluevista
last update:
06/12/2006
IBM supplies some high-performance math libraries that are particularly efficient on IBM SP-cluster systems: ESSL, PESSL, and MASS. The routines in ESSL and PESSL are threaded to run more efficiently on IBM SP-cluster systems nodes; MASS routines are single-threaded. ESSL is designed to be run on a single process, and PESSL is for math routines running across multiple parallel processes.
SCD also maintains a large collection of single-threaded math and statistics libraries useful to computational work in the geosciences. Note that these local libraries have not been designed for multiple-thread usage, so you should use them only in single-threaded codes, and you should not consider them to be thread safe.
To maximize your job's performance on IBM SP-cluster systems, SCD recommends that you use the IBM-supplied library MASS.
IBM's Mathematical Acceleration SubSystem (MASS) provides thread-safe scalar and vector libraries of tuned mathematical intrinsic functions. You access these libraries on IBM SP-cluster systems using the -lmass and -lmassv options at compile time. The IBM website Mathematical Acceleration SubSystem (MASS) MASS Version 2.7 provides details about the libraries, including performance and accuracy information and some usage instructions.
Compilation note: When compiling with -qstrict, the mass library will not automatically be invoked, since -qstrict prevents the compiler from reordering arithmetic operations for performance, which includes preventing automatic linking of mass. However, if mass is explicitly linked as a second step after the program has been compiled, the tuned functions in mass will be used.
You can access IBM's ESSL (IBM Engineering and Scientific Subroutine Library for AIX) and PESSL (IBM Parallel Engineering and Scientific Subroutine Library for AIX) on IBM SP-cluster systems. ESSL provides a variety of distributed-memory parallel-processing subroutines in key math areas; it is designed to be run on a single process. PESSL is for math routines running across multiple parallel processes. The routines in both ESSL and PESSL are threaded to optimize their performance as they run on IBM SP-cluster systems. These routines can significantly improve the performance of your codes that use Fast Fourier Transforms and linear algebra functions. They may be linked using the -lessl or -lpessl options at compile time. IBM offers documentation for both ESSL and PESSL at their Engineering and Scientific Subroutine Library web page.
Note: While the BLAS and LAPACK libraries may be available to support some packages, we recommend using the equivalent routines in ESSL, since they give much better performance.
NCAR's collection of math and statistics libraries is made available to IBM SP-cluster systems users by request. You can contact the SCD Consulting Office at consult1@ucar.edu or call 303-497-1278 to discuss your needs and request one of the libraries listed on NCAR's Mathematical and Statistical Libraries Overview page.
On bluevista, the default address size is 64-bit. Because most SCD systems are evolving in this direction, local libraries on bluevista have been built with 64-bit addressing and placed in the directory:
/usr/local/lib
- The directory /usr/local/lib/r4i4 provides 64-bit routines that use 4-byte reals and 4-byte integers.
- The directory /usr/local/lib/r8i4 provides 64-bit routines that use 8-byte reals and 4-byte integers.
To use local libraries, you need to use the -L and -l options in your compiler calls as shown in the examples provided in the individual descriptions below. Note that these local libraries have not been designed for multiple-thread usage, so you should use them in single-threaded code and should not consider them to be thread safe.
Note: 32-bit versions of libraries will only be built by special request. Contact the SCD Consulting Office at consult1@ucar.edu or call 303-497-1278 to discuss your needs.
The supported data-manipulation libraries provided for IBM SP-cluster systems users include netCDF (an interface, library, and data format for accessing and representing array-oriented scientific data), NCL (interactive data manipulation for specifying visualizations), MSS (an API and commands for archival storage operations), and NCO (netCDF Operators for manipulating netCDF datasets).
netCDF
netCDF (network Common Data Format) is an interface for array-oriented data access and a library that provides an implementation of the interface. The netCDF library also defines a machine-independent format for representing scientific data. Together, the interface, library, and format support the creation, access, and sharing of scientific data. The freely available source can be obtained from Unidata's netCDF website.
Using netCDF with codes that use 8-byte reals by default requires special attention to the data type sizes. Contact the netCDF help desk (support@unidata.ucar.edu) if you need assistance with this.
Here are some examples of how to link to the netCDF libraries:
xlf -qrealsize=8 file.f -L/usr/local/lib/r4i4 -lnetcdf xlf -q64 file.f -L/usr/local/lib/r4i4 -lnetcdfNCL
The NCAR Command Language (NCL) is an interactive, text-based language interpreter that helps you manipulate data and create NCAR Graphics images. NCL allows you to import and export data in a variety of formats (ASCII, binary, HDF, netCDF) and to interactively manipulate that data. Interactive data manipulation includes selecting region-of-interest data, thinning data, interpolating irregular grids to regular grids, removing corrupted data, and performing useful transformations. You can also customize the methods NCL uses to ingest and manipulate data. As a tool for specifying visualizations, NCL allows you to generate 1D and 2D NCAR Graphics plots in many styles with many options for annotating, labeling, and animating the visualizations. The NCL user manual provides thorough usage instructions and examples.
MSS
The MSS library on IBM SP-cluster systems provides Fortran and C APIs and commands for data storage operations on NCAR's Mass Storage System (MSS). The libmss.a subroutines allows users to perform MSS operations in their programs such as "call mswrite(...)". To link in the MSS library on your compile-load line, use the option -lmss. IBM SP-cluster systems use the DCE command interface to the MSS. When moving MSS data to or from IBM SP-cluster systems, you should use the MSS commands documented at Introduction to NCAR's Mass Storage System (MSS) and Quick reference: MSS commands by function. You may also reference the man pages on IBM SP-cluster systems for all MSS commands.
MPI, the Message Passing Interface, is a core set of library routines useful to a wide range of users of parallel computers. The MPI libraries are loaded by default when you compile your code with a compiler that has an "mp" prefix on the compiler name. We recommend that you always use the _r suffix version of the compilers, e.g. mpxlf90_r, to avoid unsatisfied externals caused by certain assumptions made by the compilers.
On bluevista, to load the 64-bit MPI library, specify compiler option -q64 or set environment variable OBJECT_MODE to 64. We recommend setting OBJECT_MODE to 64 (set the environment variable rather than specifying the compiler option) to ensure that compiler, loader, and library commands (such as "ld", "nm", and "ar") work as expected. Otherwise, each of these commands requires a special flag for use with 64-bit objects. If the flag is not specified and the object is 64-bit, the command treats the object as null.
A C shell example of setting the OBJECT_MODE environment variable to 64 would be: "setenv OBJECT_MODE 64".
The NCAR Graphics libraries and NCAR Command Language (NCL) data manipulation tools are available on IBM SP-cluster systems. In order to use NCAR Graphics or NCL, set your NCARG_ROOT environment variable to /usr/local.
The ncl command is under /usr/local/bin.
For NCAR Graphics, you must select the mathematical precision version of the NCAR Graphics libraries that you want, choosing one of the following:
setenv NCARG_LIB /usr/local/lib/r4i4 #single precision, 64-bit addressingorsetenv NCARG_LIB /usr/local/lib/r8i4 #double precision, 64-bit addressingFor NCL, you can only use the single-precision library paths. We recommend setting these in your dotfiles (.cshrc or .profile).
For more information, see NCAR Graphics and NCL.
NCARU provides efficient routines for packing and unpacking data and for converting Cray-formmatted data to IEEE format. This library also does Fortran-callable data packing. Guidance for using NCARU is provided in the document Converting Cray-style datasets for use on non-Cray computers.
Note: libncaru.a requires 8-byte reals. Because this library is no longer supported, please contact the SCD Consultants if you need this software. They will provide you a copy of the source code "as is."
Next page | Table of contents - Getting started on bluevista
If you have questions about this document, please contact SCD Customer Support. You can also reach us by telephone 24 hours a day, seven days a week at 303-497-1278. Additional contact methods: consult1@ucar.edu and during business hours in NCAR Mesa Lab Suite 39.
© Copyright 2005. University Corporation for Atmospheric Research (UCAR). All Rights Reserved.
Address of this page: http://www.scd.ucar.edu/docs/bluevista/lib.html