Getting started on blueice
last update:
03/27/2007
This document is provided for NCAR supercomputer users who are involved in moving applications from systems installed with 32-bit address space limitations, if those applications actively manage memory in the course of computations.
CISL/SCD released the lightning linux cluster system to production status on March 1, 2005; this was a large step toward the adoption of a pure 64-bit computational environment for users of NCAR's supercomputers ( http://www.cisl.ucar.edu/docs/lightning/cbr.jsp). This marked a departure from the practice of supporting both 32- and 64-bit environments within the same system, as was the case for blackforest and bluesky (decommissioned January 2005 and February 2007, respectively). CISL/SCD's newer systems, an IBM SP Power5 cluster named bluevista released in October 2006, and an IBM SP POWER5 cluster named blueice released in January 2007, were both installed as pure 64-bit systems. This means that you will find more and better support on the newer systems, including libraries, for 64-bit APIs than for 32-bit APIs.
CISL encourages its customers to convert to 64-bit APIs, although we realize that some of the older applications may be locked into 32-bit by virtue of software constructs chosen during development.
In describing a computer system, "32-bit" and "64-bit" are adjectives that might apply variously to the amount of circuitry available for the representation of integer and floating-point data or as determining the range of memory addresses that a CPU can access (232 vs. 264, 4 gigabytes vs. 16 exabytes). The terminology can also be used to describe a CPU's register size, the width of its external data bus, or even the size of (not the number of) its binary instructions.
These factors distinguish the various processor technologies between and even within manufacturers, and it is indeed possible to have features of both within the same CPU, making the 32-bit/64-bit distinction anything but definitive. However, in general discussions, a processor architecture is described as "64-bit" if it uses integer registers that are 64 bits wide and thus directly supports dealing — both internally (address bus) and externally (data bus) — with data 64 bits at a time.
The NCAR systems noted above can support a 32-bit and a 64-bit user environment simultaneously. This means that applications, object modules, and object libraries can be created and executed on 64-bit hardware that either fully exploit hardware features available or some subset.
The 64-bit/32-bit characteristic of the user environment is determined by the OBJECT_MODE environmental variable on bluesky and blueice, which are AIX systems. On these systems, if OBJECT_MODE has no defined value or has been assigned a value of 32, a user will generate 32-bit object modules during compilation, 32-bit libraries during object file archival, and 32-bit executables during link processing. On bluesky, OBJECT_MODE is not defined during the user login process, so 32-bit processing is performed (by default) unless the user specifies otherwise. In contrast, by the time that a user receives a command-line prompt on blueice, OBJECT_MODE has been set to 64. Batch jobs on both systems follow the same rules respectively if compilation and so on is performed as part of the batch job.
User environment settings on AIX systems that determine the size of the address space accessible to object modules, libraries, and executables can be overridden at compile time through the use of -q32 to set a 32-bit address space or -q64 to set a 64-bit address space.
On lightning (running the SUSE Linux OS), object mode is determined via the choice of compilers and/or compiler options as above. See the compiler man page for options that affect this setting.
Note that even while both 32-bit and 64-bit capabilities might be supported by the same system, it is certain that 64-bit executables must be built from object modules and/or libraries that are generated purely as 64-bit. The same is true for 32-bit processing. Attempting to mix OBJECT_MODEs will not work and will be flagged as an error at link time. Linker bit-mode options must exactly match the characteristics of the object modules and libraries being linked.
Address space: 64-bit applications can address 16 exabytes of data, which is vastly larger than the 4 gigabytes addressable for 32-bit applications. This is especially important for scientific applications and for clustered Symmetric MultiProcessing (SMP) systems such as blueice, bluesky, and lightning. Each of these systems provides 2 GB of memory per processor, but the number of processors per node varies from two (for lightning) up to 32 (for bluesky). There are eight processors per node on blueice. A serial 32-bit application, running on a dedicated node, can utilize all (minus OS requirements) of the memory on a lightning node, but only 1/8th of the total memory on a 32-processor bluesky node. A 64-bit application would be able to share all 64 GB on a 32-processor bluesky node or all 16 GB on a blueice node via OpenMP threads.
Numerical accuracy: 64-bit applications, if converted with no code changes from 32-bit applications, are not more accurate in numerical computation. Numerical accuracy is controlled by data type declarations and/or compiler options. It is possible to upgrade default type declarations very easily without massive code changes by utilizing the compiler flags such as -qintsize or -qrealsize. The -qautodbl flag provides a way to perform all floating point arithmetic in double precision without changing data types.
Program size: With the exception of pointer-type variables, data types in 64-bit applications are the same size as those used in 32-bit applications. For scientific applications, this makes the memory footprint of a 64-bit application larger only by the degree to which it relies on pointers. Generally, the memory increase induced by larger pointers is marginal and need not be a concern for scientific applications. The greatest impact on program size normally would arise from a blanket promotion of real or integer (or both) variable types by compiler options to extended precision. But in light of the failure of such redeclarations to improve numerical accuracy, there is little reason to pursue this. It may, in fact, adversely impact application performance. Larger memory requirements can have an impact on data allocation patterns in memory and any or all levels of data cache. Increasing the presence of data in cache can increase access times and slow down the rate of computation. An optimizing complier can do much to reduce this problem but may not always be completely successful.
Faster integer operations: Compilation in 64-bit mode activates certain 8-byte integer hardware instructions on processors utilizing PowerPC technology that will speed up integer computations. Internal to the code, this feature is accessed by operations involving 8-byte integers. The default for Fortran is 4 bytes, so either explicit declaration is required or -qintsize=8 can be specified as a complier option. OBJECT_MODE=64 or -q64 will not activate this change for integer variables. If -qintsize=8 is used, all integer variables will be 8 bytes in length. This approach can cause program size to increase and create possible impacts to cache performance as discussed above, so testing these options is needed to determine any overall performance gain or loss.
Loop indices: Since xlf 6.1 (AIX systems), loop indices declared as INTEGER, INTEGER(1), or INTEGER(4) have been promoted to INTEGER(8) by the compiler unless -qstrict_induction is specified. The use of -qintsize=8 is no longer necessary to achieve this optimization that can save program memory.
All C fundamental types other than long and pointer will be the same size in both 32-bit and 64-bit compilation modes. Inter-language calls (C calling Fortran or Fortran calling C) will work if data types are consistent across the call and the return. The call across languages will also require attention to the data-passing mechanism; whether it is pass-by-value or pass-by-reference. The following lists of data types that change with object mode for Fortran and C should be a good guide:
C type FORTRAN 32-bit FORTRAN 64-bit unsigned int INTEGER INTEGER long INTEGER INTEGER*8 signed long INTEGER INTEGER*8 unsigned long LOGICAL LOGICAL*8 long long INTEGER*8 INTEGER*8 signed long long INTEGER*8 INTEGER*8 unsigned long long LOGICAL*8 LOGICAL *8 pointer INTEGER INTEGER*8 ANSI C standard header files: The current set of standard C ( http://www.open-std.org/jtc1/sc22/wg14/) header files is:
<assert.h> <complex.h> <ctype.h> <errno.h> <fenv.h> <float.h> <inttypes.h> <iso646.h> <limits.h> <locale.h> <math.h> <setjmp.h> <signal.h> <stdarg.h> <stdbool.h> <stddef.h> <stdint.h> <stdio.h> <stdlib.h> <string.h> <tgmath.h> <time.h> <wchar.h> <wctype.h> C code that uses any header files from the system besides these should be brought into conformance with the ANSI C standard. In the conversion of legacy 32-bit applications to 64-bit addressing, it is not unusual to find that when malloc is being used, <malloc.h> is included but not <stdlib.h>. The inclusion of <stdlib.h> supersedes <malloc.h> and may be used to replace it. Please check the Unix man page for malloc for function descriptions as well as the headers that need to be included. Failure to include the required headers may lead to unpredictable results, including segmentation faults.
Fortran loc function: In 64-bit mode, the Fortran loc function returns an INTEGER*8 value. Code that assumes a return value with the default integer type may experience the same problems as indicated for <malloc.h> above since the return value will be truncated to INTEGER*4.
There are two categories of migration tools: compiler options and utilities.
The complier options most useful in debugging 32-bit-to-64-bit migration problems appear here. (Consult the xlf User's Guide for more complete information.)
-qwarn64 aids in porting code from a 32-bit environment to a 64-bit environment by detecting the truncation of an 8-byte integer pointer to 4 bytes. The -qwarn64 option uses informational messages to identify statements that may cause problems with the 32-bit-to-64-bit migration.
-qextchk sets up type-checking information for common blocks, procedure definitions, procedure references, and module data. Later, the linker can detect mismatches across compilation units by using this information.
-qinitauto initializes each byte or word (4 bytes) of storage for automatic variables to a specific value, depending on the length of the hex_value provided. This helps locate variables that are referenced before being defined. For example, by using both the -qinitauto option to initialize REAL variables with NaNs value (for example -qinitauto=FF911299) and the -qflttrap option, it is possible to identify references to uninitialized REAL variables at run time.
An AIX utility that might prove useful in the migration process is lint. The lint command checks C and C++ language source code for coding and syntax errors and for inefficient or non-portable code. Use lint to:
- Identify source code and library incompatibility.
- Enforce type-checking rules more strictly than does the compiler.
- Identify potential problems with variables.
- Identify potential problems with functions.
- Identify problems with flow control.
- Identify legal constructions that may produce errors or be inefficient.
- Identify unused variable and function declarations.
- Identify possibly non-portable code.
Tutorial Notes: WRF Software 2.0, John Michalakes, June 29, 2004
Tutorial Notes: WRF Software 2.1 Part I, by John Michalakes, 7/27/2005 (Real Media file)
Tutorial Notes: WRF Software 2.1 Part II, by John Michalakes, 7/27/2005 (Real Media file)
The IBM SP at the University of Minnesota Supercomputing Institute
IBM SP Systems Overview
Selecting the appropriate IBM SP-cluster system for your job
NCAR acquires an IBM Linux cluster for community model development
Lightning charging began March 1, 2005
How to run a large-memory job
Message passing libraries: 32-bit and 64-bit MPI
blueice main page
64-bit, from Wikipedia, the free encyclopedia
32-bit to 64-bit Migration Considerations
General Programming Concepts: Writing and Debugging ProgramsIBM Publications:
User's Guide, XL Fortran for AIX, Version 8 Release 1
C for AIX User's GuideIBM Redbooks:
RS/6000 Scientific and Technical Computing: POWER3 Introduction and Tuning Guide, SG24-5155
AIX 64-bit Performance in Focus, SG245103
AIX 5L Porting Guide, SG246034
Next page | Table of contents - Getting started on blueice
If you have questions about this document, please contact us via any of the methods (phone, email, ticket, or in person) described here: CISL Customer Support.
© Copyright 2007. University Corporation for Atmospheric Research (UCAR). All Rights Reserved.
Address of this page: http://www.cisl.ucar.edu/docs/blueice/index.html