Blackforest, bluesky users: Recompile and check results

Blackforest

During the course of validation testing of the bluesky expansion, a programming environment problem was discovered that causes correct programs to generate incorrect results. Subsequent testing of blackforest also revealed the problem to be extant there as well. Thus far, babyblue shows no evidence of problems on the test cases.

Description

The problem was first noticed in codes that use a combination of MPI and OpenMP and later isolated to the use of OpenMP. The test codes never crashed, but rather produced results that were subtly incorrect. Evidence points to the problem probably existing within the runtime environment rather than within the compiler itself. It is believed that only codes that use OpenMP were at risk, though all users should exercise caution and check their results.

The window of exposure for this failure can be bracketed by the TCG validation tests which are typically performed before and after all announced system changes.

Blackforest

Blackforest last passed the validation suite on 22 July 2003, but tests on 17 October showed blackforest failing the validation suite. After the XLF 8.1.1.0 compiler and runtime environment upgrade on 21 October, blackforest again passed the validation suite.

Several files relating to the runtime environment were changed on 21 August 2003. The validation suite was not run at this time and it is not possible to determine if this was the point at which the problem was introduced. As previously stated, the compiler and runtime environment upgrade on 21 October appears to fix the problem.

Bluesky

Bluesky was known to last pass the validation suite on 9 October 2003 immediately prior to the bluesky expansion. However, it currently fails the validation suite (possibly due to cross-contamination with blackforest during the course of the expansion).

The compiler on bluesky will be upgraded to XLF 8.1.1.0 on the morning of 27 October. It is expected that once the upgrade is complete, it will also resolve the problems on bluesky. A downtime will not be taken for this upgrade—rather it will be a "rolling upgrade" between 0600 and 1200, and users may encounter compile problems, job failures, and/or incorrect results during the course of the upgrade. Users should refrain from using the compiler during the upgrade window.

Recommendations

• SCD suggests users scrutinize blackforest results computed between 22 July and 21 October 2003 with special attention to results computed after 21 August.
• SCD suggests users carefully scrutinize bluesky results computed between 17 October and 27 October 2003.
• SCD also strongly recommends that all codes be recompiled using the new compiler and runtime environment to eliminate the possibility of problematic interactions with the previous compiler version. Blackforest codes may be recompiled immediately, and bluesky codes should be recompiled after 1200 on 27 October 2003.