SMT with CCSM
last update: 12/12/2006

Taking advantage of the Simultaneous Multi-Threading (SMT) on bluevista when running CCSM

With the current crunch on computing resources at NCAR, it is important for users to maximize the efficiency of both the release and development versions of the Community Climate System Model (CCSM) on bluevista. At this time, not all CCSM users may be taking advantage of the SMT capabilities of this platform, which can offer a roughly 20%-30% efficiency increase with minimal changes to the CCSM scripts and no changes to the code. We encourage users to read details in the bluevista user document section, "Using Simultaneous Multi-Threading (SMT).

Note that with SMT, the model will run with double the number of MPI tasks per node, and this can be utilized to either increase model throughput or decrease model cost significantly in most production runs. In what follows, we summarize three simple approaches to utilize SMT that apply to users of both the release and development versions of CCSM.

Post ccsm3_1_beta39 tags

We have introduced new machine support for "bluevista16", beginning with the CCSM tag, ccsm_3_1_beta39. The user should simply use this machine name instead of "bluevista" (for example, configure -mach bluevista16), and the generated scripts will automatically take advantage of SMT to maximizing throughput. If users are utilizing pre-ccsm3_1_beta39 tags, then they should follow the directions below for the release-based modifications.

Pre ccsm3_1_beta39 tags, increasing throughput

To maximize throughput of CCSM runs, utilizing SMT allows us to effectively double the number of MPI processes used and increase throughput by up to 30% at NO additional cost. The following modifications are required to files in your case directory:

  1. modify the file "env_pes.bluevista", doubling the number of MPI tasks (but not threads) assigned to each component;
  2. modify the file "env_mach.bluevista", changing the environment variable MAX_TASKS_PER_NODE from 8 to 16;
  3. after completion of the first two, reconfigure the case, using the commands 
                 configure -cleanall
             configure -mach bluevista
  4. after reconfiguring, change the lines in the run and test scripts from
    #BSUB -R "span[ptile=8]"
    to
    #BSUB -R "span[ptile=16]"

At this point, the case is ready to be submitted.

Pre ccsm3_1_beta39 tags, reducing model cost

The SMT capabilities also allow us to run using effectively the same number of MPI tasks processes but at about half the cost. There is some performance degradation, but the net cost savings should be around 25%. For this option, the following modifications are required to files in your case directory:

  1. modify the file "env_mach.bluevista", changing the environment variable MAX_TASKS_PER_NODE from 8 to 16;
  2. after completion of the first two, reconfigure the case, using the commands 
                 configure -cleanall
             configure -mach bluevista
  3. after reconfiguring, change the lines in the run and test scripts from
    #BSUB -R "span[ptile=8]"
    to
    #BSUB -R "span[ptile=16]"

At this point, the case is ready to be submitted.

Table of contents - Getting started on bluevista

If you have questions about this document, please contact SCD Customer Support. You can also reach us by telephone 24 hours a day, seven days a week at 303-497-1278. Additional contact methods: consult1@ucar.edu and during business hours in NCAR Mesa Lab Suite 39.

© Copyright 2005-2006. University Corporation for Atmospheric Research (UCAR). All Rights Reserved.

Address of this page: http://www.cisl.ucar.edu/docs/bluevista/ccsm.html