This document provides the basic information new users need
before they can start using this computer.
If you are new to computing at NCAR, you can greatly increase
your ability to work productively here by reading
Overview of computing at NCAR.
- Joint project between the University of Colorado and NCAR.
- As of August 1, 2007, Frost is also a TeraGrid computing
resource. Twenty-five percent (25%) of frost cycles are available to
all investigators with NSF allocations who wish to compute under the
TeraGrid project, sponsored by the National Science Foundation.
Each section of this document provides information needed to use
frost in either of two ways: (1) under an NCAR project, or (2) under a
TeraGrid project.
Frost is a single-rack, IBM Blue Gene/L system with 1024 compute
nodes. There is one I/O node for every 32 compute nodes, for a total
of 32 I/O nodes in the rack. Each compute node and I/O node is a
dual-core chip, containing two 700MHz PowerPC-440 CPUs, 512MB of
memory, and two floating-point units (FPUs) per core. Thus frost has a
total of 2048 processors capable of sustaining a peak performance of
5.734 trillion floating-point operations per second (TFLOPs). By
default, the compute nodes run in coprocessor mode (one processor
handles computation and the other handles communication).
In virtual-node mode, both processors on a node are available to run user code, but the processors share both the computation and the communication load.
The operating system is SuSE Linux Enterprise Server 9. A total of 6
TB of disk space is available.
The Frost Blue Gene/L system is a highly scalable platform for
developing, testing and running parallel MPI applications up to 2048
processors, and providing efficient computing for smaller job sizes.
More information about frost's system architecture appears in
Figure 1. Technical schematic of frost
BlueGene/L supercomputer
- Operating System: Linux 2.6
- Compilers: blrts_xlc, blrts_xlC, blrts_xlf, blrts_xlf90, blrts_xlf95
- Profilers: HPM, mp_profilers, Peekperf, Xprofiler, gprof, mpi_trace
- Debugger: Totalview
- Utilities: gmake, gcc, contact the
CISL consultants for more
information
- Software libraries:
- • Mass
- • netCDF
- • Parallel-netCDF
- • BLAS/LAPACK
- • ESSL
- Batch subsystem:
Cobalt
- System information commands: cqstat for queued and
running job status, cqsub for job
submission, partlist for partition information
NCAR frost users
Users accessing frost from inside the UCAR security perimeter must have
a CRYPTOCard. If you do not have a CISL-issued CRYPTOCard,
you can request one by submitting an
Extraview work request or calling 303-497-1200. Instructions
for navigating the UCAR security perimeter are provided in the document "Overview of
computing at NCAR" and the relevant section is
Accessing computing services.
You will need Secure SHell (SSH) on the workstation you use to
access CISL systems. Information about using SSH for
computing at NCAR is described in the document "Overview of
computing at NCAR" and the relevant section is
SSH - Secure SHell.
Use the "secure" commands (scp and sftp) on frost to transfer files.
TeraGrid frost users
TeraGrid users may access frost using ssh, gsissh, or
single-sign-on using certificates. Please see the section
on Interactive
access for details.
The General Parallel File System (GPFS) is installed on frost. The
scratch space (/ptmp) is GPFS. Home directories are NFS mounted.
The large amounts of data generated by executables should be
written to temporary disk in individual /ptmp directories
(/ptmp/$LOGNAME, where $LOGNAME is the user login name). You may
generate subdirectories on /ptmp and expect that /ptmp data will exist
for the duration of the user job or interactive login. Valuable data
on /ptmp should be moved to the MSS or other storage before a batch
job or user login session terminates.
The /ptmp file system is not backed up, and files are subject
to deletion if the system gets too full.
Table 1. Filesystems on Frost
| Path |
Soft Quota |
Hard Quota |
Use |
Backups |
Use |
| /home/[username] |
4MB |
16MB |
binaries |
~Monthly |
The /home filesystem is NFS mounted from the Blue Gene/L service node, and is not to be used for job data. |
| /ptmp/[username] |
600GB |
800GB |
Job data |
Never |
/ptmp is a GPFS filesystem which provides up to 600MB/s for job
I/O and job data. The filesystem is 6TB, and the quotas are only to
prevent abuse - please police your own use. Files are not scrubbed at present,
but under high use the administrative staff will contact the peak
users, and may potentially intervene. |
| /mnt/gpfs-wan |
(see SDSC) |
(see SDSC) |
Copy files to/from this data store and local resources |
|
The GPFS-WAN filesystem from SDSC is available as /mnt/gfps-wan,
but is only provided as a convenience for moving files between the
sites. Currently, running jobs out of this filesystem is not
supported. Note that we do not copy the entire set of TeraGrid DN's -
if your home directory on this filesystem shows up as being owned by
'nobody', the local set of DN's for your user account does not match
the set used for GPFS-WAN. Please open a ticket
at help@teragrid.org
to have us add the additional DN's for your account. |
Requesting an NCAR account
Regular accounts on Frost may be available to investigators who are
sponsored by one of the Co-Investigators on the grant used to purchase
Frost.
To get an account on Frost, have your sponsor send a request to
Henry Tufo, with the following information:
- First name
- Last name
- Email address
- Telephone number
- FAX number
- Institution name if it is not apparent from the email address
- If they need a NCAR cryptocard, and a FedEx address if so. It needs to be a complete mailing
address - FedEx will not accept P.O. Box or campus drop box.
This request will be reviewed and if approved will be forwarded on
to the appropriate NCAR team for account creation. You will receive
an email notifying you of your new account, and a document
providing an overview of Frost will be attached. A few days later you
should receive your cryptocard at the FedEx address provided in the
request. At this point you will be able to log in and use the
system.
Requesting a TeraGrid account
Information on requesting a TeraGrid account is given at:
http://www.teragrid.org/userinfo/access/allocations.php
Roaming accounts are not automatically enabled. In order to access the
Frost system, please send email to help@teragrid.org requesting that
your roaming account be enabled.
Setting up your NCAR environment
New users are placed in the bash shell by default. The csh,
ksh, and a few other less common shells (see /etc/shells) are also
available.
Basic shell startup files are provided for all users at the time
of account creation. To see the default environment for your login,
type:
printenv
when you are logged in.
If you have questions about the locations of libraries or other
software, contact CISL Consulting Services at
consult1@ucar.edu or
303-497-1249.
Changing your NCAR password and shell
NCAR users' password will be in sync with the UCAR Central
Authentication Services (UCAS) password. If you desire to change your
UCAR password, go to:
http://www.ucar.edu/csac/ssh-auth-password-help.html
Please
see Password
guidelines and instructions for changing passwords on NCAR
computers.
If you want to change your shell:
- Log on to frost
- chsh -s shell
Setting up up your TeraGrid environment - CTSS
Your TeraGrid software environment is set up to use the Coordinated
TeraGrid Software and Services (CTSS) software stack.
The following CTSSv4 kits are available in production:
- Core
- Remote Login
- Remote Compute
- Data Movement
- Application Development and Runtime Support
- Parallel Application
- Science Workflow Support
The following CTSSv4 kits will not be supported on this resource:
- Data Management
- Visualization Support
SoftEnv is a tool that makes it easy for users to modify Unix shell
environment variables as necessary to make software available. To
simplify the set up of your environment, SoftEnv uses keys that
represent individual software packages or suites of related
packages. All CTSS software may be accessed using SoftEnv keys. Your
default login software is determined by the SoftEnv keys listed in the
.soft file in your home directory. To see your environment settings,
type softenv. See
http://www.teragrid.org/userinfo/jobs/softenv.php
for information on how to use SoftEnv.
Charges for NCAR accounts
There are currently no charges for frost for regular NCAR accounts.
Charges for TeraGrid accounts
The TeraGrid uses a charging unit called a Service Unit
(SU). TeraGrid users with startup (DAC) accounts have a limited amount of
resources for running interactive jobs and are not charged. MRAC
(medium) or LRAC (large) allocation usage is charged in SUs.
The charging formula for TeraGrid projects that run on frost
is:
queue factor * machine factor * allocated cpus * wallclock time in hours
Currently the machine factor is one. The queue factors are all one.
You can view your SU usage via the TeraGrid portal web site at
https://portal.teragrid.org/gridsphere
Log in to the
TeraGrid portal using your portal login and password and go to the "My
TeraGrid" tab.
NCAR frost users
It is possible to use scp/sftp to transfer files between
frost and other computers without using key-based authentication.
However, it is much more convenient to have your SSH
public key installed to access and transfer files between
frost and other machines. Following is the method for installing keys.
- If you have not created an SSH key on your machine you
may create it by giving the command:
You may elect not to add a passphrase; just hit enter when prompted
for a passphrase. If you add a passphrase, then every invocation of
ssh/scp will ask you to key in the passphrase. We strongly suggest
creating SSH keys with passphrases and using ssh-agent to avoid
repeated prompts.
- Now copy the contents of ~/.ssh/id_rsa.pub and
append it to the file on frost ~/.ssh/authorized_keys.
Make sure your ~/.ssh file is writable only by you.
Once this is done you may transfer files from/to
your machine to/from frost by initiating the transfers
from your machine. Examples:
While logged on your local machine:
To transfer a file from your machine to frost:
scp file your_user_id@frost.ncar.teragrid.org:/ptmp/$LOGNAME/file
To get a file from frost to your machine:
scp your_user_id@frost.ncar.teragrid.org:/ptmp/$LOGNAME/file file
If your site accepts incoming scp requests to your machine,
then you may install your ssh key generated on frost onto your
your local machine and be able to initiate transfers from frost.
For TeraGrid users
If you want to transfer files between two TeraGrid resources,
we recommend using GridFTP, as described at:
http://www.teragrid.org/userinfo/data/gridftp.php
If you want to transfer files between your machine and a TeraGrid
machine, we recommend that you:
- install MyProxy and GSISSH (part of the Globus toolkit) on your
machine (Linux/Unix/Mac and through Portal and Java based in windows)
For details please
consult http://www.teragrid.org/userinfo/access/tgsso_native.php
- get a single-signon proxy certificate using the myproxy-logon method described in the document above.
To transfer files, use the following example commands:
To transfer files from your desktop to frost:
gsiscp file your_login@frost.ncar.teragrid.org:file
To transfer files from frost to your machine:
gsiscp your_login@frost.ncar.teragrid.org:file file
© Copyright 2004-2007. University Corporation for Atmospheric
Research (UCAR). All Rights Reserved.
