This document sets out SCD's policies and procedures for user-maintained software repositories that are shared on a supercomputer in a /contrib directory. SCD provides storage space for software in the /contrib directory, but the users who have permission to install it there are responsible for it.
With the availability of open source licensed software packages for a broad range of purposes, there has been a great deal of demand to have such packages made available on the computers in NCAR's supercomputing facility. There are tens of thousands of such projects hosted on sourceforge.net alone, and the volume of software packages in which NCAR's user community expresses interest far exceeds SCD's support and maintenance capabilities (especially since user interest in any individual package tends to be rather limited). While SCD can provide storage and infrastructural support for people who want to install, maintain, and support such software, SCD cannot provide maintenance or support for such packages, and SCD does not make any representations with regard to the safety, usability, or longevity of such software packages.
If you are interested in contributing software to a supercomputer at NCAR, contact the SCD Consulting Office (consult1@ucar.edu). Write permission to the SCD-supplied storage will be granted to users who conform to the guidelines in this document and are willing to agree to the conditions expressed herein. If these guidelines are violated, write permission to the /contrib directory will be revoked.
These guidelines are intended to denote the minimum expectations for acceptable use of installation privileges, in accordance with the UCAR Policies and Procedures Manual, specifically the section Access to and Use of Computer and Information Systems, and the UCAR Computer Security Advisory Committee (CSAC) Security standards for exposed hosts. (If you are outside the UCAR security perimeter, you will not be able to follow the link to "Security standards for exposed hosts;" please contact the SCD Consulting Office for assistance.)
SCD therefore poses the following guidelines for managing contributed software:
You must have permission from your management chain to undertake this activity. You might define your management chain to include your supervisor or section head, your Principal Investigator, or your advisor, as appropriate.
The software installed on /contrib should be of general use to the user community, and packages for personal use only should not be installed on /contrib. Consideration for install privileges will be given to users who support software packages that are applicable to a broader group of users.
The SCD supercomputing facility is used by about 1,000 users from NCAR and across the country. Because installed software has the potential to impact the productivity of such a large community of researchers, SCD will make installation privileges available only to people who have experience installing and supporting software for use by others. People who are not experienced in this area should seek to develop their expertise at lower-risk venues before requesting contribution privileges in this supercomputing facility.
If a software package you have installed is found to contain a security flaw, you are expected to remove the package or patch the security flaw in a timely fashion as defined by the CSAC guidelines for exposed hosts.
You are responsible for ensuring compliance with any and all licensing terms for the software you install. (Refer to the Access to and Use of Computer and Information Systems section of the UCAR Policies and Procedures Manual regarding unlicensed software.)
The Access to and Use of Computer and Information Systems section of the UCAR Policies and Procedures Manual and the CSAC services standards outline UCAR's definition of appropriate use of computing resources. The software you install should not exceed the limits imposed by UCAR's policies.
You should prepare .pkg and .softdb files for each package you install. The files should be placed in the top-level directory for the package. For more information on the contents of these files, see the Installation procedures section of this document.
Per UCAR policy, SCD reserves the right to disable or remove software with or without advance notice if is notified of or discovers that the software or the installation poses undocumented changes or risks to users, poses security risks, poses licensing issues, or is otherwise in violation of UCAR's appropriate use policy. This disabling or removal shall stay in effect until the issues are resolved.
Moreover, should space on /contrib become scarce, packages with small user bases or packages that have not been used recently may be removed to free space. A good-faith effort to notify the package maintainer will be made in advance of such removals.
A /contrib partition is provided on each of the IBM and SGI supercomputers.
All software installed under /contrib is contained in subdirectories of /contrib according to the name of the software package and its version. An example would be /contrib/emacs-19.29. An individual package belongs to the person who installed it. Other members of the /contrib group are not allowed to remove or modify packages they did not install themselves.
Inside each package directory, will be a package.softdb file and a package.pkg file. The package.softdb file documents who installed the package, how it was installed, contains usage tips and lists what, if any, support they are willing to provide. The package.pkg file documents what changes must be made to your environment to run the software out of that directory. It may be possible to use the pkg file to directly modify your environment, but that is not currently provided.
Using the example above, the /contrib/emacs-19.29 directory would contain the files, "emacs-19.29.softdb" and "emacs-19.29.pkg".
Here is a directory listing for an example directory:
/contrib /contrib/tk4.0 /contrib/tk4.0/tk4.0.softdb /contrib/tk4.0/tk4.0.pkg /contrib/tk4.0/bin /contrib/tk4.0/etc /contrib/tk4.0/lib /contrib/tk4.0/man /contrib/tk4.0/man/man1 /contrib/tk4.0/man/man3 /contrib/tk4.0/man/mann
There are two ways users can usually add contributed software to their environment:
Please note that it is up to the installer to decide whether the software is linked into the /contrib/{bin,lib,man} directories, so in some cases the second option is the only one available.
Because we anticipate a large amount of contributed software, modifying the PATH, MANPATH, and LD_LIBRARY_PATH for each package will likely be unwieldy and eventually unworkable. We have therefore provided /contrib/{bin,man,lib} directories. If you add these directories to your PATH, MANPATH, and LD_LIBRARY_PATH, you will have access to all of the symlinked contrib software. You can do this by modifying your .cshrc as follows:
setenv PATH /contrib/bin:${PATH}
setenv MANPATH /contrib/man:${MANPATH}
setenv LD_LIBRARY_PATH /contrib/lib:${LD_LIBRARY_PATH}
Note that this means you will always get all of the contributed software that has been linked into the /contrib directories. If you want to select some software packages while not loading others, you need to add each package individually.
Every software package has its own contrib subdirectory (see the Structure section of this document for information on /contrib structure). The subdirectory will contain a file named package.softdb. For example, the package pvm3.3.10.a has the file pvm3.3.10.a.softdb. Before adding the package to your environment, you should scan this file for information, particularly the "Notes" section because it may contain information on how to run the software that will be useful to you.
In addition, there will be a .pkg file in the directory as well. The instructions contained in this file will be specific to the particular package. For example, if someone had installed version 3.2 of a software package from my favorite fruit smoothy company in Boulder, the installer of the "jamba3.2" package should have added a file named "jamba3.2.pkg" to the directory /contrib/jamba3.2 that contains the following usage instructions:
set JAMBAROOT = "/contrib/jamba3.2"
prepend ${JAMBAROOT}/bin to PATH
prepend ${JAMBAROOT}/man to MANPATH
set JAMBAHOST = blackforest.ucar.edu
This is all the information you need to add the jamba3.2 package to your environment. Thus, you need to modify your .cshrc file as follows (per the instructions) to use the jamba3.2 package:
setenv JAMBAROOT /contrib/jamba3.2
setenv PATH ( /contrib/jamba3.2/bin /usr/bin /usr)
setenv MANPATH /contrib/jamba3.2/man:/usr/local/man:/usr/man
setenv JAMBAHOST blackforest.ucar.edu
When you have completed all of the preceding steps, log out and log back in. You should find the man pages and the binaries for the package without any problem.
Details on how to perform each of these steps follow.
The space available under /contrib is limited. To allow the maximum amount of software to be installed, it is important to put your source code elsewhere. We recommend that you save a compressed tarfile of your source directory to the Mass Storage System when your installation is done so that you can retrieve it easily if you want.
Note that even without source code, some packages are very large. If you do not have enough room under /contrib to install your package, you can contact the SCD Technical Consulting Group at consult1@ucar.edu. However, disk space on the supercomputers is limited, so it is unlikely that space can be found unless other software packages can be removed first.
First, make sure you create the proper directory on the supercomputer's /contrib. As a member of the contrib group, you have the right to create files under /contrib. However, once your subdirectory is created, you will own it, and the group ownership of the directory will match your normal group ownership. This will prevent other members of the contrib group from removing or modifying your package.
The directory you create should contain the name of the software and the version. The exact format used is usually adopted from the software itself. Acceptable examples include "emacs-19.29", "pvm2.4", "elm2.4pl24", and more.
Under some operating systems, your addition to a new group may not be recognized, depending on the number of groups to which you already belong. In this case, it may be necessary for you to type "newgrp contrib_group_name" before creating your top-level directory. We recommend that after creating the top-level directory, you change back to your default group (by running newgrp without arguments) and change the group ownership of the directory back to your normal group, to prevent it from being accidentally removed by another member of the contrib group. Please see the Permissions section of this document for more information.
It is important that the directory you've created and all the files under it, not be owned by the contrib group, and in general, not have group write permission. This will prevent someone from accidentally deleting your software. As soon as you've created your directory, use "ls -ld" to check the permissions on it. It should look like this:
prompt> ls -ldg test-version.release/ drwxr-xr-x 2 tomshane ncar 512 Feb 5 11:32 test-version.release//
Note that the group ownership of the directory is not contrib, there is no write permission for group, and the setgid bit is not set. If the setgid bit is set, the permissions will read "drwxr-sr-x".
By ensuring at this time that the group ownership and permissions on the directory are correctly set should prevent problems later on as you compile and install your software. However, you may want to check the group ownership and permissions on all files and subdirectories later, after your installation is done, to make sure no problem has been introduced.
All files required by your package must go into the /contrib/package directory!
Follow the instructions for your software and modify it as necessary to make sure it does not try to install under /usr/local, /opt, /pkg, or any other non-contrib directory. Further, it should not install files directly into /contrib/bin, /contrib/lib, or /contrib/man.
If you have tailored the package correctly, you should be able to compile it and direct the software to install under the /contrib/{package-version} directory. A good test to make sure that you do not have any remaining dependencies on your source directory is to rename your source directory and test the binaries under /contrib to make sure they work.
The softdb file should contain the following sections:
In the "Other Notes" section, include any tips people will need to run the program and what, if any, support you are willing to provide, e.g., whether or not you are willing to a) answer questions, b) receive bug reports, or c) neither.
Copy the file to the proper name and location. The full name of the package should be used. This will make it easier for the Systems Staff to provide scripts that list all the currently available contrib software, for example. The proper file name is "/contrib/package/package.softdb. An example would be "/contrib/emacs-19.29/emacs-19.29.softdb".
Pkg files are kept in a simple format to document the environment changes that are needed to use the package. In the following example, the package file software requires setting an environment variable, adding the directory to the user's PATH, and modifying their MANPATH and LD_LIBRARY_PATH environment variables.
Contact the SCD Consulting Office at consult1@ucar.edu if you want assistance with creating .pkg files.
No specific steps are required here, but we strongly recommend that you test your software before announcing its availability!
A special version of slink has been provided to allow contrib software to be slinked into the paths /contrib/bin, /contrib/man, /contrib/lib, /contrib/include, etc. You must use the slink script that is stored in /contrib/slink. Be careful! In this area, you have the ability to wipe out symbolic links created by other people, or to wipe out the history file, slink.conf, with potentially nasty consequences, so please be careful!
Here is an example of what needs to be done to use /contrib/slink to link the gs3.53 package into /contrib/{bin,lib,man}. Example: slinking /contrib/gs3.53:
newgrp aixcntrb (on IBM systems) or newgrp irxcntrb (on SGI systems)
co -l slink.conf
The slink.conf file should now be owned by you and have 644 permissions.
If you ever try to check out the file and find it is locked, it will report who owns the lock. Please contact that person and have them remove the lock. If they are unavailable and the lock is days old, and you are confident they are not currently using it and they probably left it locked by accident, you can break the lock on the file by typing:
rcs -u slink.conf
The owner of the lock should then receive email notifying them that the lock was broken. Then proceed to lock the file as shown above.
# gs, version 3.53, by Dave Workman, 2003FEB08
/contrib/gs3.53/bin /contrib/bin
/contrib/gs3.53/lib /contrib/lib
/contrib/gs3.53/man /contrib/man
/contrib/gs3.53/gs3.53.softdb /contrib/softdb/gs3.53.softdb
/contrib/gs3.53/gs3.53.pkg /contrib/pkg/gs3.53.pkg
This records the package information in the slink.conf file.
ci slink.conf
RCS/slink.conf,v <-- test
new revision: 1.8; previous revision: 1.7
enter log message, terminated with single '.' or end of file:
>> added link lines for gs3.53
>> ^D
The slink.conf file will not appear in /contrib now! Do not skip the next step!
co slink.conf
The slink.conf file should now be owned by you and have 444 permissions.
/contrib/slink
This will check the entire slink tree and link in your new software. For a /contrib directory of approximately 200MB, this will take 10-20 seconds.
less /contrib/slink.log less /contrib/slink.err
newgrp
Be sure to do this step after you have slinked your package into the /contrib structure, if you are going to do so. Otherwise, slink will try to create a link to your "whatis" or "windex" file, and this link will conflict with other packages.
We want to create the proper MAN databases, so that when "man -k" is run and the MANPATH refers to a directory under your package, it can search for all related references. If you do not follow this, the user may get error messages every time they run the "man -k" command while they have your package loaded.
Go to your top-level "man" subdirectory, e.g., /contrib/gs3.53/man, for the gs3.53 package. While in that directory, type:
/usr/bin/catman -M. -w
This will create a file called "windex".
In addition to creating the man databases for searching man pages with the "man -k" command in the package subdirectory, update the appropriate pages under /contrib/man, so that when someone uses the "man -k" command to search for all man pages matching a specific key, your man pages will be included in that search.
Here is an example of how to do this on a SystemV-derivative system:
cd /contrib/man /usr/bin/catman -M. -w chmod 664 windex