
Browse NETS topics: A B C D E F G H I J K L M N O P Q R S T U V W X Y Z	Search NETS:

Hypermail (NETS notes)

By Blake Caldwell (blake@ucar.edu)
updated 2003-05-21 by Pete Siemsen (siemsen@ucar.edu
updated 2003-09-26 by David Mitchell (mitchell@ucar.edu)
updated 2004-01-30 by Pete Siemsen (siemsen@ucar.edu)
updated 2005-04-08 by Jeff Custard (jcustard@ucar.edu)

Introduction
Adding a new archive
How is Hypermail Set Up?
Mailing Lists that are Currently Archived
Known Problems
Directory Structure on Netserver
Automating Hypermail
Installation
- Hypermail
- Procmail
Configuration
- Hypermail
- Procmail
Current Configuration Files
Troubleshooting
Searching
Links

Introduction

This web page goes over the general issues, installation, configuration, and troubleshooting of the Hypermail archiving system. Hypermail is a open source program that converts standard email messages into web pages. NETS has used this system to archive the mail sent to several NETS, Web100, and FRGP mailing lists. Hypermail was initially installed on netserver in March of 2000.

How is Hypermail Set Up?

Hypermail is set up on netserver to archive messages as it receives them. hmail@netserver.ucar.edu must be a member of all mailing lists that are to be archived so that it receives a copy of each message. When a message reaches hmail on netserver, it is sent to a program called Procmail through the use of a .forward file. Procmail looks at the header information of the message, usually the To or Cc lines, to determine which mailing list the message was originally sent to. As specified in the /usr/home/hmail/.procmailrc file, Procmail then sends the message to Hypermail with the appropriate configuration file. Once Hypermail is called with the correct configuration, it can archive the message into the specified directory and update the index files.

Mailing Lists that are Currently Archived

NETS lists:

csac, csnap, csnapi, malt, ncab, ne, nets, nets-fss, nets-work, ni, qwest
FRGP lists:

fmc, ftc, frgp, frgp-inv, frgp-eng, frgp-outages

Known Problems

There is one bug which I ran into involving the creation of web pages based on specified header and footer files. Hypermail appears to have a problem with the string width="100%". It substitutes a question mark for the second double quote. All web browsers can cope with the question mark in there, but not a missing double quote. So instead I put width="100% " which seems to work, even though it still puts in a question mark.

Directory Structure on Netserver

The Hypermail web tree starts out with an index page containing links to all of the mailing lists. Then there are index pages, each with links to the directories containing the messages sorted out by month and year. Each month has its own directory, which is named with the first three letters of the month, followed by the last two numbers of the year. For example jan99 and nov00. Inside these directories are the index pages Hypermail creates, and the messages themselves.

Automating Hypermail

One thing Hypermail doesn't do is group messages by a set amount of time. To get around this, I wrote a Perl script called hypermailmonthly.pl that resides in /usr/home/hmail/ and is run as a cron job every month. This script changes the directories where Hypermail will store its messages for each mailing list to reflect the change in month. It also modifies the web pages to include links to the directories of the new month.

Installation

Hypermail

First download the hypermail source. Then as root type:
mv hypermail.tar.gz /usr/src cd /usr/src gunzip hypermail.tar.gz tar -xvf hypermail.tar cd hypermail-* ./configure make make install cd .. rm hypermail.tar
Procmail
First download the procmail source. Then as root type:
mv procmail-3.15.1.tar.gz /usr/src cd /usr/src gunzip procmail-3.15.1.tar.gz tar -xvf procmail-3.15.1.tar cd procmail-3.15.1 make install

Configuration

Hypermail
Hypermail can either get its configuration options from the command line or a config file. I choose to use a config file, because it easier to keep track of. The config file can have any name and be in any directory, as long as it is specified when Hypermail is invoked. The name of each file is hmrc. followed by the name of the mailing list. For example, hmrc.nets-work is the configuration file for the nets-work@ucar.edu mailing list. These files are located in the ~hmail/configs/ directory on netserver. The configuration file specifies options defining the layout of the index pages, and the directories in which to store the messages.
A template can also be defined, in the configuration file, for the pages Hypermail is to create. This usually consists of a header and footer file, which make up the bulk of the HTML Hypermail produces. All of the NETS and FRGP mailing lists use the footer.html file for the footer and a header file like header.nets. Header and footer files are stored in /usr/src/hypermail-2b29/headers_and_footers/.
An explanation of all configuration options can be found in the example configuration file. For information on how to pass configuration options to Hypermail through the command line, see the Hypermail man page.
Procmail
All of Procmail's configuration is done through the .procmailrc in hmail's home directory. The exact format and options for .procmailrc is explained in the Procmail man page. One important thing to remember is that messages sent to a mailing list can contain the mailing list name in the To or the Cc line. Also it is important to be specific in telling Procmail what to look for. For example, to get all messages sent to the ne@ucar.edu archive, procmail would look for the "ne" string in the To or Cc lines. However, this can cause some confusion. If the message was sent to the nets@ucar.edu mailing list, Procmail would see the "ne" string and assume it is to the ne@ucar.edu mailing list. To get around this, tell Hypermail to look for the name of the archive followed by a space, @ sign, or a carriage return by putting [ @\n] after the name of the archive.

Current Configuration Files

Troubleshooting

Problem: Messages are disappearing and not being archived.
1. Look in the from log in hmail's home directory. This is Procmail's log and it will give you information on which Hypermail command was executed and any error messages that Hypermail produced. It is very long since it logs every message sent to hmail@netserver.ucar.edu. Check to see if Hypermail actually received the message first. In this log it will tell you whether it sent the message to Hypermail or if it abandoned it because of an error. If the message is not listed in the log, make sure Sendmail and the .forward file are configured correctly.
2. Look in mbox file in hmail's home directory. Whenever Procmail doesn't find any rules that apply to a certain message and is not sure what to do with it, it will dump the message into this file. Currently, there is a script that will send blake@ucar.edu and colburn@ucar.edu a copy of any message that is put into this file. If a message gets dumped into this file, it usually means that there needs to be an entry in the .procmailrc file, telling Procmail what to do with the message. If a new mailing list is added, messages sent to that mailing list will be put into this file, because Procmail doesn't know what to do with it. A couple of lines in .procmailrc telling Procmail which command to execute when it receives a message sent to the mailing list will solve this. Follow the format of the current .procmailrc.
3. Look in the system log. If Hypermail has to abort archiving a message, because it cannot set the correct permissions, it will log to syslog. This can usually be fixed by changing the directory in which the messages will be archived to be group writable. Most of the time when something like this happens, Hypermail will bounce the mail back to Procmail, so you should see a similar entry in the Procmail log file.
4. Check the other mail archives to see that the message was not put into the wrong archive. If this is the case, then you need to modify .procmailrc. Make sure to be specific in your Procmail rules and also to be careful with the order in which the rules are arranged.
Problem: Messages are being archived into the wrong month
1. See step 4 above

Searching (JC updated 2005-04-08)

To search the Hypermail archives for a string, go to netserver and do
cd /usr/web/nets/archives/internal/mailing-lists find . -type f -exec grep -i -l "str" {} \;

...where str is the string you want to find. This will search all the hypermail directories and list the files that contain the string.

If you remove the "-i", the search will become case sensitive.

If you remove the "-l", it'll show every line that matches instead of the names of files.

If you replace the . with the name of a mailing list, it'll search just that list.

EXAMPLES

To show lines that contain the string "Bob Brenner"...

find . -type f -exec grep "Bob Brenner" {} \;

To show names of files that contain the string "Bob Brenner"...

find . -type f -exec grep -l "Bob Brenner" {} \;

To show names of files in the bpop mailing list that contain the string "Bob Brenner"...

find bpop -type f -exec grep -l "Bob Brenner" {} \;

Note that the hyper-mail archives for these lists:

archive; csac; dtc-tech.html; nb; ne.html; nets-outage.html; nets.html; phones; pr.html; wc; wl.html; cg1-ext; csac.html; ithaka; nb.html; nets; nets-work;ni; phones.html; strat; wc.html; cg1-int; dtc-tech; ithaka.html; ne; nets-outage; nets-work.html; ni.html; pr; strat.html; wl

Now live in: netserver:/usr/web/nets/datacomm/mailing-lists

Links

Address comments or questions about this Web page to the Network Engineering & Telecommunications Section at nets-www@ncar.ucar.edu. The NETS is part of the Computational & Information Systems Laboratory of the National Center for Atmospheric Research, which is sponsored by the National Science Foundation and managed by the University Corporation for Atmospheric Research. This website follows the UCAR General Privacy Policy and the NCAR/UCAR/UOP Terms of Use.

Hypermail (NETS notes)

Table of Contents

NETS lists:

FRGP lists:

Problem: Messages are disappearing and not being archived.

Problem: Messages are being archived into the wrong month

Searching (JC updated 2005-04-08)