Spring 1999, Vol. 20, No. 1

Farewell to MASnet: A network era ends

System lasted more than twenty years . . .

Craig Ruff and Basil Irwin, MASnet designers Contents SCDzine Search Article index Subscribe Contact us SCD

by Juli Rew and Basil Irwin Think back into the mists of time--to a place when there was no Ethernet, no IP, no Cisco, not even an Internet... The year was 1979, and SCD (then known as the Computing Facility) was justly proud of its powerful CRAY-1A and Control Data 7600 supercomputers. Remote jobs at universities were submitted via special remote job entry (RJE) card readers at the universities, which communicated with a MODCOMP II computer attached to the CDC 7600 at NCAR. Supercomputer jobs local to NCAR were punched onto paper cards and read in by a local card reader. NCAR programmers could prepare their programs on "satellite" computers, which were a flock of small PDP11/70 and Systems Engineering Laboratories (SEL) minicomputers. However, although programmers could get e-mail over the Department of Defense's experimental ARPAnet, these NCAR computers couldn't even talk to each other yet. About this time, Basil Irwin of the SCD Networking Section began hand-drafting his ideas for a network to connect the NCAR supercomputer and the satellite computers. Since then, almost every compute server, supercomputer, and Mass Storage System (MSS) that SCD has operated has been hooked up to Irwin's brainchild, originally called the "NCAR Local Network" and later the "Mainframe and Server Network" (MASnet). MASnet's pioneering design featured many new networking techniques and withstood the test of time for nearly 20 years. MASnet was finally replaced early this year by more modern networking protocols. A layered approach Basil's design for MASnet was layered, encompassing a progression from low-level hardware to high-level software. (This presaged the layered design of today's Open Systems Interconnect -- OSI -- networking model). "Basically, it was a distributed, peer-based spooling system, as opposed to the less sophisticated client-server model still in use today)," Basil notes. Users could do network file transfers, network file printing, remote job submission, and remote graphics output. "It was a distributed system before that term was invented." The key network hardware components were HYPERchannel Network Adapter boxes from Network Systems Corporation (NSC, now a division of StorageTek). HYPERchannel was a networking technology that ran at 50 megabits per second, even before 10-megabit Ethernet was commercially available. All MASnet software was written in-house. At the low level, SCD needed to write custom device drivers on every computer for the NSC adapter boxes. Interprocess communication was not yet available on UNIX or most other operating systems, so operating system kernel codes were modified to allow a single byte of user information to be communicated between pairs of processes. At the higher level, Basil wrote the MASnet code in IFTRAN, a highly portable, structured superset of FORTRAN, whose preprocessor was supported by Dave Kennison. Marc Nelson soon wrote both an IFTRAN-to-C translator for UNIX systems and an IFTRAN-to-PL/1 translator for the NCAR MSS MSCP MVS system, automatically producing updated C and PL/1 code each time the IFTRAN source was modified. Eventually, Craig Ruff developed a native C version of the MASnet software, starting with an automated translation of the IFTRAN code using a new and improved IFTRAN-to-C translator he wrote. Over the years, SCD has attached many diverse computer platforms to MASnet, including Sols, Apollos, DEC PDPs, Pyramids, IBMs, DEC VAXs, many Cray supercomputer models, a CDC 7600, and SELs. Operating systems have included DEC RSX, DEC VMS, CRAY COS, IBM VM/CMS, IBM MVS, SEL, SCD CDC 7600, and many flavors of UNIX (e.g., ISC UNIX, SGI Irix, IBM AIX, Sun SOLARIS, and CRAY UNICOS). New platforms were added continually, including more special-purpose servers, high-speed printers, and film and microfiche recorders. Basil gladly recalls when desktop drawing programs came to rescue him from the laborious task of hand-drawing network diagrams, which got more complicated everyday. Glancing at a stack of network diagrams, he notes that the diagrams actually represent a history of computing at NCAR in terms of computers, operating systems, networks, and the documentation tools used to produce the diagrams themselves. In 1988, Basil also wrote a network-monitoring system for MASnet, called MOM (MASnet Operations Monitor) -- once again predating by many years network-monitoring systems such as Hewlett-Packard's OpenView system. MOM even came with audio and would announce over the loudspeakers when a network connection went down. MOM's dulcet tones apparently weren't too popular, though, according to George Fuentes, head of SCD's Supercomputing Systems Group. "The audio got dumped pretty quickly, although the operators still used the speakers occasionally for computer room tours." Craig and Basil also made changes to MASnet to support film and printing services as well. "We did enough film to carpet a football field through MASnet," Craig notes. Internet-enabled MASnet Ethernet-based Local Area Networking was added to SCD computers in 1986 and began to gain impetus around 1988. Because SCD now had more than one local area network, calling the HYPERchannel one the "NCAR Local Network (NLN)" was now no longer appropriate, so the name was changed to "Mainframe and Server Network (MASnet)." As originally designed, each MASnet node had to be attached to a HYPERchannel adapter. Unfortunately, these devices were expensive and complex in terms of driver software and hardware to attach to new computers. There was also tremendous demand to expand MASnet functionality beyond just those few computers that could attach to HYPERchannel, because both Local Area Networking and Wide Area Networking had come of age; this resulted in Internet capabilities being widely available to the university community. Several approaches were ultimately taken to expand MASnet's functionality beyond the bounds of HYPERchannel-attached computers. The first approach, made in 1985, was specific to VAX/VMS systems using DECnet. SCD attached a gateway system to both MASnet and DECnet, with the MASnet software split between the gateway and DECnet-only clients. To the users, the clients appeared to behave exactly as if they were natively attached to HYPERchannel. The gateway could serve an arbitrary number of clients. However, this solution was not readily extensible to Internet-only systems. So for Internet-only clients, the next approach involved two gateway systems that were essentially application-level file store-and-forward systems. Each gateway was attached to both the MASnet and the Internet. Files would arrive at a gateway via the Internet from Internet-only client systems, and the gateway would submit these files into the MASnet network. Files arriving at a gateway from MASnet would be forwarded by the gateway to the specified Internet-client via the Internet. One gateway was called the Internet Remote Job Entry System (IRJE), and the other was called the MASnet/Internet Gateway System (MIGS). The principal difference between the two gateways was the user interface of the client systems and the underlying design. Britt Bassett wrote the IRJE system, while Basil designed MIGS and Craig Ruff implemented it. Eventually, Greg Woods reimplemented the IRJE internals to use MIGS, though the IRJE user-interface remained unchanged. MIGS and IRJE went into service in 1988. With the demise of MASnet, these gateways have been modified to continue to perform their gateway functions between the Internet clients and the new software used to replace MASnet. Both MIGS and IRJE are still heavily used today for job submission, MSS access, and other computing tasks. The acronym "MIGS" now means "Mainframe/Internet Gateway Server." The third and last MASnet extension effort was to enhance MASnet so that it was completely independent of HYPERchannel hardware. This was done by emulating the HYPERchannel adapters with software that used UNIX IP-based sockets across the Internet. By 1995, any UNIX system could be directly attached to MASnet as a native MASnet node without HYPERchannel hardware, and using only conventional Internet connections. Unfortunately, the MSS MSCP MVS system could not be easily converted in this fashion and still had to rely on being attached to HYPERchannel networking equipment. Yet another gateway system had to be developed to convert HYPERchannel packets to IP packets (and vice versa) so the IP-based MASnet nodes could communicate with the HYPERchannel-based MSS MSCP. The path to UNIX For twenty years, MASnet has been amazingly successful in handling a complex network of highly heterogeneous platforms. But using only UNIX systems held the promise of reducing this complexity. UNIX utilities became available for functions previously only available under MASnet such as file transfer, e-mail, and remote login. Programmers now had powerful UNIX workstations right on the desktop. Even Cray Research replaced its operating system (COS) with its UNIX-style UNICOS operating system. Around 1990, former SCD Director Bill Buzbee set into motion a plan to convert all SCD platforms to UNIX where possible. If successful, SCD might be able to safely retire the locally developed MASnet in favor of commercial UNIX-based networking and job submission utilities. However, although the path to UNIX was the logical one, it took a long time to fully develop some of the needed functions to replace MASnet. For example, to perform job submission, the Network Queuing System (NQS) was not available on multiple platforms until well into the 1990s. The DCS project -- replacing MASnet In order to finish the process of retiring MASnet, SCD launched the Distributed Computing Services (DCS) project to replace the many MASnet functions still needed or abandon the ones no longer needed or that already had substitutes, such as network printing. For example, SCD operated a centralized Text and Graphics Server (TAGS), which provided networked printer and film output services. The advent of cheaper, faster laser printers made it possible to retire SCD's high-speed Xerox 4050 laser printers, and utilize UNIX's lpr/lpd networked file-spooling system for these distributed printers. SCD Networking Section manager Marla Meehl adds, "Although some demand for film services remained, SCD found that users could be weaned from the service in favor of electronic visualization technology." NQS finally became available as a vendor-supported job submission system, removing another barrier to the retirement of MASnet. However, there still remained the formidable job of moving all Mass Storage System functionality off the locally written MASnet and onto a more modern network standard. SCD programmers Craig Ruff, John Clyne, Peter Morreale, and Erich Thanhardt first laid the groundwork by installing the Open Software Foundation's Distributed Computing Environment (DCE) on all MASnet nodes. DCE is an industry-standard, vendor-neutral set of distributed computing technologies. Next, the DCS programmers totally rewrote all of the MASnet-based MSS commands to provide the same functionality via POSIX-similar interfaces running over DCE. The first phase replaced "metadata" commands that enable users to inspect their MSS data. The final phase included file transfer and importing/exporting of data between the MSS and foreign media. What can users expect? Will users see big changes now that MASnet is gone? Luckily, users will find that they have already adjusted to the retirement of MASnet. Changes to move SCD networking and distributed services into the next millennium have been implemented gradually, and most of the new DCS MSS commands were introduced over a year ago. Since the IRJE and MIGS user interfaces were designed to be MASnet independent to start with, replacement of MASnet was almost 100% transparent to MIGS and IRJE users, who are already enjoying faster responses from the DCS-enabled IRJE and MIGS servers. MIGS and IRJE won't go away -- MIGS now simply uses NQS for job submission and DCS for other functions instead of using MASnet. End of an era Gene Harano, head of SCD's MSS Group, notes that the MASnet software ran longer than any other piece of system software developed in-house by UCAR, including the MSS software. (In fact, pieces of MASnet software are still used internally by the MSS.) During MASnet's twenty years, it's estimated that about 100 supercomputers and servers were directly attached to MASnet, while countless other systems were served via MIGS, IRJE, and the DECnet gateway. Among these systems, MASnet enabled an estimated 50 million supercomputer jobs, supercomputer job outputs, printer jobs, film jobs, and data files before Ethernet and IP-networks were ubiquitous. All this was done by a core code of fewer than 10,000 lines that was ported to dozens of different computing platforms running eight entirely different kinds of operating systems (plus dozens of variations of UNIX) over a period of twenty years. "In our current environment for three-year life cycle on hardware purchases, I guess we got our money's worth on this project," said Marla. As the MASnet era ended, Basil sent out the following e-mail: ------------------------------------ Date: Mon, 1 Feb 1999 08:51:17 -0700 From: irwin@ucar.edu (Basil L. Irwin) To: nets@ncar.ucar.edu, hps@ncar.ucar.edu Subject: MASnet HYPERchannel adapter power-down The last NSC A-Series HYPERchannel adapter at UCAR was powered down at 8:11 am February 1, 1999. This equipment was purchased in 1977 and has been in continuous production since 1979.

Contents || SCDzine || Search || Article index || Subscribe || Contact us || SCD