[Previous] [Table of contents] [Next]

High speed network and data communications

Introduction to NETS

The Network Engineering and Technology Section (NETS) is responsible for determining networking strategy, planning, engineering, installation, operation, and maintenance of state-of-the-art networking and data communications facilities for NCAR/UCAR.

The support of these facilities includes:

NETS supports over 120 logical networks, approximately 210 monitorable network devices, and over 3,000 network-attached devices at NCAR/UCAR.

NETS also supports the networking portion of advanced projects that are a product of outside or interagency funding. International networking project support is included in this category as well. NETS maintains a transit network that supports the interconnection of several major national networks.

Another major NETS activity is the support of all SCD internal networking needs.

In summary, NETS provides a vital service to the atmospheric and oceanographic research communities by linking supercomputing resources (including mass storage systems and other data communication resources) to scientists at NCAR and throughout the university research community. These activities are essential to the effective use of NCAR/UCAR's scientific resources, and they foster the overall advancement of scientific inquiry.

The primary NETS accomplishments in FY1999 include:

More information about NETS is available at the NETS website: http://www.scd.ucar.edu/nets/

To access the rest of the NETS FY1999 Annual Scientific Report:

Local Area Network projects
Metropolitan Area Network projects
Wide Area Network projects
Other network projects

Local Area Network projects

NETS supports both NCAR/UCAR network needs as well as the special networking needs of SCD itself. LAN projects are thus further subdivided as being either NCAR/UCAR network projects or SCD network projects.

NCAR/UCAR LAN projects

UCAR network infrastructure recabling projects

A large number of network infrastructure recabling projects were completed in FY1999. The common goal of all the recabling projects is to provide each workspace with a standard set of dedicated data communications links. The overall plan calls for each workspace to be provisioned with a standard Telecommunications Outlet (TO) that connects with four Category 5e (Cat5e) twisted-pair cables, two Category 3 (Cat3) twisted-pair cables, and two pairs of multi-mode optical fiber. Additionally, intra-building (trunk) wiring must be installed to concentrate all workspace cables to intermediate and central locations.

Concurrent with recabling, each network device is delivered a dedicated 10 Mbps of bandwidth via a dedicated Ethernet packet-switch port. Such dedicated-port access offers substantial networking performance improvement over shared-media Ethernet access. The end of FY1999 has now made dedicated-port access available to all UCAR computers.

FL4 Uniform Network Project (FUN)

FUN completely rewired UCAR's FL4 facility. This involved the installation of 236 standard TOs, supporting Telecommunication Closets (TCs), and new Ethernet switches throughout the FL4 building. Each computer at FL4 is now connected to its own dedicated Ethernet packet-switch port to provide LAN access. FUN took approximately a year to complete from the beginning of the planning phase until the last network device was plugged in. FUN was successfully completed on time and under budget. For more details see: http://www.scd.ucar.edu/nets/projects/NETSprojectplans/2000.complete.projects/fun/

Millennium Infrastructure Network Technology - Foothills Lab (MINT-FL) Project

The design, planning, and specifications phase of the MINT-FL project was completed this year. MINT-FL will replace all remaining portions of the outdated wiring at the Foothills complex during FY2000. The MINT-FL project will increase desktop network access by a factor of 10 by introducing wiring and switching equipment that will deliver a dedicated 100 Mbps to the desktop, and provides for future access that will be capable of 1,000 Mbps on the copper cabling and 10 Gbps on the fiber optic cabling. The most critical element that MINT-FL provides is a wiring infrastructure able to provide network growth to the desktop and in the backbone for at least 10 years. For more information on this project, see: http://www.scd.ucar.edu/nets/projects/mintfl/

UCAR LAN Engineering Projects

GigE Retrofit Project (GERP)

GERP was a project to upgrade the UCAR LAN backbone networks from Centimegabits to Gigabits. The UCARnet backbone was previously an ATM network running at 155 Mbps. The new GERP backbone is 1,000 Mbps using Gigabit Ethernet technology. GigE is not only much faster, but it is a much simpler and more robust technology than ATM. GERP was successfully completed on time, within budget, and was virtually transparent to UCAR's network users. For more details, see: http://www.scd.ucar.edu/nets/projects/gerp/

Netflow-Feature-Card-3 Upgrade Project (NUP)

NUP was a project to upgrade all UCARnet Cisco 5500 Ethernet Switches with a new Netflow Feature Card on the supervisor engine. This enhanced feature card allows us to run additional switching algorithms as well as collect detailed statistics. This project was completed successfully on a very aggressive schedule. For more information, see: http://www.scd.ucar.edu/nets/projects/nup/

Network Engineering Routing Design (NERD)

NERD is a project to redesign the UCARnet routing infrastructure. The goal of this project is to make the router structure simpler and more robust. Substantial re-engineering has taken place and is still in progress to better define the router boundary with the external world. Previously, internal and external routing functions were mixed together on a variety of routers. One of the goals of the NERD project is to end up with a subset of gateway routers that perform only external routing functions, and a subset that perform only internal routing functions. This separation simplifies a variety of configuration issues, including security filters. This project is well on its way to completion. For more information, see: http://www.scd.ucar.edu/nets/projects/nerd/

Ethernet packet switch re-engineering project

Several substantial improvements were made to the Ethernet packet-switching capabilities at NCAR. First, and probably most important, very large numbers of user-computers were converted from shared-media Ethernet to dedicated-media Ethernet, where each computer has its own dedicated Ethernet packet-switch port. By the end of FY1999, all user-computers had the opportunity to convert to dedicated-media connections. NETS also made progress in installing GigE and Fast Ethernet (100BaseT and 100BaseF) port cards in various Cisco Ethernet packet switches, making Fast Ethernet more readily available to desktop users. Divisional servers are now usually connected to Fast Ethernet, and many specialized desktops connect to Fast Ethernet as well. Lastly, two new Cisco 6509 Ethernet switches were installed in the ML Computer Room and the FL Computer Room. These switches are the core of the GigE UCARnet backbone and also provide GigE host connectivity.

Y2K Project

NETS has worked throughout the year to ensure Y2K compliance of UCARnet networking equipment and facilities. NETS also designed and implemented a Y2K testbed so that system administrators could test system Y2K compliance. For more information on the devices and plan, see: http://www.scd.ucar.edu/nets/projects/Y2K/

Miscellaneous LAN projects

NETS cabled the ATD RAF C130 aircraft with CAT5e twisting pair wiring. This was a logistically and technically difficult project, but it will provide advanced networking on the aircraft.

NETS also recabled a number of divisional/program computer rooms to provide advanced networking infrastructure. These included the ATD, MIS, COMET, UNAVCO, COSMIC, HIRDLS, JOSS, and UNIDATA computer rooms.

NETS also cabled the newly remodeled FL1 South Atrium to the current standards.

NETS implemented the COMET satellite AWIPS NOAAPort receiving system. An AWIPS receive system consists of a satellite downlink, a series of HP workstations to ingest and distribute the data via FDDI and Ethernet networks, and a series of end-user visualization workstations. The corporate-wide benefit is that the data stream from the satellite antenna is split, and data are distributed to multiple groups throughout the UCAR campus via the IITA data server.

NETS developed and implemented a cable labeling database that is used to label all cabling components in the UCARnet infrastructure.

Network monitoring project

NETS continues to use HP Openview and Cricket as its principal monitoring tools. MRTG (Multi-Router Traffic Grapher) was replaced in FY1999 with Cricket to provide enhanced real-time web-based graphs of SNMP-accessible load statistics available from a variety of networking devices. Information on HP Openview, Cricket, and other NETS tools can be found at: http://www.scd.ucar.edu/nets/tools/

NETS statistics can be viewed at: http://netserver.ucar.edu/nets/Statistics/

NETS will also be installing Cisco NetFlow statistics systems to obtain extremely detailed information about IP traffic flows through UCAR's most critical routers. Two Sun Ultra 10 computers are in the process of being configured to support the NetFlow data gathering and analysis software.

Additional performance monitors have also been installed at the request of two national network measuring organizations. An OC3mon ATM OC3 vBNS monitor (also called a Coral monitor) has been installed at the request of SDSC's Measurement and Operations Analysis Team (MOAT). Information about MOAT and Coral are available at: http://moat.nlanr.net/

The MOAT OC3mon is in addition to the OC3mon that MCI/vBNS maintains at NCAR to gather vBNS statistics. NETS has also obtained its own OC3mon to use as a mobile ATM statistics tool.

UCAID's Internet2, in conjunction with Advanced Network and Services, Inc., installed one of their national Surveyor network monitors. Information about the Surveyor project is available at: http://www.advanced.org/surveyor/

Local serial-access project

NETS supports several terminal servers for providing serial console access to various computers and networking equipment. Serial support is also provided for the very few serial terminals remaining at UCAR. The Annex terminal servers will soon be replaced with Cisco remote access servers.

NETS CSAC support project

The NCAR/UCAR Computer Security Advisory Committee (CSAC) was chartered by the SCD Director to assess the state of computer and network security at NCAR/UCAR, and to make recommendations to assist NCAR and UCAR management in setting policies related to the security of computers and other devices attached to the NCAR/UCAR network. Membership of CSAC is composed of technical representatives located throughout the various NCAR/UCAR organizations.

NETS has been involved with CSAC because almost all security plans use various types of network-connected devices located between the networks belonging to the external world and the UCAR networks that are being protected from the external world. These network-attached devices can operate as filters and/or authentication devices operating at one or more OSI (Open Systems Interconnection) layers, usually at the Network Layer (Layer 3) and higher.

Based on CSAC recommendations, NETS recently implemented significant new gateway router filters to greatly improve network security for UCAR.

Extensive testing and extensive coordination throughout UCAR was required to implement the recommended security filters.

Multicast support activities project

Multicasting is a technology in which a single outbound stream of data can be made to arrive at multiple destinations. The data stream is multiplied in a tree-wise fashion using both software and hardware to effect the multiplication. Multicasting technology is particularly useful for videoconferencing and audioconferencing applications. NETS continues to support and enhance multicast services.

UPS project

NETS has completed installing UPS (Uninterruptable Power Supply) units into all TCs so that all networking equipment will receive standby power in the event of a power failure. UPS units also help to filter damaging power spikes. For more information, see: http://www.scd.ucar.edu/nets/projects/NETSprojectplans/1999.complete.projects/upsinstallation.html

SCD LAN projects

In addition to its overall NCAR/UCAR networking responsibilities, NETS has additional special support responsibilities for SCD. NETS handles or consults on most of the host-based SCD networking, including all supercomputing networking. NETS is also responsible within SCD for several other tasks including:

Ongoing SCD network support project

NETS is responsible for most aspects of daily operation of the SCD LANs and host-based networking. Among these responsibilities are monitoring, managing, tuning, troubleshooting, upgrading, reconfiguring, and expanding SCD LANs and host-based networking. NETS works closely with the system administrators of all SCD network-connected systems.

Supercomputing network support project

NETS supports almost all aspects of networking for all SCD supercomputers. This includes hardware, software, and routing configuration support for GigE, FDDI, ATM, and Ethernet interfaces. HiPPI (High-Performance Parallel Interface) IP routing configuration support is also provided for the supercomputer HiPPI connections.

MASnet support project

MASnet was an SCD-developed distributed remote job entry system that had been in place since 1990. MASnet had been used to submit batch jobs and return batch job output for the Cray supercomputers, to transport metadata commands for the NCAR Mass Storage System, and to submit graphical output jobs to TAGS. Originally MASnet ran only on HYPERchannel networking equipment, but it was enhanced to run over any IP network to extend its lifetime. All functions of MASnet have now been replaced, and all MASnet software and hardware were retired in February 1999.

Metropolitan Area Network projects

Front Range GigaPOP (FRGP)

The Front Range GigaPOP (FRGP) is a consortium of universities, non-profit corporations, and government agencies that are cooperating in a regional network aggregation point called the FRGP to share the costs of Wide Area Networking (WAN) services.

The current FRGP partners are UCAR/NCAR, NOAA, Colorado State University (CSU), CU-Boulder, CU-Denver, and the University of Wyoming. Additional partners are likely to join later. There are similar gigapops throughout the U.S. There are a number of advantages gained by sharing services through such a gigapop. Costs for WAN services are reduced for each partner, expertise among partners can be shared, a higher level of services can be purchased than individual institutions could afford, there is more buying power among a consortium, and there are great economies of scale.

NCAR/UCAR has agreed to provide the engineering and NOC support for the FRGP, with the service costs incurred by NCAR/UCAR being shared by all members.

NETS believes that the greater service and bandwidth obtained through the FRGP are important enough for NCAR/UCAR to participate and provide the engineering and NOC services. FRGP has agreed that NETS has the most qualified engineering and NOC staff to provide the very best engineering and NOC services for the FRGP. This is a critical service for the UCAR/NCAR staff as well as all the other partners. The FRGP will provide NCAR/UCAR's primary WAN connectivity. For more information, see: http://www.frgp.net/

BRAN project

BRAN (Boulder Research and Administrative Network) is a strategic initiative in which UCAR has joined along with CU-Boulder, NIST, NOAA, and the City of Boulder to construct and operate a private fiber network to interconnect the key facilities of these institutions and directly access certain common-carrier telecommunications facilities in Boulder. UCAR's portion of the construction costs are approximately $400,000, with capital recovery resulting from cost savings in other areas very conservatively estimated at around 8-10 years.

BRAN will include fiber that transits both the US West Boulder CO (Central Office) and the ICG Boulder POP (Point Of Presence). UCAR will greatly reduce tail circuit charges to US West for access to the Boulder Main CO (BMCO), and could access ICG without any US West charges at all. ICG is a major U.S. CLEC (Competitive Local Exchange Carrier), and direct access to its facilities has major implications for cheaper access by UCAR to national telecommunications facilities in Denver.

BRAN will also enable significant new network engineering initiatives that would otherwise not be financially feasible without the BRAN fiber. NETS already has plans to implement a direct GigE (1,000 Mbps) link between the FL and ML complexes in parallel with a legacy ATM link that will be maintained to carry voice traffic between the two sites. BRAN will also allow the newly leased FL5 building to be connected to FL/ML at GigE speeds without added link costs. SCD's Mass Storage Systems group is also contemplating using a BRAN fiber pair between ML/FL to distribute components of NCAR's Mass Storage System. NETS is contemplating an OC-12 (622 Mbps) link to NOAA and a GigE link to CU Boulder.

Construction of BRAN began in July 1999 and is scheduled to be completed by the end of calendar year 1999. For more information, see: http://www.branfiber.net/

Voice and data network project

NETS continues to support communications systems from NCAR's ML site to NCAR's Jeffco and Marshall sites. Jeffco still uses a T1 connection for voice service, though data service to Jeffco is provided by a US West Transparent LAN Service (TLS) link, which acts like a 4-Mbps full-duplex Ethernet link between ML and Jeffco. These legacy voice and data networks are in the process of being replaced with a 10-Mbps ATM data network with circuit emulation for the voice.

The Marshall site is the only NCAR/UCAR site that will rely exclusively on T1 for communications. The T1 line to Marshall provides both voice and data service between Marshall and the Mesa Lab. This network is also being upgraded to make it Y2K compliant.

Remote-working and home-access project

The legacy modem pool was replaced by digital T1 PRI access to a Cisco 5300 Remote Access Server (RAS). Each T1 PRI line provides 23 56-Kbps channels that can support analog or ISDN dial-in access. Three such lines were installed in FY1999. Long distance access via direct 1-800 lines was moved from the modem pool and overlaid on one of the PRI lines. The RAS modems were also upgraded to the new 56-Kbps V.90 modem standard when it became available. Telnet, PPP, and ARAP access are supported on the Cisco RAS devices.

NOAA atmospheric laser link project

NETS worked with NOAA to support an OC-3 ATM atmospheric link between NCAR's ML site and NOAA's new Boulder complex, which is about 1.5 miles from ML. NOAA used this link to communicate with NCAR until recently, when NOAA installed a replacement microwave link to NCAR.

Wide Area Network projects

Commodity Internet access project

This project provides NCAR/UCAR with general-purpose access to the Commodity Internet. Until recently, access was provided via an MCI non-ATM DS-3 (45-Mbps) link. As of the new fiscal year, NCAR receives its commodity Internet service from Qwest via a T3 PVC over an OC3 ATM link. NCAR is in the process of activating a secondary link to UUNET/MCI via NOAA.

vBNS and NLANR projects

The vBNS is a production ATM network provided and operated by MCI under a cooperative agreement to the NSF. The vBNS interconnects the NSF supercomputing centers and universities that have been approved via the High Performance Connections (HPC) program. For more information, see: http://www.vbns.net/

As of November 1999, 101 NSF HPC (High Performance Connection) grant universities were connected to the vBNs. NETS has continued its support of the vBNS, providing both network engineering and applications support and research.

NLANR (National Laboratory for Applied Network Research) is the umbrella project funded by NSF to facilitate the individual research projects conducted on the vBNS. NCAR/UCAR has been a major participant in several collaborative NLANR projects that demonstrate the high bandwidth capabilities of the vBNS. NCNE (National Center for Network Engineering) at PSC (Pittsburgh Supercomputer Center) subcontracts with NETS to help it provide vBNS engineering support. For more information on the NLANR project, see: http://www.nlanr.net/

A major project NCAR conducted for NLANR was to help determine the future role of the NSF in high-performance networking. The NSF requested NCAR to host an NLANR workshop titled, "Workshop on Advanced Networking Infrastructure Needs in Atmospheric and Related Sciences." For more information on the workshop, see: http://www.scd.ucar.edu/nets/projects/NETSprojectplans/1999.complete.projects/nlanr/

The Web100 project is another major sub-project of NLANR. The Web100 project is an initiative proposed by NCAR, PSC, and NCSA to fix some well-known problems with the Unix (and other) operating systems that are currently inhibiting effective utilization of national high-performance networks such as vBNS and Abilene. One of the biggest problems is the current need to manually calculate the optimal bandwidth delay product to specify a TCP window size that is large enough to avoid prematurely halting data transmission between TCP acknowledgment packets.

This issue generally isn't important for LANs, but it is important for high-performance WANS. It is difficult to determine the "bandwidth" part of the product, and right now the only effective way to obtain this is to have knowledge of the network topology, which usually means consulting with a network engineer. Furthermore, most applications don't provide a means for the user to specify this information even if it was available. The Web100 Project is seeking to solve this problem and some other related ones, and has received funding from the NSF to investigate these solutions in the context of future NSF funding to support performing the necessary work.

It is highly likely that the initial work will be done on Linux, and existing academic users of Linux will receive support to utilize this high-performance network-enabled version of Linux. For more information on Web100 and on the related workshop that NETS hosted, see: http://www.scd.ucar.edu/nets/projects/web100/

Internet2/Abilene

Internet2 has finally borne fruit in the form of the Abilene network, which is a national network for Internet2 university members constructed on fiber loaned by Qwest from its new national fiber network. Abilene is initially being constructed at 2.4-Gbps speeds and transitioned to 9.9-Gbps speeds as quickly as practical. As of November 1999, Abilene had 71 institutions connected, with over 160 institutions eligible to connect, and this year UCAR joined UCAID and is also in the process of attaching to Abilene. It is vitally important for UCAR to connect to Abilene. UCAR currently depends on the vBNS for high-performance connections to the majority of its member universities.

As of this date, the future role of the vBNS is highly uncertain and it appears that almost all universities connected to the vBNS have already connected to Abilene or will connect to Abilene sometime before May 1, 2000. The Internet2 Project website provides more information about Internet2, Abilene and UCAID at http://www.Internet2.edu/

Earth Systems Grid Project

NETS is providing network engineering support to the DOE Earth Systems Grid project. For more information on this project, see: http://www.scd.ucar.edu/css/esg/

RAP WAN project

NETS continued to provide WAN networking service to RAP for their extensive field project requirements.

NREN WAN project

NETS continued to provide WAN networking service to NASA for their NREN connections in Boulder.

Other network projects

Projects listed in this section are ones that don't neatly fit into the LAN, MAN, and WAN project classification scheme. These other projects are organized as NCAR/UCAR support projects and SCD support projects.

NCAR/UCAR projects

Westnet

Westnet is an affinity group that grew out of the NSFnet regional network called Westnet. UCAR has always been included in Westnet in an advisory role through the years. In the last three years, NETS has taken a lead role in participating in the Westnet group.

NETS is a member of the Westnet Steering Committee and leads the effort to plan and run bi-annual meetings that include technical presentations from members and vendors. Westnet provide powerful political and technical contacts with Universities that are UCAR members and share common concerns. The current Westnet members include CU-Boulder, CU-Denver, CSU, University of Wyoming, University of Utah, Utah State University, Arizona State University, University of Arizona, University of New Mexico, and New Mexico State University, Idaho State University, Denver University, South Dakota School of Mines and Technology, New Mexico Technet, Boise State University, and UCAR.

Project-tracking system

The REMEDY-based project-tracking system continued in full production in FY1999. NETS work requests and projects are opened, tracked, and closed with this project-tracking system. The use of project-tracking tools is necessary because of the large number and variety of projects. It would be impossible to manually track just the personnel assignments for these hundreds of projects, much less track progress details of so many projects.

In FY1999, NETS completed more than 1,000 work requests and over 100 projects. At the beginning of FY1999, 84 projects were in progress, and 378 work requests were queued.

NETS NCAB project

The Network Coordination and Advisory Board (NCAB) consists of appointed technical representatives from the NCAR divisions and UCAR. The purpose of NCAB is to advise NETS concerning network strategy, planning, policy, expansion, and management issues for all of NCAR and UCAR. The work of NCAB has been indispensable to the success of networking at UCAR.

Strategic plan

At the request of UCAR management, NETS formulated a comprehensive strategic plan outlining technical and budgetary requirements for UCAR networking for the next several years. This document is available at http://www.scd.ucar.edu/nets/Documents/stratplan/strategy.html

Conference and outreach support project

The NETS continues to provide networking support for classes, demonstrations, meetings, and conferences throughout NCAR/UCAR. This work involved the design, construction, configuration, and operation of the network components required by these activities.

NETS also presented seminars as part of the SCD Seminar Series, and NETS personnel attended and presented at numerous conferences, meetings, and training sessions and prepared the numerous trip reports and presentations published at: http://www.scd.ucar.edu/nets/docs/reports/trips/ http://www.scd.ucar.edu/nets/presentations/

Committee support project

NETS representatives have attended and supported the following committees:

NCAR/UCAR committees:

  • The Network Coordination Advisory Board (NCAB)
  • The Computer Security Advisory Committee (CSAC)
  • The Multi-media Advisory Group (MAG)
  • The Business Rules Advisory Committee (BRAC)
  • SCRUB Committee

SCD committees:

  • The Computer Room Planning Committee (CRPC)
  • The Machine Dependencies Committee
  • The Inventory Committee
  • The Trouble Ticket Committee (TTC)
  • SCD Desktop Advisory Committee (SDAC)

External committees:

  • NSF HPC Grant Proposal Review Panel
  • The vBNS Technical Coordination Committee
  • The Westnet Steering Committee
  • The Front Range GigaPop Management Committee (FMC)
  • The Front Range GigaPop Technical Committee (FTC)
[Previous] [Table of contents] [Next]