SCD ASR header SCD ASR header

High speed network and data communications

Introduction to Network Engineering and Technology

The Network Engineering and Technology Section (NETS) is responsible for determining networking strategy, planning, engineering, installation, operation, and maintenance of state-of-the-art networking and data communications facilities for NCAR/UCAR.

Support of these facilities requires NETS staff to:

NETS supports over 84 logical networks, approximately 178 monitored network devices, and over 3,442 network-attached devices at NCAR/UCAR.

NETS also supports the networking portion of advanced projects that are a product of outside or interagency funding. International networking project support is included in this category as well.

Another major NETS activity is the support of all SCD internal networking needs.

In summary, NETS provides a vital service to the atmospheric and oceanographic research communities by linking supercomputing resources (including mass storage systems and other data communication resources) to scientists at NCAR and throughout the university research community. These activities are essential to the effective use of NCAR/UCAR's scientific resources, and they foster the overall advancement of scientific inquiry.

The primary NETS accomplishments in FY2000 include:

More information about NETS is available at the NETS website: http://www.scd.ucar.edu/nets/

The rest of the NETS FY2000 Annual Scientific Report is organized into these sections:

Local Area Network projects

NETS supports both NCAR/UCAR network needs as well as the special networking needs of SCD itself. LAN projects are thus further subdivided as being either NCAR/UCAR network projects or SCD network projects.

NCAR/UCAR LAN projects

UCAR network infrastructure recabling projects

A large number of network infrastructure recabling projects were completed in FY2000. The common goal of all the recabling projects is to provide each workspace with a standard set of dedicated data communications links. The overall plan calls for each workspace to be provisioned with a standard Telecommunications Outlet (TO) that connects with four Category 6 (Cat6) twisted-pair cables, two Category 5e (Cat5e) twisted-pair cables, and two pairs of multi-mode optical fiber. Additionally, intra-building (trunk) wiring must be installed to concentrate all workspace cables to intermediate and central locations.

Concurrent with recabling, each network device is delivered a dedicated 100 Mbps of bandwidth via a dedicated Ethernet packet-switch port. Such dedicated-port access offers substantial networking performance improvement over shared-media Ethernet access. At the end of FY2000, dedicated-port access was available to all UCAR computers.

MINT-FL

MINT-FL is a project to completely rewire UCAR's FL1-FL3 facilities. This involves the installation of approximately 850 standard TOs, six supporting Telecommunication Closets (TCs), and new Ethernet switches throughout the FL1-FL3 buildings. Each computer at FL1-FL3 will be connected to its own dedicated 100-Mbps Ethernet packet-switch port to provide LAN access. MINT-FL began in February 2000 and will be completed April 1, 2001: http://www.scd.ucar.edu/nets/projects/mintfl/

MINT-ML

MINT-ML is a project to completely rewire UCAR's Tower B facilities. This involves the installation of standard TOs, supporting Telecommunication Closets (TCs), and new Ethernet switches throughout Tower B. Each computer at ML Tower B will be connected to its own dedicated 100-Mbps Ethernet packet-switch port to provide LAN access. MINT-ML is in the planning stages and will begin April 1, 2001: http://www.scd.ucar.edu/nets/projects/mintml/

Pearl Street

UCAR acquired a new leased site at Pearl Street and Foothills Parkway this year. NETS worked extensively on the networking design and installation which included the installation of approximately 120 telecommunications outlets and one primary communication closet.

UCAR LAN engineering projects

Network Engineering Routing Design (NERD)

NERD is a project to redesign the UCARnet routing infrastructure. The goal of this project is to make the router structure simpler and more robust. Substantial re-engineering has taken place and is still in progress to better define the router boundary with the external world. Previously, internal and external routing functions were mixed together on a variety of routers. One of the goals of the NERD project is to end up with a subset of gateway routers that perform only external routing functions, and a subset that perform only internal routing functions. This separation simplifies a variety of configuration issues, including security filters. Another goal is to replace slower routers with new faster routers. This project is well on its way to completion. For more information, see: http://www.scd.ucar.edu/nets/projects/nerd/

Ethernet packet switch re-engineering project

Several substantial improvements were made to the Ethernet packet-switching capabilities at NCAR. First, and probably most important, very large numbers of user-computers were converted from dedicated 10-Mbps Ethernet to dedicated 100-Mbps Ethernet. Also, five new Cisco 6509 Ethernet switches were installed in the network. These switches are faster and can also provide GigE host connectivity.

Miscellaneous LAN projects

NETS continued to participate in the planning of the Corporate Technical Training Center (CTTC) now to be located at the new Pearl Street (PS) location. Construction on this facility is to begin in November 2000.

NETS continued to participate in the planning of the UNIDATA remodel in FL4. Construction on this facility is to begin in November 2000.

NETS participated in the planning and implementation of the MMM computer room remodel project.

NETS continued to participate in the planning of the rewires for the COMET and MMM computer rooms.

Network monitoring project

NETS continues to use HP Openview and Cricket as its principal monitoring tools. Information on HP Openview, Cricket, and other NETS tools can be found at: http://www.scd.ucar.edu/nets/tools/

NETS statistics can be viewed at: http://netserver.ucar.edu/nets/Statistics/

NETS will also be installing cflowd statistics systems to obtain extremely detailed information about IP traffic flows through UCAR's most critical routers. Two Sun Ultra 10 computers are in the process of being configured to support the NetFlow data gathering and analysis software.

Performance monitors have also been installed at the request of two national network measuring organizations. An OC3mon ATM OC3 monitor (also called a Coral monitor) has been installed at the request of SDSC's Measurement and Operations Analysis Team (MOAT) at the FRGP. Information about MOAT and Coral are available at: http://moat.nlanr.net/

The MOAT OC3mon at the FRGP is in addition to the OC3mon that MCI/vBNS maintains at NCAR to gather vBNS statistics. MOAT also maintains an OC3mon on NCAR's vBNS link. NETS has also obtained its own OC3mon to use as a mobile ATM statistics tool.

UCAID's Internet2, in conjunction with Advanced Network and Services, Inc., installed one of their national Surveyor network monitors at the FRGP. Information about the Surveyor project is available at: http://www.advanced.org/surveyor/

Local serial-access project

NETS supports several terminal servers for providing serial console access to various computers and networking equipment. Serial support is also provided for the very few serial terminals remaining at UCAR. The Annex terminal servers are being replaced with Cisco terminal servers at both the Mesa Lab and Foothills Lab.

NETS CSAC support project

The NCAR/UCAR Computer Security Advisory Committee (CSAC) was chartered by the SCD Director to assess the state of computer and network security at NCAR/UCAR, and to make recommendations to assist NCAR and UCAR management in setting policies related to the security of computers and other devices attached to the NCAR/UCAR network. Membership of CSAC is composed of technical representatives located throughout the various NCAR/UCAR organizations.

NETS has been involved with CSAC because almost all security plans use various types of network-connected devices located between the networks belonging to the external world and the UCAR networks that are being protected from the external world. These network-attached devices can operate as filters and/or authentication devices operating at one or more OSI (Open Systems Interconnection) layers, usually at the Network Layer (Layer 3) and higher.

Based on CSAC recommendations, NETS continued to implement significant new gateway router filters to greatly improve network security for UCAR.

Extensive testing and extensive coordination throughout UCAR was required to implement the recommended security filters.

Virtual Private Networks (VPNs)

A Virtual Private Network (VPN) is a technical solution that provides secure, private connections to network applications using a public or "unsecured" medium such as the internet. With a VPN deployed across the internet, virtual private connections can be established from almost anywhere in the world. This technology is being tested by SCD to determine its viability to simplify access to internal network services from outside our firewall for university users and internal UCAR users on travel. SCD is testing a number of different solutions to determine the optimal solution. SCD is also considering any security implications of VPNs.

Multicast support activities project

Multicasting is a technology in which a single outbound stream of data can be made to arrive at multiple destinations. The data stream is multiplied in a tree-wise fashion using both software and hardware to effect the multiplication. Multicasting technology is particularly useful for videoconferencing and audioconferencing applications. NETS continues to support and enhance multicast services.

UPS project

NETS has continued installing UPS (Uninterruptable Power Supply) units into all new TCs so that all networking equipment will receive standby power in the event of a power failure. UPS units also help filter out damaging power spikes. Upgrading and maintaining these devices is an ongoing process.

Wireless

The topic of wireless ethernet has been on NETS radar for some time. However, early attempts by vendors to provide wireless ethernet were plagued by slow speeds and proprietary protocols. We have been keeping an eye on developments, and felt that with the advent of the IEEE 802.11b standard, the technology had progressed to where it would be practical to deploy into a production environment.

In recent months, a number of factors have come together to make wireless ethernet significantly more attractive to deploy than it has been in the past. Among those factors are the large increase in the number of vendors along with the increased interoperability brought about by IEEE standardization. In addition, the standard now includes a high-speed mode running at 11 Mbps. As a result of these factors, along with a widespread interest in wireless ethernet, NETS has embarked on a program to evaluate, test, and deploy wireless ethernet at UCAR. Initial testing is complete, and the deployment plan is in progress. For more information, see: http://www.scd.ucar.edu/nets/projects/wireless/

Voice over IP

Data and voice communications have traditionally been managed separately at UCAR. However, since 1986, following industry trends, there has been a transition toward the co-mingling of digital voice and traditional data communications. As voice data transitioned from analog transmission to digital transmission, the difference between the data types became negligible from a transmission point of view -- it is all just ones and zeros. Networks can move digital data no matter what the source. Great economies of scale can and have been realized by combining voice and data types onto a single network. These cost savings have been what have largely driven the merging of voice and data.

A couple of new technologies are emerging that will even further blur the difference between voice and data networks. One technology is called Voice over IP (VoIP), and the second technology is Unified Messaging. By utilizing both of these advances, UCAR will realize advanced technical capabilities to enhance the electronic "office" of the future that combines mobility with integrated voice mail, electronic mail, fax, and video technologies.

VoIP in its full implementation is the complete merging of data, voice, video, and networks. Traditional separate voice systems such as PBXs, telephone cabling infrastructure, and proprietary telephone handsets go away. These components are replaced with computer-based call managers, VoIP telephones, or your computer and voicemail (Unified Message Server) devices that are Ethernet attached just like any other computers. With a VoIP system, the same cabling infrastructure that is used for traditional network devices can be used for VoIP devices as well. Such a combination simplifies the overall cabling infrastructure and lowers total costs of installation and maintenance. The same network monitoring system can be used to monitor both VoIP and data network devices.

It is the belief of NETS and the Telecommunications Group that these are the technologies of the near future. A joint VoIP testbed is already in the planning stages to familiarize staff with this technology and assess its viability for larger scale deployment. The testbed will help determine whether UCAR should install this technology in new installations and/or transition existing facilities to this new technology.

SCD LAN projects

In addition to its overall NCAR/UCAR networking responsibilities, NETS has additional special support responsibilities for SCD. NETS handles or consults on most of the host-based SCD networking, including all supercomputing networking. NETS is also responsible within SCD for several other tasks including:

Ongoing SCD network support project

NETS is responsible for most aspects of daily operation of the SCD LANs and host-based networking. Among these responsibilities are monitoring, managing, tuning, troubleshooting, upgrading, reconfiguring, and expanding SCD LANs and host-based networking. NETS works closely with the system administrators of all SCD network-connected systems.

Supercomputing network support project

NETS supports almost all aspects of networking for all SCD supercomputers. This includes hardware, software, and routing configuration support for GigE, FDDI, and Ethernet interfaces. HiPPI (High-Performance Parallel Interface) IP routing configuration support is also provided for the supercomputer HiPPI connections.

GigE testbed

NETS provided a GigE testbed for SCD to test new GigE-attached hosts before transitioning them to an operational network.

RFP

SCD is in the process of conducting an RFP for a new supercomputer. NETS has been participating in this process, contributing both networking requirements for the system along with a set of benchmarks designed to evaluate the networking performance of the candidate systems.

SCD Visualization Lab

NETS has worked closely with VETS during the design phase of the new SCD vislab to ensure that its networking needs are well met. The vislab is expected to require a higher density of network connections compared to most SCD space. In addition, the lab will be reconfigurable between various seating arrangements. To meet these needs, NETS will be providing an increased number of wall plates, a flexible under-floor cable design allowing easy movement of network connections, and wireless connectivity. Construction on this project is expected to begin in December 2000.

Metropolitan Area Network projects

BRAN project

BRAN (Boulder Research and Administrative Network) is a strategic initiative in which UCAR has joined along with CU-Boulder, NIST, NOAA, and the City of Boulder to construct and operate a private fiber network to interconnect the key facilities of these institutions and directly access certain common-carrier telecommunications facilities in Boulder. UCAR's portion of the construction costs were approximately $400,000, with capital recovery resulting from cost savings in other areas very conservatively estimated at around 8-10 years.

All intra-Boulder MAN traffic now uses BRAN. UCAR has greatly reduced intra-Boulder circuit costs by utilizing BRAN. Active BRAN links include ML-FL, PS-FL, NOAA-ML, CU-ML. Both voice and data traffic are carried.

BRAN will also enable significant new network engineering initiatives that would otherwise not be financially feasible without the BRAN fiber. SCD's Mass Storage Systems group successfully tested using a BRAN fiber pair between ML/FL to distribute components of NCAR's Mass Storage System using both Fiber Channel and Serial HiPPI technology.

Construction of BRAN began in July 1999 and was completed by May 2000. For more information, see: http://www.branfiber.net/

Voice and data network project

NETS continues to support communications systems from NCAR's ML site to NCAR's Jeffco and Marshall sites. Jeffco still uses a T1 connection for voice service, though data service to Jeffco is provided by a US West Transparent LAN Service (TLS) link, which acts like a 4-Mbps full-duplex Ethernet link between ML and Jeffco. These legacy voice and data networks are in the process of being replaced with a 10-Mbps ATM data network with circuit emulation for voice.

The Marshall site is the only NCAR/UCAR site that currently relies exclusively on T1 for communications. The T1 line to Marshall provides both voice and data service between Marshall and the Mesa Lab. This network is being upgraded to a wireless system for voice and data.

Remote-working and home-access project

Remote access continues to be provided by digital T1 PRI access to a Cisco 5300 Remote Access Server (RAS). Each T1 PRI line provides 23 56-Kbps channels that can support analog or ISDN dial-in access. Three such lines are in place. Long distance access via direct 1-800 lines is overlaid on one of the PRI lines. Telnet, PPP, and ARAP access are supported on the Cisco RAS devices.

NETS has continued to investigate Digital Subscriber Line (DSL) Megacentral service for UCAR users. This remote access technology allows faster home access than a traditional dial-up line. Service provided by UCAR would improve performance and simplify access behind the security firewall. However, there are some security issues that NCAB and CSAC have determined are severe enough to prevent installation of ADSL service at UCAR at this time.

Wide Area Network projects

Front Range GigaPOP (FRGP)

The Front Range GigaPOP (FRGP) is a consortium of universities, nonprofit corporations, and government agencies that are cooperating in a regional network aggregation point called the FRGP to share the costs of Wide Area Networking (WAN) services.

The current FRGP partners are the BPOP, Colorado State University (CSU), CU-Boulder, CU-Denver, CU-HSC, CU-CS, CSM, DU, The State of Colorado, and the University of Wyoming. Additional partners are likely to join later. There are similar gigapops throughout the U.S. There are a number of advantages gained by sharing services through such a gigapop. Costs for WAN services are reduced for each partner, expertise among partners can be shared, a higher level of services can be purchased than individual institutions could afford, there is more buying power among a consortium, and there are great economies of scale.

NCAR/UCAR has agreed to provide the engineering and NOC support for the FRGP, with the service costs incurred by NCAR/UCAR being shared by all members.

NETS believes that the greater service and bandwidth obtained through the FRGP are important enough for NCAR/UCAR to participate and provide the engineering and NOC services. FRGP has agreed that NETS has the most qualified engineering and NOC staff to provide the very best engineering and NOC services for the FRGP. This is a critical service for the UCAR/NCAR staff as well as all the other partners. The FRGP will provide NCAR/UCAR's primary WAN connectivity. For more information, see: http://www.frgp.net/

This engineering activity has consumed a full, newly funded FTE. New and planned services include caching, news services, and statistics.

Commodity internet access project

This project provides NCAR/UCAR with general-purpose access to the commodity internet. Up until July 2000, NCAR received its commodity internet service from Qwest via a T3 PVC over an OC3 ATM link. In July, NCAR activated a secondary link to UUNET/MCI via NOAA. NCAR also converted its Qwest link to the FRGP Qwest service.

State RFP

NETS participated in the State of Colorado RFP process to obtain cost-effective commodity internet service. The RFP was awarded to C&W, and AT&T and will provide NCAR/UCAR with excellent redundant commodity internet service via the FRGP.

vBNS and NLANR projects

The vBNS is a production ATM network provided and operated by MCI under a cooperative agreement to the NSF. The vBNS interconnects the NSF supercomputing centers and universities that have been approved via the High Performance Connections (HPC) program. For more information, see: http://www.vbns.net/

NETS has continued its support of the vBNS, providing both network engineering and applications support and research.

NLANR (National Laboratory for Applied Network Research) is the umbrella project funded by NSF to facilitate the individual research projects conducted on the high-performance networks. NCNE (National Center for Network Engineering) at PSC (Pittsburgh Supercomputer Center) subcontracts with NETS to help it provide engineering support. For more information on the NLANR project, see: http://www.nlanr.net/

NLANR on-site

NETS led an NLANR on-site training session at Colorado State University on "netcasting" and "IP multicast." For more information see: http://www.scd.ucar.edu/nets/docs/reports/trips/2000/NLANR.html

Internet2/Abilene

Internet2 has finally borne fruit in the form of the Abilene network, which is a national network for Internet2 university members constructed on fiber loaned by Qwest from its new national fiber network. Abilene is constructed at 2.4-Gbps speeds and will be transitioned to 9.9-Gbps speeds as quickly as practical. As of October 2000, Abilene had 52 entities connected, many of them Gigapops, with over 182 institutions active, and this year UCAR joined UCAID and attached to Abilene via the FRGP. The Internet2 project website provides more information about Internet2, Abilene, and UCAID at: http://www.internet2.edu/

Web100

The Web100 project is described under networking technology research.

Virtual Host Interface Project (VHIP)

The VHIP project is described under networking technology research.

Earth Systems Grid project

The Earth Systems Grid project is described under networking technology research.

RAP WAN project

NETS continued to provide WAN networking service to RAP for their extensive field project requirements.

NREN WAN project

NETS continued to provide WAN networking service to NASA for their NREN connections in Boulder.

Other network projects

Projects listed in this section are ones that don't neatly fit into the LAN, MAN, and WAN project classification scheme. These other projects are organized as NCAR/UCAR support projects and SCD support projects.

NCAR/UCAR projects

Westnet

Westnet is an affinity group that grew out of the NSFnet regional network called Westnet. UCAR has always been included in Westnet in an advisory role through the years. In the last three years, NETS has taken a lead role in participating in the Westnet group.

NETS is a member of the Westnet Steering Committee and leads the effort to plan and run bi-annual meetings that include technical presentations from members and vendors. Westnet provides powerful political and technical contacts with universities that are UCAR members and that share common concerns. The current Westnet members include CU-Boulder, CU-Denver, CSU, University of Wyoming, University of Utah, Utah State University, Arizona State University, University of Arizona, University of New Mexico, New Mexico State University, Idaho State University, Denver University, South Dakota School of Mines and Technology, New Mexico Technet, Boise State University, and UCAR.

Project-tracking system

The REMEDY-based project-tracking system continued in full production in FY2000. NETS work requests and projects are opened, tracked, and closed with this project-tracking system. The use of project-tracking tools is necessary because of the large number and variety of projects. It would be unweildy to manually track just the personnel assignments for these hundreds of projects, much less track progress details of so many projects.

In FY2000, NETS completed more than 1,646 work requests and over 191 projects. At the beginning of FY2001, 69 projects were in progress, and 263 work requests were queued.

NETS has successfully transitioned to the SCD Remedy trouble ticket system.

NETS NCAB project

The Network Coordination and Advisory Board (NCAB) consists of appointed technical representatives from the NCAR divisions and UCAR. The purpose of NCAB is to advise NETS concerning network strategy, planning, policy, expansion, and management issues for all of NCAR and UCAR. The work of NCAB has been indispensable to the success of networking at UCAR.

Strategic plan

At the request of UCAR management, NETS formulated a comprehensive strategic plan outlining technical and budgetary requirements for UCAR networking for the next several years. This document is available at http://www.scd.ucar.edu/nets/Documents/stratplan/strategy.html

This document is in the process of being revised and updated.

Conference

The NETS continues to provide networking support for classes, demonstrations, meetings, and conferences throughout NCAR/UCAR. This work involved the design, construction, configuration, and operation of the network components required by these activities.

Outreach support project

NETS presented seminars as part of the SCD Seminar Series, and NETS personnel attended and presented at numerous conferences, meetings, and training sessions and prepared the numerous trip reports and presentations published at: http://www.scd.ucar.edu/nets/docs/reports/trips/ and http://www.scd.ucar.edu/nets/presentations/

Scot Colburn served on the Wheatridge High School computer and networking curriculum board.

Marla Meehl presented the keynote address at the 2000 Colorado Computational Science Fair.

Via NLANR, Basil Irwin performed an engineering site survey at the University of Tennessee at Knoxville. He then prepared an engineering report that can be used as a model for site surveys at other universities.

Also via NLANR, David Mitchell provided an onsite tutorial on multicasting for Colorado State University. A report on this tutorial appears at http://www.scd.ucar.edu/nets/docs/reports/trips/2000/NLANR.html

Committee support project

NETS representatives have attended and supported the following committees:

NCAR/UCAR committees:

SCD committees:

External committees:

Next

SCD ASR - Table of contents

Message from SCD Director Al Kellie

SCD's FY2000 science highlights

SCD: Providing support for large and small scientific research projects, no matter where they are located

SCD: A center for supercomputing resources and technologies

SCD: A center for data resources, data analysis, and emerging technologies

SCD research: Pushing the frontiers in high-performance computing for geosciences

SCD: Providing supercomputing and communications facilities and infrastructure

SCD community service activities

SCD educational activities

SCD publications and papers

SCD staff

SCD visitors and collaborators