Network engineering and telecommunications accomplishments

The Network Engineering and Telecommunications Section (NETS) is responsible for the planning, engineering, installation, operation, maintenance, strategy, and research regarding the state-of-the-art data networking and telecommunications facilities for NCAR/UCAR.

The support of these facilities includes:

Network hardware and software troubleshooting
Network error detection, load monitoring, and network tuning
Maintaining network documentation and databases
Reconstructing and reconfiguring networks to meet optimal performance needs
Expanding networks to meet increased numbers of users and increased bandwidth requirements
Consulting with network users regarding optimal network usage
Evaluating new networking technologies and equipment
Tracking and coordinating all network project activities via a work-request/work-tracking system
Managing FRGP including BiSON, NLR, and Quilt activities
Over 117 logical networks, approximately 210 monitorable network devices, and over 3,000 network-attached devices at NCAR/UCAR

NETS provides a vital service to the atmospheric and oceanographic research communities by ensuring supercomputing resources are linked to scientists at NCAR and throughout the university research community. These connectivity activities are essential to the effective use of NCAR/UCAR's scientific resources, and they foster the overall advancement of scientific inquiry.

The primary NETS accomplishments in 2005 include:

Participating in the National LambdaRail (NLR) Project
FL0 and CG1 cabling projects completed
Dark Fiber Expansion (BiSON, DREAM)
Participating in The Quilt Project

These projects, along with many other NETS 2005 accomplishments, are described in this report.

Local Area Network (LAN) projects

NETS supports both NCAR/UCAR network needs as well as the special networking needs of SCD itself. Therefore, all LAN projects are further subdivided as being either NCAR/UCAR LAN projects or SCD LAN projects.

NCAR/UCAR LAN projects

UCAR network infrastructure recabling projects:
The common goal of all UCAR recabling projects is to provide each workspace with a standard set of dedicated data communications links. The overall plan calls for each workspace to be provisioned with a standard Telecommunications Outlet (TO) that connects four Category 6e (CAT6e) twisted-pair cables and two pairs of multi-mode optical fiber. Additionally, intra-building (trunk) cabling must be installed to concentrate all workspace cables to intermediate and central locations.

Concurrent with recabling, each network device is delivered 100 Mbs of dedicated bandwidth via a dedicated Ethernet packet-switch port as well as a phone connection. Such dedicated-port access offers substantial networking performance improvements over shared-media Ethernet access.

NETS designed and installed permanent network infrastructure for CG1. NETS also participated in the CG/FL bike path design and the Jeffco hanger networking design and implementation. NETS designed and implemented network and cabling for the new FL0 building. NETS assisted in the extensive relocation and related cabling for FL4 staff. NETS participated in the design of the FL0 computer room and the FL2 3095 computer room. NETS evaluated networking cable for HIAPER and assisted with termination and testing.

Network monitoring project:
NETS continues to use HP Openview, flowscan, Prognosis and Cricket as its principal tools for network monitoring and statistics gathering. NETS is in the process of converting to Nagios from HP Openview.

Additionally, NETS has installed certain specialized network monitors. NLANR's MOAT organization has placed a GigE monitor at NCAR's Mesa Lab and installed a GigEMON at the Front Range GigaPOP. MOAT has also placed an AMP monitor at the Mesa Lab.

Local serial-access project:
NETS supports several terminal servers for providing serial console access to various computer and networking equipment. Serial support is also provided for the few serial terminals remaining at UCAR.

NETS CSAC support project:
The NCAR/UCAR Computer Security Advisory Committee (CSAC) is chartered by the ITC to assess the state of computer and network security at NCAR/UCAR, and to make recommendations to assist NCAR and UCAR management in setting policies related to the security of computers and other devices attached to the NCAR/UCAR network. CSAC membership is composed of technical representatives located throughout the various NCAR/UCAR organizations. NETS is involved with CSAC since nearly all security policies involve various types of network-connected devices located between the networks belonging to the external world and the UCAR networks that are being protected from the external world. Based on CSAC recommendations, NETS continues to implement new gateway router filters to improve network security for UCAR. NETS also cooperates on wireless, RAS, and VPN security measures.

VLAN Splitting Project/Layer 2/3 Design:
NETS is finalizing the re-engineering of our backbone to provide higher reliability and redundancy to network-based services. Previously, NETS allowed single subnets and VLANs to span across all campuses. This was a manageable design when NCAR was located in only two main campuses. The design offered increased convenience for users in being able to request any subnet to be activated anywhere at the Mesa Lab or Foothills Lab campuses.

NETS was driven to re-evaluate this design with the addition of the third major campus, Center Green, and a desire of increased reliability for VoIP and business continuity.

Completion of the project's final stage is expected before the end of the year.

Multicast support activities project:
Internet Protocol (IP) multicast is a bandwidth-conserving technology that reduces traffic by simultaneously delivering a single stream of information to thousands of corporate recipients and homes. Applications that take advantage of multicast include videoconferencing, distance learning, and distribution of scientific data. IP Multicast delivers source traffic to multiple receivers without adding any additional burden on the source or the receivers while using the least network bandwidth of any competing technology. Multicast packets are replicated in the network by routers enabled with Protocol Independent Multicast (PIM) and other supporting multicast protocols resulting in the most efficient delivery of data to multiple receivers possible. Multicast technology is used for the Access Grid video and audio conferencing applications. NETS continues to support and enhance multicast services for UCAR.

UPS project:
NETS continues to install UPS (Uninterruptible Power Supply) units into all new telecommunication closets for networking equipment to receive short-term standby power in the event of a power failure. UPS units also help filter out damaging power spikes. Upgrading, expanding, and maintaining these devices is an ongoing process.

Grounding:
NETS is in the process of grounding all communications closets and NETS hardware to eliminate static issues causing hardware and phone failures. This is a technically difficult and time-consuming project.

Wireless network access:
NETS supports wireless network access in all public areas and conference rooms, and has completed the process of deploying wireless access in all office areas as well. NETS also designed, tested, and installed a long-distance, high-speed, wireless link between Center Green and Foothills Lab, the Mesa Lab and Foothills Lab, and the Mesa Lab and Jeffco. These links provide backup connectivity for wired links.

Voice over IP (VoIP):
VoIP, in its full implementation, is the complete merging of data, voice, and video networks. Traditional separate voice systems such as PBXs, telephone cabling infrastructure, and proprietary telephone handsets are replaced with computer-based call managers, IP telephones, and IP-based voicemail devices that are Ethernet attached just like any other networked device.

With a VoIP system, the same cabling infrastructure that is used for traditional data network devices can also be used for VoIP devices. Such a combination simplifies the overall cabling infrastructure and lowers total costs of installation and maintenance. Additional, ongoing cost savings are realized through simplified moves, adds, and changes made possible through a VoIP-based system. The same network monitoring system can be used to monitor both VoIP and data network devices.

NETS implemented call forwarding off-campus and is testing field project support for VoIP. Additional business continuity has been added into the VoIP system.

SCD LAN projects

In addition to its overall NCAR/UCAR networking responsibilities, NETS has additional special support responsibilities specific to SCD. NETS handles or consults on most of the host-based SCD networking, including all supercomputing networking. NETS is also responsible within SCD for several other tasks including:

Special cabling fabrication and installation
Networking and system testbeds
External and local conference networking and system support
Special-grant project networking needs
Security

NETS designed, tested and implemented a secure supercomputing routing plan.

Metropolitan Area Network (MAN) projects

Boulder Point-Of-Presence (BPOP):
The Boulder Point-of-Presence, or BPOP, is a collaboration of institutions in the Boulder area that cooperate to provide access to wide area network services. The BPOP members are: NCAR, University of Colorado at Boulder (UCB), NOAA, Colorado State University, the City of Boulder, and the County of Boulder. BPOP equipment consists of a router, an ATM switch, and an Ethernet switch located at the UCB Telecommunications Center.

The BPOP offers two primary services to its members: access to ICG Dedicated Internet service and access to the FRGP. Access to ICG is provided via an OC12 link connected to BRAN fiber. A Gigabit Ethernet connection to the FRGP is provided over dark fiber. This link is shared by NCAR, NOAA, CSU, and the City/County of Boulder. UCB has its own OC-12 link to the FRGP. UCB and the BPOP back up each other's links to the FRGP in Denver.

BRAN project:
BRAN is the Boulder Research and Administration Network, which is an eleven-mile fiber network in Boulder built and operated by the four BRAN partners to privately interconnect their Boulder-area facilities. The BRAN partners include:

City of Boulder
University of Colorado at Boulder
National Center for Atmospheric Research
NOAA-Boulder and NIST-Boulder

UCAR has greatly reduced intra-Boulder circuit costs by utilizing BRAN fiber. Active BRAN circuits involving UCAR facilities include ML-FL, ML-FL Security, City-UCB, NOAA-ML, CU-ICG, CSU-UCB, CU-ML, UCB-FRGP (via Level3), CG-ML. BRAN carries voice and data traffic. NETS continues to actively participate in the BRAN management committee and the BRAN technical committee.

Remote-working and home-access project:
Remote access continues to be provided by four digital T1 PRI lines connected to a Cisco AS5300 Remote Access Server (RAS). Each T1 PRI line provides twenty-three 56-Kbps channels that can support analog or ISDN dial-in access. Long-distance access via direct 1-800 lines is overlaid on one of the PRI lines. Telnet, PPP, and ARAP access are supported on the Cisco RAS device. NETS is in the process of increasing security on the RAS.

Dark fiber efforts:
NETS worked with the University of Wyoming, Colorado State University, and NOAA to secure fiber from Laramie to the FRGP and to provide a northern protect fiber ring in Colorado and Wyoming. This project was successfully completed and is called BiSON. BiSON is engineered, operated and managed by NETS.

NETS worked on the Western Lights project to investigate alternative and diverse fiber paths in the western U.S. so that Utah can provision a protect path to the FRGP.

NETS continued to work with CDOT, Colorado School of Mines, and Denver University to build a fiber ring to connect these sites to the FRGP.

NETS worked with CSU, CSU-Pueblo, UC-Boulder, UC-CS, and UCDHSC to activate a southern fiber path connecting Pueblo and Colorado Springs to the FRGP with private fiber.

Wide Area Network (WAN) projects

Front Range GigaPOP (FRGP):
The Front Range GigaPOP (FRGP) is a consortium of universities, non-profit corporations, and government agencies that cooperate in an aggregation point called the FRGP to share Wide Area Networking (WAN) services connecting to the Commodity Internet, Abilene/Internet2, NLR, and to each other. UCAR operates the Front Range GigaPOP under contract to the other members. The primary FRGP networking facilities are located in the Auraria Campus computer room of the University of Colorado Denver Health Sciences Center (UCDHSC) and in the Level3 Denver POP located at 1850 Pearl Street in Denver. The current partners are:

CSM - Colorado School of Mines
CSU - Colorado State University system
DU - Denver University
FLC - Fort Lewis College
Ithaka (provider of online scientific journal articles)
NCAR - National Center for Atmospheric Research
NOAA - National Oceanic and Atmospheric Administration
State - State of Colorado
UCB - University of Colorado at Boulder
UCCS - University of Colorado at Colorado Springs
UCDHSC - University of Colorado Health Sciences Center
University of Utah
UW - University of Wyoming

There are several GigaPOPs similar to the FRGP throughout the U.S., and a number of advantages are gained by sharing services through such GigaPOPs:

Costs for WAN services are reduced for each partner
Expertise among partners can be shared
A higher level of services can be purchased than the individual institutions could afford
There is more buying power among a consortium
There are great economies of scale
Quilt membership and their negotiated discounts with Commodity Internet providers are available only to GigaPOPs

UCAR has provided the engineering and Network Operations Center (NOC) support for the FRGP, with the service costs incurred by UCAR being shared by all FRGP members. NETS believes that the greater service and bandwidth obtained through the FRGP are important enough for UCAR to operate the FRGP by providing engineering and NOC services, and the FRGP has agreed that NETS has the most qualified engineering and NOC staff to provide the very best engineering and NOC services for the FRGP.

The FRGP is a critical service for UCAR staff as well as all of the other FRGP partners, and the FRGP has proved to be an extremely successful technical project as well as an excellent collaboration with the Colorado, Wyoming, and Utah research and education community.

NETS made many significant improvements to the FRGP during 2005. These include:

Completed the BiSON project.
Developed a plan to move Abilene OC12 to a private fiber path saving $50K/year.
Developed a plan to move Qwest GigE to private fiber path saving $50K/year.
Acquired collocation space via ICG at 910 15th Street.

The Quilt:
The Quilt is a project whose participants are nonprofit advanced regional network organizations dedicated to advancing research and education in the United States by:

Providing a broad range of advanced networking services to their constituents, including network engineering, management, and operation; regional connectivity and exchange; and promotion and coordination of regional activities
Facilitating innovative and successful projects and productive working relationships

The Quilt's specific purposes and objectives are to:

Provide advanced network services to the broadest possible research and educational community
Promote end-to-end continuity, consistency, reliability, interoperability, efficiency, and cost-effectiveness in the development and delivery of advanced network services by means which, at the same time, foster innovation and reflect the diversity of its members
Represent our common interests to backbone network service providers, industry, government, standard-setting organizations, and other organizations involved in or influencing the development and delivery of advanced network services

The FRGP joined The Quilt in June 2001, and NETS represents the FRGP on the Quilt Steering Committee and the Quilt Executive Committee. NETS actively serves on the Network Facilities Project, the Regional Fiber Project, and the Peering Project. The FRGP continued to take advantage of Quilt discounted pricing for Commodity Internet services, which significantly decreased prices paid by FRGP for Commodity Internet services this past year.

National LambdaRail (NLR):
NETS has completed its connection to NLR and is in the process of activating Layer 2 switch services and Layer 3 with other NLR member sites.

vBNS+ :
The vBNS+ network is a production ATM network owned and operated by MCI. The vBNS+ interconnects the NSF supercomputing centers, universities, and other customers. UCAR remains connected to the vBNS+, and NETS continues its support of the vBNS+ as long as MCI continues to provide this service to UCAR.

Internet2/Abilene:
Internet2 operates the Abilene network, a SONET 192 (9.6 Gb/s) national backbone supporting high-performance connectivity and Internet innovation within the U.S. research university community. The set of advanced services supported include IPv6, Jumbo frame, and multicast. As of October 2005, Abilene has 44 connected entities serving 241 institutions. UCAR joined UCAID and attached to Abilene via the FRGP in FY2000.

Other network projects

Projects listed in this section are ones that don't neatly fit into the Research, LAN, MAN, or WAN project classification scheme.

Telecommunications:
NETS is responsible for all voice communications at UCAR. Responsibilities include engineering, maintenance, configuration, and operation of the telecommunications cabling plant, the VoIP system, and telephone handsets. Telephone directories and UCAR telephone reception are also NETS responsibilities. See the VoIP subsection of the LAN section above for more information on work in this area.

Westnet:
Westnet is an affinity group that grew out of the NSFnet regional network called Westnet. NETS' long-term participation continued, and NETS continues in its leadership role. NETS is a member of the Westnet Steering Committee and leads the effort to plan and run bi-annual meetings that include technical presentations from members and vendors. Westnet provides powerful political and technical contacts with Rocky Mountain area universities that are UCAR members and that share common information technology concerns. The current Westnet members include:

CU Boulder
CU Denver
Colorado State University
University of Denver
University of Wyoming
University of Utah
Utah State University
Brigham Young University
Arizona State University
University of Arizona
University of New Mexico
New Mexico State University
New Mexico TechNet
Idaho State University
Boise State University
South Dakota School of Mines and Technology
UCAR

Project-tracking system:
The project-tracking system continued in full production in FY2005. NETS work requests and projects are opened, tracked, and closed with this project-tracking system. The use of project-tracking tools is necessary because of the vast number and variety of projects. NETS is working with SCD to implement a new project-tracking system using Extraview.

Property-tracking system:
NETS used the Remedy-based asset manager system to track its small property equipment. NETS is also responsible for tracking all large NETS equipment, including the VoIP phone system. Presently, NETS tracks approximately 388 equipment items. (See NETS Inventory).

Business continuity:
It is UCAR's policy to develop and maintain a corporate-wide contingency program, known as the Business Continuity Plan (BCP). This plan aims to provide the corporation with every opportunity to withstand a catastrophic event, be it accidental, man-made, or natural, and resume total operations in an efficient and effective manner. It is expected that, with the BCP in place, management will be able to provide the swift and decisive leadership that will be necessary for a successful recovery. Also, because of the guidance provided by the BCP, it is expected that employees will be able to carry out their tasks and responsibilities efficiently and effectively. NETS actively contributes related material to this document and process.

Documentation:
NETS has an ongoing effort to create and maintain documentation in support of project management and network support. NETS created an SLA for UCARnet per ACCIS recommendations. NETS created an FRGP Conditions of Use (CoU) and a FRGP Survivability Assessment.

NETS NCAB project:
The Network Coordination and Advisory Board (NCAB) consists of appointed technical representatives from the NCAR divisions and UCAR. The purpose of NCAB is to advise NETS concerning network strategy, planning, policy, expansion, and management issues for all of NCAR and UCAR. The work of NCAB continues to be indispensable to the success of networking at UCAR. NCAB meets monthly.

Committee support

NETS representatives have attended and supported the following NCAR/UCAR committees:

The Network Coordination Advisory Board (NCAB) Marla Meehl, NETS Liaison
The Computer Security Advisory Committee (CSAC)
Advisory Committee for Central Infrastructure Service (ACCIS) - Jim Van Dyke chair

ACCIS subcommittees - Belinda Housewright

Information Technology Council (ITC)
The Computer Room Planning Committee (CRPC)

SCD committees:

The CISL Executive Committee
The SCD Security Policy Committee (SSPC)

External committees:

The Quilt Executive Committee
The Quilt Steering Committee
The Westnet Steering Committee
The Front Range GigaPOP Management Committee (FMC)
The Front Range GigaPOP Technical Committee (FTC)
The BRAN Technical Committee
The BRAN Management Committee
The NLR Board of Directors
NLR Engineering Committee
HOPI Design Team

FY2005 Annual Report