NCAR's Boulder PoP at CU's Telecommunications Building

In March 2004, NETS relocated gin, NCAR's Juniper M20 border router, to the University of Colorado's Telecommunications Building. CU Telecom, as the building is called, is situated on the ring portion of the BRAN fiber. Placing network equipment on this ring increases network reliability in case of a BRAN fiber cut. NCAR, The City and County of Boulder, Colorado State University, the University of Colorado at Boulder, and NOAA all benefit from this increased reliability.

In early 2008, the majority of gin's routing functionality was moved to the Sup720 3BXL tcom-gs-1. In early 2009, gin was decomissioned and the ICG OC-12 was replaced with a Level3 GigE DIA service, using the same demarcation point at Level3/ICG Boulder.

BPoP Network Architecture

The BPoP installation at CU Telecom now consists of:

an NCAR Cisco 6509 switch/router, tcom-gs-1, for GigE and 10-GigE connections
A pair of BiSON Movaz (Adva) wave-division multiplexers for BiSON connections (east and west)
a GigE connection to CU's tcom switch, for CU and Level3 DIA connections.

This equipment is in Rack 6, at the end of a row in the basement of CU TCOM. Connections to BPoP participants, including NCAR, are made thru GigE and 10-GigE connections to tcom-gs-1. The MOVAZ WDM equipment terminates a lambda from Longmont for a Boulder County MAN connection. All BRAN fibers are routed up into fiber trays to the BRAN rack in the row facing Rack 6, about 15 feet away. Out-of-band and console access is provided thru CU's Cyclades terminal server, neo-ts-tcom.colorado.edu

tcom-gs-1 connects to two redundant internet providers:

Level3 via CU's tcom switch, over BRAN to the Level3/ICG Boulder hub on Walnut Street.
FRGP via BiSON-A and B, both sides of BRAN to the Level3/ICG Boulder hub and on to the FRGP's Level3 colocation in Denver.

BPoP BRAN Connection Specifics

Portions of all BPoP connections are carried over BRAN fiber.

NCAR

NCAR has two 10-GigE connections to the BPoP hardware - one to Mesa Lab and one to Foothills Lab. All 6 pairs of NCAR's red-tube BRAN fiber were terminated at TCOM specifically for the BPoP relocation. RD-11/12 southbound is patched to Mesa Lab at ECOT, and RD-11/12 northbound is patched at ICG to WH-11/12 to connect Foothills lab.
A table of NCAR's BRAN fiber utilization is available.

CSU

CSU originally installed the MOVAZ multiplexer at CU TCOM to bring their connection from Fort Collins on ICG and BRAN fiber. This was eventually expanded into the full BiSON ring, which provides redundant paths into Level3 Denver Colo at 1850 Pearl. CSU no longer peers directly with the BPoP router at TCOM. However, CSU obtains Level3 DIA out of BPoP via BiSON -> tcom-gs-1 (at layer 2) -> CU's switch.

City of Boulder

The City of Boulder uses two GigE connections Etherchanneled together to provide redundancy. BRAN fibers BL-1/2 are used both north and southbound to the City's switch at the Public Safety building. From the FRGP perspective, the City and County of Boulder are both secondaries behind the NCAR router.

NOAA

NOAA has a GigE as well as a 10-GigE connection to tcom-gs-1, carried on BRAN directly to ECOT, where its cross-patched to NOAA. Like CSU, they use tcom-gs-1 (at layer 2) for access to Level3 DIA and the FRGP.

FRGP

The 10-GigE connection to the FRGP uses the BiSON-B path to Denver as primary, to avoid sharing a path and vulnerability with the Level3 DIA service. The FRGP is also reachable over the GigE BiSON-A path. The primary FRGP connection is carried on BRAN RD-9/10 southbound. NCAR's "inner ring" Red tube is patched to White at ECOT, which is spliced back to Red at Pearl Street, so it arrives at Level3/ICG from the east on RD-9/10. At Walnut, the connection is patched to ICG IRU fiber to Level3 at 1850 Pearl in Denver.

Level3 DIA

Level3 DIA service has replaced the ICG OC-12. It is patched on BRAN BL-1/2 along the same path as the original ICG OC-12. From Walnut, it rides a Level3 "crossroads" GigE backhaul to 1850 Pearl, where it lands on car1.denver1.level3.net for "HSIP" service.

BRAN fiber diagram

Troubleshooting

Emergency console access is provided by a telephone modem and CU's network, which doesn't rely on the BPoP installation. Console cables are provided to the 6509, and the MOVAZ shelves. Access neo-ts-tcom.colorado.edu remotely by dialing 303-417-0020 or by "ssh -l ncarbpop neo-ts-tcom.colorado.edu". Access details are found in the usual place.
To gain physical access to the Telecom building, contact Robert Roybal or David Lindblad or the ITS help desk. Normally, contact Robert for hardware/software issues and David for BRAN fiber issues. In any case, please notify backbone@colorado.edu as soon as practical.

Failover in case of fiber cut

Resistance to a BRAN fiber cut was the primary motivation for locating the BPoP equipment at CU TCOM. BPoP connections to the Internet are diverse: The Level3 connection is made around the North and West side of BRAN, while the FRGP connection is made around both sides of BRAN. NCAR and the City have redundant fiber connections to the BPoP, and NOAA has a backdoor connection to NCAR's Mesa Lab which provides them with a backup wireless path via FL.

If a fiber cut in the BRAN ring were to occur, one of the BPoP's Internet connections could fail. Also, one of the City's connections would fail, and the same would hold for one of NCAR's ML or FL connections. The County's MAN connection would either stay up or fail entirely, depending on where the cut occured.

Spare modules and parts for the Catalyst 6509 and the Movaz shelves are available at NCAR ML.