NCAR's Boulder PoP at CU's Telecommunications Building

In March 2004, NETS relocated gin, NCAR's Juniper M20 border router, to the University of Colorado's Telecommunications Building. CU Telecom, as the building is called, is situated on the ring portion of the BRAN fiber. Placing network equipment on this ring increases network reliability in case of a BRAN fiber cut. NCAR, The City and County of Boulder, Colorado State University,  the University of Colorado at Boulder, NOAA, and BiSON (Bi-State Optical Network) all benefit from this increased reliability.

Diagrams

BPOP Network Architecture

The BPOP installation at CU Telecom consists of: This equipment is in Rack 6. Connections to BPOP participants, including NCAR, are made thru GigE connections to tcom-gs-1. The ADVA (MOVAZ) WDM equipment connects 10Gbps waves from Denver for BiSON and NCAR's Teragrid connection. The ADVA WDM equipment also terminates lambdas from Longmont for a Boulder County MAN connection, and another lambda from Fort Collins for for BiSON (CSU, UWyo, and backup service to Denver). All BRAN fibers are routed up into fiber trays to the BRAN rack in the row facing Rack 6, about 10 feet away. Out-of-band and console access is provided thru CU's Cyclades terminal server, neo-ts-tcom.colorado.edu.

Gin connects to two redundant internet providers:



BPOP BRAN Connection Specifics

Portions of all BPOP connections are carried over BRAN fiber.

NCAR

NCAR has two connections to the BPOP hardware - a 10GE to Mesa Lab and a GigE to Foothills Lab. All 6 pairs of NCAR's red-tube BRAN fiber were terminated at TCOM specifically for the BPOP relocation. RD-11/12 southbound is patched to Mesa Lab at ECOT, and RD-11/12 northbound is patched at ICG to WH-11/12 to connect Foothills lab.
A table of NCAR's BRAN fiber utilization is available.

NCAR's Teragrid

NCAR's Mesa Lab has a direct 10G connection to Teragrid carried on a dedicated BiSON wave to Denver. The client connection is made directly onto BRAN fiber WH-1/2 to Mesa Lab.

Boulder County

Boulder County adds a wave in Longmont; its GigE LX client connection is made directly back to BRAN on BL-3/4 northbound and doesn't land on the tcom-gs-1 switch.

City of Boulder

The City of Boulder uses two GigE connections Etherchanneled together to provide redundancy. BRAN fibers BL-1/2 are used both north and southbound to the City's switch at the Public Safety building.

CSU

CSU initially installed the ADVA (Movaz) multiplexer at CU TCOM to bring their connection from Fort Collins on Level3 (ICG) and BRAN fiber. The BiSON fiber ring was completed after more ADVA (Movaz) gear was installed for the connection to Denver. Thanks to BiSON, CSU now enjoys redundant fiber connections to Boulder and Denver. The fiber from Fort Collins arrives on BRAN RD-3/4 directly from the Level3 (ex ICG) Boulder Hub. Two GigE's on a single BiSON wave to Fort Collins connect to l3-gw-1.

FRGP

After ADVA (Movaz) gear was installed on the fiber to Denver, a 2xGE connection was made to the FRGP. This wave was subsequently replaced with a 10G BiSON connection to l3-gw-1. The WDM fiber connection to the FRGP is carried the long way around the BRAN ring, to avoid sharing a path and vulnerability with the northbound BiSON fiber and the Level3 (ICG) DIA service ATM OC-12. The connection to the FRGP's Level3 collocation at 1850 Pearl in Denver is carried on BRAN RD-9/10 southbound. NCAR's "inner ring" Red tube is patched to White at ECOT, which is spliced back to Red at Pearl Street, so it arrives at the Walnut Street PoP from the east on RD-9/10. At the PoP the connection is patched to Level3 (ICG) IRU fiber to in Denver.

NOAA

NOAA has two parallel GigE connections to tcom-gs-1. The first one is carried on BRAN SL-1/2 directly to ECOT, where its cross-patched to SL-5/6 to NOAA. The second one was installed later and presumably goes around the BRAN ring but the exact path is known only to NOAA

Troubleshooting

Emergency console access is provided by a telephone modem and CU's network, which doesn't rely on the BPOP installation. Console cables are provided to the M20, the 6509, and the MOVAZ WDM boxes. The information necessary for console access is available on the OOB page.

To gain physical access to the Telecom building, information is available on the Collocation Access Procedures page.

Failover in case of fiber cut

Resistance to a BRAN fiber cut was the primary motivation for locating the BPOP equipment at CU TCOM. BPOP connections to the Internet are diverse: The Level3 ATM DIA connection is made around the North and West side of BRAN, while the Level3 collocation connection is made around the South and East side of BRAN. BiSON's fiber connection to Fort Collins parallels the Level3 ATM connection. NCAR, the City, and presumably NOAA have redundant fiber connections to the BPOP, and NOAA has a backdoor connection to NCAR's Mesa Lab to take advantage of NCAR's ML-FL wireless link.

If a fiber cut in the BRAN ring were to occur, the Level3 DIA service could fail. One leg of the BiSON ring would break, but the other should stay up. If one of the City's connections were to fail, the other should stay up. The same would hold for one of NCAR's ML or FL connections and possibly NOAA. The County's MAN connection would either stay up or fail entirely, depending on where the cut occurred.

What to do in case of a BPOP equipment failure

gin

Gin has redundant routing and SSB engines, and failover of these systems can occur automatically. The M20's FPC (Flexible PIC Carrier) and its GigE, ATM, and Tunneling PICs are not redundant, but spares are stored at NCAR in Tonic, Gin's backup router. With some reworking of VLANs (easy) and ATM circuits (harder), Tonic can be configured to replace Gin without removing it from the Mesa Lab machine room.

Tonic will need to be configured with netserver:/tftpboot/gin.conf.