This page is provided for NETS staff if they are faced with severe network problems involving one of our Cisco switches. Twice during early 2001, the entire UCAR network experienced outages due to misbehaving gigabit Ethernet ports. This document will give you the steps that have been successfully used in the past to isolate where these problem ports might be.
Problem: the network is obviously 'broken' (all connectivity seems lost; no hosts can reach anything internally or externally). You should probably start with getting console access on one of the core switches (ml-mr-c1-gs or fl2-3095-c1-gs). Even switch console access may be slow and unresponsive.
Solution: You are probably experiencing a broadcast storm of some kind. Determine the source port via:
clear counters
show mac
Look for (hopefully) a single port with exceptionally high multicast or broadcast counters. Once the port is isolated, follow the problem back device by device until the real source is found (e.g., if the problem is first noticed at ML, you might put a laptop on the console port of ml-mr-c1-gs, and using the steps outlined above, determine that the problem is coming from fl2-3095-c1-gs; you would then need to go to the console port of fl2-3095-c1-gs and keep following those steps until hopefully you can isolate an individual port (or module) that is causing the problems somewhere at FL).
In the case of a gigabit Ethernet NIC spewing out huge amounts of broadcast
traffic, you will most likely have to disable the port to restore network stability
(via either 'set port disable <mod_num/port_num>' or simply physically
disconnecting the questionable port). Refer also to the "Networking Emergency
Policy" under the NCAB policies
page for more information and guidance on this matter.