Catalyst 6xxx Backplane and Linecard Architecture

I (Pete Sakosky) created this page to piece together Cisco docs, TAC explanations, and my own observations. It covers mainly the architecture of the backplane and line cards. PFC, MSFC, and DFC functions and architecture are not within the scope of this document.

To understand the architecture of the 6500 series chassis and line cards, you need to start with the 6000 series. This is important, as conceptually, Fabric-enabled cards for the 6500 series can be thought of as a Catalyst 6000 on a line card. Also, if you are not running fabric-enabled line cards in a 6500, you are using essentially a 6000 series backplane.

The 6000 series backplane is composed of a Data bus (D bus), a Results Bus (R bus), and a Control bus (C bus). The Data bus handles the transfer of actual data traffic between modules in the chassis at a rate of 32Gbps. The Results bus handles communication between line card ports and the switching logic located on the Supervisor modules. The Control bus handles information between the line card ASICs and the Network Management Processor (NMP) on the Supervisor. The C bus is alternately referred to as the Ethernet Out-of-Band Channel (EOBC). This can be confusing with the TAC.

Bus data rates:

D Bus - 32Gbps

R Bus - 4Gbps

C Bus (EOBC)- 100Mbps half-duplex ethernet

Bus Functions:

D Bus- The Data plane
The D bus is a shared medium. When a frame is put on the bus by a line card, all ports (these are backplane ports, not line card ports) attached to the bus see the frame. Remember this! So, an analogy would be if you mailed a local letter in a small town of 9 possible destinations (8 homes and one post office) and your mail system was a "shared medium" the letter would be simultaneously received at the post office (the Supervisor) and each of the other 7 homes in the town (your home is excluded). This function, coupled with the ability to "Pipeline" is the core of the Catalyst switching architecture. "Pipelining" is the ability of line cards to switch up to 31 frames onto the D bus before the results of the first frame are returned by the forwarding engine. To continue the analogy, 31 letters in your small town could be sent by various houses before the first letter sent is fully delivered. The 32nd letter would have to wait.

R Bus- A Control plane
The R bus returns the results of the forwarding engine's logic in response to frames sent over the D bus. So, as frames from the D bus are seen by the switching logic in the supervisor, their addressing information is *concurrently* put through the switching logic. The analogy here is that the post office sees the letter in the mail, analyzes the address, and then calls up each potential destination address/house over the phone and tells it to either keep the letter or throw it away. Of course in the case of the switch, other information is sent over the R bus as well, such as header and rewrite instructions for the ports.

C Bus (EOBC)- A Control plane
The C bus is a channel used more for administrative communication between supervisors and line cards It is used for Line Control Processor (LCP) code download and communication between the Network Management Processor (NMP) and LCP's. Serial Control Protocol (SCP) is used for supervisor and line cards communication over this channel while Serial Link Protocol (SLP) is used for communication between redundant supervisors.

Hidden Cisco C Bus diagnostic commands

switch(enable)test scp [module number] [packet size] [packet count]

This command tests scp communication over the C bus from the supervisor you are logged into, to the module specified.

switch(enable)show scp failcnt

This command shows statistics of failure from the "test scp" command.

6000 Series Line Card Architecture

Line cards in the 6000 series utilize two port ASICs: the COIL ASIC and the PINNACLE ASIC. COIL ASICs are found on 10/100 ethernet boards and perform the function of buffering, QOS, and congestion management on a per-port basis. Each COIL ASIC supports 12 10/100 ports and provides 128k of buffer per port. The PINNACLE ASIC performs the same functions as the COIL, only for gigabit ethernet ports. The PINNACLE handles four gigabit ports on gigabit line cards, or in the case of 10/100 line cards, four COIL ASICs. PINNACLE provides for 512k buffering per port and puts frames onto the backplane. From the analogy above, PINNACLE is one of the "houses" in the town.

Summary of 6000 series architecture

The 6000 series has one available backplane composed of a 32Gbps Data bus, a Results bus, and a Control bus. The Results bus is a control plane, while the C bus is more of an "admin plane", thus you will never see data packets (such as CDP, SNMP, etc.) going over the R or C bus. The D bus is a shared medium that delivers a copy of every frame on it to every port. The Supervisor determines the frame destination while the frame is simultaneously being copied to every line card. Switching logic, which is centralized in the supervisor, is "shared" with the line cards over the R bus, telling the line card ASICs to either discard the frame or use specified headers and forwarding information.

Catalyst 6500 Backplane

The Catalyst 6500 backplane incorporates the 6000's 32Gbps backplane and adds a 256Gbps Crossbar Switching Fabric. It is essentially two backplanes in one chassis. To utilize the crossbar fabric a Switch Fabric Module (SFM) and fabric-enabled line cards are needed. When looking inside an empty 6500 chassis, the 32Gbps backplane connectors are on the right and the crossbar fabric connectors are on the left. Non-fabric-enabled line cards do not have connectors for the crossbar and cannot physically connect to it, while conversely, fabric-only line cards cannot physically connect to the 32Gbps bus.

Switch Fabric Module and Crossbar Fabric

The SFM is essentially a 16 port "switch" with each port connecting to a line card. The 6500 chassis provides two crossbar ports per slot, with each port clocked at 8Gbps. The 256Gbps backplane is derived from 16 ports x 8Gbps x 2 (these are full-duplex ports). If you do the math you can see that there are more potential crossbar ports (18 in a 6509) than SFM ports (16). This is not a problem however, as most line cards will not use both crossbar ports. There are two options for backplane redundancy with a 6500 chassis: redundant SFMs or the "native" redundancy of fabric-enabled cards. Redundant SFMs are straightforward and switch off if one fails. The native redundancy of fabric-enabled cards refers to the fact that should the crossbar fabric fail in a chassis full of fabric-enabled cards, the backplane will failover to the 32Gbps bus. The obvious limitation there being your max backplane capacity drops from 256 to 32Gbps.

6500 Series Line Card Architecture

Non fabric-enabled - Utilize the 32Gbps bus only and do not support distributed switching.

Fabric-enabled - Utilize both the crossbar fabric and the 32Gbps bus by having one port on each. 16Gbps max throughput to the Crossbar fabric. Supports optional Distributed Forwarding Card (DFC) and has one on-board 16Gbps bus.

Fabric-only - Utilize only the crossbar fabric via both crossbar ports available to the chassis slot. 32Gbps max throughput to the crossbar fabric. DFC daughter card is embedded with two on-board 16Gbps busses.

With 6500 series line cards you still see the COIL and PINNACLE ASICs. The difference is that PINNACLE now signals to the local 16Gbps switching fabric. The MEDUSA ASIC is introduced to arbitrate frames onto the Crossbar fabric. Fabric-enabled line cards have one 16Gbps bus serving all ports with a MEDUSA ASIC serving as the gateway to the Crossbar fabric. Fabric-only line cards have two 16Gbps busses, with each bus having their own MEDUSA and 16Gbps channel to the Crossbar fabric. In the case of the fabric-only cards, each bus handles half of the available line card ports.

Summary of 6500 series architecture

The 6500 offers the standard 32Gbps backplane originally offered by the 6000 and adds the ability to use a 256Gbps Crossbar fabric with an additional SFM and fabric-enabled line cards ( Note- Cisco is hinting that the release of the Supervisor III module in Q2 2003 will do away with the need for SFM modules to be installed in the chassis to utilize the Crossbar fabric, thus saving at least one slot in the chassis). Fabric-enabled line cards are themselves a small model of the 6000 chassis, with each line card having an on-board 16Gbps bus. A notable difference however is that the R bus and C bus are not present on the line cards.