Chapter 12 – Troubleshooting Ethernet LANs

• The chapter focuses on the processes of verification and troubleshooting 
• Verification refers to the process of confirming whether a network is working as designed 
• Troubleshooting refers to the follow-on process that occurs when the network is not working as designed, determining the real reason why its not working correctly so that it can be fixed 

This chapter looks at four key technical topics that matter to verifying and troubleshooting LANs: 

•  analyzing switch interfaces and cabling 
•  predicting where switches will forward frames 
•  troubleshooting port-security
•  analyzing VLANs and VLAN trunks 

Problem isolation and documentation:  

• Problem isolation refers to the process of taking what you know about an issue, confirming that there is a problem and determining which device or cable is the problem 
• This step works best when person documents what they find 

Resolve or escalate: 

• Resolving the problem means fixing that problem that is the root of the network shutdown or device issues 
• If you cannot find the root cause of the problem, escalate the problem! 
• Find and escalate the technical support or management support depending on whether the next step requires more technical expertise 

Verify or monitor: 

• You have a problem, you isolate it, document it, determine root cause, and resolve it 
• Now you need to verify that it really worked 

Troubleshooting on the exams

1. Analyzing switch interface status and statistics 
2. Analyzing where switches will forward frames 
3. Analyzing port-security
4. Analyzing VLANs and VLAN trunking 

1. Analyzing switch interface status and statistics: 

• Finding out whether each switch interfaces works, and if working, whether any statistics reveal any problems 

# show interfaces status 
# show interfaces
# show interfaces description

Those two commands list two code status names line status and protocol status. 

• the line status generally refers to whether the layer 1 is working 
• the protocol status refers to whether layer 2 is working 

Interface speed and duplex issues

• interfaces can be configured to use specific speed with speed (10, 100, 1000) interface subcommand 
• interfaces can be configured to use specific duplex with duplex (half, full) interface subcommand 
• with both configured, switch or router disables IEE standard auto negotiation process on that interface 

Auto-negotiation: 

• when auto negotiation works on both devices = they agree to the fastest speed supported and use full duplex if supported on both devices or half duplex if it is not  
• when auto negotiation works on just one device = the device using auto negotiation chooses default duplex setting based on the current speed 

Rules for auto-negotiation: 

• If the speed is not known use 10Mbps and half duplex 
• If the switch senses speed without auto negotiation 
  • if speed is 10 or 100Mbps use half duplex 
  • if speed is 1000Mbps use full duplex 

To identify duplex mismatch problems, check duplex settings on each end of the link and watch for incrementing collision and late collision counters 

Common layer 1 problems on working interfaces: 

• when the interface reaches the connect up/up state the switch considers the interface to be working 
• whenever the physical transmission has problems, receiving device might receive frame whose bits have changed values 
• these frames do not pass the error detection logic in FCS field in Ethernet trailer. 
• receiving device discards the frame and counts it as some kind of input error. 
• Cisco switches list this error as CRC error (Cyclic redundancy check) 

• number of input errors, and number of CRC errors, are few counters in the output of show interfaces command 
• challenge is to know which counters are important, which counters show that a problem is happening, and which ones are normal and of no concern 

Following list explains in short each of these counters: 

• runts: frames that did not meet the minimum frame size requirements (64 bytes including 18byte destination MAC, source MAC, type and FCS). Can be caused by collisions 
• giants: frames that exceed the maximum frame size requirement (1518 bytes, including the 18byte destination MAC, source MAC, type and FCS) 
• CRC: received frames that did not pass the FCS math, also can be caused by collisions 
• Frame: received frames that have an illegal format, for example, ending with a partial byte, also can be caused by collisions 
• Packet output: total number of packets (frames) forwarded out the interface 
• Output errors: total number of packets (frames) that the switch port tried to transmit but experienced a problem 
• Collisions: counter of all collisions that occur when the interface is transmitting frame 
• Late collisions: often point to a duplex mismatch) 

• many of these counters occur as part of the CSMA/CD process used when half duplex is enabled 
• collisions occur as a normal part of the half duplex logic imposed by CSMA/CD 
• one problem, called late collisions, points to the classic duplex mismatch problem! 
• excessive interference on the cable can cause input error counters to keep growing larger, especially the CRC counter. 
• if the CRC errors grow, but the collisions counters do not, problem might be interference on the cable 

2) Predicting where switches will forward frames: 

• section looks at the key part of the troubleshooting process for Ethernet LANs: predicting where frames should go in a LAN so that you can compare what should happen versus what is actually happening in a LAN 

Predicting the contents of the MAC address table: 

• switches learn MAC addresses, and then use the entries in the MAC address table to make forwarding/filtering decision for each frame 
• to know how a particular switch will forward the frame, you need to examine the MAC address table 
• the MAC table entries define where frames will flow 

Analyzing the forwarding path: 

• troubleshooting revolves around three big ideas:  
  • predicting what should happen 
  • determining what is happening that is different that what should be 
  • figuring out why that different behavior is happening 
   

This section discusses how to look at what is actually happening in a VLAN based on MAC address tables 

Following list summarizes switch forwarding logic including LAN switching features: 

1. process functions on the incoming interface, if the interface is up/up 

• if configured, apply port security logic to filter frame as appropriate 
• if the port is an access port, determine the interface’s access VLAN 
• if the port is a trunk, determine the frame’s tagged VLAN 

2. make forwarding decision. Look for the frame’s destination MAC address in MAC address table. If the destination MAC is… 

• found (unicast), forward the frame out the only interface listed in the matched address table entry 
• not found (unicast), flood the frame out all other access ports (except the incoming port) in the same VLAN, plus out trunks that have not restricted the VLAN from that trunk 
• broadcast, flood the frame, with the same rules as the previous step 

For an example of this process, consider frame sent by Barney to its default gateway R1 (0200.5555.5555). 

1. input interface processing 
   • the port does not happen to have port security enabled 
   • SW1 receives the frame on Fa0/12 interface, an access port in VLAN10 
2. make forwarding decision: SW1 looks in its MAC address table for entries in VLAN10 
   • SW1 finds an entry for 0200.5555.5555, associated with VLAN 10, outgoing Gi0/1, so SW1 forwards the frame only out interface Gi0/1 
3. frame Barney with destination address of R1 is on its way to SW2.  
   • the port does not happen to have port security enabled 
   • SW2 receives the frame on its G0/2, a trunk, the frame lists a tag of VLAN10 (SW2 removes 802.1Q header) 
4. make a forwarding decision: SW2 looks for its MAC table for entries in VLAN10 
   • SW2 finds an entry and forwards the frame to the Fa0/13 

3. Analyzing port security operations on an interface: 

• any analysis of the forwarding process should consider any security features that might discard some frames or packets 
• port security has three basic features based on which determines which frames to filter: 
  • limit which specific MAC addresses can send and receive frames on a switch interface, discarding frames from other MAC addresses 
  • limit the number of MAC addresses using the interface, discarding frames from other MAC addresses learned after the maximum limit is reached 
  • combination of the previous two points 

The first port security troubleshooting step should be to find which interfaces have port security enabled 

Then determine as to whether any violations are currently occurring 

General process to find port security issues is as follows: 

• identify all interfaces on which port security is enabled (show running-config or show port-security) 
• determine whether a security violation is occurring based in part of the violation mode of the interface’s port security configuration 
  • shutdown: the interface will be in an err-disabled state, and the port security port status will be secure-down 
  • restrict: the interface will be in a connected state, the port security port status will be secure-up, but the show port-security interface command will show an incrementing violations counter 
  • protect: the interface will be in a connected state, and the show port-security interface command will not show an incrementing violations counter 
• in all cases, compare the port security configuration to the diagram and to the Last Source Address field in the output of the show port-security interface command 

Troubleshooting shutdown mode and err-disabled recovery: 

• troubleshooting shutdown and err-disabled interface state from the determine whether a security violation is occurring list above: 
• this state verifies that the interface has been configured to use port security, that violation has occurred, and that no traffic is allowed on the interface at the present time 
• this interface state implies that the shutdown violation mode is used, because it is the only one of the three port security modes that causes the interface to be disabled !!! 
• to recover from an err-disabled state, interface must be shut down with shutdown command, and then enabled with the no shutdown command 

• output of show port-security interface command lists helpful troubleshooting things 
• port status field and secure-shutdown means that the interface is disabled for all traffic as a result of a violation while in port security shutdown mode. 
• also means that the interface state should be err-disabled 
• the violation counter at the bottom, does not keep incrementing 
• once the first violating frame triggers IOS to move port to err-disabled state, IOS ignores any other frames until engineer uses shutdown and no shutdown command 
• Last source Address: VLAN can show which MAC address violated the rule!!! 

Troubleshooting restrict and protect modes: 

• restrict and protect violation modes take much different approach to securing ports 
• these modes still discard offending traffic, but the interface remains in an up/up state, and in a port security state of secure-up 
• as a result, port continues to forward good traffic and discard offending traffic 
• such approach can be challenging when troubleshooting 
• basically, in order to solve this problem, we need to know about this rule that interface is up and up but something is wrong 
• #show port-security interface command reveals whether protect mode has discarded frames using the last source address item in the output. 
• port security protect mode does not increase violation count of bad MAC addresses 
• port security restrict mode does increase violation counter and syslog messages 

4. Analyzing VLANs and VLAN trunks: 

• Four potential issues are as follows: 
  • identify all access interfaces and their assigned access VLANs and reassign into correct VLANs as needed 
  • determine whether the VLANs both exist and are active on each switch. If not, configure and activate the VLANs to resolve problems as needed 
  • check the allowed VLAN lists, on the switches on both ends of the trunk, and ensure that the lists of allowed VLANs are the same 
  • check for incorrect configuration settings that result in one switch operating as a trunk with the neighboring switch no operating as trunk 
   

Ensuring that the right access interfaces are in the right VLANs: 

• to ensure that each access interface has been assigned to the correct VLAN, engineers need to determine which interfaces are access and which are trunk 
• also determine the assigned access VLANs on each interface, and compare the information to the blueprint of the network 

• start with #show vlan and #show vlan brief commands 
• those two commands list all known VLANs and the access interfaces assigned to VLAN 
• those two commands do not list operational trunks! 

Access VLANs not being defined: 

• switches do not forward frames for VLANs that are not configured or configured but disabled (shutdown) 
• section shows best ways to tell that a switch knows that a particular VLAN exists, and if exists, determines the state of the VLAN 
• VLAN can be defined to switch in two ways: 
  • using the vlan number global configuration command 
  • or it can be learned from another switch using VTP 
• #show vlan command always lists all VLANs known to the switch. However, show running-config does not