Chapter 12 – Troubleshooting Ethernet LANs

  • The chapter focuses on the processes of verification and troubleshooting 
  • Verification refers to the process of confirming whether a network is working as designed 
  • Troubleshooting refers to the follow-on process that occurs when the network is not working as designed, determining the real reason why its not working correctly so that it can be fixed 

This chapter looks at four key technical topics that matter to verifying and troubleshooting LANs: 

  •  analyzing switch interfaces and cabling 
  •  predicting where switches will forward frames 
  •  troubleshooting port-security
  •  analyzing VLANs and VLAN trunks 

Problem isolation and documentation:  

  • Problem isolation refers to the process of taking what you know about an issue, confirming that there is a problem and determining which device or cable is the problem 
  • This step works best when person documents what they find 

Resolve or escalate: 

  • Resolving the problem means fixing that problem that is the root of the network shutdown or device issues 
  • If you cannot find the root cause of the problem, escalate the problem! 
  • Find and escalate the technical support or management support depending on whether the next step requires more technical expertise 

Verify or monitor: 

  • You have a problem, you isolate it, document it, determine root cause, and resolve it 
  • Now you need to verify that it really worked 

Troubleshooting on the exams

  1. Analyzing switch interface status and statistics 
  2. Analyzing where switches will forward frames 
  3. Analyzing port-security
  4. Analyzing VLANs and VLAN trunking 

1. Analyzing switch interface status and statistics: 

  • Finding out whether each switch interfaces works, and if working, whether any statistics reveal any problems 
# show interfaces status 
# show interfaces
# show interfaces description

Those two commands list two code status names line status and protocol status. 

  • the line status generally refers to whether the layer 1 is working 
  • the protocol status refers to whether layer 2 is working 
Table 12-2 LAN Switch Interface Status Codes 
Line Status 
Protocol 
Status 
administratively down 
down 
down 
up 
down 
up 
down 
down 
down 
(err-disabled) 
up 
Interface 
Status 
disabled 
notconnect 
notconnect 
err-disabled 
connected 
Typical Root Cause 
The shutdown command is configured on the 
interface. 
No cable; bad cable; wrong cable pinouts; 
speed mismatch; neighboring device is (a) 
powered off, (b) shutdown, or (c) error 
disabled. 
Not expected on LAN switch physical 
interfaces. 
Port security has disabled the interface. 
The interface is working.

Interface speed and duplex issues

  • interfaces can be configured to use specific speed with speed (10, 100, 1000) interface subcommand 
  • interfaces can be configured to use specific duplex with duplex (half, full) interface subcommand 
  • with both configured, switch or router disables IEE standard auto negotiation process on that interface 

Auto-negotiation: 

  • when auto negotiation works on both devices = they agree to the fastest speed supported and use full duplex if supported on both devices or half duplex if it is not  
  • when auto negotiation works on just one device = the device using auto negotiation chooses default duplex setting based on the current speed 

Rules for auto-negotiation: 

  • If the speed is not known use 10Mbps and half duplex 
  • If the switch senses speed without auto negotiation 
    • if speed is 10 or 100Mbps use half duplex 
    • if speed is 1000Mbps use full duplex 
PCI 
Figure 12-3 
SWI 
Autonegotiation 
Fails 
GiO/2 
SW2 
Autonegotiation 
Is Disabled 
speed 100 
duplex full 
0200.0101.0101 
Conditions to Create a Duplex Mismatch Between SWI and SW2

To identify duplex mismatch problems, check duplex settings on each end of the link and watch for incrementing collision and late collision counters 

Common layer 1 problems on working interfaces: 

  • when the interface reaches the connect up/up state the switch considers the interface to be working 
  • whenever the physical transmission has problems, receiving device might receive frame whose bits have changed values 
  • these frames do not pass the error detection logic in FCS field in Ethernet trailer. 
  • receiving device discards the frame and counts it as some kind of input error. 
  • Cisco switches list this error as CRC error (Cyclic redundancy check) 
Example 12-3 Interface Counters for Layer 1 Problems 
SWI# show interfaces faO/13 
! lines omitted for brevity 
Received 284 broadcasts (O multicast) 
runts , O giant* , O throttles 
O input errors, O CRC, O frame, O overrun, O ignored 
O watchdog, 281 multicast, O pause input 
O input packets with dribble condition detected 
5226 packets output, 10849674 bytes, O underruns 
o 
O 
O 
O 
O 
colligioriÖ, 1 interface resets 
output errors C) 
unknown protocol drops 
babbles , 
late collision, O deferred 
lost carrier, O no carrier, O PAUSE output 
output buffer failures, O output buffers swapped out
  • number of input errors, and number of CRC errors, are few counters in the output of show interfaces command 
  • challenge is to know which counters are important, which counters show that a problem is happening, and which ones are normal and of no concern 

Following list explains in short each of these counters: 

  • runts: frames that did not meet the minimum frame size requirements (64 bytes including 18byte destination MAC, source MAC, type and FCS). Can be caused by collisions 
  • giants: frames that exceed the maximum frame size requirement (1518 bytes, including the 18byte destination MAC, source MAC, type and FCS) 
  • CRC: received frames that did not pass the FCS math, also can be caused by collisions 
  • Frame: received frames that have an illegal format, for example, ending with a partial byte, also can be caused by collisions 
  • Packet output: total number of packets (frames) forwarded out the interface 
  • Output errors: total number of packets (frames) that the switch port tried to transmit but experienced a problem 
  • Collisions: counter of all collisions that occur when the interface is transmitting frame 
  • Late collisions: often point to a duplex mismatch) 
  • many of these counters occur as part of the CSMA/CD process used when half duplex is enabled 
  • collisions occur as a normal part of the half duplex logic imposed by CSMA/CD 
  • one problem, called late collisions, points to the classic duplex mismatch problem! 
  • excessive interference on the cable can cause input error counters to keep growing larger, especially the CRC counter. 
  • if the CRC errors grow, but the collisions counters do not, problem might be interference on the cable 

2) Predicting where switches will forward frames: 

  • section looks at the key part of the troubleshooting process for Ethernet LANs: predicting where frames should go in a LAN so that you can compare what should happen versus what is actually happening in a LAN 

Predicting the contents of the MAC address table: 

  • switches learn MAC addresses, and then use the entries in the MAC address table to make forwarding/filtering decision for each frame 
  • to know how a particular switch will forward the frame, you need to examine the MAC address table 
  • the MAC table entries define where frames will flow 
SWI MAC Table 
VLAN Address 
SW2 MAC Table 
VLAN Address 
10 
10 
10 
0200.2222.2222 
0200. 5555.5555 
SWI 
Interface 
FaO/9 
FaO/12 
GiO/1 
GWI 
10 
10 
10 
0200 .2222 .2222 
0200 .5555. 5555 
Interface 
GiO/2 
GiO/2 
FaO/13 
FaO/9 
Fred 
FaO/12 
Barney 
0200.2222.2222 
GiO/2 
sw2 
FaO/13 
GiO/1 0200.5555.5555 
RI 
Figure 12-5 Predictions for MAC Table Entries on SWI and SW2
Example 12-4 Examining SWI and SW2 Dynamic MAC Address Table Entries 
SWI# show mac address - table dynamic 
Mac Address Table 
VI an 
10 
10 
10 
Mac Address 
0200.2222 . 2222 
0200.5555. 5555 
Type 
DYNAMIC 
DYNAMIC 
DYNAMIC 
SW2# show mac address- table dynamic 
Mac Address Table 
VI an 
10 
10 
10 
Mac Address 
0200.2222 . 2222 
5555. 
0200. 
5 5 55 
Type 
DYNAMIC 
DYNAMIC 
DYNAMIC 
Ports 
FaO/9 
Fa0/12 
Gi0/1 
Ports 
Gi0/2 
Gi0/2 
FaO/13

Analyzing the forwarding path: 

  • troubleshooting revolves around three big ideas:  
    • predicting what should happen 
    • determining what is happening that is different that what should be 
    • figuring out why that different behavior is happening 
     

This section discusses how to look at what is actually happening in a VLAN based on MAC address tables 

Following list summarizes switch forwarding logic including LAN switching features: 

  1. process functions on the incoming interface, if the interface is up/up 
  • if configured, apply port security logic to filter frame as appropriate 
  • if the port is an access port, determine the interface’s access VLAN 
  • if the port is a trunk, determine the frame’s tagged VLAN 
  1. make forwarding decision. Look for the frame’s destination MAC address in MAC address table. If the destination MAC is… 
  • found (unicast), forward the frame out the only interface listed in the matched address table entry 
  • not found (unicast), flood the frame out all other access ports (except the incoming port) in the same VLAN, plus out trunks that have not restricted the VLAN from that trunk 
  • broadcast, flood the frame, with the same rules as the previous step 

For an example of this process, consider frame sent by Barney to its default gateway R1 (0200.5555.5555). 

  1. input interface processing 
    • the port does not happen to have port security enabled 
    • SW1 receives the frame on Fa0/12 interface, an access port in VLAN10 
  2. make forwarding decision: SW1 looks in its MAC address table for entries in VLAN10 
    • SW1 finds an entry for 0200.5555.5555, associated with VLAN 10, outgoing Gi0/1, so SW1 forwards the frame only out interface Gi0/1 
  3. frame Barney with destination address of R1 is on its way to SW2.  
    • the port does not happen to have port security enabled 
    • SW2 receives the frame on its G0/2, a trunk, the frame lists a tag of VLAN10 (SW2 removes 802.1Q header) 
  4. make a forwarding decision: SW2 looks for its MAC table for entries in VLAN10 
    • SW2 finds an entry and forwards the frame to the Fa0/13 

3. Analyzing port security operations on an interface: 

  • any analysis of the forwarding process should consider any security features that might discard some frames or packets 
  • port security has three basic features based on which determines which frames to filter: 
    • limit which specific MAC addresses can send and receive frames on a switch interface, discarding frames from other MAC addresses 
    • limit the number of MAC addresses using the interface, discarding frames from other MAC addresses learned after the maximum limit is reached 
    • combination of the previous two points 

The first port security troubleshooting step should be to find which interfaces have port security enabled 

Then determine as to whether any violations are currently occurring 

General process to find port security issues is as follows: 

  • identify all interfaces on which port security is enabled (show running-config or show port-security) 
  • determine whether a security violation is occurring based in part of the violation mode of the interface’s port security configuration 
    • shutdown: the interface will be in an err-disabled state, and the port security port status will be secure-down 
    • restrict: the interface will be in a connected state, the port security port status will be secure-up, but the show port-security interface command will show an incrementing violations counter 
    • protect: the interface will be in a connected state, and the show port-security interface command will not show an incrementing violations counter 
  • in all cases, compare the port security configuration to the diagram and to the Last Source Address field in the output of the show port-security interface command 

Troubleshooting shutdown mode and err-disabled recovery: 

  • troubleshooting shutdown and err-disabled interface state from the determine whether a security violation is occurring list above: 
  • this state verifies that the interface has been configured to use port security, that violation has occurred, and that no traffic is allowed on the interface at the present time 
  • this interface state implies that the shutdown violation mode is used, because it is the only one of the three port security modes that causes the interface to be disabled !!! 
  • to recover from an err-disabled state, interface must be shut down with shutdown command, and then enabled with the no shutdown command 
Example 12-5 Using Port Security to Define Correct MAC Addresses of Particular 
Interfaces 
! The first command lists all interfaces on which port security has been enabled, 
! and the violation mode, under the heading "Security Action. 
SWI# show port- security 
Secure Port MaxSecureAddr CurrentAddr 
SecurityVi01ation 
( Count ) 
( Count ) 
1 
( Count ) 
1 
Total Addresses in System (excluding one 
Max Addresses limit in System (excluding 
mac 
one 
per 
mac 
1 
port ) 
per port) 
ecurity Actio 
$hu t doyp 
8192 
! The next command shows the err-disabled state , 
SWI# show interfaces FaO/13 status 
implying a security violation. 
Port 
FaO/13 
Name 
Status 
VI an 
rr-disab1Åä 1 
Duplex 
auto 
Speed Type 
auto 10/100BaseTX 
! The next command's output has shading for several of the most important facts. 
SWI# show port-security interface FaO/13
Port Security 
Port Status 
Violation Mode 
Aging Time 
Aging Type 
SecureStatic Address Aging 
Maximum MAC Addresses 
Total MAC Addresses 
Conf igured MAC Addresses 
Sticky MAC Addresses 
Last Source Address : Vlan 
Security Violation Count 
. Enabled 
Secure - shutdow+ 
Shutdown 
. O mins 
. Absolute 
Disabled 
1 
1 
1 
0200.3333 .3333:2 
1
  • output of show port-security interface command lists helpful troubleshooting things 
  • port status field and secure-shutdown means that the interface is disabled for all traffic as a result of a violation while in port security shutdown mode. 
  • also means that the interface state should be err-disabled 
  • the violation counter at the bottom, does not keep incrementing 
  • once the first violating frame triggers IOS to move port to err-disabled state, IOS ignores any other frames until engineer uses shutdown and no shutdown command 
  • Last source Address: VLAN can show which MAC address violated the rule!!! 

Troubleshooting restrict and protect modes: 

  • restrict and protect violation modes take much different approach to securing ports 
  • these modes still discard offending traffic, but the interface remains in an up/up state, and in a port security state of secure-up 
  • as a result, port continues to forward good traffic and discard offending traffic 
  • such approach can be challenging when troubleshooting 
  • basically, in order to solve this problem, we need to know about this rule that interface is up and up but something is wrong 
  • #show port-security interface command reveals whether protect mode has discarded frames using the last source address item in the output. 
  • port security protect mode does not increase violation count of bad MAC addresses 
  • port security restrict mode does increase violation counter and syslog messages 

4. Analyzing VLANs and VLAN trunks: 

  • Four potential issues are as follows: 
    • identify all access interfaces and their assigned access VLANs and reassign into correct VLANs as needed 
    • determine whether the VLANs both exist and are active on each switch. If not, configure and activate the VLANs to resolve problems as needed 
    • check the allowed VLAN lists, on the switches on both ends of the trunk, and ensure that the lists of allowed VLANs are the same 
    • check for incorrect configuration settings that result in one switch operating as a trunk with the neighboring switch no operating as trunk 
     

Ensuring that the right access interfaces are in the right VLANs: 

  • to ensure that each access interface has been assigned to the correct VLAN, engineers need to determine which interfaces are access and which are trunk 
  • also determine the assigned access VLANs on each interface, and compare the information to the blueprint of the network 
Table 12-3 Commands That Can Find Access Ports and VLANs 
EXEC Command 
show vlan brief 
show vlan 
show vlan id num 
show interfaces type 
number switchport 
show mac address-table 
Description 
Lists each VLAN and all interfaces assigned to that VLAN (but 
does not include operational trunks) 
Lists both access and trunk ports in the V LAN 
Identifies the interface's access V LAN and voice V LAN, plus the 
configured and operational mode (access or trunk) 
Lists MAC table entries, including the associated V LAN
  • start with #show vlan and #show vlan brief commands 
  • those two commands list all known VLANs and the access interfaces assigned to VLAN 
  • those two commands do not list operational trunks! 

Access VLANs not being defined: 

  • switches do not forward frames for VLANs that are not configured or configured but disabled (shutdown) 
  • section shows best ways to tell that a switch knows that a particular VLAN exists, and if exists, determines the state of the VLAN 
  • VLAN can be defined to switch in two ways: 
    • using the vlan number global configuration command 
    • or it can be learned from another switch using VTP 
  • #show vlan command always lists all VLANs known to the switch. However, show running-config does not 
Table 12-6 
Command 
shutdown 
Commands for Catalyst Switch Configuration 
no shutdown 
switchport port-security 
violation {protect I restrict I 
shutdown} 
speed {auto 1 10 1 100 1 1000} 
duplex {auto I full I half} 
Description 
Interface subcommands that administratively disable and 
enable an interface, respectively 
Interface subcommand that tells the switch what to do if 
an inappropriate MAC address tries to access the network 
through a secure switch port 
Interface subcommand that manually sets the interface speed 
Interface subcommand that manually sets the interface duplex
Table 12-7 Chapter 12 EXEC Command Reference 
Command 
Description 
show mac address-table [dynamic I Displays the MAC address table. The static option displays 
static] [address hw-addrl [interface information about the restricted or static settings. 
interface-idl Ivlan vlan-idl 
show port-security [interface 
interface-idl [addressl 
show interfaces [type numberl 
show interfaces description 
show interfaces [type numberl 
status 
show interfaces Vype numberl 
switchport 
show interfaces ltype numberl 
trunk 
show vlan brief, show vlan 
show vlan id num 
show vtp status 
Displays information about security options configured 
on an interface. 
Displays detailed information about interface status, 
settings, and counters. 
Displays one line of information per interface, with a 
two-item status (similar to the show interfaces command 
status), and includes any description that is configured on 
the interfaces. 
Displays summary information about interface status and 
settings, including actual speed and duplex, a single-item 
status code, and whether the interface was autonegotiated. 
Displays a large variety of configuration settings and 
current operational status, including VLAN trunking 
details, access and voice VLAN, and native VLAN. 
Lists information about the currently operational trunks 
(or just for the trunk listed in the command) and the 
VLANs supported on those trunks. 
Lists each VLAN and all interfaces assigned to that VLAN 
but does not include trunks. 
Lists both access and trunk ports in the VLAN. 
Lists the current VTP status, including the current mode.

Leave a Reply