CCIE TO BE

TO BE

2009-03-03T05:57:00.000-08:00

On my way back home from San Jose. I want to thank those that follow this blog and especially IPexpert for all the help. Will right more soon :)

-CCIE# 23707

PIM NBMA, DR and RPF issues

2009-02-25T18:08:00.000-08:00

Below is the topology. RIP is running everywhere, PIM-SM on all interfaces and everyone has R4 at 192.168.100.4 as the static RP.

R1 has the following config on its LAN interface:

interface Ethernet0/0
 ip address 192.168.0.1 255.255.255.0
 ip pim sparse-mode
 ip igmp join-group 239.0.0.1

Let's ping from R6:

R6#ping 239.0.0.1 re 5  

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 239.0.0.1, timeout is 2 seconds:
.....
R6#

Hmmm....what gives? Let's look at R4:

R4#sho ip pim neighbor
PIM Neighbor Table
Neighbor          Interface                Uptime/Expires    Ver   DR
Address                                                            Prio/Mode
192.168.34.3      Ethernet0/0              03:29:50/00:01:39 v2    1 / S
192.168.100.2     Serial0/0                02:25:22/00:01:38 v2    1 / S
192.168.100.5     Serial0/0                02:25:22/00:01:39 v2    1 / DR S
192.168.100.1     Serial0/0                02:25:22/00:01:38 v2    1 / S

R4#sho ip mroute 239.0.0.1 | be \(
(*, 239.0.0.1), 00:24:31/00:02:33, RP 192.168.100.4, flags: S
  Incoming interface: Null, RPF nbr 0.0.0.0
  Outgoing interface list:
    Serial0/0, 192.168.100.2, Forward/Sparse, 00:24:31/00:02:33

(192.168.56.6, 239.0.0.1), 00:02:03/00:02:45, flags: T
  Incoming interface: Serial0/0, RPF nbr 192.168.100.5
  Outgoing interface list:
    Serial0/0, 192.168.100.2, Forward/Sparse, 00:02:03/00:00:57

R4#

Well, it looks R2 is showing up in the OIL, but why isn't R1? It is a PIM neighbor afterall. The reason is because R2 has won the DR election and has the right to forward traffic. So it is the neighbor that sends PIM joins to R4. R1 receives the traffic, but it comes in on its LAN interface and thus fails the RPF check.

R1#debug ip mpacket
IP multicast packets debugging is on
03:40:21: IP(0): s=192.168.56.6 (Ethernet0/0) d=239.0.0.1 id=197, ttl=251, prot=1, len=114(100), not RPF interface
03:40:23: IP(0): s=192.168.56.6 (Ethernet0/0) d=239.0.0.1 id=198, ttl=251, prot=1, len=114(100), not RPF interface

It is important to remember we have at least two ways to resolve this:

1) Make R1 the DR

R1(config)#int e0/0
R1(config-if)#ip pim dr-priority 3000

R6#ping 239.0.0.1 re 1

Type escape sequence to abort.
Sending 1, 100-byte ICMP Echos to 239.0.0.1, timeout is 2 seconds:

Reply to request 0 from 192.168.100.1, 60 ms
R6#

R1(config-if)#^Z
03:41:47: IP(0): s=192.168.56.6 (Serial0/0) d=239.0.0.1 (Ethernet0/0) id=207, ttl=252, prot=1, len=100(100), mforward

2) Static mroute to R2 for 192.168.56.6

R1(config)#int e0/0
R1(config-if)#no ip pim dr-priority 3000
R1(config-if)#exit
R1(config)#ip mroute 192.168.56.0 255.255.255.0 192.168.0.2

Make sure to clear mroutes otherwise previous state may mislead you :)

R4#clear ip mroute *

R6#ping 239.0.0.1 re 1

Type escape sequence to abort.
Sending 1, 100-byte ICMP Echos to 239.0.0.1, timeout is 2 seconds:

Reply to request 0 from 192.168.100.1, 56 ms
R6#

This is one of those labs where I had no idea where I was going and I ended up with a nice troubleshooting scenario. If multicast is one your weaknesses than I highly recommend digging in and making something happen. Debug ip mpacket works best with "no ip mroute-cache" on your interfaces. In this scenario, I started troubleshooting on R5, then worked my way around to resolve the issue :)

PIM Forwarder and the Assert Mechanism

2009-02-23T20:26:00.000-08:00

I know, it's a cool name for a band, huh? Ladies and gentlemen...PIM Forwarder and the Assert Mechanism! Anyways, I always get confused about PIM DR and PIM Forwarder so this is to clear up my confusion. Here we take a look at PIM Forwarder and how to verify the assert process is working.

Here is the topology:

Here is what I have enabled:
-RIP on all interfaces
-ip multicast-routing on all routers
-ip pim sparse-dense on all interfaces
-ip igmp join-group 239.0.0.1 on R5 ethernet

For debugging:
-no ip mroute-cache
-debug ip mpacket
-ping

Scenario 1: R2 is the PIM Forwarder based on highest IP

From R4 we ping twice:

R4#ping 239.0.0.1 re 2

Type escape sequence to abort.
Sending 2, 100-byte ICMP Echos to 239.0.0.1, timeout is 2 seconds:

Reply to request 0 from 192.168.0.5, 20 ms
Reply to request 0 from 192.168.0.5, 20 ms
Reply to request 1 from 192.168.0.5, 8 ms

On R1 and R2 we see the following:

R1#
*Mar 2 02:05:36.795: IP(0): s=192.168.34.4 (Serial0/1) d=239.0.0.1 (Ethernet0/0) id=70, ttl=253, prot=1, len=100(100), mforward
*Mar 2 02:05:36.799: IP(0): s=192.168.34.4 (Ethernet0/0) d=239.0.0.1 id=70, ttl=252, prot=1, len=114(100), not RPF interface
*Mar 2 02:05:38.787: IP(0): s=192.168.34.4 (Ethernet0/0) d=239.0.0.1 id=71, ttl=252, prot=1, len=114(100), not RPF interface

R2#
*Mar 1 02:25:00.567: IP(0): s=192.168.34.4 (Serial0/1) d=239.0.0.1 (Ethernet0/0) id=70, ttl=253, prot=1, len=100(100), mforward
*Mar 1 02:25:00.571: IP(0): s=192.168.34.4 (Ethernet0/0) d=239.0.0.1 id=70, ttl=252, prot=1, len=114(100), not RPF interface
*Mar 1 02:25:02.559: IP(0): s=192.168.34.4 (Serial0/1) d=239.0.0.1 (Ethernet0/0) id=71, ttl=253, prot=1, len=100(100), mforward

Notice that each router sent the first packet onto the LAN and R5 responded to both. We can tell because R4 got two replies. What also happened is that R1 and R2 each saw that very same packet on their LAN interfaces. Immediately the PIM Assert process took over. Because both routers have the same AD (90) and metric (2) to the source, R2 won the right to forward based on highest IP.

Next we see that the second packet only gets forwarded by R2. Here we see that R2 has the A (Assert Winner) flag in its mroute entry. R1 has pruned that same interface.

R2#sho ip mroute 239.0.0.1 192.168.34.4 | be \(
(192.168.34.4, 239.0.0.1), 00:00:39/00:02:26, flags: T
  Incoming interface: Serial0/1, RPF nbr 192.168.23.3
  Outgoing interface list:
    Ethernet0/0, Forward/Sparse-Dense, 00:00:39/00:00:00, A

R1#sho ip mroute 239.0.0.1 192.168.34.4 | be \(
(192.168.34.4, 239.0.0.1), 00:01:27/00:01:34, flags: PT
  Incoming interface: Serial0/1, RPF nbr 192.168.13.3
  Outgoing interface list:
    Ethernet0/0, Prune/Sparse-Dense, 00:01:27/00:01:32

Scenario 2: R1 is the PIM Forwarder based on lowest AD

Now we change R1's AD for RIP below the default of 120:

R1(config)#router rip
R1(config-router)#distance 89

We see the same behavior from R4's perspective but now R1 has won the Assert process and is forwarding group 239.0.0.1 onto the LAN:

R4#ping 239.0.0.1 re 2

Type escape sequence to abort.
Sending 2, 100-byte ICMP Echos to 239.0.0.1, timeout is 2 seconds:

Reply to request 0 from 192.168.0.5, 12 ms
Reply to request 0 from 192.168.0.5, 12 ms
Reply to request 1 from 192.168.0.5, 8 ms
R4#

R1#sho ip mroute 239.0.0.1 192.168.34.4 | be \(
(192.168.34.4, 239.0.0.1), 00:00:07/00:02:54, flags: T
  Incoming interface: Serial0/1, RPF nbr 192.168.13.3
  Outgoing interface list:
    Ethernet0/0, Forward/Sparse-Dense, 00:00:07/00:00:00, A

R1#

How Route-Reflector clusters prevent loops

2009-02-23T15:12:00.000-08:00

This is the topology I used to get familiar with the concept:

The idea is fairly easy to understand. You never want to learn routes from someone who learned them from you (directly or indirectly). I made the peers one by one to step through the process.

Here is the route on R1:

R1#sho ip bgp 200.0.0.0
BGP routing table entry for 200.0.0.0/8, version 12
Paths: (1 available, best #1, table Default-IP-Routing-Table)
Advertised to update-groups:
3
100, (Received from a RR-client)
6.6.6.6 (metric 2) from 6.6.6.6 (6.6.6.6)
Origin IGP, metric 0, localpref 100, valid, internal, best
R1#

Now on R2, we see the first case of the origintaor-id as set by R1. And we also see the beginning of the cluster-list:

R2#sho ip bgp 200.0.0.0
BGP routing table entry for 200.0.0.0/8, version 9
Paths: (1 available, best #1, table Default-IP-Routing-Table)
Advertised to update-groups:
2
100
6.6.6.6 (metric 3) from 1.1.1.1 (1.1.1.1)
Origin IGP, metric 0, localpref 100, valid, internal, best
Originator: 6.6.6.6, Cluster list: 1.1.1.1
R2#

R2 appends itself to the cluster-list before advertising to R5:

R5#sho ip bgp 200.0.0.0
BGP routing table entry for 200.0.0.0/8, version 12
Paths: (1 available, best #1, table Default-IP-Routing-Table)
Advertised to update-groups:
2
100
6.6.6.6 (metric 2) from 2.2.2.2 (2.2.2.2)
Origin IGP, metric 0, localpref 100, valid, internal, best
Originator: 6.6.6.6, Cluster list: 2.2.2.2, 1.1.1.1
R5#

Eventually, these are the messages we get on R6 and R2, respectively.

R6#
*Mar 1 00:44:55.807: BGP(0): 5.5.5.5 rcv UPDATE about 201.0.0.0/8 -- DENIED due to: ORIGINATOR is us;
*Mar 1 00:44:55.811: BGP(0): 5.5.5.5 rcv UPDATE about 200.0.0.0/8 -- DENIED due to: ORIGINATOR is us;

R2#
*Mar 1 00:53:39.075: BGP(0): 3.3.3.3 rcv UPDATE about 201.0.0.0/8 -- DENIED due to: CLUSTERLIST contains our own cluster ID;
*Mar 1 00:53:39.083: BGP(0): 3.3.3.3 rcv UPDATE about 200.0.0.0/8 -- DENIED due to: CLUSTERLIST contains our own cluster ID;

My new favorite IOS message

2009-02-23T12:12:00.000-08:00

I don't know what my previous one was, but this is the new one:


R1(config-if)#traffic-shape rate 64000 ?    
 <0-100000000>  bits per interval, sustained
 
R1(config-if)#traffic-shape rate 64000 640
less than 1000 bits in an interval doesn't make sense
R1(config-if)#

Watch the RIP metric when summarizing redistributed routes

2009-02-14T23:34:00.000-08:00

I was reading through the GS archives and saw this interesting issue about the metrics of summarized routes after being redistributed into RIP.

Scenario:

R1---RIP---R2---OSPF---R5---5.5.5.5/32

R2 is redistributing OSPF to RIP as follows:

R2#sho run | sec router rip
router rip
 version 2
 redistribute ospf 1 metric 2
 network 192.168.0.0
 no auto-summary

R1 has the following route:

R1#sho ip route | sec 5.0.0.0
     5.0.0.0/32 is subnetted, 1 subnets
R       5.5.5.5 [120/2] via 192.168.0.2, 00:00:11, FastEthernet0/0

1) Manual Summary

R2(config)#int f0/0
R2(config-if)#ip summary-address rip 5.0.0.0 255.0.0.0 

R1#sho ip route | sec 5.0.0.0
R    5.0.0.0/8 [120/3] via 192.168.0.2, 00:00:01, FastEthernet0/0

The metric increased by 1.

2) Auto-summary

R2(config-if)#router rip
R2(config-router)#auto-summary 

R1#sho ip route | sec 5.0.0.0
R    5.0.0.0/8 [120/2] via 192.168.0.2, 00:00:05, FastEthernet0/0

The metric is the same as when redistributed.

CCIE Assessor Lab Review

2009-02-14T13:50:00.000-08:00

I don't know how much I can say about this so I will keep it brief. I purchased both assessor labs, one for today and one for tomorrow. I just completed the first one in about 2 hours. That left a good chunk of time to verify and run the assessment. I only missed two tasks and they were very simple mistakes.

My one worry was that it would take awhile to get used to the topology and the web interface. I spent about 30 minutes last night reading the user guide and it was smooth transition getting used to the GUI and the controls. This should not worry you.

I redrew a diagram and kept a task/point tracker. I read the lab before I started and first glanced seemed to be pretty easy. There are some things that will leave you scratching your head and that is good. The best part: There were no errors or typos in any tasks or drawings! :)

The telnet sessions are Java based and you have to open one in each window and then arrange them on your screen. I opened R1 first, the moved on so they were arranged in my taskbar in order. I don't expect many difference for tomorrow's session, so hopefully I do good.

OSPF filtering issue when virtual-links are present

2009-02-13T10:42:00.000-08:00

Here is the topology I will start off with:

R4 has two INTER-area routes to 1.1.1.1:

R4#sho ip route 1.1.1.1
Routing entry for 1.1.1.1/32
  Known via "ospf 1", distance 110, metric 4, type inter area
  Last update from 192.168.45.5 on Serial1/0, 00:00:11 ago
  Routing Descriptor Blocks:
    192.168.45.5, from 5.5.5.5, 00:00:11 ago, via Serial1/0
      Route metric is 4, traffic share count is 1
  * 192.168.34.3, from 2.2.2.2, 00:00:11 ago, via Serial1/1
      Route metric is 4, traffic share count is 1

If we want to filter the path from R2 through 192.168.34.3 we could do it this way:

R4(config)#access-list 1 permit 1.1.1.1
R4(config)#access-list 2 permit 2.2.2.2
R4(config)#route-map OSPF deny 10
R4(config-route-map)#match ip address 1
R4(config-route-map)#match ip route-source 2
R4(config-route-map)#route-map OSPF permit 20
R4(config-route-map)#router ospf 1
R4(config-router)#distribute-list route-map OSPF in
R4(config-router)#^Z
      
R4#sho ip route 1.1.1.1        
Routing entry for 1.1.1.1/32
  Known via "ospf 1", distance 110, metric 4, type inter area
  Last update from 192.168.45.5 on Serial1/0, 00:00:12 ago
  Routing Descriptor Blocks:
  * 192.168.45.5, from 5.5.5.5, 00:00:12 ago, via Serial1/0
      Route metric is 4, traffic share count is 1

But let's say we have a task that asks us to create a new area attached to R4 as follows:

Now we need two virtual-links and look at was happened to our route 1.1.1.1.

R4(config)#router ospf 1                 
R4(config-router)#area 1 virtual-link 5.5.5.5          
R4(config-router)#area 1 virtual-link 2.2.2.2

*Mar  3 01:31:13.935: %OSPF-5-ADJCHG: Process 1, Nbr 5.5.5.5 on
OSPF_VL2 from LOADING to FULL, Loading Done
*Mar  3 01:31:16.979: %OSPF-5-ADJCHG: Process 1, Nbr 2.2.2.2 on
OSPF_VL3 from LOADING to FULL, Loading Done

R4#sho ip route 1.1.1.1
Routing entry for 1.1.1.1/32
  Known via "ospf 1", distance 110, metric 4, type intra area
  Last update from 192.168.34.3 on Serial1/1, 00:00:00 ago
  Routing Descriptor Blocks:
  * 192.168.45.5, from 1.1.1.1, 00:00:00 ago, via Serial1/0
      Route metric is 4, traffic share count is 1
    192.168.34.3, from 1.1.1.1, 00:00:00 ago, via Serial1/1
      Route metric is 4, traffic share count is 1

What gives? Well now we are learning 1.1.1.1 as an INTRA-area route so the router-ID advertising the LSA has changed. We are now learning the route from type-1 LSAs originated by R1 directly in Area 0. If we filter based on router-id we will lose both paths so now we need to filter based on next-hop:

R4(config)#access-list 3 permit 192.168.34.3
R4(config)#no route-map OSPF
R4(config)#route-map OSPF deny 10
R4(config-route-map)#match ip add 1
R4(config-route-map)#match ip next-hop 3
R4(config-route-map)#route-map OSPF pe 20  
R4(config-route-map)#^Z

R4#sho ip route 1.1.1.1
Routing entry for 1.1.1.1/32
  Known via "ospf 1", distance 110, metric 4, type intra area
  Last update from 192.168.45.5 on Serial1/0, 00:00:02 ago
  Routing Descriptor Blocks:
  * 192.168.45.5, from 1.1.1.1, 00:00:02 ago, via Serial1/0
      Route metric is 4, traffic share count is 1

All of this change could of course been prevented had we read ahead :-)

OSPF on unnumbered links

2009-02-12T15:43:00.000-08:00

I was reviewing the OSPF chapter in the CCIE exam guide today and something irked me. It said that OSPF neighbors will become adjacent if one or both of the neighbors are using unnumbered interfaces between them. I swear this was not case as I had experienced before so I labbed it up.

R3#sho ip ospf ne

Neighbor ID     Pri   State           Dead Time   Address         Interface
2.2.2.2           0   FULL/  -        00:00:37    192.168.23.2    Serial1/1
4.4.4.4           0   FULL/  -        00:00:39    192.168.34.4    Serial1/0
R3#

R3(config)#int s1/0
R3(config-if)#ip unnumbered lo 0
R3(config-if)#
*Mar  2 06:31:01.600: %OSPF-5-ADJCHG: Process 1, Nbr 4.4.4.4 on Serial1/0
from FULL to DOWN, Neighbor Down: Interface down or detached

The adjcency will not come back up. Let's configure R4:

R4(config)#int s1/1
R4(config-if)#ip unnumbered lo 0
R4(config-if)#
*Mar  2 06:33:14.288: %OSPF-5-ADJCHG: Process 1, Nbr 3.3.3.3 on Serial1/1
from LOADING to FULL, Loading Done

There we go! If one side is unnumbered, the other side needs to be also. I am running 12.4(7) so maybe this was not the case awhile ago, but right now it seems so. There are a few other mistakes in this chapter, especially in the beginning quiz - SO QUESTION (LAB) EVERYTHING!

SNMP - sending traps to specific hosts

2009-02-12T10:41:00.000-08:00

This was an issue I ran into awhile ago. I was trying to send BGP traps to one host, and PIM traps to another. As you can see below, BGP traps were getting sent to both hosts when I used version 1.

When I had version 2c specified, traps were only sent to the host configured for BGP. I do not know if this is difference in the protocol, but it is something you may want to be aware of if you need to send traps to different hosts.

Version 1, traps get sent to both hosts:

R1#sho run | inc snmpsnmp-server enable traps bgp
snmp-server enable traps pim
snmp-server host 2.2.2.2 public  bgp
snmp-server host 3.3.3.3 public  pim

R1#clear ip bgp *
R1#
00:11:49: %BGP-5-ADJCHANGE: neighbor 172.12.14.4 Down User reset
00:11:49: SNMP: Queuing packet to 2.2.2.2
 00:11:49: SNMP: V1 Trap, ent bgp, addr 172.12.12.1, gentrap 6, spectrap 2 
 bgpPeerEntry.14.172.12.14.4 = 00 00  
 bgpPeerEntry.2.172.12.14.4 = 1
00:11:49: SNMP: Queuing packet to 3.3.3.3
00:11:49: SNMP: V1 Trap, ent bgp, addr 172.12.13.1, gentrap 6, spectrap 2 
  bgpPeerEntry.14.172.12.14.4 = 00 00  
 bgpPeerEntry.2.172.12.14.4 = 1
00:11:49: SNMP: Packet sent via UDP to 2.2.2.2
00:11:49: SNMP: Packet sent via UDP to 3.3.3.3

Version 2c, traps get sent to one as desired:

R1#sho run | inc snmpsnmp-server enable traps bgp
snmp-server enable traps pim
snmp-server host 2.2.2.2 version 2c public  bgp
snmp-server host 3.3.3.3 version 2c public  pim

R1#clear ip bgp *
R1#
00:13:09: %BGP-5-ADJCHANGE: neighbor 172.12.14.4 Down User reset
R1#
00:13:09: SNMP: Queuing packet to 2.2.2.2
00:13:09: SNMP: V2 Trap, reqid 21, errstat 0, erridx 0 
 sysUpTime.0 = 78967 
  snmpTrapOID.0 = bgpTraps.2 
 bgpPeerEntry.14.172.12.14.4 = 00 00  
 bgpPeerEntry.2.172.12.14.4 = 1
00:13:09: SNMP: Packet sent via UDP to 2.2.2.2
R1#

Messin' around with multicast boundary

2009-02-11T15:14:00.000-08:00

I got a multicast lab in dynamips going so I thought I would just play around with some lesser known commands and learn how they actually work.

Here is the topology:

R5---R6---R1---R2---R3---R4

R1 = MA and RP for 232/8, 233/8, 234/8


R4#show ip pim rp mapping 
PIM Group-to-RP Mappings

Group(s) 232.0.0.0/8
  RP 1.1.1.1 (?), v2v1
    Info source: 1.1.1.1 (?), elected via Auto-RP
         Uptime: 00:10:08, expires: 00:02:48
Group(s) 233.0.0.0/8
  RP 1.1.1.1 (?), v2v1
    Info source: 1.1.1.1 (?), elected via Auto-RP
         Uptime: 00:10:08, expires: 00:02:47
Group(s) 234.0.0.0/8
  RP 1.1.1.1 (?), v2v1
    Info source: 1.1.1.1 (?), elected via Auto-RP
         Uptime: 00:10:08, expires: 00:02:46

R4 has the following on Loopback 0:


interface Loopback0
 ip address 4.4.4.4 255.255.255.255
 ip pim sparse-mode
 ip igmp join-group 233.0.0.1
 ip igmp join-group 234.0.0.1

R3 has set up a multicast boundary as follows:


access-list 1 permit 232.0.0.0 0.255.255.255
access-list 1 permit 233.0.0.0 0.255.255.255
 
interface Serial1/0
 ip address 192.168.34.3 255.255.255.0
 ip pim sparse-mode
 ip multicast boundary 1

Now R3 only allows PIM joins that are in 232/8 or 233/8.


R3#sho ip mroute 234.0.0.1
Group 234.0.0.1 not found
R3#

Let's ping 233.0.0.1:


R6#ping 233.0.0.1 re 100

Type escape sequence to abort.
Sending 100, 100-byte ICMP Echos to 233.0.0.1, timeout is 2 seconds:
......................................................................
..........

Whoa now, what gives? Well...remember we only allowed 2 groups...what does Auto-RP use to propagate messages? Group 224.0.1.40! So even if you start passing traffic to 233.0.0.1 after you enable the boundary, eventually R3 will lose state for the Auto-RP discovery group and R4 will lose the RP information. All multicast traffic will then fail the RPF check.

So here is our modified ACL on R3:


R3#sho run | inc access
access-list 1 permit 224.0.1.40
access-list 1 permit 233.0.0.0 0.255.255.255
access-list 1 permit 232.0.0.0 0.255.255.255

224.0.1.39 is what the MA's listen to so we don't need to worry about that for this example. Now we can ping:


R6#ping 233.0.0.1 re 2  

Type escape sequence to abort.
Sending 2, 100-byte ICMP Echos to 233.0.0.1, timeout is 2 seconds:

Reply to request 0 from 192.168.34.4, 212 ms
Reply to request 0 from 192.168.34.4, 216 ms
Reply to request 1 from 192.168.34.4, 184 ms
Reply to request 1 from 192.168.34.4, 184 ms

Now this seems a little inefficient, right? Why should R4 even know about the RP if R3 is going to prevent mroute state from being created for 234.0.0.1 on that interface. If we could prevent R4 from learning that RP information, that would be great. Well on R3 we can modify the boundary as follows:


R3(config)#int s1/0               
R3(config-if)#ip multicast boundary 1 filter-autorp

Now R3 only sends RP information for the groups permitted in the ACL:


R4#show ip pim rp mapping 
PIM Group-to-RP Mappings

Group(s) 232.0.0.0/8
  RP 1.1.1.1 (?), v2v1
    Info source: 1.1.1.1 (?), elected via Auto-RP
         Uptime: 00:00:03, expires: 00:02:55
Group(s) 233.0.0.0/8
  RP 1.1.1.1 (?), v2v1
    Info source: 1.1.1.1 (?), elected via Auto-RP
         Uptime: 00:00:03, expires: 00:02:53
R4#

Multicast TTL-Threshold

2009-02-11T09:28:00.000-08:00

Maybe I am misunderstanding some things, but documents and books always say that the TTL of a packet must be higher than the threshold to be forwarded. From the 12.4 command reference:

ip multicast ttl-threshold

Usage Guidelines

"Only multicast packets with a TTL value greater than the threshold are forwarded out the interface."

Oh yeah?! I guess it depends on when you look at the TTL. Consider the network:

R1----R2----R3----R4

PIM-DM is enabled everywhere.
R4 has joined 239.0.0.1
R1 is sending pings which have 255 TTL when sent from R1.
R2 receives the PING, decrements the TTL to 254 before sending to R3.

So if we set TTL threshold to 254 on R2's interface to R3, it should block it right? No:


R2(config)#int s1/0
R2(config-if)#ip multicast ttl-threshold 254

R1#ping 239.0.0.1   

Type escape sequence to abort.
Sending 1, 100-byte ICMP Echos to 239.0.0.1, timeout is 2 seconds:

Reply to request 0 from 192.168.34.4, 164 ms
R1#

The router will still pass packets that have a TTL equal to the threshold if it was the router that decremented the TTL to reach that value. Here we see 255 will fail:


R2(config)#int s1/0
R2(config-if)#ip multicast ttl-threshold 255

R1#ping 239.0.0.1

Type escape sequence to abort.
Sending 1, 100-byte ICMP Echos to 239.0.0.1, timeout is 2 seconds:
.
R1#

IE Mock Lab 4 Review

2009-02-08T10:32:00.000-08:00

If you plan on taking this lab, don't read this post as it may contain some spoilers. I took this lab yesterday and did okay, though I could have done a lot better. I got a 73, but I finished in about 4 hours. After verifying the whole the lab for the next 2 hours, I didn't really make any major changes. In fact, the only error I noticed was that I had an OSPF key configured wrong. Turns out, there was more...

"Do not configure anything on SW3 for this task." This refers to every thing in this task!

In a hub spoke topology, we were only allowed one map statement on one of the spokes. This means we need that map statement for L3/L2 resolution to the other spoke, then rely on INARP for spoke-to-hub resolution. On the hub, I mapped by local IP for self-ping (which was not required) which in effect turns of INARP for that IP. The bottom line is INARP has to be enabled on one end. My mappings did show up dynamically, but the grader said this wouldn't work after a reboot. I am going to lab this up again and verify.

I missed 3 tasks (out of 13) in the IGP section. One was impossible (IMO) but by looking at the SG it appears the task itself was worded incorrectly, had to do with summarizing in OSPF on R3 and R5. The SG has them summarizing SW2 and R5 which are in the same area so it would have worked. The task said to summarize R3 and R5, which are not in the same area. Another IGP task required a tunnel with a new adressing. I think this violates the rule (clearly stated at the beginning) that we are not allowed to add any addresses. Lastly, I failed to redistribute a BB link into an IGP.

My traffic filter in the security task was fine except I didn't allow IGMP, which then caused me to miss one multicast task. Two Birds, One stone. Meh.

The other two tasks I missed involved TFTP: Limiting access to a router's config via SNMP, and TFTP boot. These were definitely doable, I was just unfamiliar with a couple commands and configured "half-solutions" which are just as good as "no-solutions" :-)

This lab was rated a 9 (from what I here the real thing is about a 7 or so) and recommended by IE to take within the final month of preparation. The grade report wasn't too detailed but then again, there was not a whole lot to explain. The tasks I did miss, were very simple mistakes. There is also a report that says how well you did in relation to other people who took this lab. For example, I got an 11% in NAT which means 89% people also got this right. Each task has a breakdown like this.

I try not to put too much stock in that though. I just want to learn from my mistakes and work on time management. I know a couple people who failed miserably on mocks and then passed the lab. And I am sure there are people who did great on mocks, then failed the real thing. I would rather be part of that first group :-)

Overlapping/Duplicate AS-External-LSA IDs

2009-02-04T11:06:00.000-08:00

I was reading OSPF: Anatomy of an Internet Routing Protocol by John T. Moy today and I came across an issue with AS-external LSA Link-State IDs. The LSA uses the network address as the identifier. If one router was to generate multiple Type 5 LSA's with the same network number but different masks, only 1 would be advertised because the LSA ID would be the same.

The book was published in 1998 and at the time there was no way of dealing with this. After doing this lab, I realized there was a way and it had since been documented in Appendix E of RFC 2328:

RFC 2328 Appendix E

Here I create 3 static routes, that all end up with the same network number and would normally have the same LSA ID:


R1(config)#ip route 192.9.0.0 255.255.0.0 Null0  
R1(config)#ip route 192.9.0.0 255.255.254.0 Null0
R1(config)#ip route 192.9.0.0 255.255.255.0 Null0
R1(config)#router ospf 1
R1(config-router)#redistribute static subnets

Let's see what the LSA IDs are:


R1#sho ip osp database | inc 192.9
192.9.0.0       1.1.1.1         246         0x80000001 0x00933F 0
192.9.0.255     1.1.1.1         149         0x80000001 0x00933F 0
192.9.1.255     1.1.1.1         234         0x80000001 0x00834F 0
R1#

R1#sho ip ospf database external 192.9.0.0

            OSPF Router with ID (1.1.1.1) (Process ID 1)

                Type-5 AS External Link States

  LS age: 14
  Options: (No TOS-capability, DC)
  LS Type: AS External Link
  Link State ID: 192.9.0.0 (External Network Number )
  Advertising Router: 1.1.1.1
  LS Seq Number: 80000003
  Checksum: 0x8F41
  Length: 36
  Network Mask: /16
        Metric Type: 2 (Larger than any link state path)
        TOS: 0 
        Metric: 20 
        Forward Address: 0.0.0.0
        External Route Tag: 0

The router gives the last 2 networks the broadcast address of that respective network as the Link State ID. The /16 network got the network address as the ID. I wonder if order of operations has anything to do with it


R1(config)#no ip route 192.9.0.0 255.255.0.0 Null0
R1(config)#no ip route 192.9.0.0 255.255.254.0 Null0
R1(config)#no ip route 192.9.0.0 255.255.255.0 Null0
R1(config)#ip route 192.9.0.0 255.255.255.0 Null0

Ok, so now the /24 is the only in there and it is using 192.9.0.0 as its ID:


R1#sho ip osp database external 192.9.0.0

            OSPF Router with ID (1.1.1.1) (Process ID 1)

                Type-5 AS External Link States

  LS age: 36
  Options: (No TOS-capability, DC)
  LS Type: AS External Link
  Link State ID: 192.9.0.0 (External Network Number )
  Advertising Router: 1.1.1.1
  LS Seq Number: 80000001
  Checksum: 0x933F
  Length: 36
  Network Mask: /24
        Metric Type: 2 (Larger than any link state path)
        TOS: 0 
        Metric: 20 
        Forward Address: 0.0.0.0
        External Route Tag: 0

What happens if we add a /16 now?


R1(config)#ip route 192.9.0.0 255.255.0.0 Null0

R1#sho ip osp database external 192.9.0.0

            OSPF Router with ID (1.1.1.1) (Process ID 1)

                Type-5 AS External Link States

  LS age: 12
  Options: (No TOS-capability, DC)
  LS Type: AS External Link
  Link State ID: 192.9.0.0 (External Network Number )
  Advertising Router: 1.1.1.1
  LS Seq Number: 80000002
  Checksum: 0x9140
  Length: 36
  Network Mask: /16
        Metric Type: 2 (Larger than any link state path)
        TOS: 0 
        Metric: 20 
        Forward Address: 0.0.0.0
        External Route Tag: 0

R1#

The /16 stold the ID from the /24!


R1#sho ip osp database | inc 192.9       
192.9.0.0       1.1.1.1         45          0x80000002 0x009140 0
192.9.0.255     1.1.1.1         45          0x80000001 0x00933F 0

How OSPF transmit capability can prevent virtual-link routing loops

2009-02-03T10:16:00.001-08:00

I ran into the command "capability transit" some time ago but never really understood how it worked. The explanation in the RFC and the DocCD may seem pretty vague unless you understand what issues cause it to be necessary or desirable. It is on by default so you probably will never have any issues with it, but I find it an interesting feature to look into. And by doing so, you tend to learn more about how OSPF works.

In this lab, I turn it off so we can see what issues arise. We will focus on R2's path to R4's loopback of 4.4.4.4. Each router's interface IP address ends ends with the router number so we can tell easily where traffic is flowing. Here is the topology:

I disabled capability transit on all routers, but I found that in this lab R2 is where the action is, so that might be only place we need to do it:


router ospf 1
  no capability transit

Now we begin...

R1 has a virtual link to R3 in order to connect area 234 to area 0. This works fine. R3 has become an ABR and R2 will use R3 to get to R4's loopback:


R2#sho ip route 4.4.4.4
Routing entry for 4.4.4.4/32
Known via "ospf 1", distance 110, metric 66, type inter area
Last update from 192.168.23.3 on Serial1/0, 00:00:07 ago
Routing Descriptor Blocks:
* 192.168.23.3, from 3.3.3.3, 00:00:07 ago, via Serial1/0
  Route metric is 66, traffic share count is 1

Now let's say R2 needs to add a network to area 2 as follows


R2(config)#int lo 0
R2(config-if)#ip address 2.2.2.2 255.255.255.255
R2(config-if)#ip ospf 1 area 2

Since R2 does not have an interface in area 0 we can build a virtual-link to R1:


R2(config)#router ospf 1                    
R2(config-router)#area 123 virtual-link 1.1.1.1 

R1(config)#router ospf 1              
R1(config-router)#area 123 virtual-link 2.2.2.2
*Mar  1 00:59:19.191: %OSPF-5-ADJCHG: Process 1, Nbr 2.2.2.2 on
OSPF_VL3 from LOADING to FULL, Loading Done

Perfect, right?

Let's take a look at that route towards R4 again:


R2#sho ip route 4.4.4.4
Routing entry for 4.4.4.4/32
  Known via "ospf 1", distance 110, metric 194, type inter area
  Last update from 192.168.12.1 on Serial1/1, 00:00:04 ago
  Routing Descriptor Blocks:
  * 192.168.12.1, from 3.3.3.3, 00:00:04 ago, via Serial1/1
      Route metric is 194, traffic share count is 1

Oh-no...Let's trace:


R2#trace 4.4.4.4      

Type escape sequence to abort.
Tracing the route to 4.4.4.4

  1 192.168.12.1 72 msec 24 msec 8 msec
  2 192.168.12.2 56 msec 20 msec 84 msec
  3  *  *  *
  4  *  *  *

We have a loop all-right.To fix it, on R2:


R2(config)#router ospf 1
R2(config-router)#capability transit

R2#clear ip ospf process
Reset ALL OSPF processes? [no]: yes
R2#

Few moments later:


R2#sho ip route 4.4.4.4
Routing entry for 4.4.4.4/32
  Known via "ospf 1", distance 110, metric 66, type inter area
  Last update from 192.168.23.3 on Serial1/0, 00:00:18 ago
  Routing Descriptor Blocks:
  * 192.168.23.3, from 3.3.3.3, 00:00:18 ago, via Serial1/0
      Route metric is 66, traffic share count is 1

From cisco.com

"The OSPF Area Transit Capability feature provides an OSPF Area Border Router (ABR) with the ability to discover shorter paths through the transit area for forwarding traffic that would normally need to travel through the virtual-link path."

OSPF Area Transit Capability

So in this case, we have allowed R2 to use it direct path to R3 instead of it's own path through the backbone area. We have basically made area 123 a transit area that can carry traffic to destinations not in it's own area. We are flowing from Area 0 (R2 is an ABR now) to Area 123 to Area 234!

Since this command is enabled by default on recent IOS versions, I am not sure you would ever run into this issue in the lab. However, it is still an interesting feature and it is always good to know what's really going on under the hood :-)

3560 QoS: Per-port per-vlan policing

2009-02-02T11:59:00.000-08:00

I know the name is scary, but I do dig Catalyst QoS. This is the second of back-to-back posts on the subject. This is one is a little more complex than classification and decided on a Visio for it:

Per-van policing in the 3560s is different from the 3550s because there is no "match VLAN" clause available. Instead you create hierarchical policies and attach them to the SVI.

Here is the scenario:

VLAN100 will be policed to 64k (192.168.100.0/24)
VLAN200 Will be policed to 128k (192.168.200.0/24)

Because of bursts, I was not able to get these exact rates, but you will see how these policies are applied and the effect they have on traffic flow. Plus you can always play with the burst sizes on your own :)

Here is the tracker I created on R2:


access-list 1 permit 192.168.100.1
access-list 1 permit 192.168.100.3
access-list 2 permit 192.168.200.5
!
class-map match-any VLAN100
 match access-group 1
class-map match-any VLAN200
 match access-group 2
!
policy-map TRACKER
 class VLAN100
 class VLAN200
!
interface Ethernet0/0
 no ip address
 load-interval 30
 full-duplex
!
interface Ethernet0/0.100
 encapsulation dot1Q 100
 ip address 192.168.100.2 255.255.255.0
 service-policy input TRACKER
!
interface Ethernet0/0.200
 encapsulation dot1Q 200
 ip address 192.168.200.2 255.255.255.0
 service-policy input TRACKER

All configuration is being done on SW2. There really is not an order of operations to follow, but basically you just need to make sure class-maps and policy-maps are created before you apply them. The logical flow is what you want to get used to. Otherwise you will be jumping into and out of classes and policies, reconfiguring them like I did :)

At our child (aka "second") level we have a class-map that matches the interface and we have our policer. The interface matching here is whats is referred into in the first clause of "per-port per-vlan" policing.


class-map match-all TRUNK
  match input-interface  FastEthernet0/13
!
policy-map VLAN100-POLICER
  class TRUNK
    police 64000 12000 exceed-action drop
policy-map VLAN200-POLICER
  class TRUNK
    police 128000 24000 exceed-action drop

As far as I know, this "bottom" or "second" level class-map can only match input-interface. And this second level policy must be a policer.

Now, at the parent level we create a new class to match IP traffic and then apply our child polices below that. This top-level class must match an ACL (match protocol ip gave me errors when applying the policy).


access-list 100 permit ip any any
!
class-map match-all IP
  match access-group 100
!
policy-map VLAN100-PARENT
  class IP
   set ip precedence 1
   service-policy VLAN100-POLICER
policy-map VLAN200-PARENT
  class IP
   set ip precedence 2
   service-policy VLAN200-POLICER

Notice that I have the "set ip precedence" clause in our parent policies. These first level policies are required to have an action. You will get an error message stating this if you try to apply it to the SVI without an action:

SW2(config)#int vlan 100
SW2(config-if)#service-policy input VLAN100-PARENT
%QoS: No action is configured in the policymap VLAN100-PARENT classmap IP, or it is being modified.

So make sure you have set or trust clause in there. Now we can apply them to the SVIs:


mls qos
!
interface FastEthernet0/13
 mls qos vlan-based
!
interface Vlan100
 no ip address
 service-policy input VLAN100-PARENT
!
interface Vlan200
 no ip address
 service-policy input VLAN200-PARENT

From R1, R3 and R5 I will send a bunch of pings to R2:


R1#ping 192.168.100.2 re 1000000
R3#ping 192.168.100.2 re 1000000
R5#ping 192.168.200.2 re 1000000

Let's look at R2 after a few minutes.


R2#sho policy-map interface e0/0.100 | section VLAN100
    Class-map: VLAN100 (match-any)
      107819 packets, 12722642 bytes
      30 second offered rate 50000 bps
      Match: access-group 1
        107819 packets, 12722642 bytes
        30 second rate 50000 bps

R2#sho policy-map interface e0/0.200 | section VLAN200
    Class-map: VLAN200 (match-any)
      156873 packets, 18511014 bytes
      30 second offered rate 107000 bps
      Match: access-group 2
        156873 packets, 18511014 bytes
        30 second rate 107000 bps

We don't see the limits of 64k and 128k being reached, but the drops on the senders indicate that policing is working. And we can also tell VLAN 200 is getting roughly twice the bandwidth that VLAN 100 is getting. We could get closer to the limit by adjusting the burst sizes appropriately.

Key things to remember:

Child classes use match input-interface
Child policies use police
Parent classes match ACL (I think you can also match dscp, maybe others)
Parent policies must have an action (e.g. set or trust)
Apply parent policies to SVI

I strongly recommend getting your hands dirty with these configurations if you want to master them. I read a lot about switch qos, but it wasn't until I started playing around with scenarios like this that I got a better understanding of how to do it and what is required. If we truly understand what each QoS method does, then we should have no trouble deciphering what we are asked to do on the lab :)

3560 QoS: VLAN-Based Classification

2009-02-02T09:08:00.001-08:00

This is a topic I learned about while reading blogs over at IE. Here is the original:

Comparing Traffic Policing Features in the 3550 and 3560 switches

I have the following topology:

R1----|
R3---SW1---SW2---R2
R5----|

R1,R3 are in vlan 100, 192.168.100.0/24
R5 is in vlan 200, 192.168.200.0/24

R2 is on a trunked port with the following configuration:


interface Ethernet0/0.100
 encapsulation dot1Q 100
 ip address 192.168.100.2 255.255.255.0
 ip accounting precedence input
 no snmp trap link-status
!
interface Ethernet0/0.200
 encapsulation dot1Q 200
 ip address 192.168.200.2 255.255.255.0
 ip accounting precedence input
 no snmp trap link-status

On SW2 we will enable vlan-based qos and then mark traffic based on ACLs. First we make the ACLs:


ip access-list extended ICMP
 permit icmp any any
ip access-list extended TCP
 permit tcp any any

Next we make our class-maps and policy-maps:


class-map match-all ICMP
  match access-group name ICMP
class-map match-all TCP
  match access-group name TCP

policy-map VLAN
  class TCP
   set ip precedence 5
  class ICMP
   set ip precedence 3

Next enable mls qos, vlan-based qos and apply the policy to an SVI. Note that the SVI does not need an IP address:


mls qos

int f0/13
 interface FastEthernet0/13
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 50
 switchport mode trunk
 mls qos vlan-based

int vlan 100
 service-policy input VLAN
int vlan 200
 service-policy input VLAN

Now run some tests. Here I Ping and Telnet from R5, telnet from R1 and then ping from R3:


R5#ping 192.168.200.2 rep 100

Type escape sequence to abort.
Sending 100, 100-byte ICMP Echos to 192.168.200.2, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (100/100), round-trip min/avg/max = 1/3/4 ms
R5#

R5#telnet 192.168.200.2
Trying 192.168.200.2 ... Open

R2>exit

[Connection to 192.168.200.2 closed by foreign host]
R5#

R1#telnet 192.168.100.2 
Trying 192.168.100.2 ... Open

R2>exit

[Connection to 192.168.100.2 closed by foreign host]
R1#

R3#ping 192.168.100.2 re 50

Type escape sequence to abort.
Sending 50, 100-byte ICMP Echos to 192.168.100.2, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (50/50), round-trip min/avg/max = 1/3/4 ms
R3#

Verify on R2:


R2#sho int precedence 
Ethernet0/0.100 
  Input
    Precedence 3:  50 packets, 5900 bytes
    Precedence 5:  46 packets, 2953 bytes
Ethernet0/0.200 
  Input
    Precedence 3:  100 packets, 11800 bytes
    Precedence 5:  15 packets, 969 bytes
R2#

TCP Load Balancing, Destination NAT

2009-02-01T19:41:00.000-08:00

The "ip nat inside destination" command can be used to split up the load from what looks like one global destination, to several inside hosts. This behaves very much like server load balancing, at least without all the health checks.

Below is the topology. I have static default routes from R1, R2, and R3 pointing to R4. R7 has a static route to each serial link.

Here is R4's config:


interface FastEthernet0/0
ip address 192.168.0.4 255.255.255.0
ip nat inside
ip virtual-reassembly
!
interface Serial1/0
ip address 192.168.45.4 255.255.255.0
ip verify unicast reverse-path
ip nat outside
ip virtual-reassembly
serial restart-delay 0
!
interface Serial1/1
ip address 192.168.46.4 255.255.255.0
ip verify unicast reverse-path
ip nat outside
ip virtual-reassembly
!
ip route 0.0.0.0 0.0.0.0 192.168.45.5
ip route 0.0.0.0 0.0.0.0 192.168.46.6
!
ip nat pool POOL 192.168.0.1 192.168.0.3 prefix-length 24 type rotary
ip nat inside destination list 10 pool POOL
!
access-list 10 permit 192.168.45.10
access-list 10 permit 192.168.46.10

From R7 we will verify:


R7#telnet 192.168.45.10  
Trying 192.168.45.10 ... Open

R1>
R1>exit

[Connection to 192.168.45.10 closed by foreign host]
R7#telnet 192.168.45.10
Trying 192.168.45.10 ... Open

R2>exit

[Connection to 192.168.45.10 closed by foreign host]
R7#telnet 192.168.45.10
Trying 192.168.45.10 ... Open

R3>exit

[Connection to 192.168.45.10 closed by foreign host]
R7#telnet 192.168.46.10
Trying 192.168.46.10 ... Open

R1>exit

[Connection to 192.168.46.10 closed by foreign host]
R7#telnet 192.168.46.10
Trying 192.168.46.10 ... Open

R2>exit

[Connection to 192.168.46.10 closed by foreign host]
R7#

R4's NAT table:


R4#sho ip nat translations
Pro Inside global      Inside local       Outside local      Outside global
tcp 192.168.45.10:23   192.168.0.1:23     200.0.0.7:51519    200.0.0.7:51519
tcp 192.168.46.10:23   192.168.0.1:23     200.0.0.7:64139    200.0.0.7:64139
tcp 192.168.46.10:23   192.168.0.2:23     200.0.0.7:11691    200.0.0.7:11691
tcp 192.168.45.10:23   192.168.0.2:23     200.0.0.7:62913    200.0.0.7:62913
tcp 192.168.45.10:23   192.168.0.3:23     200.0.0.7:17295    200.0.0.7:17295

I used two links just to show the flexibility of this configuration. I was playing around with route-map NAT failover/LB and then decided to work on this scenario.

NTP - How long is too long?

2009-02-01T11:24:00.001-08:00

This is how long I waited for NTP to sync today:

R2(config)#ntp server 136.10.4.4
R2(config)#^Z
Feb 1 19:26:53.915: %SYS-5-CONFIG_I: Configured from console by console

Feb 1 19:37:11.852: NTP Core(NOTICE): Clock is synchronized.

More than 10 minutes. It should be noted that the clocks were only seconds apart to begin with. Code on these routers is 12.4(22)T. I don't know if I have ever waited so long but it's unbelievably ridiculous.

Then I enable authentication:

R4(config)#ntp authentication-key 1 md5 ipexpert

R2(config)#ntp authentication-key 1 md5 ipexpert
R2(config)#ntp trusted-key 1
R2(config)#ntp authenticate
R2(config)#ntp server 136.10.4.4 key 1

Feb 1 19:45:02.628: NTP Core(INFO): key (1) added.
Feb 1 19:45:02.752: NTP Core(INFO): key (1) marked as trusted.
Feb 1 19:45:03.276: NTP Core(INFO): system event 'event_clock_reset' (0x05) status 'sync_alarm, sync_unspec, 10 events, event_peer/strat_chg' (0xC0A4)
Feb 1 19:45:03.276: NTP Core(NOTICE): Clock synchronization lost.

Peers never come up, I get this every so often (debug ntp all):

.Feb 1 19:45:47.852: NTP message sent to 136.10.4.4, from interface 'Loopback0' (136.10.2.2).
.Feb 1 19:45:47.852: NTP message received from 136.10.4.4 on interface 'Loopback0' (136.10.2.2).
.Feb 1 19:45:47.852: NTP Core(DEBUG): ntp_receive: message received
.Feb 1 19:45:47.852: NTP Core(DEBUG): ntp_receive: peer is 0x674B9DF8, next action is 1.
.Feb 1 19:45:47.852: NTP Core(NOTICE): ntp_receive: dropping message: crypto-NAK.

.Feb 1 19:50:52.852: NTP message sent to 136.10.4.4, from interface 'Loopback0' (136.10.2.2).
.Feb 1 19:50:52.852: NTP message received from 136.10.4.4 on interface 'Loopback0' (136.10.2.2).
.Feb 1 19:50:52.852: NTP Core(DEBUG): ntp_receive: message received
.Feb 1 19:50:52.852: NTP Core(DEBUG): ntp_receive: peer is 0x674B9DF8, next action is 1.
.Feb 1 19:50:52.852: NTP Core(NOTICE): ntp_receive: dropping message: crypto-NAK.

Here we are still

.Feb 1 19:58:19.851: NTP message sent to 136.10.4.4, from interface 'Loopback0' (136.10.2.2).
.Feb 1 19:58:19.851: NTP message received from 136.10.4.4 on interface 'Loopback0' (136.10.2.2).
.Feb 1 19:58:19.851: NTP Core(DEBUG): ntp_receive: message received
.Feb 1 19:58:19.851: NTP Core(DEBUG): ntp_receive: peer is 0x674B9DF8, next action is 1.
.Feb 1 19:58:19.851: NTP Core(NOTICE): ntp_receive: dropping message: crypto-NAK.

So, for kicks on the master I do this:

R4(config)#ntp authenticate
R4(config)#ntp trusted-key 1

I now get a new message on R2:

.Feb 1 20:01:20.851: NTP message sent to 136.10.4.4, from interface 'Loopback0' (136.10.2.2).
.Feb 1 20:01:20.851: NTP message received from 136.10.4.4 on interface 'Loopback0' (136.10.2.2).
.Feb 1 20:01:20.851: NTP Core(DEBUG): ntp_receive: message received
.Feb 1 20:01:20.851: NTP Core(DEBUG): ntp_receive: peer is 0x674B9DF8, next action is 1.
.Feb 1 20:01:20.851: NTP Core(DEBUG): receive: packet given to process_packet

This looks promising:

R2#
.Feb 1 20:03:30.851: NTP message sent to 136.10.4.4, from interface 'Loopback0' (136.10.2.2).
.Feb 1 20:03:30.851: NTP message received from 136.10.4.4 on interface 'Loopback0' (136.10.2.2).
.Feb 1 20:03:30.851: NTP Core(DEBUG): ntp_receive: message received
.Feb 1 20:03:30.851: NTP Core(DEBUG): ntp_receive: peer is 0x674B9DF8, next action is 1.
.Feb 1 20:03:30.851: NTP Core(DEBUG): receive: packet given to process_packet
.Feb 1 20:03:30.851: NTP Core(DEBUG): Peer becomes reachable, poll set to 6.
.Feb 1 20:03:30.851: NTP Core(INFO): peer 136.10.4.4 event 'event_reach' (0x84) status 'unreach, conf, auth, 1 event, event_reach' (0xE014)

TA-DA!

.Feb 1 20:06:43.851: NTP Core(NOTICE): Clock is synchronized.

I have never had to enable trusted-key on the master before. Watch this:

R4(config)#no ntp trusted-key 1

Back on R2:

R2#
Feb 1 20:07:47.851: NTP message sent to 136.10.4.4, from interface 'Loopback0' (136.10.2.2).
Feb 1 20:07:47.851: NTP message received from 136.10.4.4 on interface 'Loopback0' (136.10.2.2).
Feb 1 20:07:47.851: NTP Core(DEBUG): ntp_receive: message received
Feb 1 20:07:47.851: NTP Core(DEBUG): ntp_receive: peer is 0x674B9DF8, next action is 1.
Feb 1 20:07:47.851: NTP Core(INFO): system event 'event_clock_reset' (0x05) status 'sync_alarm, sync_unspec, 15 events, event_peer/strat_chg' (0xC0F4)
Feb 1 20:07:47.851: NTP Core(NOTICE): Clock synchronization lost.
.Feb 1 20:07:47.851: NTP Core(NOTICE): ntp_receive: dropping message: crypto-NAK.

Maybe something has changed in this T train but looks like we need "ntp trusted-key" on the Master now. I am not an NTP guru by any means but if you look at some of my other ntp blogs, you will see I didn't need this command. Note that I only needed "trusted-key" on the Master, not "ntp authenticate" even though I showed it above. Removing it did not cause sync loss. Something to keep in mind if you find yourself singing the NTP blues.

Oh, and while you are waiting for the sync - go configure something else in the meantime!

IPexpert Volume 3 Mock Lab 1 - Take 2

2009-01-31T17:48:00.000-08:00

I did this lab again today mainly to see how much I improved since the first time. If your curious, here was my original post:

IPexpert Volume 3 Mock Lab 1 - Take 1

That was just over 5 months ago, and I more than doubled my score and finished in about half the time. I got a 91 this time, missing 3 tasks. The first one was a grading script error. The second one was a bonehead mistake because the task said to prevent odd routes and I blocked odd (BGP task).

The last one was tricky and I skipped it because I did not know how to complete it without messing up another task. It was 2 points vs 3 points and I took the 3-pointer. I will explain what the issue was and how to resolve it.

The first task had you allow telnet only on port 3005 of R9. Then you create a privilege 15 user named cisco with a password cisco. The next task says that the user cisco should only be allowed to do show commands and not configure anything. Menus are not allowed.

Well....since user cisco is a level 15 user he can do anything he wants. And he HAS to be a level 15 user according to the first task. The solution was to configure AAA which basically ignores privilege levels that are assigned to username commands. Now, when user cisco logs in, he is actually in level 1 and he cannot get to configuration mode (without an enable password). Do you think this violates the previous task?

Anyways, it felt good to know that I have retained a lot of info. I'm going to do another mock lab tomorrow morning from IPexpert (Before the Super Bowl of course!). Then next week I have an IE mock lab and another proctor lab session scheduled. The week after that, it will be Cisco Assessor Labs on the 14th and 15th (if my schedule gets accepted).

That leaves one more weekend of nothing which I plan on just reviewing and tying up loose ends. Probably play around on the home lab most of the time. Then the next weekend I will be in San Jose :-)

RSPAN between 3550 and 3560 - Multiple Sources

2009-01-31T10:33:00.000-08:00

Topology is as follows:

R5----SW1----SW2----SW4----R4/R6

R4 and R6 are on VLAN 300, 192.168.250.0/24 subnet
R5 is on VLAN 100, connected to port f0/5 of SW1
Inter-switch links are dot1q trunks
I will set up RSPAN between the switches and use debug ip packet with an ACL to verify.

3550 is the source:

SW4(config)#vlan 999
SW4(config-vlan)#remote-span
SW4(config)#monitor session 1 source vlan 300 rx
SW4(config)#monitor session 1 destination remote vlan 999 reflector-port f0/12

3560 is connected to the monitor:

SW1(config)#monitor session 1 source remote vlan 999
SW1(config)#monitor session 1 destination interface f0/12

On R5 We can verify like this:

R5(config)#access-list 1 permit 192.168.250.4 0.0.0.0
R5(config)#access-list 1 permit 192.168.250.6 0.0.0.0
R5(config)#no service timestamps debug
R5#debug ip packet 1 detail
IP packet debugging is on (detailed) for access list 1
IP: s=192.168.250.4 (Ethernet0/0), d=224.0.0.5, len 80, rcvd 0, proto=89
IP: s=192.168.250.6 (Ethernet0/0), d=224.0.0.5, len 80, rcvd 0, proto=89
IP: s=192.168.250.4 (Ethernet0/0), d=224.0.0.5, len 80, rcvd 0, proto=89
IP: s=192.168.250.6 (Ethernet0/0), d=224.0.0.5, len 80, rcvd 0, proto=89

Here we can see EIGRP packets from VLAN 300, which verifies our monitoring is working. The only place I specified "remote-span" under a VLAN was the source 3550. However, I have read that that this required on all switches that carry the remote-span VLAN.

Let's add a source on SW2, where R2 is plugged into f0/2. We will put it on a different VLAN just to prove it is working:

SW2(config)#int f0/2
SW2(config-if)#sw a v 150
SW2(config)#vlan 999
SW2(config-vlan)#remote-span
SW2(config)#monitor session 1 source interface f0/2
SW2(config)#monitor session 1 destination remot vlan 999

If we jump to R5, we won't see any packets from R2...hmm...oh yeah, the ACL!

R5(config)#access-list 1 permit 192.168.0.2 0.0.0.0

There we go:

IP: s=192.168.0.2 (Ethernet0/0), d=224.0.0.2, len 48, rcvd 0
UDP src=1985, dst=1985
IP: s=192.168.0.2 (Ethernet0/0), d=224.0.0.5, len 88, rcvd 0, proto=89
IP: s=192.168.250.4 (Ethernet0/0), d=224.0.0.5, len 80, rcvd 0, proto=89
IP: s=192.168.0.2 (Ethernet0/0), d=224.0.0.2, len 48, rcvd 0
UDP src=1985, dst=1985
IP: s=192.168.250.6 (Ethernet0/0), d=224.0.0.5, len 80, rcvd 0, proto=89

Looks like we got HSRP packets from R2 and OSPF packets from R4 and R6.

Key things to remember:

-Reflector port needed on 3550
-remote-span command used under the RSPAN VLAN. In this example, I only did it on the source, but I would verify that you need it on all devices with this VLAN.
-To allow destination port to connect back to the network use "ingress" keyword on session destination command

EIGRP Bounded updates

2009-01-30T10:17:00.000-08:00

I was reading about EIGRP in Routing TCP/IP Volume 1 by Jeff Doyle and focusing on the comparisons between it and distance vector and link-state protocols. One characteristic of EIGRP that sets it part from other protocols is that updates are "bounded" meaning that they are only sent to the "affected" neighbors. I was trying to find a way to see this behavior in action so I created this summarization scenario.

R4 is in the middle of the star with R3,R5 and R7 at the edges:

R4-R5 = 192.168.45.0/24
R4-R7 = 192.168.47.0/24
R4-R3 = 192.168.34.0/24

R4 is advertising a summary of 192.168.44.0/22 to R3.
If a new link was brought up in the /22 range, R4 will not send an update to R3.

Here it is in action:

R3#debug eigrp packets update
R4#debug eigrp packets update
R5#debug eigrp packets update

R3#sho ip route eigrp
D 192.168.44.0/22 [90/2681856] via 192.168.34.4, 00:02:54, Serial1/1

R7 is off of R4 serial 1/1, R3 is off of R4 serial 1/0:

R4#sho ip eigrp ne
IP-EIGRP neighbors for process 1
H Address Interface Hold Uptime SRTT RTO Q Seq
(sec) (ms) Cnt Num
1 192.168.47.7 Se1/1 10 00:11:55 75 450 0 8
2 192.168.34.3 Se1/0 13 00:30:39 106 636 0 8
0 192.168.45.5 Se1/2 12 00:32:16 106 636 0 8
R4#

Let's add a new loopback on R5 in the range of the summary:

R5(config)#int lo 1
R5(config-if)#ip address 192.168.46.5 255.255.255.0
R5(config-if)#router eigrp 1
R5(config-router)#network 192.168.46.5 0.0.0.0

*Mar 1 20:02:59.075: EIGRP: Enqueueing UPDATE on Serial1/0 iidbQ un/rely 0/1 serno 8-8
*Mar 1 20:02:59.079: EIGRP: Enqueueing UPDATE on Serial1/0 nbr 192.168.45.4 iidbQ un/rely 0/0 peerQ un/rely 0/0 serno 8-8
*Mar 1 20:02:59.087: EIGRP: Sending UPDATE on Serial1/0 nbr 192.168.45.4
*Mar 1 20:02:59.095: AS 1, Flags 0x0, Seq 8/20 idbQ 0/0 iidbQ un/rely 0/0 peerQ un/rely 0/1 serno 8-8
*Mar 1 20:02:59.235: EIGRP: Received UPDATE on Serial1/0 nbr 192.168.45.4
*Mar 1 20:02:59.235: AS 1, Flags 0x0, Seq 26/8 idbQ 0/0 iidbQ un/rely 0/0 peerQ un/rely 0/0

R5 sends an update to R4. R4 only sends it to R7:

*Mar 1 20:02:59.515: EIGRP: Received UPDATE on Serial1/2 nbr 192.168.45.5
*Mar 1 20:02:59.519: AS 1, Flags 0x0, Seq 8/20 idbQ 0/0 iidbQ un/rely 0/0 peerQ un/rely 0/0
*Mar 1 20:02:59.563: EIGRP: Enqueueing UPDATE on Serial1/1 iidbQ un/rely 0/1 serno 11-11
*Mar 1 20:02:59.567: EIGRP: Enqueueing UPDATE on Serial1/1 nbr 192.168.47.7 iidbQ un/rely 0/0 peerQ un/rely 0/0 serno 11-11
*Mar 1 20:02:59.575: EIGRP: Sending UPDATE on Serial1/1 nbr 192.168.47.7
*Mar 1 20:02:59.579: AS 1, Flags 0x0, Seq 24/8 idbQ 0/0 iidbQ un/rely 0/0 peerQ un/rely 0/1 serno 11-11
*Mar 1 20:02:59.583: EIGRP: Enqueueing UPDATE on Serial1/0 iidbQ un/rely 0/1 serno 11-11
*Mar 1 20:02:59.587: EIGRP: Enqueueing UPDATE on Serial1/0 nbr 192.168.34.3 iidbQ un/rely 0/0 peerQ un/rely 0/0 serno 11-11
*Mar 1 20:02:59.587: EIGRP: Enqueueing UPDATE on Serial1/2 iidbQ un/rely 0/1 serno 11-11
*Mar 1 20:02:59.591: EIGRP: Enqueueing UPDATE on Serial1/2 nbr 192.168.45.5 iidbQ un/rely 0/0 peerQ un/rely 0/0 serno 11-11
*Mar 1 20:02:59.591: EIGRP: Sending UPDATE on Serial1/2 nbr 192.168.45.5
*Mar 1 20:02:59.591: AS 1, Flags 0x0, Seq 26/8 idbQ 0/0 iidbQ un/rely 0/0 peerQ un/rely 0/1 serno 11-11

And of course on R3 we see nothing. What's really interesting is we see R4 "Enqueuing" the update but never actually sending it as it does to R5 and R7.

I am still not sure of one thing though. Is this a fundamental characteristic of EIGRP itself or the fact that we are summarizing? I cannot think of another scenario where this "bounded" update scenario would take place without summarization. If you can, please drop a comment.

OSPFv3 Neighbors do not need to be on same subnet

2009-01-27T13:39:00.000-08:00

Check it out:

R2 F0/0 <-----> F0/0 R3

Here is R2's config:


R2#sho run int f0/0
Building configuration...

Current configuration : 153 bytes
!
interface FastEthernet0/0
 no ip address
 duplex auto
 speed auto
 ipv6 address 2001:2::2/64
 ipv6 address FE80::2 link-local
 ipv6 ospf 1 area 0
end

Here is R3's config:


R3#sho run int f0/0
Building configuration...

Current configuration : 153 bytes
!
interface FastEthernet0/0
 no ip address
 duplex auto
 speed auto
 ipv6 address 2001:3::3/64
 ipv6 address FE80::3 link-local
 ipv6 ospf 1 area 0
end

R2's show commands:


R2#sho ipv6 ospf ne   

Neighbor ID     Pri   State           Dead Time   Interface ID    Interface
3.3.3.3           1   FULL/DR         00:00:35    4               FastEthernet0/0

R2#sho ipv6 route  
IPv6 Routing Table - 5 entries
Codes: C - Connected, L - Local, S - Static, R - RIP, B - BGP
       U - Per-user Static route
       I1 - ISIS L1, I2 - ISIS L2, IA - ISIS interarea, IS - ISIS summary
       O - OSPF intra, OI - OSPF inter, OE1 - OSPF ext 1, OE2 - OSPF ext 2
       ON1 - OSPF NSSA ext 1, ON2 - OSPF NSSA ext 2
C   2001:2::/64 [0/0]
     via ::, FastEthernet0/0
L   2001:2::2/128 [0/0]
     via ::, FastEthernet0/0
O   2001:3::/64 [110/10]
     via ::, FastEthernet0/0
L   FE80::/10 [0/0]
     via ::, Null0
L   FF00::/8 [0/0]
     via ::, Null0

R2 can now ping 2001:3::3


R2#ping 2001:3::3

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 2001:3::3, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 8/20/56 ms
R2#

This is possible because neighbors are known by their router-ids and link-local addresses are used as next hops, not the actual interface addresses.

Dynamic ARP Inspection with NON-DHCP hosts

2009-01-26T20:11:00.000-08:00

The Dynamic ARP Inspection concept is well understood, but sometimes the commands and requirements can be hard to remember. This scenario shows how DAI works with DHCP snooping to block ARP requests from untrusted ports and how NON-DHCP clients can still be apart of the network.

R1,R3 and R5 are all on VLAN100, connected to switch SW1:

R1 = Static host
R3 = DHCP Server
R5 = DHCP client

SW1 has ARP Inspection and DHCP snooping enabled already, with trust enabled on the port connected to R3.

SW1#sho run | inc snoop|arp
ip dhcp snooping vlan 100
ip dhcp snooping
ip arp inspection vlan 100
 ip dhcp snooping trust

R5 gets an IP address from R3 and now we have the following entry on SW1:

SW1#sho ip dhcp snooping binding 
MacAddress         IpAddress   Lease(sec) Type           VLAN  Interface
------------------ ----------- ---------- -------------  ----  ---------------
00:00:00:00:00:05  192.168.0.5 86381      dhcp-snooping  100   FastEthernet0/5
Total number of bindings: 1

R5 tries to ping R1 but can't:


R5#ping 192.168.0.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.0.1, timeout is 2 seconds:

*Jan  7 09:36:20.361: IP: tableid=0, s=192.168.0.5 (local), d=192.168.0.1
            (Ethernet0/0), routed via RIB
*Jan  7 09:36:20.361: IP: s=192.168.0.5 (local), d=192.168.0.1 (Ethernet0/0),
             len 100, sending
*Jan  7 09:36:20.361:     ICMP type=8, code=0
*Jan  7 09:36:20.361: IP ARP: creating incomplete entry for IP address:
             192.168.0.1 interface Ethernet0/0
*Jan  7 09:36:20.361: IP ARP: sent req src 192.168.0.5 0000.0000.0005,
                 dst 192.168.0.1 0000.0000.0000 Ethernet0/0

On SW1 we see this:


SW1#debug arp 
07:43:49: %SW_DAI-4-DHCP_SNOOPING_DENY: 1 Invalid ARPs (Res) on Fa0/1, vlan 100.
([0000.0000.0001/192.168.0.1/0000.0000.0005/192.168.0.5/07:43:49 UTC Mon Mar 1 1993])

SW1 is not allowing the ARP reply from R1 because the port is untrusted in the arp inspection configuration and R1's address is not in the DHCP snooping database. We can see the request make it on R1:


R1#
*Mar  2 00:31:09.685: IP ARP: rcvd req src 192.168.0.5 0000.0000.0005,
             dst 192.168.0.1 Ethernet0/0
*Mar  2 00:31:09.685: IP ARP: sent rep src 192.168.0.1 0000.0000.0001,
                 dst 192.168.0.5 0000.0000.0005 Ethernet0/0

But R5 never gets the reply. For NON-DHCP hosts we can create an ARP ACL and apply it to the DAI configuration:


SW1(config)#arp access-list ARP-TEST  
SW1(config-arp-nacl)#permit ip host 192.168.0.1 ?
  mac  Sender MAC address
SW1(config-arp-nacl)#permit ip host 192.168.0.1 mac ?
  H.H.H  Sender MAC address
  any    Any MAC address
  host   Single Sender host
SW1(config-arp-nacl)#permit ip host 192.168.0.1 mac host 0000.0000.0001
SW1(config-arp-nacl)#exit
SW1(config)#ip arp inspection filter ARP-TEST vlan 100

Now let's ping:


R5#ping 192.168.0.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.0.1, timeout is 2 seconds:
.!!!!
Success rate is 80 percent (4/5), round-trip min/avg/max = 8/9/12 ms
R5#

There is another option for the DAI filter and that is "static".


SW1(config)#ip arp inspection filter ARP-TEST vlan 100  ?
  static  Apply the ACL statically

If we applied this argument to the command, DAI would only check the ARP ACL and not fallback to the DHCP snooping database. That would prevent R5 ARPs from being allowed:


SW1(config)#ip arp inspection filter ARP-TEST vlan 100  static 

R5#ping 192.168.0.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.0.1, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)
R5#

Check debugs on SW1:


SW1#
07:52:53: %SW_DAI-4-ACL_DENY: 1 Invalid ARPs (Req) on Fa0/5, vlan 100.
([0000.0000.0005/192.168.0.5/0000.0000.0000/192.168.0.1/07:52:53 UTC Mon Mar 1 1993])

Requests are being denied inbound on f0/5 now.

RSH/RCP - quick and easy

2009-01-26T15:16:00.000-08:00

This is one of those topics that probably won't be in the exam, but it can't hurt to learn it if its easy enough.

On R3, I have:

R3#sho run | inc rcmd
ip rcmd remote-username R3
ip rcmd source-interface Loopback0

On R5, I have:

R5#sho run | inc rcmd
ip rcmd rsh-enable
ip rcmd remote-host cisco 172.16.0.3 R3 enable
ip rcmd source-interface Loopback0

On R3:

R3#rsh 172.16.0.5 /user cisco sho run int lo0

Building configuration...

Current configuration : 63 bytes
!
interface Loopback0
ip address 172.16.0.5 255.255.255.255
end

R3#

Now Let's do some RCP file copying:

R5(config)#ip rcmd rcp-enable
R5(config)#^Z
R5#copy run r5test.txt
Destination filename [r5test.txt]?
Erase flash: before copying? [confirm]n
Verifying checksum... OK (0xFD5B)
2714 bytes copied in 4.856 secs (559 bytes/sec)
Rack1R5#

Copy from R3:

R3#copy rcp://cisco@172.16.0.5/R5test.txt flash:
Destination filename [R5test.txt]?
Accessing rcp://cisco@172.16.0.5/R5test.txt...
Erase flash: before copying? [confirm]n!
Verifying checksum... OK (0xFD5B)
2714 bytes copied in 0.644 secs (4214 bytes/sec)
R3#

Key things to remember:

-Server side has two names in that rcmd command
-First one must match /user on client
-Second one must match client hostname or client "remote-username" command

IPexpert Volume 3 Mock Lab 9 Review

2009-01-25T10:51:00.000-08:00

This lab was actually pretty fun, though I made a lot of mistakes. I was short on time so I did not have any time to verify. I had a previous conflict in schedule so I had to take an hour+ off in the middle of the lab. There was a little bit of everything here from IPv6 redistribution, routing loops (if your not careful), mls qos, hierarchical MQC, and some interesting multicast stuff.

Here's a summary of what I missed:

IGP

Forgot to add "no-summary" to an NSSA ABR. The task said "no intra-area" routes, and I guess I saw "no inter-area" instead.

I needed to traffic engineer OSPF to influence path selection in two directions, and I only did one way. I was going to come back after all the redistribution tasks, and I did not have time.

R1 was to only accept RIP routes from BB1. Without using authentication, the way to do this would be to make RIP AD 255, then use another neighbor-specific distance command for BB1. I missed this.

BGP

I had to prevent BB1/BB2 routes from being exchange to each other. Usually you would use an as-path filter, but the task did not allow this. I used community no-export, which I knew was over-filtering but for some reason I still used it. I should have just used community values like a tag, and then drop them on the way to each BBR.

I also had to find out what timers BB1 was using without looking at the config. I thought if I debugged keepalives I could tell. This does not work if your router has lower configured timer values. The peers use the lower value. The answer was to make your timers really high and then see what is negotiated. This is something I have read before but for some reason it didn't stick. I shall never forget again.

Multicast

I missed all 3 multicast tasks which was surprising because I am usually strong in this area. We need to make R6 an RP for the GLOP address ending with a 1. I used 233.0.0.1 but the middle octets are supposed to be the AS number (5051). Also, my multicast rate limiting statement wasn't specific enough because I didn't use a source list. And then I forget "filter-autorp" at the end of my multicast boundary statement. There was a lot more than this to configure but these items cost me the points.

Services

On DHCP, I forgot to disable dhcp conflict logging which I need to start remembering to do. I never disable it and I never have any problems, but the PG always has it disabled.

Security

Finally I missed a VTY security task to limit "telnet" access to only certain hosts. I made the ACL but forgot the transport input telnet.

One more volume 3 lab to go, which I start in a few hours. Next weekend I plan on doing Lab 1 again. This is the one I bombed on back in July when I was a wee little CCIE wannabe. It's been long enough for me to forget the details of that lab, so I want to see how much I have improved.

Renumbering IPv6 with ease via ipv6 general-prefix

2009-01-23T09:46:00.000-08:00

This is rather neat IPv6 feature that eases renumbering. We define a general prefix globally and then assign interface addresses based on that interface. Should you change providers or ever have to renumber the network, all you have to do is change the general prefix. Here's how it works:


R5(config)#ipv6 general-prefix TEST 2001:5::/48           
R5(config)#
R5(config)#int s1/0
R5(config-if)#ipv6 address TEST 2001:5::/48 eui-64 

R5#sho ipv6 interface s1/0 | inc :
  IPv6 is enabled, link-local address is FE80::E1B8:5FF:FE4C:9CDD 
  Global unicast address(es):
    2001:5::, subnet is 2001:5::/48 [GEN]
    2001:5::E1B8:5FF:FE4C:9CDD, subnet is 2001:5::/48 [EUI]
  Joined group address(es):
    FF02::1
    FF02::2
    FF02::1:FF00:0
    FF02::1:FF4C:9CDD
  ND DAD is enabled, number of DAD attempts: 1
R5#

We now have an IPv6 address assigned based on the EUI-64 method. The address is 2001:5::E1B8:5FF:FE4C:9CDD. Now suppose we need to change our prefix to 2001:6.


R5(config)#no ipv6 general-prefix TEST 2001:5::/48        
R5(config)#ipv6 general-prefix TEST 2001:6::/48   
R5(config)#
R5(config)#^Z
R5#sho ipv6 interface s1/0 | inc :
  IPv6 is enabled, link-local address is FE80::E1B8:5FF:FE4C:9CDD 
  Global unicast address(es):
    2001:6::, subnet is 2001:6::/48 [GEN]
    2001:6::E1B8:5FF:FE4C:9CDD, subnet is 2001:6::/48 [EUI]
  Joined group address(es):
    FF02::1
    FF02::2
    FF02::1:FF00:0
    FF02::1:FF4C:9CDD
  ND DAD is enabled, number of DAD attempts: 1
R5#

Image if we had more interfaces, this would make things so much easier. Especially considering each interface would have its own subnet. Imagine if we had interfaces on the 2001:5:0:1, 2001:5:0:2 (and so on) networks. We could change all of these to /48 prefix 2001:6:0:x:/64 with a couple commands. When you do change the general prefix, it does not overwrite the already configured one. This way you can have two prefixes during transition and eventually remove the older one as we did above.

CBAC with APPFW

2009-01-22T15:44:00.000-08:00

I have begun my goal of reading the entire 12.4 Security Configuration Guide. I likely won't read it all because many things are probably unrelated to CCIE R&S, but you never really can tell. Especially since the blueprint has "Other Security Features" on it. This configuration is part of CBAC and so I thought I would test a small scenario.

R4----s1/0 R5----R6

R4 is the http server and R6 is the client. Here is how I set them up to verify it's working:

R4#copy run test.html
Destination filename [test.html]?
Erase flash: before copying? [confirm]
Erasing the flash filesystem will remove all files! Continue? [confirm]
Erasing device... eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee ...erased
Erase of flash: complete
Verifying checksum... OK (0x7071)
1942 bytes copied in 4.628 secs (420 bytes/sec)
R4#
R4#dir
Directory of flash:/

1 -rw- 1942 test.html

7864316 bytes total (7862308 bytes free)
R4#conf t
R4(config)#ip http path flash:

R4 is setup, let's test R6 the client:

R6#copy http://192.168.45.4/test.html flash:
Destination filename [test.html]?
Erase flash: before copying? [confirm]
Erasing the flash filesystem will remove all files! Continue? [confirm]
Erasing device... eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee ...erased
Erase of flash: complete
Loading http://192.168.45.4/test.html !
Verifying checksum... OK (0x7071)
1942 bytes copied in 0.688 secs (2823 bytes/sec)
R6#

Good, so we know that works. Now we can configure R5 as the HTTP Application FW. This does require CBAC as well as some new appfw commands which I have never used. There are MANY more options besides this, so I suggest you read the DocCD for a more in depth explanation. I just wanted to get the gist of it here:

ip inspect name APPFW appfw HTTPFW
ip inspect name APPFW http
!
appfw policy-name HTTPFW
application http
strict-http action allow alarm
content-length minimum 1945 action reset alarm
port-misuse tunneling action reset

interface Serial1/0
description TO R4
ip inspect APPFW out

Notice the minimum content length is 1945 byes. This will prevent R6 from copying the file via HTTP (test.html is 1942 bytes as we can see above):

6#copy http://192.168.45.4/test.html flash:
Destination filename [test.html]?
Erase flash: before copying? [confirm]n
%Error opening http://192.168.45.4/test.html (I/O error)
R6#

Jump to R5 and see the message:

R5#
*Mar 2 05:34:02.708: %APPFW-4-HTTP_CONT_TYPE_SIZE: Sig:11 Content size 1942 out of range - Reset - Content size out-of-bounds from 192.168.56.6:25101 to 192.168.45.4:80

If we change the minimum content legth to 1942, everything works as expected:

R5(config)#appfw policy-name HTTPFW
R5(cfg-appfw-policy)#application http
R5(cfg-appfw-policy-http)#content-length minimum 1942 action reset alarm

R6#copy http://192.168.45.4/test.html flash:
Destination filename [test.html]?
%Warning:There is a file already existing with this name
Do you want to over write? [confirm]y
Erase flash: before copying? [confirm]n
Loading http://192.168.45.4/test.html !
Verifying checksum... OK (0x7071)
1942 bytes copied in 0.396 secs (4904 bytes/sec)
R6#

AS_SET not used in AS Path length comparison

2009-01-20T12:57:00.000-08:00

I was reading Chapter 3 today of Routing TCP/IP Volume 2 and it says that AS_SET is not considered when determining shortest AS_PATH. So I decided to lab it and see for myself. R4 is learning the 192.168.0.0/16 aggregate from R5 and R7 each with differing AS_SET lengths.


R4#sho ip bgp | be Net
   Network          Next Hop     Metric LocPrf Weight Path
*  192.168.0.0/16   192.168.47.7      0             0 7 {8,900} i
*>                  192.168.45.5      0             0 5 {6,600,6000,3033} i

The longest one is winning! AS_SET does count as 1 AS by the way.


R4#sho ip bgp 192.168.0.0
BGP routing table entry for 192.168.0.0/16, version 4
Paths: (2 available, best #2, table Default-IP-Routing-Table)
Flag: 0x820
  Advertised to update-groups:
     1         
  7 {8,900}, (aggregated by 7 192.168.78.7)
    192.168.47.7 from 192.168.47.7 (192.168.78.7)
      Origin IGP, metric 0, localpref 100, valid, external
  5 {6,600,6000,3033}, (aggregated by 5 192.168.56.5)
    192.168.45.5 from 192.168.45.5 (192.168.56.5)
      Origin IGP, metric 0, localpref 100, valid, external, best
R4#

All other things being equal it looks like the most recent path is winning. If we clear BGP on R5, R7 would be the most recent:


R5#clear ip bgp *
R5#
R5#
*Mar  1 01:34:11.475: %BGP-5-ADJCHANGE: neighbor 192.168.45.4 Down User reset
*Mar  1 01:34:11.479: %BGP-5-ADJCHANGE: neighbor 192.168.56.6 Down User reset
*Mar  1 01:34:11.667: %BGP-5-ADJCHANGE: neighbor 192.168.56.6 Up 
*Mar  1 01:34:12.287: %BGP-5-ADJCHANGE: neighbor 192.168.45.4 Up 

R4#sho ip bgp | be Net   
   Network          Next Hop     Metric LocPrf Weight Path
*  192.168.0.0/16   192.168.45.5      0             0 5 {6,600,6000,3033} i
*>                  192.168.47.7      0             0 7 {8,900} i
R4#

For more details, you can read this document which I am sure we have all seen by now. But little things like this may be forgotten:

BGP PATH SELECTION

IPexpert Volume 3 Mock Lab 8 Review

2009-01-19T19:53:00.000-08:00

At first read through, this lab appears very difficult because the number of routing protocol domains. 2 EIGRP domains, 3 OSPF domains, 1 RIP domain and almost every router and switch running 2 or 3 protocols. I attacked this by creating ACLs matching every set of networks, example:

ip access-list standard EIGRP134
ip access-list standard EIGRP24
ip access-list standard OSPF1
ip access-list standard OSPF2
ip access-list standard OSPF3
ip access-list standard RIP

Each ACL contained the networks in that domain. I then altered distance on each border router as needed so I could force the router to learn a route from that direction. In OSPF you can only specify one distance command so I had to merge the RIP and EIGRP ACL's in one case. The goal was to prevent route-feedback by ensuring that routes were learned through the best protocol to begin with. It took me about an hour but it worked great.

Another task had me configure clustering which is not as hard as it seems. A few commands on the commander and I was done. I had to read through the DocCD to figure some stuff out.

Side note: If you need to ping your own interface on a frame-relay task but are not allowed to use "frame-relay map", you can use Multilink interface and run MLPPPoFR.

BGP Section was pretty convoluted and I did not complete it. The main section revolved around using prepends so that distant ASes would disallow certain networks. I thought I could do this, but I did a bad job of reading ahead so I did not have the required confederations for this task. I did not fell like going back and re-doing BGP.

Another new command: "no service disable-ip-fast-frag"

IPv6 was pretty easy, I used tunnels to get everywhere.

QOS: Misunderstood the Flow Based WRED task, instead configured WRED + WFQ in a policy-map. I also configured the Be wrong and forgot adaptive shaping in a FRTS task.

Last task said to keep traffic stats for a host that might be under a DoS attack. I used accounting, but the PG has ip source-track. I should have got this, as I was just reviewing this topic in the DocCD last week.

I am not doing as good as I want to be doing. The last few labs in Volume 3 have been tougher, but there's nothing that I should not be able to get or find in the DocCD at this point. It's just a matter of staying focused and keeping the skills sharp.

IPexpert Volume 3 Mock Lab 7 Review

2009-01-18T19:58:00.000-08:00

This is a very challenging lab. I missed quite a few things, and there was a LOT of troubleshooting involved when things wouldn't work.

To begin with, all switches have dot1q trunk links to each other. However SW2 and SW3 are using flex-links. At first nothing seems wrong, then all of a sudden in the IGP section, SW1 becomes unbearably slow and R4 and R7 keep dropping EIGRP adjacencies. I noticed the RIP and IP INPUT processes on R1 were eating up the CPU. RIP and EIGRP packets were being looped over and over and over because STP does not run over Flex links! I shut the links down and attacked it later.

Another task asked to configure MLPPP over Frame Relay without using a multilink group. I created a multilink interface, but the answer was to use ppp multilink on a virtual-template and forget about the multilink interface.

BGP, Multicast, IP Services and IPv6 were pretty easy. I was glad because I had already spent 4+ hours getting through IGP. I did miss the HSRP task because they wanted the highest group possible. I used 255 but you were supposed to switch to version 2 and use group 4096!

Missed some delicate stuff regarding QoS. Byte counts were easy enough configure but the task said you should assume packet sizes of 100. This means you needed to adjust queue-limits also, which I did not do.

Another tricky one was a CQ to MQC conversion task. They displayed the CQ conversion as using a TCP syslog port. If you use NBAR to match syslog, it only uses UDP buy default. So you had to create a custom port-map. Tough one to see right away.

There was a login task that asked me to enable SSH for VTY lines. I forgot to create the key so it never would have worked. I should have verified this by attempting to login via SSH.

Finally, when I went back to the flex-link tasks I just used "switchport trunk allowed vlan none" to get it to work. The PG pruned even VLANs off from SW3 to SW2, then pruned the odd ones from SW2 to SW4. Then they shut the link from SW1 to SW2. Anyways, there were probably a number of ways to do it. It didn't really matter as long as you have connectivity.

Next up: Lab 8 tomorrow.

IPexpert Volume 3 Mock Lab 6 Review

2009-01-17T20:08:00.000-08:00

This lab took me about 7 hours to complete, verify and grade. There a few things I did not think I would get, but I ended up with solutions for everything and pretty much all of them worked. There was a task for something I had never heard of, SSG, which I got by looking in the DocCD and browsing the context sensitive help. The question mark is your friend!

Also another tricky one was R5 had a new loopback that needed to be NATTED based on the outgoing interface. I spent a good chunk of time on this but once I figured it out, it was pretty basic. I just had to match interfaces and addresses in a route-map, and use the route-map on a few NAT statements.

Here's part of it:

access-list 55 permit 55.55.55.55
route-map VLAN15 permit 10
match ip address 55
match interface FastEthernet0/0
ip nat inside source route-map VLAN15 interface FastEthernet0/0 overload

Here's what I missed:

-7 Task 1.1, 1.2 Switching

Didn't create vlan 400 on CAT1 or CAT2 and didn't make CAT4 root for that vlan. Vlan 400 did not have any hosts but was used as a native vlan. The task said to make CAT4 root for any vlans you have to create on that switch. CAT4 had no hosts, but nevertheless we had to create the vlan and make it root. This task involved a lot of stuff so to lose points for a couple unnecessary things is a bummer.

-3 Task 5.1 Multicast

IGMP Filter task said to deny 227.0.0.43 - 227.0.0.99 but only use a permit statement. Silly me included 227.0.0.1-42 and 100-255. I completely forgot this was denying all the 224/8, 225/8, etc groups.

-3 Task 8.1 QOS

I don't know if I would have got this wrong but I kind of misunderstood it. R9 has a Fastethernet connection while BB3 has an Ethernet. The task said to base your MQC percentages of off BB3's link speed. Well, I used "bandwidth 10000" under R9's interface so all the percentages worked out. The SG modified the percentages themselves. For example, the task said to give SMTP 25%, so the SG gave it bandwidth 2500 as opposed to 25000.

Other differences:

Task 3.3 - SG had an extra OSPF VL between R2 and R6 in area 246. Not really needed but probably a good idea.

Task 4.4 - SG used cost-community to influence path selection, I used "set origin" in a route-map. So much easier!

Well that's it for tonight. Lab 7 tomorrow and lab 8 on Monday. I am almost done with all the IPexpert material. I have watched and/or listened to all the bootcamp stuff at least once or twice as well. I bought an IE mock lab for next month, and I am planning on doing the Cisco Assessor labs as well.

SNAT: Making it work?

2009-01-16T23:36:00.001-08:00

This is a poorly documented feature and I really just played around with it until I got it to work. If you see anything missing or unnecessary, please comment. The one thing I worry about is I am using secondary addresses which may or may not be allowed in the lab. If you know another way, PLEASE let me know. Other than that, it was all kind of patchwork but it does the job :-)

Here is the topology:

R6 will be our test host who will telnet to R4 at 4.4.4.4. If all goes well, after we shut the link from R1 to R2 (whos is HSRP Active), R6 session will stay up. We will then look at the NAT translation table on R2 and R3.

Here is the configuration for R2:

interface FastEthernet0/0
ip address 10.0.0.2 255.255.255.0
ip nat inside
standby 1 ip 10.0.0.1
standby 1 priority 105
standby 1 preempt
standby 1 name SNAT
standby 1 track Serial1/0

interface Serial1/0
ip address 172.12.23.202 255.255.255.0 secondary
ip address 172.12.12.2 255.255.255.0
ip nat outside

ip nat Stateful id 1
redundancy SNAT
mapping-id 10

ip nat pool POOL 172.12.23.1 172.12.23.254 prefix-length 24
ip nat inside source list LAN pool POOL mapping-id 10 overload

R3 is pretty much the same except for the IP addresses:

interface FastEthernet0/0
ip address 10.0.0.3 255.255.255.0
ip nat inside
standby 1 ip 10.0.0.1
standby 1 preempt
standby 1 name SNAT
standby 1 track Serial1/0

interface Serial1/0
ip address 172.12.23.203 255.255.255.0 secondary
ip address 172.12.13.3 255.255.255.0
ip nat outside

ip nat Stateful id 1
redundancy SNAT
mapping-id 10

ip nat pool POOL 172.12.23.1 172.12.23.254 prefix-length 24
ip nat inside source list LAN pool POOL mapping-id 10 overload

I had to put secondary addresses on the serial links. These routers need to share an address space so they can use the same address to translate and so R1 and R4 no how to reach the translated address range. This secondary address range is being advertised in ospf:

R1#sho ip route 172.12.23.0
Routing entry for 172.12.23.0/24
Known via "ospf 1", distance 110, metric 128, type intra area
Last update from 172.12.12.2 on Serial1/0, 00:07:58 ago
Routing Descriptor Blocks:
* 172.12.13.3, from 172.12.35.3, 00:07:58 ago, via Serial1/1
Route metric is 128, traffic share count is 1
172.12.12.2, from 2.2.2.2, 00:07:58 ago, via Serial1/0
Route metric is 128, traffic share count is 1

Also note that the HSRP group name "SNAT" is referenced in the stateful NAT configuration. The mapping ID is then referenced in the NAT statement itself.

Let's telnet from R6 to R4, we will first verify that we route through R2:

R6#telnet R4
Translating "R4"
% Unknown command or computer name, or unable to find computer address
R6#telnet 4.4.4.4
Trying 4.4.4.4 ... Open

R4#
R4#!here we are!

Shut the interface on R1 to R2:

R1(config)#int s1/0
R1(config-if)#shut

Check back on R4. This may take awhile because HSRP still has to failover:

R4#
R4#
R4#!Hey we're still alive!
R4#
R4#exit

[Connection to 4.4.4.4 closed by foreign host]
R6#trace 4.4.4.4

Type escape sequence to abort.
Tracing the route to 4.4.4.4

1 10.0.0.3 56 msec 48 msec 60 msec
2 172.12.13.1 132 msec 68 msec 104 msec
3 172.12.14.4 148 msec * 184 msec

We are going through R3! If we did not have SNAT, our session would have dropped when R4 noticed that our address has changed.

Let's look at our address translations:

R2#sho ip nat translations
Pro Inside global Inside local Outside local Outside global
udp 172.12.23.5:37518 10.0.0.6:37518 4.4.4.4:33441 4.4.4.4:33441
udp 172.12.23.5:39661 10.0.0.6:39661 4.4.4.4:33442 4.4.4.4:33442
udp 172.12.23.5:42398 10.0.0.6:42398 4.4.4.4:33437 4.4.4.4:33437
udp 172.12.23.5:36656 10.0.0.6:36656 4.4.4.4:33439 4.4.4.4:33439
udp 172.12.23.5:39090 10.0.0.6:39090 4.4.4.4:33438 4.4.4.4:33438
udp 172.12.23.5:35099 10.0.0.6:35099 4.4.4.4:33440 4.4.4.4:33440

R3#sho ip nat translations
Pro Inside global Inside local Outside local Outside global
udp 172.12.23.5:37518 10.0.0.6:37518 4.4.4.4:33441 4.4.4.4:33441
udp 172.12.23.5:39661 10.0.0.6:39661 4.4.4.4:33442 4.4.4.4:33442
udp 172.12.23.5:42398 10.0.0.6:42398 4.4.4.4:33437 4.4.4.4:33437
udp 172.12.23.5:36656 10.0.0.6:36656 4.4.4.4:33439 4.4.4.4:33439
udp 172.12.23.5:39090 10.0.0.6:39090 4.4.4.4:33438 4.4.4.4:33438
udp 172.12.23.5:35099 10.0.0.6:35099 4.4.4.4:33440 4.4.4.4:33440
R3#

Exactly the same! Have no idea where these ports came from, but let's watch closer at the interaction between R2 and R3.

R2#clear ip nat translation *
R3#clear ip nat translation *

R6#telnet 4.4.4.4
Trying 4.4.4.4 ... Open

R4#

Here we go:

R2#sho ip nat translations
Pro Inside global Inside local Outside local Outside global
tcp 172.12.23.6:47684 10.0.0.6:47684 4.4.4.4:23 4.4.4.4:23

R3#sho ip nat translations
Pro Inside global Inside local Outside local Outside global
tcp 172.12.23.6:47684 10.0.0.6:47684 4.4.4.4:23 4.4.4.4:23

Some more commands:


R2#sho ip snat distributed
Stateful NAT Connected Peers

SNAT: Mode IP-REDUNDANCY :: STANDBY
    : State READY
    : Local Address 10.0.0.2
    : Local NAT id 1
    : Peer Address 10.0.0.3
    : Peer NAT id 1
    : Mapping List 10 

R3#sho ip snat distributed 

Stateful NAT Connected Peers

SNAT: Mode IP-REDUNDANCY :: ACTIVE
    : State READY
    : Local Address 10.0.0.3
    : Local NAT id 1
    : Peer Address 10.0.0.2
    : Peer NAT id 1
    : Mapping List 10 
R3#

R3 has already been updated and is ready to take over when needed.

Troubleshooting PIM-SM issues on a LAN segment

2009-01-16T19:59:00.000-08:00

Below is the topology for this lab. R1 is the Mapping Agent and the RP. PIM-SM is enabled everywhere except the link between R1 and R3.

All routers also have the following debug command:

debug ip pim 239.0.0.1

Let's take a look at what happens R5 joins group 239.0.0.1

R5(config)#int f0/0
R5(config-if)#ip igmp join-group 239.0.0.1
Mar 1 00:51:50.599: PIM(0): Check RP 1.1.1.1 into the (*, 239.0.0.1) entry

R4#ping 239.0.0.1 re 5 sou s1/0

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 239.0.0.1, timeout is 2 seconds:
Packet sent with a source address of 172.12.14.4
.....
R4#

Hmmm....a quick check of the RP mapping and everyone knows about 1.1.1.1 (R1) as the RP. Let's take a look at the mroute table on R1:

R1#sho ip mrou 239.0.0.1 | be \(
(*, 239.0.0.1), 00:01:38/stopped, RP 1.1.1.1, flags: SP
Incoming interface: Null, RPF nbr 0.0.0.0
Outgoing interface list: Null

(172.12.14.4, 239.0.0.1), 00:01:38/00:01:46, flags: PT
Incoming interface: Serial1/2, RPF nbr 0.0.0.0
Outgoing interface list: Null

R1 is seeing the packets from R4 but it's outgoing interface list is NULL. Let's take a look at R2's mroute table:

R2#sho ip mrou 239.0.0.1 | be \(
(*, 239.0.0.1), 00:03:27/00:02:57, RP 1.1.1.1, flags: SP
Incoming interface: Serial1/0, RPF nbr 172.12.12.1
Outgoing interface list: Null

NULL also, what gives? Let's wait and see if we get any debugs on R2:

R2#
00:55:42: PIM(0): Received v2 Join/Prune on FastEthernet0/0 from 172.12.25.6, not to us
00:55:42: PIM(0): Building Periodic Join/Prune message for 239.0.0.1

Interesting...It appears that R6 has become the DR for this segment and is responsible for sending (*,G) joins to the RP. R2 is hearing them, but ignoring them...why? What exactly is in the packet that tells R2 its not for us. Well since this is a dynamips lab, we can find out!

Here is a screenshot of the packet capture:

We can see that when R6 sends this join it is using a multicast address of 224.0.0.13. But inside of the PIM packet we can see R6 specifies an upstream neighbor of 172.12.25.3 which is R3.

Also on R6 we see the following debug messages:

R6#
*Mar 1 01:02:48.847: PIM(0): Building Periodic Join/Prune message for 239.0.0.1
*Mar 1 01:02:48.847: PIM(0): Insert (*,239.0.0.1) join in nbr 172.12.25.3's queue
*Mar 1 01:02:48.851: PIM(0): Building Join/Prune packet for nbr 172.12.25.3
*Mar 1 01:02:48.855: PIM(0): Adding v2 (1.1.1.1/32, 239.0.0.1), WC-bit, RPT-bit, S-bit Join
*Mar 1 01:02:48.859: PIM(0): Send v2 join/prune to 172.12.25.3 (FastEthernet0/0)

Can we fix this? Of course!

R6(config)#ip mroute 1.1.1.1 255.255.255.255 172.12.25.2

*Mar 1 01:05:52.019: PIM(0): Building Periodic Join/Prune message for 239.0.0.1
*Mar 1 01:05:52.019: PIM(0): Insert (*,239.0.0.1) join in nbr 172.12.25.2's queue
*Mar 1 01:05:52.023: PIM(0): Building Join/Prune packet for nbr 172.12.25.2
*Mar 1 01:05:52.027: PIM(0): Adding v2 (1.1.1.1/32, 239.0.0.1), WC-bit, RPT-bit, S-bit Join
*Mar 1 01:05:52.027: PIM(0): Send v2 join/prune to 172.12.25.2 (FastEthernet0/0)

Ping now:

R4#ping 239.0.0.1 re 5 sou s1/0

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 239.0.0.1, timeout is 2 seconds:
Packet sent with a source address of 172.12.14.4

Reply to request 0 from 172.12.25.5, 244 ms
Reply to request 1 from 172.12.25.5, 104 ms
Reply to request 2 from 172.12.25.5, 72 ms
Reply to request 3 from 172.12.25.5, 52 ms
Reply to request 4 from 172.12.25.5, 44 ms

But wait! There's one more solution. We can make R2 the DR for the segment (Remove the mroute on R6 and clear the mroute table on R2):

R2(config)#int f0/0
R2(config-if)#ip pim dr-priority 300000

01:07:09: PIM(0): Changing DR for FastEthernet0/0, from 172.12.25.6 to 172.12.25.2 (this system)
01:07:09: %PIM-5-DRCHG: DR change from neighbor 172.12.25.6 to 172.12.25.2 on interface FastEthernet0/0 (vrf default)
01:07:09: PIM(0): Check RP 1.1.1.1 into the (*, 239.0.0.1) entry
01:07:09: PIM(0): Building Triggered Join/Prune message for 239.0.0.1
01:07:09: PIM(0): Insert (*,239.0.0.1) join in nbr 172.12.12.1's queue
01:07:09: PIM(0): Building Join/Prune packet for nbr 172.12.12.1
01:07:09: PIM(0): Adding v2 (1.1.1.1/32, 239.0.0.1), WC-bit, RPT-bit, S-bit Join
01:07:09: PIM(0): Send v2 join/prune to 172.12.12.1 (Serial1/0)

R2#sho ip mrou 239.0.0.1 | be \(
(*, 239.0.0.1), 00:01:28/00:02:31, RP 1.1.1.1, flags: SJC
Incoming interface: Serial1/0, RPF nbr 172.12.12.1
Outgoing interface list:
FastEthernet0/0, Forward/Sparse, 00:01:28/00:02:31

It's always good to have more than one solution up your sleeve :)

Sending Logs as SNMP Traps

2009-01-16T09:13:00.000-08:00

I have been reading Chapter 9 of Routing TCP/IP Vol. II this week. It has a good overview of the non-core topics such as snmp, rmon, ntp, etc. I recommend it for anyone struggling with these topics or just wanting a concise review.

This example shows how to configure a router to send logs to an snmp-server and verify it.

The first thing you must do is configure a server. Without this, you want be able to see any debugging because the router won't send any packet out.

R8(config)#snmp-server host 4.4.4.4 public syslog
R8(config)#snmp-server enable traps syslog
R8(config)#logging console warnings
R8(config)#logging buffered 16384 debugging

I have also decided to buffer the logs instead of send them to the console. This is not required just a way to keep everything less cluttered on the console screen. What this configuration does is buffer all log messages of debugging level or lower. Then these messages are sent via SNMP to the server at 4.4.4.4. Let's debug snmp packets, then a quick shutting/no shutting of an interface will give us some messages to view:

R8#debug snmp packets
SNMP packet debugging is on

R8(config)#int f0/0
R8(config-if)#no snmp trap link-status
R8(config-if)#shut
R8(config-if)#no shut

R8#sho logging | beg Log Buffer
Log Buffer (16384 bytes):

*Mar 2 18:33:38.565: %LINK-5-CHANGED: Interface FastEthernet0/0, changed state to administratively down
*Mar 2 18:33:40.117: %PIM-5-DRCHG: DR change from neighbor 0.0.0.0 to 192.168.8.8 on interface FastEthernet0/0 (vrf default)
*Mar 2 18:33:40.641: %LINK-3-UPDOWN: Interface FastEthernet0/0, changed state to up
*Mar 2 18:33:40.669: SNMP: Queuing packet to 4.4.4.4
*Mar 2 18:33:40.669: SNMP: V1 Trap, ent ciscoSyslogMIB.2, addr 192.168.78.8, gentrap 6, spectrap 1
clogHistoryEntry.2.58 = LINK
clogHistoryEntry.3.58 = 4
clogHistoryEntry.4.58 = UPDOWN
clogHistoryEntry.5.58 = Interface FastEthernet0/0, changed state to up
clogHistoryEntry.6.58 = 15322065
*Mar 2 18:33:40.921: SNMP: Packet sent via UDP to 4.4.4.4
*Mar 2 18:33:42.513: %SYS-5-CONFIG_I: Configured from console by console

I disabled snmp trap link-status to show that we are not using this feature to send traps. Notice the entry labeled clogHistoryEntry.5.58, this is exactly the same message as our logging message a few lines up. We only get linkup messages with this basic config.

To modify the configuration we use the logging history command:

R8(config)#logging history ?
<0-7> Logging severity level
alerts Immediate action needed (severity=1)
critical Critical conditions (severity=2)
debugging Debugging messages (severity=7)
emergencies System is unusable (severity=0)
errors Error conditions (severity=3)
informational Informational messages (severity=6)
notifications Normal but significant conditions (severity=5)
size Set history table size
warnings Warning conditions (severity=4

R8(config)#logging history size 2

Here I set the history size to 2 so I am able to view the last 2 messages sent with the following command:

R8#sho logging history
Syslog History Table:2 maximum table entries,
saving level notifications or higher
149 messages ignored, 11 dropped, 0 recursion drops
57 table entries flushed
SNMP notifications enabled, 45 notifications sent
entry number 58 : LINK-3-UPDOWN
Interface FastEthernet0/0, changed state to up
timestamp: 15322065
entry number 59 : SYS-5-CONFIG_I
Configured from console by console
timestamp: 15345618

Pretty basic scenario. It is important to remember this is different from the usual way of sending linkup/linkdown traps. Here, we are not using "snmp-server enable traps snmp linkup linkdown" or the interface command "snmp trap link-status".

Also, I think I figured out why linkdowns are not being sent, if I manually configure the logging level to "notifications" it works:

R8(config)#logging history notifications
R8(config)#int f0/0
R8(config-if)#shut

R8#sho logging

*Mar 2 18:46:38.885: SNMP: Packet sent via UDP to 4.4.4.4
*Mar 2 18:46:39.981: %LINK-5-CHANGED: Interface FastEthernet0/0, changed state to administratively down
*Mar 2 18:46:40.057: SNMP: Queuing packet to 4.4.4.4
*Mar 2 18:46:40.061: SNMP: V1 Trap, ent ciscoSyslogMIB.2, addr 192.168.78.8, gentrap 6, spectrap 1
clogHistoryEntry.2.72 = LINK
clogHistoryEntry.3.72 = 6
clogHistoryEntry.4.72 = CHANGED
clogHistoryEntry.5.72 = Interface FastEthernet0/0, changed state to administratively down
clogHistoryEntry.6.72 = 15399999
*Mar 2 18:46:40.309: SNMP: Packet sent via UDP to 4.4.4.4
*Mar 2 18:46:40.981: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/0, changed state to down
*Mar 2 18:46:41.033: SNMP: Queuing packet to 4.4.4.4
*Mar 2 18:46:41.033: SNMP: V1 Trap, ent ciscoSyslogMIB.2, addr 192.168.78.8, gentrap 6, spectrap 1
clogHistoryEntry.2.73 = LINEPROTO
clogHistoryEntry.3.73 = 6
clogHistoryEntry.4.73 = UPDOWN
clogHistoryEntry.5.73 = Line protocol on Interface FastEthernet0/0, changed state to down
clogHistoryEntry.6.73 = 15400099

Not sure why I needed that because according to the "show logging history", notifications were the default level. However, it appears they aren't because the command shows up in the config:

R8#sho run | inc logging his
logging history size 2
logging history notifications
R8#

Basic MSDP configuration

2009-01-15T10:23:00.000-08:00

This is a short MSDP scenario designed to get familiar with the command to enable it and where you would use it. Below is the toplogy.

There are two domains, each with an RP. We seperate the domains by using the following commands on R3 and R4:

R3(config)#int s1/1
R3(config-if)#ip pim bsr-border

R4(config)#int s1/0
R4(config-if)#ip pim bsr-border

R2 and R4 have already been configured as the BSR and RP's for their respective domains. Let's verify on R1 and R5:

R1#sho ip pim rp mapping
PIM Group-to-RP Mappings

Group(s) 224.0.0.0/4
RP 2.2.2.2 (?), v2
Info source: 2.2.2.2 (?), via bootstrap, priority 0, holdtime 150
Uptime: 18:21:34, expires: 00:02:13

R5#sho ip pim rp map
PIM Group-to-RP Mappings

Group(s) 224.0.0.0/4
RP 4.4.4.4 (?), v2
Info source: 4.4.4.4 (?), via bootstrap, priority 0, holdtime 150
Uptime: 18:19:56, expires: 00:01:52

R1 and R8 have already joined group 225.0.0.1. Let's see what happens when R6 sends a ping to this group:

R6#ping 225.0.0.1 re 10

Type escape sequence to abort.
Sending 10, 100-byte ICMP Echos to 225.0.0.1, timeout is 2 seconds:

Reply to request 0 from 192.168.78.8, 192 ms
Reply to request 1 from 192.168.78.8, 192 ms
Reply to request 2 from 192.168.78.8, 100 ms
Reply to request 3 from 192.168.78.8, 84 ms
Reply to request 4 from 192.168.78.8, 112 ms
Reply to request 5 from 192.168.78.8, 104 ms

Only R8 responds. This is because the PIM joins from Domain 1 never get sent to the RP in Domain 2. Thus R4 never knows to forward to R3. Let's configure MSDP between R2 and R4:

R2(config)#ip msdp peer 4.4.4.4 connect-source loopback 0

R4(config)#ip msdp peer 2.2.2.2 connect-source loopback 0

It may take a moment but we will see this message:

*Mar 1 19:56:14.343: %MSDP-5-PEER_UPDOWN: Session to peer 2.2.2.2 going up

If we debug we would see this:

R4#debug ip msdp de
MSDP Detail debugging is on

*Mar 1 19:56:15.263: MSDP(0): Received 3-byte TCP segment from 2.2.2.2
*Mar 1 19:56:15.263: MSDP(0): Append 3 bytes to 0-byte msg 1170 from 2.2.2.2, qs 1
*Mar 1 19:56:15.643: MSDP(0): Sent entire mroute table, mroute_cache_index = 0, Qlen = 0
*Mar 1 19:56:15.647: MSDP(0): start_index = 0, sa_cache_index = 0, Qlen = 0
*Mar 1 19:56:15.651: MSDP(0): Sent entire sa-cache, sa_cache_index = 0, Qlen = 0
*Mar 1 19:56:16.275: MSDP(0): Received 3-byte TCP segment from 2.2.2.2
*Mar 1 19:56:16.275: MSDP(0): Append 3 bytes to 0-byte msg 1171 from 2.2.2.2, qs 1

Notice that R4 sent R2 its entire mroute table. Let's check the mroute table on R2:

R2#sho ip mroute 225.0.0.1 | be \(\*
(*, 225.0.0.1), 00:04:59/00:03:27, RP 2.2.2.2, flags: S
Incoming interface: Null, RPF nbr 0.0.0.0
Outgoing interface list:
Serial1/0, Forward/Sparse, 00:04:59/00:03:27

(192.168.56.6, 225.0.0.1), 00:01:47/00:01:12, flags: M
Incoming interface: Serial1/1, RPF nbr 192.168.23.3
Outgoing interface list:
Serial1/0, Forward/Sparse, 00:01:47/00:03:27

R2 now knows about the source of R6 and has even populated its OIL. The M flag tells us this is an MSDP created entry. Let's ping from R6:

R6#ping 225.0.0.1 re 5

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 225.0.0.1, timeout is 2 seconds:

Reply to request 0 from 192.168.78.8, 188 ms
Reply to request 0 from 192.168.12.1, 284 ms
Reply to request 0 from 192.168.12.1, 268 ms
Reply to request 1 from 192.168.12.1, 132 ms
Reply to request 1 from 192.168.78.8, 184 ms
Reply to request 2 from 192.168.12.1, 132 ms
Reply to request 2 from 192.168.78.8, 132 ms
Reply to request 3 from 192.168.12.1, 100 ms
Reply to request 3 from 192.168.78.8, 100 ms
Reply to request 4 from 192.168.12.1, 96 ms
Reply to request 4 from 192.168.78.8, 100 ms

Well that's it for now. You can have more complex scenarios with multiple domains (DocCD says MBGP is required for that) but the basics are easy to get down.

STP: UplinkFast and BackBoneFast

2009-01-14T12:56:00.000-08:00

This is I lab made to get familiar with the two STP features Uplinkfast and Backbonefast. RSTP (802.1w) includes these features but I don't seem to get the BackboneFast behavior when using "spanning-tree mode rapid-pvst+".

These features are similar but they are used to provide fast convergence in different scenarios depending on where the failure is in the STP toplogy.

Here is Topology #1 with SW1 configured as Root for VLAN13 where R1 and R4 reside. The Red Xes mark where STP is blocking:

Without Uplinkfast or Backbonefast enabled, lets see how long it takes STP to converge if port 13 on SW2 is shut:

R4 starts the ping while we shut the port on SW2

R4#ping 2001:13::1 re 1000000 Type escape sequence to abort. Sending 1000000, 100-byte ICMP Echos to 2001:13::1, timeout is 2 seconds: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!................!!!!!!

We missed about 16 pings and our topology has now converged. Let's enable UplinkFast on SW2, SW3 and SW4 and test again:

SW2(config)#spanning-tree uplinkfast
SW3(config)#spanning-tree uplinkfast
SW4(config)#spanning-tree uplinkfast

When you re-enable Port 13 on SW2, it takes awhile to come back up. It doesn't move through the LIS and LRN states according to the output but it will come up and you will see this message:

02:31:17: %SPANTREE_FAST-7-PORT_FWD_UPLINK: VLAN0013 FastEthernet0/13 moved to Forwarding (UplinkFast)

Now Let's ping from R4 again and the shut port 13 on SW2:

R4#ping 2001:13::1 re 1000000 Type escape sequence to abort. Sending 1000000, 100-byte ICMP Echos to 2001:13::1, timeout is 2 seconds: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!

One ping lost - WOW! Very fast. Let's build upon this failed scenario and look our new topology. Now we have SW2 forwarding through SW4 on it's way to the root SW1.

Let's see how long it takes to converge if we shut port 13 on SW4:

R4#ping 2001:13::1 re 1000000 Type escape sequence to abort. Sending 1000000, 100-byte ICMP Echos to 2001:13::1, timeout is 2 seconds: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.........!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

We miss about 9 pings here. What happened? Well in short, when SW4 path to root went down, it started thinking that it was the new root. This caused a new STP election to occur and SW4 finally had to wait until it heard the new SW1 BPDUs from SW1 > SW3 > SW2 > SW4.

BackboneFast can speed up this process, when SW2 starts hearing this inferior BPDUS from SW4 (who is cliaming to be root when port 13 goes down") and special query process takes place to speed up convergence. Let's enable it and check the pings again. This goes on all switches.

SW1(config)#spann backbonefast
SW2(config)#spann backbonefast
SW3(config)#spann backbonefast
SW4(config)#spann backbonefast

R4#ping 2001:13::1 re 1000000

Type escape sequence to abort.
Sending 1000000, 100-byte ICMP Echos to 2001:13::1, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Rapid Spanning Tree Protocol (802.1w) includes these features natively. In failover scenario #2 (shutting port 13 on SW4), only backbonefast provided quick convergence while Rapid-PVST+ did not. If have any ideas why, please let me know.

IPv6 NAT-PT

2009-01-14T10:27:00.000-08:00

This is a very simple IPv6 NAT-PT scenario. Here is the topology and addressing:

R1 is an IPv6 only host and R2 is an IPv4 only host.
R1 should use address 2001:23::2 to reach R2.
R2 should use 192.168.13.1 to reach R1.
R3 will be doing NAT-PT

Assign addresses per the diagram. The rest of the configuration is on R3.


R3(config)#int e0/0
R3(config-if)#ipv6 nat
R3(config-if)#int e0/1
R3(config-if)#ipv6 nat
R3(config)#ipv6 nat v4v6 source 192.168.23.2 2001:23::2
R3(config)#ipv6 nat v6v4 source 2001:13::1 192.168.13.1
R3(config)#ipv6 nat prefix 2001:23::/96

Remember to assing default gateways on R1 and R2:


R1(config)#ipv6 route 0::/0 2001:13::3

R2(config)#ip route 0.0.0.0 0.0.0.0 192.168.23.3

Let's ping from R1 while debugging on R3:

R3#debug ipv6 nat
IPv6 NAT-PT debugging is on

R1#ping 2001:23::2 re 1

Type escape sequence to abort.
Sending 1, 100-byte ICMP Echos to 2001:23::2, timeout is 2 seconds:
!
Success rate is 100 percent (1/1), round-trip min/avg/max = 4/4/4 ms
R1#

R3#
*Mar 1 13:51:12.323: IPv6 NAT: icmp src (2001:13::1) -> (192.168.13.1), dst (2001:23::2) -> (192.168.23.2)
*Mar 1 13:51:12.327: IPv6 NAT: src (192.168.23.2) -> (2001:23::2), dst (192.168.13.1) -> (2001:13::1)
R3#

Now let's try the other way:

R2#ping 192.168.13.1 re 1

Type escape sequence to abort.
Sending 1, 100-byte ICMP Echos to 192.168.13.1, timeout is 2 seconds:
!
Success rate is 100 percent (1/1), round-trip min/avg/max = 8/8/8 ms
R2#

R3#
*Mar 1 13:53:00.991: IPv6 NAT: src (192.168.23.2) -> (2001:23::2), dst(192.168.13.1) -> (2001:13::1)
*Mar 1 13:53:00.995: IPv6 NAT: icmp src (2001:13::1) -> (192.168.13.1), dst (2001:23::2) -> (192.168.23.2)
R3#

You can view the translations on R3:


R3#sho ipv6  nat translations
Prot  IPv4 source              IPv6 source
      IPv4 destination         IPv6 destination
---   ---                      ---
      192.168.23.2             2001:23::2

---   192.168.13.1             2001:13::1
      192.168.23.2             2001:23::2

---   192.168.13.1             2001:13::1
      ---                      ---

That's it!

Finding out port numbers with NBAR show commands

2009-01-13T10:56:00.000-08:00

I had a filtering task that said to allow H323 Traffic to a specific vlan. Well...what ports does H323 use? I could not find it on the DocCD but I remembered a show command that will let us know:


R1#sho ip nbar port-map h323 
port-map h323       udp 1300 1718 1719 1720 11720 
port-map h323       tcp 1300 1718 1719 1720 11000 - 11999

Sweet!

PPP Authentication with MD5

2009-01-12T10:08:00.000-08:00

I had a task this weekend that asked to authenticate PPP via Md5. I did a context sensitive help and saw this:


R2(config-if)#ppp authentication ?
chap        Challenge Handshake Authentication Protocol (CHAP)
eap         Extensible Authentication Protocol (EAP)
ms-chap     Microsoft Challenge Handshake Authentication Protocol (MS-CHAP)
ms-chap-v2  Microsoft CHAP Version 2 (MS-CHAP-V2)
pap         Password Authentication Protocol (PAP)

Doesn't look like there is an Md5 option...or is there? I looked up the ppp authentication commands in the DocCD:

-12.4 Mainline
-Master Index
-Cisco IOS Master Command List, All Releases
-ppp authentication MWP-147, SEC-1481

Click the SEC-1481 link

Now is where I used the browser search to look for "Md5." Not sure if this is possible in the lab so you may have to quickly scan with your eyes. The only hit comes up under "ppp eap local" command. You will see this phrase:

"In local mode, the EAP session is authenticated using the MD5 algorithm and obeys the same authentication rules as does Challenge Handshake Authentication Protocol (CHAP)."

Voila!

So now that we know what mode we need everything else is easy, and it works just like CHAP. On both sides:


username R5 password cisco

interface Serial1/1
ip address 150.100.25.2 255.255.255.0
encapsulation ppp
ppp authentication eap
ppp eap password 0 cisco
ppp eap local

Always verify just to make sure it's working:


R2#debug ppp authentication

*Mar  1 00:34:34.779: Se1/1 PPP: Using default call direction
*Mar  1 00:34:34.783: Se1/1 PPP: Treating connection as a dedicated line
*Mar  1 00:34:34.783: Se1/1 PPP: Session handle[9700001A] Session id[32]
*Mar  1 00:34:34.787: Se1/1 PPP: Authorization required
*Mar  1 00:34:34.967: Se1/1 EAP: O REQUEST  IDENTITY id 50 len 5
*Mar  1 00:34:35.015: Se1/1 EAP: I REQUEST  IDENTITY id 19 len 5
*Mar  1 00:34:35.015: Se1/1 EAP: O RESPONSE IDENTITY id 19 len 7 from "R2"
*Mar  1 00:34:35.123: Se1/1 EAP: I RESPONSE IDENTITY id 50 len 7 from "R5"
*Mar  1 00:34:35.127: Se1/1 EAP: O REQUEST  MD5 id 51 len 24 from "R2"
*Mar  1 00:34:35.131: Se1/1 EAP: I REQUEST  MD5 id 20 len 24 from "R5"
*Mar  1 00:34:35.147: Se1/1 EAP: Using hostname from unknown source
*Mar  1 00:34:35.151: Se1/1 EAP: Using password from interface EAP
*Mar  1 00:34:35.151: Se1/1 EAP: O RESPONSE MD5 id 20 len 24 from "R2"
*Mar  1 00:34:35.435: Se1/1 EAP: I RESPONSE MD5 id 51 len 24 from "R5"
*Mar  1 00:34:35.451: Se1/1 PPP: Sent CHAP LOGIN Request
*Mar  1 00:34:35.455: Se1/1 EAP: I SUCCESS id 20 len 4
*Mar  1 00:34:35.463: Se1/1 PPP: Received LOGIN Response PASS
*Mar  1 00:34:35.475: Se1/1 PPP: Sent LCP AUTHOR Request
*Mar  1 00:34:35.483: Se1/1 PPP: Sent IPCP AUTHOR Req
*Mar  1 00:34:35.495: Se1/1 LCP: Received AAA AUTHOR Response PASS
*Mar  1 00:34:35.499: Se1/1 IPCP: Received AAA AUTHOR Response PASS
*Mar  1 00:34:35.499: Se1/1 EAP: O SUCCESS id 51 len 4
*Mar  1 00:34:35.507: Se1/1 PPP: Sent CDPCP AUTHOR Request
*Mar  1 00:34:35.519: Se1/1 CDPCP: Received AAA AUTHOR Response PASS
*Mar  1 00:34:35.543: Se1/1 PPP: Sent IPCP AUTHOR Request

*Mar  1 00:34:36.503: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/1, changed state to up

You can see above that we have incoming and outgoing MD5 reposnses and the requests pass.

IPexpert Volume 3 Mock Lab 5 Review

2009-01-11T18:55:00.000-08:00

I just finished this lab in about 5 hours. I spent 1 hour verifying and found some mistakes. I ended up with a 73 and every single mistake except the BGP task should have been fixed. You will see below how easy these were.

I have still got some work to do in terms of fully understanding the requirements. I failed to make sure R7's extra loopbacks were in every routers table and for some reason IPv6 RIP failed when the script checked it. I logged back in and everything was fine.

Here are the mistakes:

-4 3.6 Redistribution
Failed connectivity to R7's loopback 2 and 3 addresses. I did not test reachability to these - only verified loopback 1.

-4 7.2 and 7.3 OSPFv3
I tested the results after the lab and it works fine. I wonder if the script shuts ports down for another VRRP failover task and then doesn't wait long enough for STP to forward, I don't know. Simple OSPFv3 and RIP redistribution.

-3 4.3 BGP Conditional Routing
Could not get this to work.

-3 5.3 Multicast
The argument is kbps and I put 64000 instead of 64. DOH!

-4 6.1 System Management
I had privilege 15 along with my username command and the script says this is wrong. The menu needs to display "show interfaces" and "show clock" and privilege 15 is not required for this.

-3 6.3 DHCP
The problem is that I filtered with a lot of caution so RIP routes do not enter EIGRP domain through OSPF. There were no requirements that stated full reachability is needed when various interfaces are shut.

-3 6.5 IP Accounting
I missed this because I thought you could only configure 1 list so I used a funky wildcard to match 2 subnets. I swear that on my first try the second list overwrote the first. Oh well, now I know.

-3 8.3 MQC
I forgot the bandwidth command on the serial interface of R7 and R8. This was a very easy MQC task, give 30% to telnet and 20% to smtp. The task said percentage of "interface bandwidth" and they are clocked at 2M.

Well that's it for the graded Mock Labs from IPexpert. There are still 5 more ungraded ones that I plan on doing. I think I will redo Mock Lab 1, I got a 41 on this about 6 months ago and am curious to know how much I have learned since then.

I also plan on doing mock labs from some other companies. I am going to do 1 or 2 from IE and I am debating on doing the CCIE Assessor labs. If you have any recommendations, please let me know.

IPexpert Volume 3 Mock Lab 4 Review

2009-01-10T21:05:00.000-08:00

Just took a dump on this lab. Lots of little mistakes. My problem was that many of the tasks were configuration heavy, mixing and matching totally unrelated options.

-4 2.2 PPP MD5 Authentication
I had this working right but the script said I needed "ppp eap identity " commands on each side. My link came up without them and I debugged PPP auth to verify it was authenticating.

-4 3.2 OSPF
A load of OSPF configuration and I was not supposed to use a network statement on areas 256, 96 and 97. I used a network statement for R9's loopback costing me the 4 points. There were probably over 50 commands for this task and 1 command cost me.

-3 3.3 OSPF Authentication
Grading script was wrong. You didn't need a VL between Cat1 and Cat3 and the script was checking for one.

-3 7.3 IPv6 Security
Grading script was wrong. I had filters on all IPv6 interfaces but the script was looking for it on the wrong interface.

-3 4.4 BGP Path Selection
I had to engineer a next hop solution and I used the "set ip next-hop" in a neighbor route-map. The Script didn't use a ping or trace to verify the solution so it did not know that my solution worked. Instead it came up with a bogus explanation.

-3 5.1 Multicast
What I do not like about the grading script is once it finds an error - it doesn't continue so you never get to see how the rest of the task would have been checked. In this case I did not put PIM in the loopbacks of the multicast routers. The task said all interfaces so I messed up on this.

-3 5.2 Multicast - Sink RP
I guess your supposed to deny the RP groups in an ACL on your mapping statement when configuring a sink RP. Makes sense otherwise you get the chicken/egg problem.

-2 6.2 System Management
This kind of crap pisses me off. I had to enable load-interval 60 on all interfaces and I forgot it on the port channels and loopback. Good grief.

-3 6.4 ECN
ip tcp ecn. I had no clue on this one.

-3 6.5 Local DNS
This is where the lab is screwed up. In this task R5 needs to telnet to R2 by name. But in a later task we have to block IPv4 telnet to R2 and only allow IPv6 via LOOPBACK IPV6 ADDRESS. The script does not use /source-interface and so it fails. I am going to bring this up with IPexpert.

-3 8.1 MQC
We were supposed to prevent R1 from sending unreachables without an interface command or rate-limit. The SG uses CPP to block them, pretty nifty. I used local policy routing to NULL 0 - probably not allowed but I could not think of another way.

-3 9.2 CBAC
The ACL the SG wants is supposed to be "strict". I allowed RIP and BGP and the SG only allows RIP. I guess BGP is included in the tcp router-traffic command but I will have to verify this. I actually thought about this but not enough to have me change it. I figured if I got it wrong I will be forced to learn a little more.

-3 9.4 TCP intercept
I did not configure any one-minute low/high options. I don't think the question asked for this, so I am not sure why they are there.

Well that's about it for today. The total score was 60 but I am pretty satisfied. I was able to browse the DocCD for some tasks I never heard of. This includes disabling the RFC 2217 option for telnet, MRM (which I got right), PPP EAP authentication, round robin DNS and some others.

This lab was heavy on the configuration and 4 point-tasks. This sucks because one little thing ruins the whole task. Plus, there was a bunch of little interesting topics on this lab I will probably blog about this week.

Blocking traffic to random unicast MAC addresses

2009-01-09T13:24:00.000-08:00

Ran into this command today. Never even knew about it:

Rack1SW1(config)#int f0/22
Rack1SW1(config-if)#switchport ?
access Set access mode characteristics of the interface
backup Set backup for the interface
block Disable forwarding of unknown uni/multi cast addresses
host Set port host
mode Set trunking mode of the interface
nonegotiate Device will not engage in negotiation protocol on this
interface
port-security Security related command
priority Set appliance 802.1p priority
private-vlan Set the private VLAN configuration
protected Configure an interface to be a protected port
trunk Set trunking characteristics of the interface
voice Voice appliance attributes

Rack1SW1(config-if)#switchport block unicast
Rack1SW1(config-if)#switchport block multicast

From the DocCD:

"By default, the switch floods packets with unknown destination MAC addresses out of all ports. If unknown unicast and multicast traffic is forwarded to a protected port, there could be security issues. To prevent unknown unicast or multicast traffic from being forwarded from one port to another, you can block a port (protected or nonprotected) from flooding unknown unicast or multicast packets to other ports."

Configuring Port Blocking

IPexpert Volume 2 Section 15 Review

2009-01-04T20:10:00.000-08:00

This lab took quite awhile, there were a lot of things I would have had to ask the proctor about. I just finished grading and its been about 7 hours since I started. I missed 4 tasks for sure and a couple other solutions differed from the SG but I believe they worked just fine.

These are the ones I missed:

-4 Task 1.3 Switching
Needed a bridge-group on R8 to bridge between dot1q sub-interfaces. I had a tough time understanding what was required and it really looked like a typo so I peaked in the SG. I didn't even read the entire solution once I saw "bridge-group" I knew what I needed.

-3 Task 8.1 VRRP
The very beginning of the lab says to use "open standards" as needed. I didn't even think about that when configuring this task and used HSRP instead.

-2 Task 8.2 TFTP
The task said to make an "IOS file" called BACKUP.bin available. I just copied running-config to BACKUP.bin not realizing that the file needed to be an IOS image. The actual command was tftp-server with an alias option for the bin file.

-3 Task 8.4 GRE Tunnel
This task required a GRE tunnel and I really misundertood this. First I configured mobile ARP, then NAT, but alas it was GRE.

There were a couple other tasks requiring filtering that I disagreed with the SG. No big deal but I am pretty sure my solutions were worked fine. Also I needed to set the DE bit on all traffic "from BB1". The SG created a de-list that matched the input interface. I used MQC to match a class then set the DE bit with "set fr-de" in a policy map.

Volume 2 is in the books now. I have already done a few Volume 3 labs and I will probably concentrate on these from now on. I will also browse through the Volume 2 labs again trying to solve the problems in my head.

IPexpert Volume 2 Section 14 Review

2009-01-03T16:59:00.000-08:00

I just completed this lab in about 4 hours. I spent some time before my session started drawing diagrams and reading through the lab. I find this helps me save session time in case I take too long. Plus it gives me substantial time to grade the lab, verify solutions and even test out the solution guide if needed.

I only missed a few tasks, and some of these were because I was unfamiliar with commands and I made the solution too difficult.

-3 Task 7.3 BGP
I needed to provide redundant BGP connectivity after filtering some routes. I misunderstood this task, we needed an aggregate with as-set and I left it off.

-2 Task 7.5 BGP
I had to prevent AS paths of 16 or longer. I created an enormously large as-path ACL when all that was required was max-as limit. I know this command but I just had a brain fart.

-3 Task 10.1 CQ
Custom Queuing task had some extra stuff to throw you off. Very tricky ;-)

-3 Task 11.3 Multicast
For rate limiting multicast I used normal CAR, but there actually exists a special multicast rate-limit command.

There were several peculiar service tasks that I am getting the hang of now. I had to compress the config, decreases the telnet timeout, and some other stuff. Browsing the DocCD as well as the context sensitive help as helped me with these kinds of tasks.

A little less than 2 months to go now. I have been doing pretty well on 8 hour labs. Lately I have working on improving my verification habits. After a lab, I force myself to review almost every single task, even things like VLAN assignments. I always find a couple errors and I have been reducing my bonehead mistakes by a lot.

One more Volume 2 lab to go then I will probably stick with doing my own labs and Volume 3 graded mock labs.

NAT on a Stick

2009-01-02T09:36:00.001-08:00

NAT on a stick can get pretty confusing. Here is a lab I put together with the help of an example on Cisco site. I don't know if it is accessible through the DocCD so here is the link:

Network Address Translation on a Stick

The topology is a little different because I am using routers without any cable devices:

R2 is a host on the 10.0.0.0/24 network. It is using 10.0.0.1 (R1) as the gateway. R1 then NATs this address to 192.168.2.X before sending the packet on its way to R3. That's the basic rundown but the configuration is a little more complex.

First things first...Since 192.168.2.0 will be our translated address range make sure R4 and R3 both have routes to this range. R3 will use 192.168.1.1 as the next hop and R4 will use 100.0.0.3.

The rest of the configuration is on R1. Assign two addresses to R1 and configure it as our inside interface. 10.0.0.1 is used as a gateway address for hosts on the LAN and 192.168.1.1 is used to communicate with R3.

interface FastEthernet0/0
ip address 192.168.1.1 255.255.255.0 secondary
ip address 10.0.0.1 255.255.255.0
ip nat inside

Create the loopback that will be used as our outside interface. Keep in mind we are using a /30 network, you will see why a little later on.

interface Loopback0
ip address 10.0.1.1 255.255.255.252
ip nat outside

Next define an ACL to match our inside hosts then configure the NAT pool and NAT statements:

access-list 10 permit 10.0.0.0 0.0.0.255
ip nat pool NAT 192.168.2.100 192.168.2.200 prefix-length 24
ip nat inside source list 10 pool NAT

Now this is where it get's a little tricky. I am trying to do this in a logical order, but in reality I just have to memorize what needs to be configured to finish this thing off.

Since the 192.168.2.0 network does not exist on any interface we tell R1 that it exists of F0/0 like this:

ip route 192.168.2.0 255.255.255.0 FastEthernet0/0

Finally, we add our policy routing configuration. Remember the ACL has to match traffic in both directions:

access-list 100 permit ip any 192.168.2.0 0.0.0.255
access-list 100 permit ip 10.0.0.0 0.0.0.255 any

route-map NAT-LOOP permit 10
match ip address 100
set ip next-hop 10.0.1.2

interface FastEthernet0/0
ip policy route-map NAT-LOOP

So now R1 knows to policy route any traffic coming from 10.0.0.0/24 or going towards 192.168.2.0/24. The next hop address is 10.0.1.2 which technically exists on the Loopback 0 network. Since our loopback has the NAT outside statement, translation occurs here.

Let's test:

R2#ping 100.0.0.4

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 100.0.0.4, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 24/60/112 ms
R2#

R1#show ip nat translations
Pro Inside global Inside local Outside local Outside global
icmp 192.168.2.100:8 10.0.0.2:8 100.0.0.4:8 100.0.0.4:8
--- 192.168.2.100 10.0.0.2 --- ---
R1#

Key things to remember about NAT on a Stick:

-Ensure upstream routers have to routes back to the NAT (outside) address
-ACL for policy routing is 2-way
-Loopback is used for outside interface, but the NAT pool is on a separate network.
-Use a route pointing to the LAN interface to tell the router where the outside network resides.

There are probably some variations of this configuration that will work. I am going to play around with some now, but that should be enough to get started.

HSRP and Redirects

2009-01-01T19:35:00.001-08:00

HSRP is fairly easy to understand and configure but the more you dig into something there is always something new you are bound to find. This goes for me today with HSRP redirection. Honestly, I never really bothered to look into the topic too much but reading through the DocCD today kind of piqued my curiosity.

Here is the topology for this lab which I borrowed from the configuration guide. I would post a link but that would be too easy, I figured I'd let you practice finding it :)

The topology is big but my focus is on what R1 does based on what it knows about the HSRP status of all the other routers. Since all routers are doing OSPF, R1 learns about 172.16.34.0/24 from R3 and R4 and it learns about 172.16.8.0/24 from R8.

R9 is NOT doing OSPF and using the standby group 1 address of 192.168.1.100 as its default gateway. In normal operation R1 would send redirects to R9 if it received packets for the R8 or the R3/R4 network. When HSRP is in use (with a default config), it only sends redirects for the R8 network.

Before we test, enable "debug ip icmp" on all routers as well as "debug standby events" on R1. Let's see what happens when R9 pings 172.168.8.8:

R9#ping 172.16.8.8 re 1

Type escape sequence to abort.
Sending 1, 100-byte ICMP Echos to 172.16.8.8, timeout is 2 seconds:
!
Success rate is 100 percent (1/1), round-trip min/avg/max = 124/124/124 ms
R9#
*Mar 1 01:29:20.207: ICMP: redirect rcvd from 192.168.1.100- for 172.16.8.8 use gw 192.168.1.8
*Mar 1 01:29:20.263: ICMP: echo reply rcvd, src 172.16.8.8, dst 192.168.1.9

This is what we see on R1:

R1#
*Mar 1 01:30:15.771: ICMP: Use HSRP virtual address 192.168.1.100 as ICMP src
*Mar 1 01:30:15.775: ICMP: redirect sent to 192.168.1.9 for dest 172.16.8.8, use gw 192.168.1.8

Now look what happens when we ping 172.16.34.3:

R9#ping 172.16.34.3 re 1

Type escape sequence to abort.
Sending 1, 100-byte ICMP Echos to 172.16.34.3, timeout is 2 seconds:
!
Success rate is 100 percent (1/1), round-trip min/avg/max = 112/112/112 ms
*Mar 1 01:30:54.407: ICMP: echo reply rcvd, src 172.16.34.3, dst 192.168.1.9

The ping succeeded of course but we did not get a redirect. The packet actually hops from R1 to R4, then the reply goes to R9. On R1 we see this:

R1#
*Mar 1 01:30:56.187: ICMP: redirect not sent to 192.168.1.9 for dest 172.16.34.3
*Mar 1 01:30:56.191: ICMP: 192.168.1.4 does not contain an active HSRP group

The reason R1 does not send a redirect is because R4 is not active for any groups. For all R1 knows, R4 is not active for a reason and thus should not send redirects for it. But how does R1 know this? It keeps track of all the HSRP messages it hears. We can view this as follows:

R1#sho standby redirect
Interface Redirects Unknown Adv Holddown
FastEthernet0/0 enabled enabled 30 180

Active Hits Interface Group Virtual IP Virtual MAC
local 0 FastEthernet0/0 1 192.168.1.100 0000.0c07.ac01
192.168.1.3 0 FastEthernet0/0 3 192.168.1.200 0000.0c07.ac03

Passive Hits Interface Expires in
192.168.1.2 0 FastEthernet0/0 179.856
192.168.1.4 4 FastEthernet0/0 162.824
R1#

Notice that R3 is an Active router for a group. R1 would send a redirect if R3 was listed as the next hop, but in this case R3 and R4 are equal costs and R4 is being picked. In spite of all this we can trick R1 into sending redirects by making R4 Active for a group. Let's enable group 4 only on R4 to ensure it becomes Active, then ping again from R9:

R4(config)#int f0/0
R4(config-if)#standby 4 ip 192.168.1.204

R1#sho stan re active
Active Hits Interface Group Virtual IP Virtual MAC
local 0 FastEthernet0/0 1 192.168.1.100 0000.0c07.ac01
192.168.1.3 0 FastEthernet0/0 3 192.168.1.200 0000.0c07.ac03
192.168.1.4 0 FastEthernet0/0 4 192.168.1.204 0000.0c07.ac04

R9#ping 172.16.34.3 re 1

Type escape sequence to abort.
Sending 1, 100-byte ICMP Echos to 172.16.34.3, timeout is 2 seconds:
!
Success rate is 100 percent (1/1), round-trip min/avg/max = 80/80/80 ms
R9#
*Mar 1 01:35:44.703: ICMP: redirect rcvd from 192.168.1.100- for 172.16.34.3 use gw 192.168.1.204
*Mar 1 01:35:44.739: ICMP: echo reply rcvd, src 172.16.34.3, dst 192.168.1.9

We got a redirect! What's interesting about this behavior is that R1 knows the difference between a network behind HSRP routers and a network behind a non-HSRP router. It knows this by learning which routers are sending HSRP messages and comparing them to the next hops in its route table. So if you every get a tricky task about HSRP and redirects, I hope this helps shed some light on it.

Before I go, here is some more food for thought:

R1(config-if)#standby redirect ?
advertisement Redirect advertisement messages
timers Adjust redirect timers
unknown Redirect to non-HSRP routers

Making a VLAN IPv6 only

2008-12-23T13:04:00.000-08:00

Here is the simple topology for this lab. R1 and R2 are on VLAN 12. VLAN12 needs to be IPv6 only. We test this my assigning IPv4 and IPv6 addresses to both routers and then pinging.

R1---SW1---SW2---R2

R1:

IPv4: 192.168.12.1/24
IPv6: 2001::1/64

R2:

IPv4: 192.168.12.2/24
IPv6: 2001::2/64

Making a vlan IPv6 only requires more configuration than I previously thought. This was my first attempt. On all switches:

mac access-list extended IPv6
permit any any 0x86DD 0x0
vlan access-map IPv6only 10
action forward
match mac address IPv6
vlan filter IPv6only vlan-list 12

So R1 pings R2:

R1#ping 192.168.12.2

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.12.2, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)
R1#ping 192.168.12.2

But wait, let's remove the filter, ping, add the filter back, and ping again.

SW1(config)#no vlan filter IPv6only vlan-list 12

R1#ping 192.168.12.2

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.12.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/3/4 ms

SW1(config)#vlan filter IPv6only vlan-list 12

R1#ping 192.168.12.2

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.12.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/4 ms

R1 can still ping. What happened? Well the original filter wasn't blocking IP, it was only blocking ARP packets. Remember MAC access-lists do not have an implicit deny for the IP ethertype but they do have an implicit deny for all the other ethertypes. So once we removed the filter and allowed ARP through, R1 was able to ping R2 when the filtered was applied.

To make the vlan IPv6 only I had to specify a drop action in an empty access-map statement:

SW1(config)#vlan access-map IPv6only 20
SW1(config-access-map)# action drop

R1#ping 192.168.12.2

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.12.2, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)
R1#

But wait, let's check out spanning-tree:

SW1#sho spanning-tree vlan 12 | inc root
This bridge is the root
SW2#show spanning-tree vlan 12 | inc root
This bridge is the root

This is bad because both switches forward out all ports when they think they are root. If we had multiple links between these switches, we would have a loop. You may start seeing these messages:

SW2#
01:28:49: %SW_MATM-4-MACFLAP_NOTIF: Host 00b0.6410.3901 in vlan 12 is flapping between port Fa0/13 and port Fa0/14
01:28:49: %SW_MATM-4-MACFLAP_NOTIF: Host 0007.eb14.4f81 in vlan 12 is flapping between port Fa0/13 and port Fa0/14

We need to allow STP bpdu's in our original MAC access-list. Do this now:

SW1(config)#mac access-list extended IPv6
SW1(config-ext-macl)#permit any any lsap 0xAAAA 0x0

Now we see SW2 blocking on the port f0/14 (for VLANs 1 and 12):

SW2#sho span | inc BLK
Fa0/14 Altn BLK 19 128.16 P2p
Fa0/14 Altn BLK 19 128.16 P2p

Verify R1 can ping R2 via IPv6 and not IPv4:

R1#ping 192.168.12.2

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.12.2, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)
R1#ping 2001::2

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 2001::2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 0/1/4 ms
R1#

I used 0xAAAA because this what lsap type PVST uses. I don't know where I got this but I think I saw it on GS somehwere. I have also seen 0x4242 used but I think this is for normal STP (802.1d). In any case, only the 0xAAAA worked for me.

Extended Range VLAN - FAIL

2008-12-22T15:28:00.000-08:00

I was reading an old post on CCIE talk about extended range vlans and I learned something new. The post is here: CCIE Talk

If a Catalyst switch has any routed ports then it uses an extended vlan as an "internal vlan" for that port. Why? I don't know but it's something to take caution with if you run into any issues.

Check it out:

SW1#sho vlan internal usage

VLAN Usage
---- --------------------

Now let's create a routed port:

SW1(config)#int f0/24
SW1(config-if)#no sw

Now we have VLAN 1006 taken up:

SW1#sho vlan internal usage

VLAN Usage
---- --------------------
1006 FastEthernet0/24

SW1#

What happens if we try to create or modify VLAN 1006? Let's see:

SW1(config)#vlan 1006
SW1(config-vlan)#exit
% Failed to create VLANs 1006
VLAN(s) not available in Port Manager.
%Failed to commit extended VLAN(s) changes.
00:06:13: %PM-4-EXT_VLAN_INUSE: VLAN 1006 currently in use by FastEthernet0/24
00:06:13: %SW_VLAN-4-VLAN_CREATE_FAIL: Failed to create VLANs 1006: VLAN(s) not available in Port Manager
SW1(config)#

FAIL. What if we were supposed to use VLAN 1006? Shut it down, enable VLAN 1006, then re-enable the port.

SW1(config)#int f0/24
SW1(config-if)#shut
SW1(config)#do show vlan internal usa

VLAN Usage
---- --------------------

SW1(config)#vlan 1006
SW1(config-vlan)#exit
SW1(config)#int f0/24
SW1(config-if)#no shut
SW1(config-if)#exit
SW1(config)#do show vlan internal usa

VLAN Usage
---- --------------------
1007 FastEthernet0/24

The switch uses the next number.

IP Source Guard

2008-12-19T15:54:00.000-08:00

I was reading through the 3560 Configuration guide looking for things to lab and I came up with this. I already had DHCP snooping configured from my last lab so it was real easy.

Topology:

R1---SW1---R3

R1 has an address via DHCP:

R1#show ip int brief | ex unas
Ethernet0/0 192.168.12.1 YES DHCP up up

R1 can ping R3 on it's subnet:

R1#ping 192.168.12.3

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.12.3, timeout is 2 seconds:
.!!!!
Success rate is 80 percent (4/5), round-trip min/avg/max = 1/3/4 ms

Sw1 has dhcp snooping already enabled. Here we configure IP source guard:

SW1(config)#int f0/1
SW1(config-if)#ip verify source

Now on R1 if we change the IP address, we cannot ping anymore:

R1(config)#int e0/0
R1(config-if)#ip address 192.168.12.100 255.255.255.0
R1(config-if)#^Z

R1#ping 192.168.12.3

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.12.3, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)

Verify IP source guard is in effect on SW1:

SW1#show ip verify source
Interface Filter-type Filter-mode IP-address Mac-address Vlan
--------- ----------- ----------- --------------- -----------------
Fa0/1 ip active deny-all 12

Set R1 to get address via DHCP:

R1(config)#int e0/0
R1(config-if)#ip address dhcp
*Mar 1 02:53:06.259: %DHCP-6-ADDRESS_ASSIGN: Interface Ethernet0/0 assigned DHCP address 192.168.12.4, mask 255.255.255.0, hostname R1

Now R1 can ping again:

R1#ping 192.168.12.3

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.12.3, timeout is 2 seconds:
.!!!!
Success rate is 80 percent (4/5), round-trip min/avg/max = 1/3/4 ms

Verify on SW1:

SW1#show ip verify source
Interface Filter-type Filter-mode IP-address Mac-address Vlan
--------- ----------- ----------- --------------- -----------------
Fa0/1 ip active 192.168.12.4 12

You can also configure static bindings, but I will probably do that in another blog :)

SLA - Sending a Constant Bitrate

2008-12-17T14:25:00.000-08:00

Everytime I study QoS I think about ways to generate a constant rate of traffic from a router. I always test using pings but I never really know at what rate data is being pushed through. With SLA, I can configure a somewhat rudimentary way of doing this.

Suppose I want a router R1 to send 64K to R7 (off in the distance).

Let's figure out the data size and frequency. There are probably multiple ways to do this depending on frequency and request-data-size but here is how I do it:

My load-interval on R7 is going to be 30 seconds so 1,920,000 (64,000 x 30) bits need to flow through every 30 second interval. Now if I send data a 1 second intervals, then I need to send 64000 bits every second. 64000 bits = 8000 bytes.

Formula using 1 second frequency intervals:

Load-interval X desired bitrate = total bits per interval
total bits per interval / 8 = request-data-size

Here is the config:

R1(config)#ip sla monitor 1
R1(config-sla-monitor)#type echo protocol ipIcmpEcho 150.100.56.7
R1(config-sla-monitor-echo)#request-data-size 8000
R1(config-sla-monitor-echo)#frequency 1

On R7 I created this tracker:

ip sla monitor responder

access-list 100 permit icmp host 150.100.12.1 any

class-map match-all SLA
match access-group 100

policy-map TRACK-SLA
class SLA

interface FastEthernet0/0
service-policy input TRACK-SLA

Now Let's start the SLA monitor on R1:

R1(config)#ip sla monitor schedule 1 life forever start-time now

Now on R7 we use the show policy-map interface command to see the bit rate. It takes a little while but it should peak near 64000 bps give or take 1000.

After 750 packets we have 65K:

R7#sho policy-map interface
FastEthernet0/0

Service-policy input: TRACK-SLA

Class-map: SLA (match-all)
750 packets, 1027500 bytes
30 second offered rate 65000 bps
Match: access-group 100

Class-map: class-default (match-any)
91 packets, 6548 bytes
30 second offered rate 0 bps, drop rate 0 bps
Match: any

Now several minutes later we are still at 65K:

R7#sho policy-map interface
FastEthernet0/0

Service-policy input: TRACK-SLA

Class-map: SLA (match-all)
2784 packets, 3814080 bytes
30 second offered rate 65000 bps
Match: access-group 100

Class-map: class-default (match-any)
280 packets, 20188 bytes
30 second offered rate 0 bps, drop rate 0 bps
Match: any

Parser View

2008-12-17T10:23:00.000-08:00

I was reading this pdf called "1001 things to do with a Cisco Router" and I came across this topic. I have done it before while doing the ISCW but here it is again.

FIRST, ENABLE AAA:

R4#conf t
Enter configuration commands, one per line. End with CNTL/Z.
R4(config)#aaa new-model

SET ENABLE PASSWORD:

R4(config)#enable secret cisco
R4(config)#^Z

SWITCH TO VIEW MODE:

R4#en view
Password:
R4#
*Mar 2 23:03:20.352: %PARSER-6-VIEW_SWITCH: successfully set to view 'root'.
R4#

NOW WE CAN CREATE THE VIEW:

R4(config)#parser view operator
R4(config-view)#?
View commands:
commands Configure commands for a view
default Set a command to its defaults
exit Exit from view configuration mode
no Negate a command or set its defaults
secret Set a secret for the current view
R4(config-view)#commands exec include ping
% Password not set for the view operator
R4(config-view)#secret operator
R4(config-view)#commands exec include ping
R4(config-view)#commands exec include show hardware
R4(config-view)#commands exec include show interface
R4(config-view)#commands exec include show ver
R4(config-view)#exit

LOG IN TO THE VIEW:

R4#en view operator
Password:

*Mar 2 23:05:41.212: %PARSER-6-VIEW_SWITCH: successfully set to view 'operator'.

R4#show ?
flash: display information about flash: file system
hardware Hardware specific information
interfaces Interface status and configuration
parser Display parser information
slot0: display information about slot0: file system
slot1: display information about slot1: file system
version System hardware and software status

ALSO, YOU CAN ADD THE VIEW TO THE USER:

R4(config)#username operator view operator password operator

DHCP Snooping - missing command?

2008-12-15T14:59:00.000-08:00

I was having a hard time with this awhile ago because I could not get an address even when I enabled "trust" on the server port. However, after looking through the PG on Mock Lab 3 and discussion in the cisco channel on freenode I found out the issue.

I needed this command on the server:

R2(config)#int e0/0
R2(config-if)#ip dhcp relay information trusted

Now my binding database is populated after about 9 months!

SW1#show ip dhcp snooping binding
MacAddress IpAddress Lease(sec) Type VLAN Interface
------------------ --------------- ---------- ------------- ---- --------------------
00:07:EB:14:4F:81 192.168.12.1 86312 dhcp-snooping 12 FastEthernet0/1
Total number of bindings: 1

SW1#

IPexpert Volume 3 Mock Lab 3 Review

2008-12-14T20:24:00.000-08:00

I finished this lab in a little more than 6 hours. It was a graded lab through Proctor Labs and I got an 82. This a very challenging lab because there was some dot1q tunneling involved and it affected reachability if you didn't prune VLANs properly due to l2portguard errors. Also, there was an IPv6 tunneling section which I got right. In fact, I got 100% on IGP, BGP and Multicast for a total of 44 points.

Here are the mistakes I made:

-3 1.2 Switching
Did not enable trust on the trunk ports after I enabled DHCP snhooping.

-2 2.3 Frame Relay
After I did some NAT R4 could no longer ping R2 over the Frame-relay.

-3 6.1 VRRP
I used group 1 instead of group 24. BONEHEAD mistake.

-4 6.5 IOS Services
Some NAT stuff. I think I got this right but...oh well.

-3 8.1 QoS
PBR config was supposedly on the wrong interface. I am arguing this one with the script writers.

-3 9.2 Security
I got the URL string wrong for blocking NIMDA.

All in all I felt pretty good. I had been practicing tunneling last night and I don't think I would have done as well or finished as fast if I hadn't. I gained a lot of confidence this round. There were some things I did not think I would be able to figure out upon the initial read-through. However, once I turned off the TV, I was in a pretty good groove :-)

IPexpert Volume 2 Section 13 Reveiw - Part II

2008-12-13T17:54:00.001-08:00

I just finished the lab that I started last week. I don't really have an estimate of how well I did because I had several conflicts with the PG. My guess is around 70 - 80. It was definitely the hardest full scale practice lab I have done to date. Not so much in configuration, just in understanding what the PG was trying to say. Here are some examples:

R2 should show L2 circuit IDs when viewing "debug traces." This was in the security section and the answer was to create an ACL that permitted everything and log-input for ICMP. Easy enough and I thought about that solution but I thought there might have been another way - alas, there wasn't.

CAT1 should only allow PC with mac address 0001.0001.0001 and IP 10.10.10.1 on port f0/10. I thought about ARP inspection along with port security but the answer was only port security. Not sure if I would gotten it wrong if I had the port security right but borked on the ARP inspection which was not required.

There was also an auto-command task that forced me to override what I had already configured for my VTY lines in previous tasks. I would have had to ask the proctor about this as there was no way to have the two solutions (line password and local authentication) without AAA. And my AAA solution was not allowing auto-command to work.

There was a total of 58 tasks on this lab which is incredible because I don't think I have had a lab that had as many as 40 yet. Each task was 1 or 2 points and there was a lot to configure. I don't know if I would have completed it in 8 hours - I took my time writing emails to support during the lab.

In any case, this was a very challenging lab and I think I will re-do this one in a couple months if I have time.

Auto-install, eh?

2008-12-13T16:05:00.000-08:00

While doing IPexpert Volume 2 Section 13 I ran into a task that said:

"There is a high chance you will be replacing your current R4 router with another high-end router. The admin of R4 has saved its configuration on a TFTP server whose IP address is 136.10.12.100. Make sure the new router will automatically configure itself."

So I started browsing through the DocCD for auto-install when it hit me...how exactly is this supposed to work? Not knowing the exact details about auto-install I knew that this should be a simple task since it was only 1 point.

Well the new router needs to know about address 136.10.12.100 somehow...but when it has no config it has no address. What I figured was that the new router will send a broadcast on it's frame-relay interface which happens to connect to R2. In fact the 136.10.12.100 network is also on R2's ethernet interface.

So I configured a helper address on R2's frame-relay interface pointing to 136.10.12.100. The PG agreed! 1 task, 1 command, 1 point :-)

ECMP Multicast Load Splitting

2008-12-11T11:32:00.000-08:00

This is a pretty simple concept. By default when two paths to the RP exist, the router sends a join to the one with the highest IP address. When you enable multicast multipath, the router will send joins up multiple paths depending on Source address (this hash is modifiable in some IOS)

Here is the topology:

R4 has joined group 239.0.0.1. R5, R6 and R7 are all sending pings to this address. Before enabling multipath, this is what R1's mroute table looks like (it's actually bigger I am omitting output for the sake of brevity):

R1#show ip mroute | be \(

(*, 239.0.0.1), 00:00:09/stopped, RP 2.2.2.2, flags: SJC
Incoming interface: Serial1/3, RPF nbr 150.100.21.2
Outgoing interface list:
FastEthernet0/0, Forward/Sparse, 00:00:09/00:02:50

(6.6.6.6, 239.0.0.1), 00:00:07/00:02:58, flags: JT
Incoming interface: Serial1/3, RPF nbr 150.100.21.2
Outgoing interface list:
FastEthernet0/0, Forward/Sparse, 00:00:07/00:02:52

(150.100.56.5, 239.0.0.1), 00:00:05/00:02:58, flags: JT
Incoming interface: Serial1/3, RPF nbr 150.100.21.2
Outgoing interface list:
FastEthernet0/0, Forward/Sparse, 00:00:05/00:02:54

(150.100.56.6, 239.0.0.1), 00:00:07/00:02:58, flags: JT
Incoming interface: Serial1/3, RPF nbr 150.100.21.2
Outgoing interface list:
FastEthernet0/0, Forward/Sparse, 00:00:07/00:02:52

(150.100.56.7, 239.0.0.1), 00:00:10/00:02:57, flags: JT
Incoming interface: Serial1/3, RPF nbr 150.100.21.2
Outgoing interface list:
FastEthernet0/0, Forward/Sparse, 00:00:10/00:02:49

Notice that it has sent joins only on Serial 1/3. Thus, R2 only sends multicast traffic for 239.0.0.1 out of this interface. R2 OIL looks like this:

R2#show ip mroute 239.0.0.1 | sec Outgoing

Outgoing interface list:
Serial1/3, Forward/Sparse, 00:34:58/00:02:44

Outgoing interface list:
Serial1/3, Forward/Sparse, 00:13:39/00:02:44

Outgoing interface list:
Serial1/3, Forward/Sparse, 00:13:39/00:02:46

Outgoing interface list:
Serial1/3, Forward/Sparse, 00:13:39/00:02:45

Let's enable multicast multipath on R1:

R1(config)#ip multicast multipath

Now we can see Joins have been sent out of both interfaces:

R1#show ip mroute | be \(
(*, 239.0.0.1), 00:00:01/stopped, RP 2.2.2.2, flags: SJC
Incoming interface: Serial1/3, RPF nbr 150.100.21.2
Outgoing interface list:
FastEthernet0/0, Forward/Sparse, 00:00:01/00:02:58

(6.6.6.6, 239.0.0.1), 00:00:01/00:02:58, flags: JT
Incoming interface: Serial1/3, RPF nbr 150.100.21.2
Outgoing interface list:
FastEthernet0/0, Forward/Sparse, 00:00:01/00:02:58

(150.100.56.5, 239.0.0.1), 00:00:01/00:02:58, flags: J
Incoming interface: Serial1/2, RPF nbr 150.100.12.2
Outgoing interface list:
FastEthernet0/0, Forward/Sparse, 00:00:01/00:02:58

(150.100.56.6, 239.0.0.1), 00:00:01/00:02:58, flags: JT
Incoming interface: Serial1/3, RPF nbr 150.100.21.2
Outgoing interface list:
FastEthernet0/0, Forward/Sparse, 00:00:01/00:02:58

(150.100.56.7, 239.0.0.1), 00:00:00/00:02:59, flags: J
Incoming interface: Serial1/2, RPF nbr 150.100.12.2
Outgoing interface list:
FastEthernet0/0, Forward/Sparse, 00:00:00/00:02:59

R2's OIL now looks like this:

R2#show ip mroute 239.0.0.1 | section Outg

Outgoing interface list:
Serial1/3, Forward/Sparse, 00:03:03/00:03:26

Outgoing interface list:
Serial1/3, Forward/Sparse, 00:01:31/00:03:26

Outgoing interface list:
Serial1/2, Forward/Sparse, 00:01:02/00:03:25

Outgoing interface list:
Serial1/3, Forward/Sparse, 00:01:31/00:03:26

At first I wasn't sure if hashing is done on source or source/group, but I found out by sending to different groups from the same address to see if it splits up. From what I can tell it uses the source to hash, so one source sending to multiple groups will not get split.

R1#show ip mroute | be \(

(150.100.100.5, 238.0.0.1), 00:00:04/00:02:55, flags: JT
Incoming interface: Serial1/2, RPF nbr 150.100.12.2
Outgoing interface list:
FastEthernet0/0, Forward/Sparse, 00:00:04/00:02:55

(150.100.100.5, 239.0.0.2), 00:00:49/00:02:17, flags: JT
Incoming interface: Serial1/2, RPF nbr 150.100.12.2
Outgoing interface list:
FastEthernet0/0, Forward/Sparse, 00:00:49/00:02:56

(150.100.100.5, 239.0.0.3), 00:00:45/00:02:17, flags: JT
Incoming interface: Serial1/2, RPF nbr 150.100.12.2
Outgoing interface list:
FastEthernet0/0, Forward/Sparse, 00:00:50/00:02:50

There is another train of IOS where you can select what to hash on, but my IOS doesn't have it.

Key thing to remember:

-Enabling multipath causes Joins to get sent towards the RP on more than one interface. This is what causes the load-splitting. Careful not to confuse this with the downstream sending of traffic, by default the router will send it out all interfaces (in the OIL) anyway!

IPv6 Tunneling - ISATAP

2008-12-10T15:27:00.001-08:00

R2, R5 and R6 connected via an IPv4 frame-relay network.
There is no PVC in use between R5 and R6.
Each device has a loopback 192.168.x.x where x is router number.
The goal here is to allow the remote IPv6 networks to communicate over the IPv4 cloud.

Below are the configs.

Loopback 100 = tunnel endpoint
Loopback 101 = "remote" network

R6:

interface Loopback100
ip address 192.168.6.6 255.255.255.255

interface Loopback101
no ip address
ipv6 address 2001:600::6/64

interface Tunnel1
ipv6 address 2001:200::/64 eui-64
tunnel source Loopback100
tunnel mode ipv6ip isatap

R5:

interface Loopback100
ip address 192.168.5.5 255.255.255.255

interface Loopback101
no ip address
ipv6 address 2001:500::5/64

interface Tunnel1
ipv6 address 2001:200::/64 eui-64
tunnel source Loopback100
tunnel mode ipv6ip isatap

Static routes on R5 and R6:

R5(config)#ipv6 route 2001:600::/64 tunnel 1 fe80::5efe:c0a8:0606

R6(config)#ipv6 route 2001:500::/64 tunnel 1 fe80::5efe:c0a8:0505

Where did I get these next hops? Well when you create an ISATAP tunnel they are created in a modified eui-64 format. Take a look

at R5:

R5#show ipv6 interface brief tun 1
Tunnel1 [up/up]
FE80::5EFE:C0A8:505
2001:200::5EFE:C0A8:505

When the router decides to route a packet out of that tunnel interface, it calculates the Ipv4 next hop address from the last 32 bits of the modified eui-64 address. In this case C0A8:505 converts to 192.168.5.5. R6 sends all packets destined for 2001:500::/64 to 192.168.5.5.

Key things to remember:

-The tunnel source address must be reachable by remote routers
-There is no manually specified tunnel destination
-You must specify the tunnel interface and link layer address in static routes

Multicast - IGMP Profile

2008-12-10T11:54:00.000-08:00

Here is the topology for this lab:

R2 is the RP and will be sending multicast pings.
R3 is the PIM DR for the 192.168.135.0 segment.
We will prevent R5 from joining group 239.0.0.1.

To deny IGMP joins on a switch, we use the IGMP filter and profile commands.

First, create the profile:

SW1(config)#ip igmp profile 1
SW1(config-igmp-profile)#deny
SW1(config-igmp-profile)#range 239.0.0.1 239.0.0.5
SW1(config-igmp-profile)#exit

Then attach it to the port:

SW1(config)#int f0/5
SW1(config-if)#ip igmp filter 1

Now we can test by having R1 and R5 join a group in the range 239.0.0.1 - 239.0.0.5

R1(config)#int e0/0
R1(config-if)#ip igmp join-group 239.0.0.1

R5(config)#int e0/0
R5(config-if)#ip igmp join-group 239.0.0.1

Let's debug on SW1 and see what happens:

SW1#debug ip igmp filter
event debugging is on

SW1#
03:26:30: IGMPFILTER: igmp_filter_process_pkt(): checking group 239.0.0.1 from Fa0/5: deny
03:26:31: IGMPFILTER: igmp_filter_process_pkt() checking group from Fa0/3 : no profile attached
03:26:33: IGMPFILTER: igmp_filter_process_pkt() checking group from Fa0/1 : no profile attached

No let's check R3 for any joined groups:

R3#show ip igmp groups
IGMP Connected Group Membership
Group Address Interface Uptime Expires Last Reporter
239.0.0.1 Ethernet0/0 00:09:28 00:02:30 192.168.135.1
224.0.1.40 Ethernet0/1 00:29:57 00:02:09 192.168.23.2
224.0.1.40 Ethernet0/0 00:30:01 00:02:37 192.168.135.3

Just to make sure, we can verify that only R1 responds to pings:

R2#ping 239.0.0.1

Type escape sequence to abort.
Sending 1, 100-byte ICMP Echos to 239.0.0.1, timeout is 2 seconds:

Reply to request 0 from 192.168.135.1, 8 ms
R2#

L2protocol Tunneling - An STP Example

2008-12-09T19:51:00.001-08:00

This is a short lab designed to help me get familiar with l2protocol tunneling, specifically tunneling STP. We are also going tunnel CDP and VTP. What's neat about this is that we will alter the STP topology without using priority or changing mac addresses. Also, SW1 will see two switches as CDP neighbors on one port.

Here is the topolgy:

Currently SW4 is root with SW2 is blocking f0/16. This works best with SW4 or SW3 as root.

SW2# show spanning-tree blockedports
Name Blocked Interfaces List
-------------------- ------------------------------------
VLAN0001 Fa0/16
Number of blocked ports (segments) in the system : 1

We can use l2protocol tunneling to create a logical loop between SW1, SW3 and SW4 and force the link between SW3 and SW4 to block. Logically that would look like this:

Physically we would have this, with SW2 not being a part of the VTP domain, any CDP relationship or STP topology:

This might be a practical case where SW2 was a service provider switch/cloud. SW1, SW3, and SW4 would then be remote switches with SW3 and SW4 having a backdoor connection.

Now for the configuration. SW1, SW3 and SW4 configure their links as trunks:

SW1(config)#int f0/13
SW1(config-if)#sw t e d
SW1(config-if)#sw mo t
SW1(config-if)#no shut

Repeat this on ports f0/16 and f0/19 of SW3 and SW4. SW2 has the following configuration:

SW2(config-if)#int rang f0/13, f0/16, f0/19
SW2(config-if-range)#swit mode dot1q-tunnel
SW2(config-if-range)#l2protocol-tunnel cdp
SW2(config-if-range)#l2protocol-tunnel stp
SW2(config-if-range)#l2protocol-tunnel vtp

Now let's verify some things. First, we can see SW3 and SW4 as CDP neighbors to SW1:

SW1#show cdp ne | be De
Device ID Local Intrfce Holdtme Capability Platform Port ID
SW4 Fas 0/13 156 S I WS-C3550-2Fas 0/16
SW3 Fas 0/13 158 S I WS-C3550-2Fas 0/16
R1 Fas 0/1 129 R S I 3640 Eth 0/0
SW1#

Notice they are both on interface f0/13. No SW2 in sight! Now let's see who's blocking between SW3 or SW4:

SW3# show spanning-tree blockedports | be VLAN
VLAN0001 Fa0/19
Number of blocked ports (segments) in the system : 1
SW3#

SW3 is blocking the connection between SW4. Perfect, just what we wanted.

This lab is designed as a little confidence booster. L2protocol tunneling is one of my weaknesses. I think because I recognize how complex it can get and it makes me worry (Ever since doing IPexpert V10 Volume 1 Lab 5). Practicing labs like this can help build confidence and gain familiarity with the configurations as well.

Multicast Heartbeat - Generating SNMP Traps

2008-12-09T09:15:00.000-08:00

This was a topic I ran into while browsing the multicast configuration guide today. I had dynamips up and running so I created a small lab.

Topology:

R1---R2---R5---R7

R1 is sending traffic to 225.0.0.7
R2 is the BSR/RP
R5 is will be configured for hearbeat
R7 has "ip igmp join-group 225.0.0.7" on one of its interfaces.

The commands to enable multicast heartbeat are very simple:

R5(config)#snmp-server host 9.9.9.9 traps public ipmulticast
R5(config)#snmp-server enable traps ipmulticast

R5(config)#ip multicast heartbeat 225.0.0.7 ?
<1-100> Minimal number of intervals where the heartbeats must be seen

R5(config)#ip multicast heartbeat 225.0.0.7 1 ?
<1-100> Number of intervals to monitor for heartbeat

R5(config)#ip multicast heartbeat 225.0.0.7 1 2 ?
<10-3600> Length of intervals in seconds

R5(config)#ip multicast heartbeat 225.0.0.7 1 2 10
R5(config)#

You will see this message:

R5#
*Mar 1 00:29:58.523: MHBEAT(0): Unless packets sent to 225.0.0.7 are seen in 1 out of 2 intervals of 10 seconds, an SNMP trap may be emitted.

Let's debug SNMP packets and the heartbeat so we can see the trap:

R5#debug snmp packets
SNMP packet debugging is on
R5#debug ip mhbeat
IP multicast heartbeat debugging is on

Now on R1 start sending packets, then stop:

R1#ping 225.0.0.7 re 2

Type escape sequence to abort.
Sending 10, 100-byte ICMP Echos to 225.0.0.7, timeout is 2 seconds:

Reply to request 0 from 150.100.56.7, 160 ms
Reply to request 1 from 150.100.56.7, 148 ms
R1#

Let's check R5. After a short while we see the following:

*Mar 1 00:38:48.555: MHBEAT(0): SNMP Trap for missing heartbeat
*Mar 1 00:38:48.575: SNMP: Queuing packet to 9.9.9.9
*Mar 1 00:38:48.575: SNMP: V1 Trap, ent ciscoExperiment.2.3.1, addr 150.100.56.5, gentrap 6, spectrap 1
ciscoIpMRouteHeartBeatEntry.2.225.0.0.7 = 0.0.0.0
ciscoIpMRouteHeartBeatEntry.3.225.0.0.7 = 10
ciscoIpMRouteHeartBeatEntry.4.225.0.0.7 = 2
ciscoIpMRouteHeartBeatEntry.5.225.0.0.7 = 0
*Mar 1 00:38:48.827: SNMP: Packet sent via UDP to 9.9.9.9

For reference, here is the link to the DocCD:

IP Multicast Heartbeat

PPP - Negotiated address via DHCP

2008-12-08T15:45:00.000-08:00

This kind of task may seem more difficult than it really is. I, in fact, spent way too long one morning/afternoon/evening trying to get this scenario to work. Turns out my server did not have a route back to the requester's subnet. So here it is without all the crap (ok, some of it) I went through:

Topology:

R5---R2---R1

R5 to R2 is PPP.
R5 needs to negotiate its address.
R1 is to supply this address.

R2-R5: 150.100.25.x/24
R1-R2: 150.100.12.x/24

R5 config is EASY:

interface Serial0/1
ip address negotiated

R2 is also easy, we configure it's interface to supply the address via DHCP and then specify a DHCP server:

R2(config)#int s1/1
R2(config-if)#peer default ip address dhcp
R2(config-if)#exit
R2(config)# ip dhcp-server 150.100.12.1

On R1 we configure the pool and everything is cool, right?

R1(config)#ip dhcp pool R5
R1(dhcp-config)#network 150.100.25.0 /24
R1(dhcp-config)#exit
R1(config)#ip dhcp excluded-address 150.100.25.1 150.100.25.4
R1(config)#ip dhcp excluded-address 150.100.25.6 150.100.25.255

Let's check R5, to see if it got an address:

R5#show ip int brief | inc l1/1
Serial1/1 unassigned YES IPCP up up

Nothing! Let's do some debugging on R1 with an ACL to match DHCP packets:

R1(config)#access-list 150 pe udp any any eq bootpc
R1(config)#access-list 150 pe udp any any eq bootps
R1(config)#access-list 150 pe udp any eq bootpc any
R1(config)#access-list 150 pe udp any eq bootps any

R1#debug ip packet 150 detail
IP packet debugging is on (detailed) for access list 150

*Mar 1 00:15:27.995: IP: s=150.100.12.1 (local), d=150.100.25.2, len 328, unroutable
*Mar 1 00:15:27.999: UDP src=67, dst=67

R1 has no route to 150.100.25.0/24 yet! Let's configure one and then manually shut/no shut the interface on R5:

R1(config)#ip route 150.100.25.0 255.255.255.0 150.100.12.2

R1#debug ip dhcp server events

*Mar 1 00:19:27.263: DHCPD: Sending notification of DISCOVER:
*Mar 1 00:19:27.263: DHCPD: htype 1 chaddr 0000.0c07.79e1
*Mar 1 00:19:27.267: DHCPD: circuit id 00000000
*Mar 1 00:19:27.267: DHCPD: Seeing if there is an internally specified pool class:
*Mar 1 00:19:27.271: DHCPD: htype 1 chaddr 0000.0c07.79e1
*Mar 1 00:19:27.271: DHCPD: circuit id 00000000
*Mar 1 00:19:28.411: DHCPD: Adding binding to radix tree (150.100.25.5)
*Mar 1 00:19:28.415: DHCPD: Adding binding to hash tree
*Mar 1 00:19:28.419: DHCPD: assigned IP address 150.100.25.5 to client 0063.6973.636f.2d31.3530.2e31.3030.2e32.352e.322d.5365.7269.616c.312f.31.
*Mar 1 00:19:28.495: DHCPD: Sending notification of ASSIGNMENT:
*Mar 1 00:19:28.499: DHCPD: address 150.100.25.5 mask 255.255.255.0
*Mar 1 00:19:28.499: DHCPD: htype 1 chaddr 0000.0c07.79e1
*Mar 1 00:19:28.503: DHCPD: lease time remaining (secs) = 86400
*Mar 1 00:20:17.647: DHCPD: checking for expired leases.
*Mar 1 00:22:17.647: DHCPD: checking for expired leases.
*Mar 1 00:24:17.647: DHCPD: checking for expired leases.

Now check R5:

R5#show ip int bri s1/1
Interface IP-Address OK? Method Status Protocol
Serial1/1 150.100.25.5 YES IPCP up up
R5#

*** IMPORTANT ***

R1 needs a route back to the 150.100.25.0/24 subnet. In this case I have a default route from R1 toward R2. This is EXTREMELY important. I wasted many minutes of my life trying to get this thing to come up. My DHCP configuration was correct but the DHCP server did not have a route back to the requester!

Two vlans, One Port, No trunk

2008-12-08T13:07:00.000-08:00

I recall a task somewhere I don't remember where we needed two vlans on one port but no trunk...in this case you can use a voice vlan for your second vlan. It is very easy to test:

Topology:

R1---SW1---SW2---R2

R1's interface:

interface Ethernet0/0

interface Ethernet0/0.2
encapsulation dot1Q 2
ip address 139.1.2.101 255.255.255.0

SW1:

interface FastEthernet0/1
switchport access vlan 11
switchport voice vlan 2

SW2:

SW2#show cdp ne | in R2
R2 Fas 0/2 135 R S I 3640 Eth 0/0
SW2#

Rack1SW2#show run int f0/2
Building configuration...

Current configuration : 83 bytes
!
interface FastEthernet0/2
switchport access vlan 2
switchport mode access

So let's ping from R1 to R2 (139.1.2.2)

Rack1R1#ping 139.1.2.2

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 139.1.2.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 4/4/4 ms
R1#

IPv6 - Stateless autoconfig

2008-12-08T08:26:00.000-08:00

Logical Topology:

R6------SW2

R6 is in vlan 6.
SW2 get its address for SVI 6 via stateless autoconfiguration.
R6 will be advertising the prefix for SW2 to use to build it's address.
R6 already has an IPv6 address configured: 2001:cc1e:1:6::6/64

Also, a good command to run here is "debug ipv6 nd".

Rack1R6#debug ipv6 nd
ICMP Neighbor Discovery events debugging is on

Rack1SW2#debug ipv6 nd
ICMP Neighbor Discovery events debugging is on

Before we do anything let's see what debugging gives us on R6:

Rack1R6#
*Mar 1 00:42:14.219: ICMPv6-ND: Sending RA to FF02::1 on Ethernet0/1
*Mar 1 00:42:14.219: ICMPv6-ND: MTU = 1500
*Mar 1 00:42:14.219: ICMPv6-ND: prefix = 2001:CC1E:1:6::/64 onlink autoconfig
*Mar 1 00:42:14.219: ICMPv6-ND: 2592000/604800 (valid/preferred)
Rack1R6#

We can see that R6 is already advertising it's prefix for hosts on this segment to use. Look at the output of the debug. We have

1) All nodes multicast address FF02::1, this is the destination of the RA advertisement
2) MTU of 1500
3) Prefix advertised by R6 2001:CC1E:1:6::/64
4) Valid and Preferred Lifetime 2592000/604800

All we need to do on SW2 is configure the SVI for autoconfiguration:

SW2#conf t
SW2(config)#int vlan 6
SW2(config-if)#ipv6 address ?
WORD General prefix name
X:X:X:X::X IPv6 link-local address
X:X:X:X::X/<0-128> IPv6 prefix
autoconfig Obtain address using autoconfiguration

SW2(config-if)#ipv6 address autoconfig

Notice that SW2 immediately sends an RS message asking for information about this segment:

00:19:39: ICMPv6-ND: Sending RS on Vlan6
00:19:39: ICMPv6-ND: Received RA from FE80::205:32FF:FE22:E442 on Vlan6
00:19:39: ICMPv6-ND: Sending NS for 2001:CC1E:1:6:21D:45FF:FEC0:F443 on Vlan6
00:19:39: ICMPv6-ND: Autoconfiguring 2001:CC1E:1:6:21D:45FF:FEC0:F443 on Vlan6
00:19:40: ICMPv6-ND: DAD: 2001:CC1E:1:6:21D:45FF:FEC0:F443 is unique.
00:19:40: ICMPv6-ND: Sending NA for 2001:CC1E:1:6:21D:45FF:FEC0:F443 on Vlan6
00:19:40: ICMPv6-ND: Address 2001:CC1E:1:6:21D:45FF:FEC0:F443/64 is up on Vlan6

It also receives the prefix, calcualtes its global unicast address and performs DAD. Now let's check the interface on SW2:

SW2#show ipv6 interface
Vlan6 is up, line protocol is up
IPv6 is enabled, link-local address is FE80::21D:45FF:FEC0:F443
Global unicast address(es):
2001:CC1E:1:6:21D:45FF:FEC0:F443, subnet is 2001:CC1E:1:6::/64 [PRE]
valid lifetime 2591864 preferred lifetime 604664
Joined group address(es):
FF02::1
FF02::2
FF02::1:FFC0:F443
MTU is 1500 bytes
ICMP error messages limited to one every 100 milliseconds
ICMP redirects are enabled
ND DAD is enabled, number of DAD attempts: 1
ND reachable time is 30000 milliseconds
ND advertised reachable time is 0 milliseconds
ND advertised retransmit interval is 0 milliseconds
ND router advertisements are sent every 200 seconds
ND router advertisements live for 1800 seconds
ND advertised default router preference is Medium
Hosts use stateless autoconfig for addresses.
Rack1SW2#

There are several adjustments we can make on the timers. Let's look at R6:

R6(config-if)#ipv6 nd ?
advertisement-interval Send an advertisement interval option in RA's
dad Duplicate Address Detection
managed-config-flag Hosts should use DHCP for address config
ns-interval Set advertised NS retransmission interval
other-config-flag Hosts should use DHCP for non-address config
prefix Configure IPv6 Routing Prefix Advertisement
ra-interval Set IPv6 Router Advertisement Interval
ra-lifetime Set IPv6 Router Advertisement Lifetime
reachable-time Set advertised reachability time
suppress-ra Suppress IPv6 Router Advertisements

Here we can set various parameters such as the advertisement interval (200 seconds default) and the RA lifetime.

More information on these options can be found in the addressing section non the IPv6 configuration guide on the DocCD:

Implementing IPv6 Addressing and Basic Connectivity

BGP - fast-external-fallover

2008-12-06T19:23:00.000-08:00

This feature allows the router to bring a BGP session down when the interface to that peer goes down. If you don't want this or are asked to not allow this to happen, you can disable it:

R1 has a neighbor:

R1#show ip bgp sum | be Ne
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
136.10.12.2 4 200 188 188 38 0 0 01:56:59 4

R1(config)#int f0/0
R1(config-if)#shut

*Dec 7 03:16:21.270: %BGP-5-ADJCHANGE: neighbor 136.10.12.2 Down Interface flap

We can prevent R1 from tearing the session down by disabling fast-external-fallover:

R1#show ip bgp sum | be Ne
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
136.10.12.2 4 200 196 194 50 0 0 00:00:03 4

R1(config)#router bgp 100
R1(config-router)#no bgp fast-external-fallover
R1(config-router)#int f0/0
R1(config-if)#shut
R1(config-if)#

*Dec 7 03:19:41.386: %LINK-5-CHANGED: Interface FastEthernet0/0, changed state to administratively down
*Dec 7 03:19:42.386: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/0, changed state to down
R1(config-if)#^Z

R1#show ip bgp sum | be Ne
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
136.10.12.2 4 200 196 194 54 0 0 00:00:50 4

Still up:

R1#show ip bgp sum | be Ne
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
136.10.12.2 4 200 196 195 54 0 0 00:01:29 4

Now the session will come down when the hold time expires. Some things to remember:

-Only works for directly-connected EBGP peers (hence the word "external" in the command)
-I tested with ebgp-multihop peers and it does not have any effect
-Keepalives are use to bring session down
-Also configurable per-interface with ip bgp fast-external-fallover

IPexpert Volume 2 Section 13 Review - PART I

2008-12-06T18:25:00.000-08:00

Well I am 2/3 of the way done here with a couple hours to go, but I am going to finish this next week. I have a terrible cold or something...I don't know, maybe it's all the Hot Pockets I ate this week. Whatever it is...I AM DEFEATED for the day.

I semi-graded this thing and I must say this is the TRICKIEST/HARDEST lab of them all. There are a total of 58 tasks, each worth 1 or 2 points and a few worth 3. This is the longest lab I have ever done to date. I am not entirely sure I would have finished in 8 hours...if I did, I wouldn't have been able to grade or verify much.

I think I missed about 7 or 8 tasks for about 15 or so points so far. Definitely a failed effort, but there were some good lessons learned. Here is a summary of what I had to configure:

-Fallback bridging. I actually got this right

-Only allow NetBIOS over TCP/IP in vlan 999. Used a VACL but I didn't what ports to match for netbios. I used range 135 - 139 but I don't know if this is right.

-Make sure CAT1 never becomes root for VLAN 999. The PG disabled STP for this VLAN, I used bpdufilter on the ports in VLAN 999. The PG was probably more correct.

-If R4 detects PVC states other than invalid, active or inactive - notify the trap receiver. What traps are these??

-Then there was a task that had me configure a secondary address 192.168.80.33/27 on an interface that already belonged to 192.168.80.0/24. Then you were supposed to filter out RIP routes on this subnet - HUH? I have no idea if this was a typo or what but the PG was really bad at explaining this one. I am not going into more detail - see it for yourself :-)

-OSPF task that had two different authentication keys on the same interface. This was a little tricky but I got it to work. I remember seeing this on GS so that helped a lot. You had to use neighbor statements on the spokes instead of the hub.

Anyways. this lab is truly a mind-number. Just the kind of trickery to expect on the lab, I assume. If you think you are hot stuff - try this one ;-)

Mobile ARP

2008-11-29T13:32:00.000-08:00

Well seeing as how I just missed a 4-point task on mobile ARP, I thought now was a good time to learn it. It's actually very simple and pretty cool once you get it working.

Topology is a little confusing so here it is in 2 parts:

PHYSICAL:

R1---CAT1===dot1q===CAT2---R8

LOGICAL:

VLAN 100---R1---R2---R5---R7---R8---VLAN 200

The task says that users on R8's LAN occasionally mover over to R1's LAN. They still need access to the network. What we do is configure R1 to listen for ARP packets from R8's subnet (VLAN 200).

We can test this by creating an SVI for VLAN 100 and giving it an IP in R8's subnet. When it tries to contact anyone, we will see a mobile route appear in R1's route table. This route then gets redistributed into the routing protocol (OSPF in this case). It appears as a /32 route so the longer match wins over any other route advertisement of VLAN 200.

All configuration is on R1:

1) CREATE THE ACL

R1(config)#access-list 8 permit 172.31.80.0 0.0.0.255

2) CONFIGURE MOBILE ARP ON INTERFACE

R1(config)#interface f0/0
R1(config-if)# ip mobile arp access-group 8

3) REDISTRIBUTE

R1(config)#router ospf 1
R1(config-router)# redistribute mobile subnets

4) VERIFY

Ceate an SVI on VLAN 100 with VLAN 200 IP address:

CAT1(config)#int vlan 100
CAT1(config-if)#ip address 172.31.80.100 255.255.255.0
CAT1(config-if)#^Z
CAT1#ping 172.31.80.8

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.31.80.8, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 20/219/1016 ms

Run debug and show commands on R1:

R1#debug ip mobile
IP mobility events debugging is on
Nov 29 21:17:14.138: Local MobileIP: route add 172.31.80.100

R1#show ip route mobile
172.31.0.0/16 is variably subnetted, 11 subnets, 6 masks
M 172.31.80.100/32 [3/1] via 172.31.80.100, 00:11:05, FastEthernet0/1
R1#

Traceroute shows how many hops we are actually going through:

CAT1#trace 172.31.80.8

Type escape sequence to abort.
Tracing the route to 172.31.80.8

1 172.31.10.1 4 msec 4 msec 0 msec
2 172.31.12.2 0 msec 4 msec 0 msec
3 172.31.100.5 12 msec 8 msec 12 msec
4 172.31.200.7 12 msec 8 msec 12 msec
5 172.31.78.8 12 msec * 8 msec
CAT1#

IPexpert Volume 2 Section 12 Review

2008-11-29T11:32:00.000-08:00

I woke up late for this one but I still finished in plenty of time. Probably about 4 hours. I made some serious mistakes though that I completely overlooked. It was in the BGP section, I didn't configure a confederation...so that may have ruined 2 or 3 tasks - not real sure how to gauge the impact. You should have seen the look on my face when I saw the PG.

With that included I missed 6 tasks for about 15 points:

-8 Tasks 8.1 - 8.3
I knew we were using private AS numbers so I immediately thought configuring a confederation. However, I did not deduce that from the task requirements so I didn't bother. Reviewing it, I completely overlooked R1's task of peering with R2 is AS 200. AS 200 should have been the confederation....BIG BOOBOO. Completely unacceptable.

-4 Task 9.3 IOS Services
Completely missed this "Mobile ARP" section. I had a NAT solution that does what I thought the task asks. I have no idea how to configure mobile arp and I guess it's time to learn. I wonder if anyone even uses it...

-2 Task 11.2 DHCP
I used "no ip bootp server" for the DHCP router not respond to bootp requests. However, the answer was "ip dhcp bootp ignore"

-1 Task 14.3 Multicast
Configured MRM incorrectly. I used the DocCD for this and was what you could call "way off." It was a 1 point task and I was not too concerned.

My goal from here on out is to keep my score above 80 while improving my "process." That includes verifying everything, making notes and a point tracker, refraining from marking the actual lab docs (which I heard you cannot do), and moving through the DocCD.

Missing the BGP confederation is something that should never happen. I am lucky there were not more tasks dependent on it. Everything else was filtering of some sort. Who knows, I may have been marked of on the entire BGP section (20 points).

One thing I worry about is that I have not really been challenged during Layer 2 configurations. I pretty much breeze through VTP, trunking, and other topics, but I know there are topics that will get me (QoS, tunneling). For these I rely on the DocCD and make my own labs. That being said, Volume 1 Section 5 has an extremely difficult tunneling lab that I need to review.

BGP - Conditional route injection

2008-11-28T20:36:00.000-08:00

Topology

R5----R7

R5 is advertising 10.34.19.0/26 to R7
Configure R7 to inject 10.34.19.48/28

1) MAKE PREFIX-LISTS

ip prefix-list EXIST seq 5 permit 10.34.19.0/26
ip prefix-list INJECT 5 permit 10.34.19.48/28
ip prefix-list SOURCE seq 5 permit 192.168.5.5/32

2) MAKE ROUTE-MAPS

route-map INJECT permit 10
set ip address prefix-list INJECT

route-map EXIST permit 10
match ip address prefix-list EXIST
match ip route-source prefix-list SOURCE

3) CONFIGURE BGP

route bgp 567
bgp inject-map INJECT exist-map EXIST

4) VERIFY

R5#show ip bgp nei 192.168.7.7 advertised-routes | begin Net
Network Next Hop Metric LocPrf Weight Path
*> 10.34.19.0/26 192.168.2.2 0 200 0 24 1 i

R7#show ip bgp injected-paths | begin Net
Network Next Hop Metric LocPrf Weight Path
*>i10.34.19.48/28 192.168.5.5 0 200 0 24 1 i

Things to remember:

- Must use Prefix-lists, NOT ACLs
- Injected route must a subset of am aggregate already in the table
- Use "set" command for inject-map, not "match"
- I commonly forget the "prefix-list" argument when configuring the maps
- inject-map Command is a bgp command, not per-neighbor

IPexpert Volume 2 Section 11 Review

2008-11-28T17:05:00.000-08:00

I just completed this lab in about 4 or 5 hours. I spent the first hour (before my session even started) reading the lab, redrawing the L3 topology and making a task checklist. This actually took me about a half hour. I got an estimated score of 89, missing 4 tasks for 11 points. Two were easy, but the other two...well, just proof that I need to review the DocCD :-)

Here are the misses:

-3 Task 5.4 EIGRP
Routes should be dropped from inactive neighbors in half the default time. I used hold time command, but the PG had "timers nsf route-hold 120" as the answer. I need to review NSF.

-3 Task 6.1 RIP
I forgot to enable v2-broadcast on one interface. BONEHEAD!

-1 Task 8.7 BGP
Completely misunderstood the aggregation task. BONEHEAD #2!

-4 Task 10.2 DNS
We needed to create a domain list with "ip domain list ipexpert.net". I just used the domain-name command. I am not familiar at all with how DNS resolution works on Cisco routers so I need to review this.

Over the last few months I have increased my speed and efficiency dramatically. Time does not seem to be an issue anymore. When I started studying in the spring, I was taking so long on full scale mock labs, I stopped doing them. Many commands I know by heart, but occasionally I misunderstand a question or just have the wrong command like the EIGRP section above.

Now that I have more time, I use it to prep before I start. This includes reviewing and drawing the topology. I want the processes I use on the practice lab to be just like the ones on the real lab. That way nothing is new and I can get in my comfort zone. I definitely "feel" ready for the real thing, but that doesn't mean I am.

I can still think of some topics that would give me a hard time, unfortunately I haven't seen to many of these lately...but I know they are there...waiting to get me ;-)

What I'm thankful for - CCIE study edition

2008-11-27T15:51:00.000-08:00

In the spirit of this holiday I thought I'd make a list of things that make labbing, studying and the overall process of preparing for the CCIE Lab exam a lot easier. In no particular order:

- Vendor Wars. Great deals are among us!

- Groupstudy. It's taken some criticism lately, but it still remains the best place to brainstorm with fellow CCIE candidates and those that have already passed. And don't forget the archives!

-Ebay. Couldn't have built my lab without it!

- Debug ip packet + ACL. Excellent when you need to debug ip packet but you've got sub-second ospf hello timers. Believe me, I found out the hard way!

- Show access-lists. Easy way to see if you get a hit on an ACL

- Static routes. Just for workarounds while you get reachability in place. I had a ppp task where the router needed an address via dhcp (proxy) but the server couldn't respond unicast to the destination subnet (no route) and no routing protocols were in place yet. Remember to remove them after!

- Colored pencils for each routing protocol. I prefer light green for OSPF, dark green for EIGRP, orange for BGP, black for RIP. I only have 4 colored pencils in my apartment and I put them to good use.

- Regular expressions for parsing show commands with the include, exclude and section arguments.

- Route tagging. Really makes route redistribution a whole lot easier to deal with.

- Master Command index on the DocCD. Nothing short of a life saver sometimes :)

- Trees. A whole lot of scratch paper going on around here...

- And last but not least...Blogrolls!

Well that's my list for now. What are you thankful for?

OSPF - Lowest IP address in OSPF is used as forwarding address on ASBR

2008-11-25T16:51:00.000-08:00

I was labbing some NSSA today and I was wondering how the OSPF ASBR chose the forward address since it seem to appear on the opposite side of traffic flow. Example:

R3----SW1----R6

SW1 is the ASBR (redistributing it's loopback). Traffic was flowing through R6, but the address pointing to R3 was the "forward address". It doesn't really matter as long as the link is in OSPF but it can impact metric calculations since SW1's interface cost to R3 will be included.

I did a short search on GS and RFC2328 but could not find anything. I had a guess it was the lowest IP in OSPF on the router and it turns out that is right:

SW1#show run | sec router ospf
router ospf 1
router-id 11.11.11.11
log-adjacency-changes
area 2 nssa
redistribute connected metric-type 1 subnets route-map con2ospf
network 2.0.0.1 0.0.0.0 area 2
network 192.168.37.7 0.0.0.0 area 2
network 192.168.67.7 0.0.0.0 area 2

SW1#show ip int bri
Interface IP-Address OK? Method Status Protocol
FastEthernet0/0 192.168.37.7 YES manual up up
FastEthernet2/0 192.168.67.7 YES manual up up
Loopback2 2.0.0.1 YES manual up up
Loopback100 100.100.100.100 YES manual up up

SW1#show ip ospf database external

OSPF Router with ID (11.11.11.11) (Process ID 1)
SW1#show ip ospf database ns
SW1#show ip ospf database nssa-external

OSPF Router with ID (11.11.11.11) (Process ID 1)

Type-7 AS External Link States (Area 2)

LS age: 243
Options: (No TOS-capability, Type 7/5 translation, DC)
LS Type: AS External Link
Link State ID: 100.100.100.100 (External Network Number )
Advertising Router: 11.11.11.11
LS Seq Number: 80000009
Checksum: 0x8EBF
Length: 36
Network Mask: /32
Metric Type: 1 (Comparable directly to link state metric)
TOS: 0
Metric: 20
Forward Address: 2.0.0.1
External Route Tag: 0

SW1#