Sunday, October 26, 2008

Debugging an E-BGP multihop scenario

I have the following frame-relay hub and spoke topology:

R5----R2----R6

R2 is the hub, all routers are in the 150.100.100.0/24 subnet.

R2 = 150.100.100.2
R5 = 150.100.100.5
R6 = 150.100.100.6

Please note that R2 has one multipoint subinterface connected to R5 and R6. Blogspot doesn't like text drawings so I must draw it like above.

All routers are in sub-AS bgp confederations. R2 can only peer with one, and R5 and R6 must peer with each other.

The peers will not come up without ebgp-multihop configured, but suppose we forgot that. What kind of debugging could we do to lead us to that conclusion?

1) debug ip bgp

R6#
*May 25 04:46:18.587: BGP: 150.100.100.5 open active, local address 150.100.100.6
*May 25 04:46:48.587: BGP: 150.100.100.5 open failed: Connection timed out; remote host not responding, open active delayed 29387ms (35000ms max, 28% jitter)

This debug command shows us that BGP never completes the Active state. RFC 1771 tells us this about the Active state:

"In this state BGP is trying to acquire a peer by initiating a transport protocol connection."

So our TCP connection is not completing. Do you we have IP connectivity to R5? Sure:

R6#ping 150.100.100.5

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 150.100.100.5, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 100/102/108 ms
R6#


So now we can look higher up the protocol stack (e.g. filtering), or maybe the problem is still in the IP layer. In this case we have no ACL's applied.

2) debug ip packet detail

What I am looking for here is some packets from R5, sourced from 150.100.100.5. Debugging shows that I am getting none! However it also shows that I am getting ICMP type 11 messages from R2 immediately after I send a packet to R5:

R6#
*May 25 04:51:44.539: IP: tableid=0, s=150.100.100.6 (local), d=150.100.100.5 (Serial0/1/0), routed via FIB
*May 25 04:51:44.539: IP: s=150.100.100.6 (local), d=150.100.100.5 (Serial0/1/0), len 44, sending
*May 25 04:51:44.539: TCP src=24713, dst=179, seq=1584149779, ack=0, win=16384 SYN
*May 25 04:51:44.563: IP: tableid=0, s=150.100.100.2 (Serial0/1/0), d=150.100.100.6 (Serial0/1/0), routed via RIB
*May 25 04:51:44.563: IP: s=150.100.100.2 (Serial0/1/0), d=150.100.100.6 (Serial0/1/0), len 56, rcvd 3
*May 25 04:51:44.563: ICMP type=11, code=0


Seems that R2 is telling us something about our packet sent to R5.

3) debug ip icmp

*May 25 04:53:23.839: ICMP: time exceeded rcvd from 150.100.100.2

Here we get our answer. At this point we realize that our tcp syn packets sent to R5 have an IP TTL of 1, and thus are getting dropped by R2.

Do you know any other commands that would help you come to this conclusion?

IPexpert Volume 2 Section 6 Review

This lab took me about 4.5 hours. I woke up late so I had only 5 hours of session time left, but I finished it. There were some bonehead mistakes because I was rushing it, notably in Multicast and BGP. I ended up with a maximum score of 83, maybe even less.

Here were the mistakes:

-3 task 3.5. Forgot max-paths command on R4 and R9. I used variance 2, the PG has variance 128. I misunderstood the question, using 2 for variance, instead of max-paths.

-3 task 4.2. Didn't get BGP reachability. CAT4 does not do BGP, but is in the middle of the BGP ASes and we are not supposed to redistribute into IGP. So I skipped this. PG had CAT4 being added to BGP. I didn't even think of this. I started thinking about tunnels, but just skipped this task instead.

-5 task 5.2 and 5.3. Completely froze on this multicast task, I didn't get reachability and skipped it. Turns out I didn't have auto-rp listener on. Maybe if I spent more time troubleshooting I could have got it.

-3 task 7.3. I had IPv6 reachability but I used default routes generated in RIPng and OSPFv3. The PG uses summary routes. Not sure if I would have missed this or not, but the beginning of the lab says do not use default routes.

-3 task 9.4. I didn't configure AAA for case sensitive authentication. I guess I didn't take the task literal enough.

I am definitely getting better at topics like QoS, IOS services and Security but as you can see I am still making some easy mistakes. I hope these bad habits don't make their way into the exam. One thing I do not like about online rack time is that you have to go on their schedule. I can't do it on the weekday because I work. The weekends are scrunched into 10pm-6am, 6am-2pm, and 2pm-10pm. This was a 6-2 session and I just could not get up that early today. Anyways, it's going to take some discipline to get my study habits in gear.

Sunday, October 19, 2008

Extended ping with TOS byte

Here is the scenario:

R2---R6---R7

R6 is only supposed to allow traffic with an IP precedence level of critical to R7. Easy right? Well with extended ping you can verify that it is working.

On R6 we the following ACL applied to the interface towards R7:

access-list 101 permit ospf any any
access-list 101 permit ip any any precedence critical log
access-list 101 deny ip any any log


The only traffic being allowed is ospf to maintain the adjacency with R7 and "critical" IP traffic.

For our extended ping we need to find the hex value of the critical precedence. Critical is precedence 5 which in the ToS byte would break out to 101 000 00. We can align it like this:

1010 0000

This is equal to hex value 0xA0 (the first four bits are 10 which is A, the last 4 are 0). So here is our first ping which fails:

R2#ping 150.100.220.7

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 150.100.220.7, timeout is 2 seconds:
U.U.
Success rate is 0 percent (0/4)


Now we send the extended ping with ToS value 0xA0

R2#ping
Protocol [ip]:
Target IP address: 150.100.220.7
Repeat count [5]:
Datagram size [100]:
Timeout in seconds [2]:
Extended commands [n]: y
Source address or interface:
Type of service [0]: 0xA0
Set DF bit in IP header? [no]:
Validate reply data? [no]:
Data pattern [0xABCD]:
Loose, Strict, Record, Timestamp, Verbose[none]:
Sweep range of sizes [n]:
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 150.100.220.7, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 16/18/20 ms
R2#

On R6 we can verify the matches:

R6#show access-lists 101
Extended IP access list 101
10 permit ospf any any (1 match)
20 permit ip any any precedence critical log (5 matches)
30 deny ip any any log (44 matches)

IPexpert Volume 2 Section 5 Review

I finished this lab in under 5 hours. I quickly ran through the PG and got an estimated score of 87 give or take a few tasks. There were only couple tough ones, but other than that it was smooth sailing. There was interesting igmp snooping task which I got right thanks to the Doc CD.

Here is a summary of the mistakes:

-4 task 1.2. Didn't configure vlan 100 as native vlan. Didn.t configure flex-link. Completely forgot about the native vlan, while the flex-link configuration I didn't pick up on. The task said to make sure "if one link fails, the other will start working, but not until then." I figured this was juts normal STP, but it wasn't.

-3 task 2.2. R9-R6 addresses were off. R6 and R9 have 2 back-to-back links. Question said to use the address space of a 172.16.69.0/30. I assigned the same addresses to both links. R6 used .1 on both interfaces, R9 used .2 on both interfaces. This worked perfectly because WAN interfaces can overlap. However the PG wanted you to split up the /30 into two sets of /31 subnets. The question was very unclear, I would have asked proctor.

-3 task 5.2. Didn't use static rp on R7. R8 was supposed to use R7 as the RP (overriding auto-rp). I used a static rp-address on R8 but didn't do the same on R7. I was really unsure about what to do on R7, I thought R7 was supposed to use R5 as the RP, so I didn't think we should have configured R7 that way. But that's the only to make R7 the RP for R8.

-3 task 8.1. I used the wrong burst sizes, the task said 16K which I thought was bits, the command wants bytes so I put 2000. Should have been 16000. I hate these types of questions...

These are some alternate solutions that I used, that differed from the PG:

3.3 I used eigrp stub with default settings, PG used "eigrp stub connected"
4.2 I used a supress-map instead of community attribute to prevent specific routes from being advertised.
6.2 I changed sdm to routing extended-match, PG just had extended match. This was for wccp
to be enabled on a switch.
8.4 I used a class-map that macthes icmp and packet length, then used a policy-map to drop it. The PG use PBR which is what my first thought was.

Saturday, October 18, 2008

Config mode lock - auto and manual

I stumbled upon this command while browsing the doc cd. It allows you to lock global configuration mode so only 1 user can access it at a time. Here is how it works.

R6-----R7

On R6 we have the following:

R6(config)#configuration mode exclusive auto
R6(config)#username cisco pass cisco
R6(config)#line vty 0 4
R6(config-line)#login local


If you exit out of global config mode now and go back, you will see this message:

R6#conf t
Enter configuration commands, one per line. End with CNTL/Z.
*Oct 19 03:42:03.710: Configuration mode locked exclusively. The lock will be cleared once you exit out of configuration mode using end/exit


On R7 we telnet to R6:

R7#telnet 200.0.0.6
Trying 200.0.0.6 ... Open
CCC Blah, Blah, Blah... If you are not an authorized user, go away!

User Access Verification

Username: cisco
Password:
R6>en


Next we try to enter global config mode but we are denied:

R6#conf t
Configuration mode locked exclusively by user 'unknown' process '3' from terminal '0'. Please try later.
R6#

Back on R6 console we can verify the lock is in place:

R6(config)#do show configuration lock
Parser Configure Lock
---------------------
Owner PID : 3
User : unknown
TTY : 0
Type : EXCLUSIVE
State : LOCKED
Class : EXPOSED
Count : 1
Pending Requests : 0
User debug info : configure terminal
R6(config)#


You can also configure a manual lock with the following command:

R6(config)#configuration mode exclusive manual


With this command, another user can still enter config mode:

R7#telnet 200.0.0.6
Trying 200.0.0.6 ... Open
CCC Blah, Blah, Blah... If you are not an authorized user, go away!

User Access Verification

Username: cisco
Password:
R6>en
Password:
R6#conf t
Enter configuration commands, one per line. End with CNTL/Z.
R6(config)#


To lock the mode we have to manually lock it upon entering global config mode:

R6#conf term ?
lock Lock configuration mode


R6#conf term lock
Enter configuration commands, one per line. End with CNTL/Z.
R6(config)#
*Oct 19 03:55:16.806: Configuration mode locked exclusively. The lock will be cleared once you exit out of configuration mode using end/exit
R6(config)#


Now R7 console we are still in global config mode, we didn't get kicked out. But if we leave, we can't go back:

R6(config)#
R6(config)#exit
R6#conf t
Configuration mode locked exclusively by user 'unknown' process '3' from terminal '0'. Please try later.
R6#

IPexpert Volume 2 Section 4 Review

I just completed this lab and I got a self-graded estimated 81 out of 100. The points I missed were not very tough. I will detail them below:

-3 task 2.3. I didn't even attempt ppp over frame-relay with multilink. To be honest, WAN tasks like this always seem so trivial. I just got a serial link up between R2 and R5 and ran with it. I was gonna come back and fix it, but by the time I was done, I didn't feel like it.

-3 task 3.4. R1 was only supposed to send 3 routes to CAT1. Well, this is how it was until I started redistributing on R8 and adding a new loopback to R8. These routes were supposedly not be included in the advertised routes to CAT1. It's time to ask the proctor when the task states "advertise 3 routes" but doesn't say which 3 routes.

-4 task 4.1. My solution worked but didn't match the PG. The PG used the local-as option and remove private-as to make all AS appear as AS8888 to CAT1. I used confederations along with local-as. Maybe I would have gotten the points anyway, but my solution was pretty convoluted here.

-3 task 8.1. I got the policy map right on, but it was applied to the multilink interface that I was supposed to create in task 2.3. So lesson learned is to not screw up a task, on which another task may depend.

-3 task 8.4. Incorrect rate-limit access-list. My first experience using a rate-limit ACL with the mask option. After reviewing the DocCD and the PG, it makes sense now. My solution was way off. I needed to match precedence 1, 3, 5 and 7. The mask should have been the hex value for binary 10101010 which is AA. I thought the binary mask values were supposed to be masking the actual precedence bit values (which they are not).

-3 task 9.1. Forget to apply ACL to second interface. I only applied the ACL to f0/0, it should have also gone on multlink 1 of R9. Careless mistake.

Overall, I think I did good considering I hadn't labbed at all in the last 2 weeks plus. When you have time like I do (5+ months remaining until lab exam) it's good to take a week or two off to see how well you retain things. Somethings I was a little rusty on were AAA security and some OSPF/NBMA stuff which surprised me.

BGP - peer session templates

I had a task to configure bgp timers on R2 for the peering session to R1. I was not to use "timers bgp" or any neighbor commands withe the word "timers". I immediately thought of peer-session templates which was something I came across while reading the DocCD one day. Here is how it works. We are on R2:

R2(config)#router bgp 2456

R2(config-router)#template ?
peer-policy Template configuration for policy parameters
peer-session Template configuration for session parameters

R2(config-router)#template peer-session TEMPLATE-R1

R2(config-router-stmp)#timers 30 90
R2(config-router-stmp)#exit

R2(config-router)#neighbor 150.21.21.1 inherit ?
peer-policy Inherit a peer-policy template
peer-session Inherit a peer-session template

R2(config-router)#neighbor 150.21.21.1 inherit peer-session TEMPLATE-R1


Now we must clear bgp then debug bgp keepalives to verify. Here we can see that R2 is sending keepalives to R1 every 30 seconds, while the other peers are still at 60 seconds.

*Oct 18 18:28:15.486: BGP: 150.21.21.1 sending KEEPALIVE (io)
*Oct 18 18:28:15.486: BGP: 150.21.21.1 received KEEPALIVE, length (excl. header) 0
*Oct 18 18:28:46.166: BGP: 150.21.21.1 sending KEEPALIVE (io)
*Oct 18 18:28:46.166: BGP: 150.21.21.1 received KEEPALIVE, length (excl. header) 0
*Oct 18 18:29:01.166: BGP: 150.20.25.5 sending KEEPALIVE (io)
*Oct 18 18:29:01.490: BGP: 150.20.25.5 received KEEPALIVE, length (excl. header) 0
*Oct 18 18:29:09.946: BGP: 150.20.246.4 sending KEEPALIVE (io)
*Oct 18 18:29:09.950: BGP: 150.20.246.4 received KEEPALIVE, length (excl. header) 0
*Oct 18 18:29:15.654: BGP: 150.20.246.6 sending KEEPALIVE (io)
*Oct 18 18:29:15.654: BGP: 150.20.246.6 received KEEPALIVE, length (excl. header) 0
*Oct 18 18:29:15.654: BGP: 150.21.21.1 sending KEEPALIVE (io)
*Oct 18 18:29:15.654: BGP: 150.21.21.1 received KEEPALIVE, length (excl. header) 0
*Oct 18 18:29:46.166: BGP: 150.21.21.1 sending KEEPALIVE (io)
*Oct 18 18:29:46.166: BGP: 150.21.21.1 received KEEPALIVE, length (excl. header) 0
*Oct 18 18:30:01.166: BGP: 150.20.25.5 sending KEEPALIVE (io)
*Oct 18 18:30:01.490: BGP: 150.20.25.5 received KEEPALIVE, length (excl. header) 0
*Oct 18 18:30:09.946: BGP: 150.20.246.4 sending KEEPALIVE (io)
*Oct 18 18:30:09.946: BGP: 150.20.246.4 received KEEPALIVE, length (excl. header) 0
*Oct 18 18:30:15.654: BGP: 150.20.246.6 sending KEEPALIVE (io)
*Oct 18 18:30:15.654: BGP: 150.20.246.6 received KEEPALIVE, length (excl. header) 0
*Oct 18 18:30:15.654: BGP: 150.21.21.1 sending KEEPALIVE (io)
*Oct 18 18:30:15.658: BGP: 150.21.21.1 received KEEPALIVE, length (excl. header) 0
*Oct 18 18:30:46.166: BGP: 150.21.21.1 sending KEEPALIVE (io)
*Oct 18 18:30:46.166: BGP: 150.21.21.1 received KEEPALIVE, length (excl. header) 0

BGP - Allowas-in with number of occurrences

I ran into this BGP issue on IPexpert volume 2 Lab 4 today. Cat 1 is in AS 500. There are 3 other ASes, but CAT1 needs to see all of these as AS8888. Confederations right? For 2 of the ASes that is right, but the task says NOT to use a sub-as for AS78. Here is the AS map:

As500----As100----AS2456====AS78

AS2456 has 2 connections (R5 and R6) to AS78 (R7).

On R5 and R6 I have this ( I am allowed to use confederations on AS 2456):

router bgp 2456
bgp confederation identifier 8888
neighbor 150.20.56.7 remote-as 8888


on R7 I have this:

router bgp 78
neighbor 150.20.56.5 remote-as 8888
neighbor 150.20.56.5 local-as 8888
neighbor 150.20.56.5 allowas-in
neighbor 150.20.56.6 remote-as 8888
neighbor 150.20.56.6 local-as 8888
neighbor 150.20.56.6 allowas-in


Works great so far:

R7#show ip bgp sum | be Neigh
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
150.10.8.8 4 78 55 64 64 0 0 00:35:46 1
150.20.56.5 4 8888 112 121 64 0 0 00:01:07 5
150.20.56.6 4 8888 109 116 64 0 0 00:01:10 5


But notice the prefix I am learning from R8 (ibgp peer with R7). That prefix does not get installed by R5 and R6 because it has AS 8888 in the path. Here is the debug from R6:

R6#debug ip bgp update
*Oct 18 17:46:11.270: BGP(0): 150.20.56.7 rcv UPDATE about 200.200.200.0/24 -- DENIED
R6(config-rout due to: AS-PATH contains our own AS;


How do we allow R6 and R5 to accept this route? By using "allowas-in" option of the neighbor command. However, because we don't want R6 to learn routes that passed from R5 to R7 we set a maximum on the number of occurrences of the AS, which is 1.

R5 and R6:

router bgp 2456
neighbor 150.20.56.7 allowas-in 1


Now here is the same debug on R6:

*Oct 18 17:56:48.366: BGP(0): Revise route installing 1 of 1 routes for 200.200.200.0/24 -> 150.20.56.7(main) to main IP table

Let's take a look at CAT1 in AS 500

Cat1#show ip bgp | begin Net

Network Next Hop Metric LocPrf Weight Path
*> 200.200.200.0 150.20.110.1 0 8888 8888 78 ?


This is not good! We can get rid of AS 78 by doing this on R7:

R7(config)#router bgp 78
R7(config-router)#neighbor 150.20.56.5 local-as 8888 no-prepend replace-as
R7(config-router)#neighbor 150.20.56.6 local-as 8888 no-prepend replace-as


Now let's take a look:

Cat1#show ip bgp | beg Net

Network Next Hop Metric LocPrf Weight Path
*> 200.200.200.0 150.20.110.1 0 8888 8888 ?

Wednesday, October 8, 2008

Status update

I just got back from California last night. I was there for about 5 days, the highlight of course was the Dodgers game 3 win over the Cubs, clinching the series win. It was the most exciting sporting event I have ever been to and was well worth it whatever I spent (I stopped counting).

In LA, people are notorious for arriving in the 3rd inning and leaving in the 7th. They were still arriving late (traffic was horrible as usual), but 20 minutes after the game, people were still in the stands as the players celebrated with everyone. It was an awesome experience I never will forget.

That being said, it's time to get back to the lab. For now, here is where I stand:

I have completed Volume 1 of IPexper'ts blended learning solution. I have gone back and labbed the multicast and qos sections again, and plan on doing them a third time. I will also review the security and ACL sections and probably a few others.

I have now started Volume 2 and will start Section 4, once I get rack time scheduled. I also plan on doing some graded mock labs (volume 3) in between the volume 2 labs. I still have more than 5 months away (which I may move up depending on how I feel) from taking the lab but confidence is pretty high.

back to the lab...