Debugging an E-BGP multihop scenario

I have the following frame-relay hub and spoke topology:

R5----R2----R6

R2 is the hub, all routers are in the 150.100.100.0/24 subnet.

R2 = 150.100.100.2
R5 = 150.100.100.5
R6 = 150.100.100.6

Please note that R2 has one multipoint subinterface connected to R5 and R6. Blogspot doesn't like text drawings so I must draw it like above.

All routers are in sub-AS bgp confederations. R2 can only peer with one, and R5 and R6 must peer with each other.

The peers will not come up without ebgp-multihop configured, but suppose we forgot that. What kind of debugging could we do to lead us to that conclusion?

1) debug ip bgp

R6#
*May 25 04:46:18.587: BGP: 150.100.100.5 open active, local address 150.100.100.6
*May 25 04:46:48.587: BGP: 150.100.100.5 open failed: Connection timed out; remote host not responding, open active delayed 29387ms (35000ms max, 28% jitter)

This debug command shows us that BGP never completes the Active state. RFC 1771 tells us this about the Active state:

"In this state BGP is trying to acquire a peer by initiating a transport protocol connection."

So our TCP connection is not completing. Do you we have IP connectivity to R5? Sure:

R6#ping 150.100.100.5

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 150.100.100.5, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 100/102/108 ms
R6#

So now we can look higher up the protocol stack (e.g. filtering), or maybe the problem is still in the IP layer. In this case we have no ACL's applied.

2) debug ip packet detail

What I am looking for here is some packets from R5, sourced from 150.100.100.5. Debugging shows that I am getting none! However it also shows that I am getting ICMP type 11 messages from R2 immediately after I send a packet to R5:

R6#
*May 25 04:51:44.539: IP: tableid=0, s=150.100.100.6 (local), d=150.100.100.5 (Serial0/1/0), routed via FIB
*May 25 04:51:44.539: IP: s=150.100.100.6 (local), d=150.100.100.5 (Serial0/1/0), len 44, sending
*May 25 04:51:44.539: TCP src=24713, dst=179, seq=1584149779, ack=0, win=16384 SYN
*May 25 04:51:44.563: IP: tableid=0, s=150.100.100.2 (Serial0/1/0), d=150.100.100.6 (Serial0/1/0), routed via RIB
*May 25 04:51:44.563: IP: s=150.100.100.2 (Serial0/1/0), d=150.100.100.6 (Serial0/1/0), len 56, rcvd 3
*May 25 04:51:44.563: ICMP type=11, code=0

Seems that R2 is telling us something about our packet sent to R5.

3) debug ip icmp

*May 25 04:53:23.839: ICMP: time exceeded rcvd from 150.100.100.2

Here we get our answer. At this point we realize that our tcp syn packets sent to R5 have an IP TTL of 1, and thus are getting dropped by R2.

Do you know any other commands that would help you come to this conclusion?

Configuring multipoint subinterface so the interface status reflects status of the PVC

On a physical frame-relay interface, if the opposite end goes down, the local interface will remain up/up. When using multipoint subinterfaces this is not the case. When the remote interface goes down (taking the dlci with it), the local ends puts its interface in a down/down state.

R1, R3 and R5 connect via full mesh frame-relay, subnet 190.1.135.0/24

R1 dlci 103 maps to R3 dlci 301
R1 dlci 105 maps to R5 dlci 501
R3 dlci 305 maps to R5 dlci 503

Configure all routers on the physical interfaces.

Here is the outlook so far from R3:

R3#show ip int brief serial 1/0
Interface IP-Address OK? Method Status Protocol
Serial1/0 190.1.135.1 YES manual up up
R3#

Now Let's shut the physical interfaces On R5 and R1:

R5(config)#int s0/0
R5(config-if)#shut

R1(config)#int s0/0
R1(config-if)#shut

R3 still has its interface up/up:

R3#show ip int brief serial 1/0
Interface IP-Address OK? Method Status Protocol
Serial1/0 190.1.135.1 YES manual up up

If we want R3's interface to go down when R5 and R1 are no longer available we need to use multipoint subinterface. Let's create one on R3 and move the config over:

R3(config)#interface Serial1/0
R3(config-if)# no ip address 190.1.135.1 255.255.255.0
R3(config-if)# no frame-relay map ip 190.1.135.1 301
R3(config-if)# no frame-relay map ip 190.1.135.5 305
R3(config-if)#int s1/0.3 multipoint
R3(config-subif)# ip address 190.1.135.1 255.255.255.0
R3(config-subif)# frame-relay map ip 190.1.135.1 301 broadcast
R3(config-subif)# frame-relay map ip 190.1.135.5 305 broadcast
R3(config-subif)# no frame-relay inverse-arp

Bring up R5 and R1 again and now we have:

R3#show ip int brief s1/0.3
Interface IP-Address OK? Method Status Protocol
Serial1/0.3 190.1.135.1 YES manual up up

Shut down R5 and R3 is stil up but look at the debug frame-relay lmi. The status of PVC 305 is 0x0 which is inactive

R3#show ip int brief s1/0.3
Interface IP-Address OK? Method Status Protocol
Serial1/0.3 190.1.135.1 YES manual up up

*Mar 1 08:27:43.170: Serial1/0(in): Status, myseq 115, pak size 45
*Mar 1 08:27:43.170: RT IE 1, length 1, type 0
*Mar 1 08:27:43.170: KA IE 3, length 2, yourseq 115, myseq 115
*Mar 1 08:27:43.170: PVC IE 0x7 , length 0x6 , dlci 301, status 0x2 , bw 0
*Mar 1 08:27:43.170: PVC IE 0x7 , length 0x6 , dlci 302, status 0x0 , bw 0
*Mar 1 08:27:43.170: PVC IE 0x7 , length 0x6 , dlci 304, status 0x2 , bw 0
*Mar 1 08:27:43.170: PVC IE 0x7 , length 0x6 , dlci 305, status 0x0 , bw 0

Now let's shut down R1 and take a look at R3 again, notice pvc 301 is now status 0x0. and the interface is down/down on R3

*Mar 1 08:30:03.170: Serial1/0(in): Status, myseq 129, pak size 45
*Mar 1 08:30:03.170: RT IE 1, length 1, type 0
*Mar 1 08:30:03.170: KA IE 3, length 2, yourseq 129, myseq 129
*Mar 1 08:30:03.170: PVC IE 0x7 , length 0x6 , dlci 301, status 0x0 , bw 0
*Mar 1 08:30:03.170: PVC IE 0x7 , length 0x6 , dlci 302, status 0x0 , bw 0
*Mar 1 08:30:03.170: PVC IE 0x7 , length 0x6 , dlci 304, status 0x2 , bw 0
*Mar 1 08:30:03.174: PVC IE 0x7 , length 0x6 , dlci 305, status 0x0 , bw 0

R3#show ip int brief s1/0.3
Interface IP-Address OK? Method Status Protocol
Serial1/0.3 190.1.135.1 YES manual down down

OSPF - dead time expiring

I was setting up a frame-relay lab a few moments ago and I ran into strange issue. R5 and R6 are going to connect via EBGP with their loopbacks. To connect to each other's loopbacks I was using OSPF over the Frame cloud.

However, I kept getting this message on R6:

R6#
Mar 1 00:34:18.895: %OSPF-5-ADJCHG: Process 1, Nbr 5.5.5.5 on Serial1/1 from FULL to DOWN, Neighbor Down: Dead timer expired
R6#
Mar 1 00:34:46.323: %OSPF-5-ADJCHG: Process 1, Nbr 5.5.5.5 on Serial1/1 from LOADING to FULL, Loading Done
R6#

These are point-to-multipoint links by the way. I started monitoring the neighbor relationship to se if hello's were being received. I was alarmed when I saw this:

R6#show ip ospf neighbor

Neighbor ID Pri State Dead Time Address Interface
5.5.5.5 0 FULL/ - 00:01:08 172.14.45.5 Serial1/1
4.4.4.4 0 FULL/ - 00:01:50 172.14.45.4 Serial1/1

The dead time crept well below 1:20 which is the period during which I should have received a hello. I ran "debug ip ospf packet" on R6 and was not receiving any hellos from R5. But how was the adjacency forming in the first place! It did however receive replies when R6 initiated the process. This was immediately after the neighbor went down.

So I did some more searching on R5 and found this:

R5#show run int s1/1
Building configuration...

Current configuration : 217 bytes
!
interface Serial1/1
ip address 172.14.45.5 255.255.255.0
encapsulation frame-relay
ip ospf network point-to-multipoint
serial restart-delay 0
frame-relay map ip 172.14.45.6 506
no frame-relay inverse-arp
end

What's the problem? No broadcast statement on the frame-relay map command!

Frame relay - debug frame events during inverse arp

I was curious about the output of this debugging command so I decided to dive a little further. Here is what I found.

R1#debug frame-relay events

LMI status received about which DLCIs are active:

*Mar 1 00:11:23.903: Serial1/0: preparing IP inarp on 102
*Mar 1 00:11:23.903: Serial1/0: preparing IP inarp on 105
*Mar 1 00:11:23.907: Serial1/0: preparing IP inarp on 104

I received an INARP:

*Mar 1 00:11:24.015: Serial1/0: FR ARP input
*Mar 1 00:11:24.019: datagramstart = 0x7A0124E, datagramsize = 34

*Mar 1 00:11:24.019: FR encap = 0x18610300

*Mar 1 00:11:24.019: 80 00 00 00 08 06 00 0F 08 00 02 04 00 09 00 00

*Mar 1 00:11:24.023: 0A 00 00 02 18 61 0A 00 00 01 01 02 00 00

*Mar 1 00:11:24.023:
*Mar 1 00:11:24.023: Serial1/0: inarp received on 102
R1#
R1#show frame-relay map Serial1/0 (up): ip 10.0.0.2 dlci 102(0x66,0x1860), dynamic, broadcast,, status defined, active

Analysis:

Datagram size: 34 bytes. The total size of this frame packet is 34 bytes. In Wireshark this be the "bytes on the wire"

FR Encap: FR encap = 0x18610300. The 1861 is the upper (0x18) and lower (0x61) DLCI values. The 0x18 is hex which in binary is 00011000 or 24 in decimal. But remember in this octet only the first 6 bits are used for the DLCI and the rest are in the 2nd octet. That means 000110XXXX will be our DLCI value where XXXX is filled in by the first 4 bits of the 2nd octet. So far we have 000110XXXX = 96 (not 24 or 6!). The 2nd octet is 0x61 which is 01100001 in binary. We are concerned with the first 4 bits and we replace the XXXX with 0110 which leaves us with 0001100110 or 102. 102 is what we have in our show frame-relay map output.

The second half of the FR Encap line is 0300 which is the control code and some padding.

In the next 2 lines we have the hex output of our packet. Below is a screenshot of the packet:

Troubleshooting OSPF over Frame-Relay Part 1

Scenario:
Full mesh of PVCs between 3 routers: R4 R5 and R6
Frame-relay map statements DO NOT have broadcast statement
Adjacencies do not form.

This is from debug ip packet on R6:

R6#debug ip packet
IP packet debugging is on
*Mar 1 01:00:26.367: IP: s=172.12.45.6 (local), d=224.0.0.5 (Serial1/1), len 76, sending broad/multicast
*Mar 1 01:00:26.371: IP: s=172.12.45.6 (local), d=224.0.0.5 (Serial1/1), len 76, encapsulation failed

After enabling broadcast on the frame maps, adjacencies came up

Solution:
Point-to-multipoint ospf networks need broadcast keyword on frame-relay map.
Without it you will see "encapsulation failed" when the router tries to send multicast hellos.