SNMP - sending traps to specific hosts

This was an issue I ran into awhile ago. I was trying to send BGP traps to one host, and PIM traps to another. As you can see below, BGP traps were getting sent to both hosts when I used version 1.

When I had version 2c specified, traps were only sent to the host configured for BGP. I do not know if this is difference in the protocol, but it is something you may want to be aware of if you need to send traps to different hosts.

Version 1, traps get sent to both hosts:

R1#sho run | inc snmp
snmp-server enable traps bgp
snmp-server enable traps pim
snmp-server host 2.2.2.2 public  bgp
snmp-server host 3.3.3.3 public  pim

R1#clear ip bgp *
R1#
00:11:49: %BGP-5-ADJCHANGE: neighbor 172.12.14.4 Down User reset
00:11:49: SNMP: Queuing packet to 2.2.2.2
 00:11:49: SNMP: V1 Trap, ent bgp, addr 172.12.12.1, gentrap 6, spectrap 2 
 bgpPeerEntry.14.172.12.14.4 = 00 00  
 bgpPeerEntry.2.172.12.14.4 = 1
00:11:49: SNMP: Queuing packet to 3.3.3.3
00:11:49: SNMP: V1 Trap, ent bgp, addr 172.12.13.1, gentrap 6, spectrap 2 
  bgpPeerEntry.14.172.12.14.4 = 00 00  
 bgpPeerEntry.2.172.12.14.4 = 1
00:11:49: SNMP: Packet sent via UDP to 2.2.2.2
00:11:49: SNMP: Packet sent via UDP to 3.3.3.3

Version 2c, traps get sent to one as desired:

R1#sho run | inc snmp
snmp-server enable traps bgp
snmp-server enable traps pim
snmp-server host 2.2.2.2 version 2c public  bgp
snmp-server host 3.3.3.3 version 2c public  pim

R1#clear ip bgp *
R1#
00:13:09: %BGP-5-ADJCHANGE: neighbor 172.12.14.4 Down User reset
R1#
00:13:09: SNMP: Queuing packet to 2.2.2.2
00:13:09: SNMP: V2 Trap, reqid 21, errstat 0, erridx 0 
 sysUpTime.0 = 78967 
  snmpTrapOID.0 = bgpTraps.2 
 bgpPeerEntry.14.172.12.14.4 = 00 00  
 bgpPeerEntry.2.172.12.14.4 = 1
00:13:09: SNMP: Packet sent via UDP to 2.2.2.2
R1#

Sending Logs as SNMP Traps

I have been reading Chapter 9 of Routing TCP/IP Vol. II this week. It has a good overview of the non-core topics such as snmp, rmon, ntp, etc. I recommend it for anyone struggling with these topics or just wanting a concise review.

This example shows how to configure a router to send logs to an snmp-server and verify it.

The first thing you must do is configure a server. Without this, you want be able to see any debugging because the router won't send any packet out.

R8(config)#snmp-server host 4.4.4.4 public syslog
R8(config)#snmp-server enable traps syslog
R8(config)#logging console warnings
R8(config)#logging buffered 16384 debugging

I have also decided to buffer the logs instead of send them to the console. This is not required just a way to keep everything less cluttered on the console screen. What this configuration does is buffer all log messages of debugging level or lower. Then these messages are sent via SNMP to the server at 4.4.4.4. Let's debug snmp packets, then a quick shutting/no shutting of an interface will give us some messages to view:

R8#debug snmp packets
SNMP packet debugging is on

R8(config)#int f0/0
R8(config-if)#no snmp trap link-status
R8(config-if)#shut
R8(config-if)#no shut

R8#sho logging | beg Log Buffer
Log Buffer (16384 bytes):

*Mar 2 18:33:38.565: %LINK-5-CHANGED: Interface FastEthernet0/0, changed state to administratively down
*Mar 2 18:33:40.117: %PIM-5-DRCHG: DR change from neighbor 0.0.0.0 to 192.168.8.8 on interface FastEthernet0/0 (vrf default)
*Mar 2 18:33:40.641: %LINK-3-UPDOWN: Interface FastEthernet0/0, changed state to up
*Mar 2 18:33:40.669: SNMP: Queuing packet to 4.4.4.4
*Mar 2 18:33:40.669: SNMP: V1 Trap, ent ciscoSyslogMIB.2, addr 192.168.78.8, gentrap 6, spectrap 1
clogHistoryEntry.2.58 = LINK
clogHistoryEntry.3.58 = 4
clogHistoryEntry.4.58 = UPDOWN
clogHistoryEntry.5.58 = Interface FastEthernet0/0, changed state to up
clogHistoryEntry.6.58 = 15322065
*Mar 2 18:33:40.921: SNMP: Packet sent via UDP to 4.4.4.4
*Mar 2 18:33:42.513: %SYS-5-CONFIG_I: Configured from console by console

I disabled snmp trap link-status to show that we are not using this feature to send traps. Notice the entry labeled clogHistoryEntry.5.58, this is exactly the same message as our logging message a few lines up. We only get linkup messages with this basic config.

To modify the configuration we use the logging history command:

R8(config)#logging history ?
<0-7> Logging severity level
alerts Immediate action needed (severity=1)
critical Critical conditions (severity=2)
debugging Debugging messages (severity=7)
emergencies System is unusable (severity=0)
errors Error conditions (severity=3)
informational Informational messages (severity=6)
notifications Normal but significant conditions (severity=5)
size Set history table size
warnings Warning conditions (severity=4

R8(config)#logging history size 2

Here I set the history size to 2 so I am able to view the last 2 messages sent with the following command:

R8#sho logging history
Syslog History Table:2 maximum table entries,
saving level notifications or higher
149 messages ignored, 11 dropped, 0 recursion drops
57 table entries flushed
SNMP notifications enabled, 45 notifications sent
entry number 58 : LINK-3-UPDOWN
Interface FastEthernet0/0, changed state to up
timestamp: 15322065
entry number 59 : SYS-5-CONFIG_I
Configured from console by console
timestamp: 15345618

Pretty basic scenario. It is important to remember this is different from the usual way of sending linkup/linkdown traps. Here, we are not using "snmp-server enable traps snmp linkup linkdown" or the interface command "snmp trap link-status".

Also, I think I figured out why linkdowns are not being sent, if I manually configure the logging level to "notifications" it works:

R8(config)#logging history notifications
R8(config)#int f0/0
R8(config-if)#shut

R8#sho logging

*Mar 2 18:46:38.885: SNMP: Packet sent via UDP to 4.4.4.4
*Mar 2 18:46:39.981: %LINK-5-CHANGED: Interface FastEthernet0/0, changed state to administratively down
*Mar 2 18:46:40.057: SNMP: Queuing packet to 4.4.4.4
*Mar 2 18:46:40.061: SNMP: V1 Trap, ent ciscoSyslogMIB.2, addr 192.168.78.8, gentrap 6, spectrap 1
clogHistoryEntry.2.72 = LINK
clogHistoryEntry.3.72 = 6
clogHistoryEntry.4.72 = CHANGED
clogHistoryEntry.5.72 = Interface FastEthernet0/0, changed state to administratively down
clogHistoryEntry.6.72 = 15399999
*Mar 2 18:46:40.309: SNMP: Packet sent via UDP to 4.4.4.4
*Mar 2 18:46:40.981: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/0, changed state to down
*Mar 2 18:46:41.033: SNMP: Queuing packet to 4.4.4.4
*Mar 2 18:46:41.033: SNMP: V1 Trap, ent ciscoSyslogMIB.2, addr 192.168.78.8, gentrap 6, spectrap 1
clogHistoryEntry.2.73 = LINEPROTO
clogHistoryEntry.3.73 = 6
clogHistoryEntry.4.73 = UPDOWN
clogHistoryEntry.5.73 = Line protocol on Interface FastEthernet0/0, changed state to down
clogHistoryEntry.6.73 = 15400099

Not sure why I needed that because according to the "show logging history", notifications were the default level. However, it appears they aren't because the command shows up in the config:

R8#sho run | inc logging his
logging history size 2
logging history notifications
R8#

Auto-install, eh?

While doing IPexpert Volume 2 Section 13 I ran into a task that said:

"There is a high chance you will be replacing your current R4 router with another high-end router. The admin of R4 has saved its configuration on a TFTP server whose IP address is 136.10.12.100. Make sure the new router will automatically configure itself."

So I started browsing through the DocCD for auto-install when it hit me...how exactly is this supposed to work? Not knowing the exact details about auto-install I knew that this should be a simple task since it was only 1 point.

Well the new router needs to know about address 136.10.12.100 somehow...but when it has no config it has no address. What I figured was that the new router will send a broadcast on it's frame-relay interface which happens to connect to R2. In fact the 136.10.12.100 network is also on R2's ethernet interface.

So I configured a helper address on R2's frame-relay interface pointing to 136.10.12.100. The PG agreed! 1 task, 1 command, 1 point :-)

Logging XML - A short DocCD adventure

I had a task today in IPexpert volume 2 section 7 that asked me to configure R5 such that while I am telnetted into it, it will display error messages in XML. huh?

I had no idea what command this was and while it turned out to be simple here's how I found it:

1. Main DocCD URL:
http://www.cisco.com/web/psa/products/index.html

2. Select Cisco IOS Software > 12.4 > 12.4 Mainline

3. Master Index

4. Cisco IOS Master Command List, All Releases

5. Now here is where I used the browser search to look for XML in EACH page (didn't take but 1 minute). By the time I go to section labeled "is-type through mwi-server" I had found the following commands:

logging buffered xml
logging console xml
logging monitor xml

Which one is it? Well it's not buffered because we want the logs displayed on the screen. And it's probably not console because we are going to be logged in through the vty. So I used the logging monitor xml and then test:

R5(config)#logging monitor xml

From any other router:

R1#
R1#telnet 192.168.5.5
Trying 192.168.5.5 ... Open


User Access Verification

Password:
R5>en
Password:
R5#conf t
Enter configuration commands, one per line. End with CNTL/Z.
R5(config)#int f0/1
R5(config-if)#shut
R5(config-if)#

hmm...nothing...oh wait!

R5(config-if)#
R5(config-if)#exit
R5(config)#exit
R5#term mon
R5#conf t
Enter configuration commands, one per line. End with CNTL/Z.
R5(config)#int f0/1
R5(config-if)#no shut
R5(config-if)#
LINK3UPDOWN*Nov 8 20:07:21.098FastEthernet0/1up

That's better!

SSH for VTY access

This is a very simple lab for SSH access:

R1 and R2 connected to same LAN
R1 will be the client, R4 the server

Let's enable SSH on R2:

R2(config)#ip domain-name theseblues.com

R2(config)#crypto key generate rsa
The name for the keys will be: R2.theseblues.com
Choose the size of the key modulus in the range of 360 to 2048 for your
General Purpose Keys. Choosing a key modulus greater than 512 may take
a few minutes.

How many bits in the modulus [512]:
% Generating 512 bit RSA keys, keys will be non-exportable...[OK]

R2(config)#
*Mar 1 00:18:50.259: %SSH-5-ENABLED: SSH 1.99 has been enabled
R2(config)#
R2(config)#line vty 0 4
R2(config-line)#login local
R2(config-line)#transport input ssh
R2(config-line)#exit
R2(config)#username dh password blues

Let's SSH from R1:

R1#ssh ?
-c Select encryption algorithm
-l Log in using this user name
-m Select HMAC algorithm
-o Specify options
-p Connect to this port
-v Specify SSH Protocol Version
WORD IP address or hostname of a remote system

R1#ssh -l dh 192.168.0.2

Password:

R2>en
% No password set
R2>

Forgot the enable password! You know what to do...

NTP authentication

This is scenario 3 dealing with NTP and in this post I will set up NTP authentication. Remember that the purpose is for the client to authenticate the server. First will make sure NTP is synchronized on the client. Next, we will turn on authentication on the client only until the clocks become unsynchronized. Finally, we will configure the server with the appropriate key and make sure the client is synchronized again.

We use the same topology as the other NTP scenarios:
R4, R5 and R6 connected via full mesh frame-relay on subnet 172.12.45.0/24

Let's make sure R6, the master, is synchronized first:

R6#show clock
20:14:37.215 MDT Mon Jun 2 2008

R6#show ntp status | inc Clock is
Clock is synchronized, stratum 2, reference is 127.127.7.1

R4:

R4#show ntp status | in Clock
Clock is synchronized, stratum 3, reference is 172.12.45.6

and R5:

R5#show ntp status | inc Clock
Clock is synchronized, stratum 3, reference is 172.12.45.6

Let's configure authentication on R4:

R4(config)#ntp authenticate
R4(config)#ntp authentication-key 1 md5 cisco
R4(config)#ntp trusted-key 1
R4(config)#ntp server 172.12.45.6 key 1

R4#debug ntp validity
NTP peer validity debugging is on
R4#debug ntp authentication
NTP authentication debugging is on

In a few moments we get these debug messages:

R4(config)#
Jun 3 03:24:49.759: NTP: packet from 172.12.45.6 failed validity tests 10
Jun 3 03:24:49.763: Authentication failed
R4(config)#

What's strange is that R4 is still synchronized with R6:

R4#show ntp status
Clock is synchronized, stratum 3, reference is 172.12.45.6

So I waited about 10 minutes (it seemed that long) and now R4 is unsynchronized:

R4#show ntp status | inc Clock
Clock is unsynchronized, stratum 16, no reference clock

Now let's configure NTP authentication on R6:

R6(config)#ntp authenticate
R6(config)#ntp authentication-key 1 md5 cisco

On R4 I will set up some debugging, a lot of the output doesn't make much sense, but here it is:

R4#debug ntp packets
NTP packets debugging is on

!! HERE R4 TRANSMITS A PACKET WITH AUTHENTICATION-KEY 1

.Jun 3 03:36:10.711: NTP: xmit packet to 172.12.45.6:
.Jun 3 03:36:10.715: leap 3, mode 3, version 3, stratum 0, ppoll 64
.Jun 3 03:36:10.715: rtdel 0D4E (51.971), rtdsp 05CD (22.659), refid AC0C2D06 (172.12.45.6)
.Jun 3 03:36:10.719: ref CBEF37C1.C8A9606E (20:23:45.783 MDT Mon Jun 2 2008)
.Jun 3 03:36:10.719: org CBEF3A6A.BF8D3204 (20:35:06.748 MDT Mon Jun 2 2008)
.Jun 3 03:36:10.723: rec CBEF3A6A.C784D317 (20:35:06.779 MDT Mon Jun 2 2008)
.Jun 3 03:36:10.723: xmt CBEF3AAA.B614DB93 (20:36:10.711 MDT Mon Jun 2 2008)
.Jun 3 03:36:10.727: Authentication key 1

!! HERE R4 RECEIVES A PACKET WITH AUTHENTICATION-KEY 1

.Jun 3 03:36:10.771: NTP: rcv packet from 172.12.45.6 to 172.12.34.4 on FastEthernet0/1:
.Jun 3 03:36:10.775: leap 0, mode 4, version 3, stratum 2, ppoll 64
.Jun 3 03:36:10.779: rtdel 0000 (0.000), rtdsp 0002 (0.031), refid 7F7F0701 (127.127.7.1)
.Jun 3 03:36:10.779: ref CBEF3A89.50E1C8AB (20:35:37.315 MDT Mon Jun 2 2008)
.Jun 3 03:36:10.783: org CBEF3AAA.B614DB93 (20:36:10.711 MDT Mon Jun 2 2008)
.Jun 3 03:36:10.783: rec CBEF3AAA.C3941C78 (20:36:10.763 MDT Mon Jun 2 2008)
.Jun 3 03:36:10.787: xmt CBEF3AAA.C39DD400 (20:36:10.764 MDT Mon Jun 2 2008)
.Jun 3 03:36:10.791: inp CBEF3AAA.C565D38F (20:36:10.771 MDT Mon Jun 2 2008)
.Jun 3 03:36:10.791: Authentication key 1

R4 is now synchronized:

R4#show ntp status
Clock is synchronized, stratum 3, reference is 172.12.45.6
nominal freq is 250.0000 Hz, actual freq is 250.0004 Hz, precision is 2**18
reference time is CBEF3AAA.C565D38F (20:36:10.771 MDT Mon Jun 2 2008)
clock offset is 22.8825 msec, root delay is 59.68 msec
root dispersion is 15897.92 msec, peer dispersion is 15875.02 msec

Let's check R5 to see if it is still synchronized:

R5#show ntp status | inc Clock
Clock is synchronized, stratum 3, reference is 172.12.45.6

As you can see, R5 is still synchronized because it is not requesting authentication from R6. R4 on the other sends the request with an authentication key and requires that the time-source, R6, does so as well.

Well, this lab went okay. I have had many problems with ntp authentication before. Initially, I thought I would have problems again when R4 was not un-synchronizing with R6 when we first set up a key on R4. Well, it took a few minutes, so I guess patience is this key. You can't afford to be too patient though, you only have 8 hours on lab day :)

NTP Access group

I will call this NTP blog NTP Scenario 2: NTP with access-group filtering.

R4, R5, and R6 are connected via full mesh frame-relay
R6 will be the NTP server.

In scenario 1 we set up R4 to sync to R6. In this scenario we
will prevent R5 from syncing with R6 by using ntp access-group command.

I have never used this command before, so I hope I am understanding it correctly. If not, please let me know how else to test this command.

On R6 we have the following ACL:

access-list 1 permit 172.12.45.4
access-list 1 permit 127.127.7.1

172.12.45.4 is the address of R4. 127.127.7.1 is the IP address that a cisco router uses as it's reference when you make the router an NTP master. This must be added to the ACL or it will become unsynchronized with itself.

In global config mode we enter:

ntp access-group serve-only 1

Let's check the current times:

R5#show clock
16:15:04.031 UTC Sat May 31 2008

R6#show clock
16:31:45.687 UTC Sat May 31 2008

R5 is about 15 minutes behind. I don't know a way to debug on R6 to make sure it's working, so what I will do is wait about 10 minutes then add R5's IP address to ACL. I think 10 minutes is sufficient to prove that R6 is not allowing R5 to sync to it.

...
...

Well it's almost 20 minutes now. Let's check if R4 is synchronized:

R4#show ntp status | inc Cloc
Clock is synchronized, stratum 3, reference is 172.12.45.6

How about R5:

R5#sho ntp status | inc Clock
Clock is unsynchronized, stratum 16, no reference clock

Some quick show clocks:

R4#show clock
16:49:06.505 UTC Sat May 31 2008

R5#show clock
16:32:29.755 UTC Sat May 31 2008

R6#show clock
16:49:12.335 UTC Sat May 31 2008

R5 is still unsynchronized and more than 15 minutes behind R6. Let's add R5's address to ACL 1 on R6 and see how long it takes to sync...

R6(config)#access-list 1 permit 172.12.45.5

We'll debug on R5:

R5#debug ntp sync
NTP clock synchronization debugging is on
R5#sho ntp status | inc Clock
Clock is unsynchronized, stratum 16, no reference clock
R5#
May 31 16:33:54.039: NTP: synced to new peer 172.12.45.6

Took less than a minute!

This is the second of many NTP scenarios to come. It really seems so simple, but I have always had problems understanding NTP server/peer relationships and authentication configurations. So these will be the topics of future NTP blogs. Hopefully I (and you!) will become NTP masters in time for CCIE :)

Basic NTP Configuration

I will call this NTP Scenario 1: Basic NTP with no authentication. There will be more blogs about NTP but this is just to get it started.

R4, R5, and R6 are connected via full mesh frame-relay
R6 will be the NTP server.

First set the clock on R6:

R6#clock set 15:47:30 31 May 2008
*May 31 15:47:30.003: %SYS-6-CLOCKUPDATE: System clock has been updated from 01:
05:19 UTC Fri Mar 1 2002 to 15:47:30 UTC Sat May 31 2008, configured from consol
e by console.

Verify the time with show clock:

R6#show clock
15:47:36.651 UTC Sat May 31 2008

Configure R6 as the master along with stratum:

R6#conf t
R6(config)#ntp master ?
<1-15> Stratum number


R6(config)#ntp master 3
R6(config)#^Z

Verify that R6 is synced with itself:

R6#show ntp associations

address ref clock st when poll reach delay offset disp
*~127.127.7.1 127.127.7.1 2 22 64 1 0.0 0.00 15875.
* master (synced), # master (unsynced), + selected, - candidate, ~ configured

R6#show ntp status
Clock is synchronized, stratum 3, reference is 127.127.7.1
nominal freq is 250.0000 Hz, actual freq is 250.0000 Hz, precision is 2**24
reference time is CBEBF22C.1583DCD8 (15:50:04.084 UTC Sat May 31 2008)
clock offset is 0.0000 msec, root delay is 0.00 msec
root dispersion is 15875.02 msec, peer dispersion is 15875.02 msec

Now it's time to configure the clients, let's first check the time on R4:

R4#show clock
*03:03:44.823 UTC Fri Mar 1 2002
R4#conf t
R4(config)#ntp server 172.12.45.6
R4(config)#^Z
R4#show clock
*03:04:52.047 UTC Fri Mar 1 2002
R4#

Notice the time is 6 years behind the current date, I have noticed that NTP will not sync when the times are so far apart (Not sure how this works, but let's manually 'push' R4 a little closer)

R4#clock set 15:30:40 31 May 2008

Now we're about 15 minutes behind. Let's do some debugging:

R4#debug ntp sync
NTP clock synchronization debugging is on

.May 31 15:31:43.107: NTP: synced to new peer 172.12.45.6

Noe that we're synced let's verify on R6 and R4:

R4#show clock
15:59:42.466 UTC Sat May 31 2008

R6#show clock
15:59:41.135 UTC Sat May 31 2008

Looks good. I will use this topology for other NTP labs so stay tuned...