Troubleshooting IPsec VPN

Now that I have introduced the theory and configuration of IPsec VPNs in my previous posts, it's time to dive into troubleshooting and examine the most common VPN problems. In fact, troubleshooting VPNs is one of the most difficult tasks in network administration. There are so many factors that can cause the IKE negotiations to fail and the tunnel not establishing.

I would strongly advice against looking at the running configuration. It is easy to become blind to your own mistakes, and, as a result, impossible to pinpoint minor configuration errors. Using debugs will provide more intuitive information, but there is a price. The output is noisy and can overwhelm the device as well as the reader.

Now, let's look at some common problems (not an exhaustive list). Again, using the same example topology.

Testing Connectivity

Well, this is not exactly a problem but a common mishap. It's possible that the configuration is correct but the administrator is improperly testing the connection. Ideally, the VPN connection is tested from the end devices inside the local networks. However, sometimes administrators try to verify the tunnel creation by pinging between the peer devices. In this case, the pings will not be encrypted, nor should they be. The key is the proxy IDs, which defines the "interesting traffic" to be encrypted and initially triggers the tunnel creation. If the ping is issued from the peer itself, by default, it will source the packets from the closest interface towards the destination. Referring to the topology, the ping from ASA01 would contain the following information:

• source: 192.0.2.1
• destination 198.51.100.1

This does not match the crypto ACL.

access-list L2L extended permit ip 10.1.1.0 255.255.255.0 192.168.1.0 255.255.255.0

Instead, the ping must be manually sourced from the local LAN interface and destined towards the remote LAN.

ASA01# ping inside 192.168.1.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.1.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/10 ms

ASA01# show crypto ipsec sa detail
interface: outside
    Crypto map tag: DEMOVPN, seq num: 10, local addr: 192.0.2.1

      access-list L2L extended permit ip 10.1.1.0 255.255.255.0 192.168.1.0 255.255.255.0
      local ident (addr/mask/prot/port): (10.1.1.0/255.255.255.0/0/0)
      remote ident (addr/mask/prot/port): (192.168.1.0/255.255.255.0/0/0)
      current_peer: 198.51.100.1

      #pkts encaps: 34, #pkts encrypt: 34, #pkts digest: 34
      #pkts decaps: 34, #pkts decrypt: 34, #pkts verify: 34
[...]

IKE Phase 1 Fail

If a VPN tunnel does not come up, verifying Phase 1 is typically the first troubleshooting step. The show crypto isakmp sa is a handy command for this purpose.

ASA01# show crypto isakmp sa

IKEv1 SAs:

   Active SA: 1
    Rekey SA: 0 (A tunnel will report 1 Active and 1 Rekey SA during rekey)
Total IKE SA: 1

1   IKE Peer: 198.51.100.1
    Type    : user            Role    : initiator
    Rekey   : no              State   : MM_WAIT_MSG2

The output indicates that the Initiator is stuck in MM_WAIT_MSG2 state. This only means that the Initiator has sent its IKE policies to the peer, but has not received a response. Therefore, there are several potential causes, for example the Receiver having no return route/link down, or simply ISAKMP not enabled on the correct interface. It's impossible to tell. Even the debugs (debug crypto isakmp sa and debug crypto ipsec sa) do not provide much further assistance, but merely state that the tunnel establishment failed.

ASA01#
%ASA-4-752012: IKEv1 was unsuccessful at setting up a tunnel.  Map Tag = DEMOVPN.  Map Sequence Number = 10.
%ASA-3-752015: Tunnel Manager has failed to establish an L2L SA.  All configured IKE versions failed to establish the tunnel. Map Tag= DEMOVPN.  Map Sequence Number = 10.

Let's see, if ASA02 offers more clues.

ASA02#
%ASA-7-713906: IP = 192.0.2.1, All SA proposals found unacceptable

Indeed, it does. 'All SA proposals found unacceptable' is the line to look for in the noisy debug output. It clearly indicates that the peers could not agree on the IKE policy to negotiate the IKE Phase 1 SA. ASA02 did not accept the proposal(s) sent by ASA01.

So let's verify the IKE policies.

ASA01# show run crypto ikev1
crypto ikev1 policy 10
 authentication pre-share
 encryption des
 hash sha
 group 2
 lifetime 86400

ASA02# show run crypto isakmp
crypto isakmp policy 15
 authentication pre-share
 encryption aes
 hash sha
 group 2
 lifetime 86400

The encryption methods do not match (des != aes).

IKE Phase 2 Fail

How does a failure in IPsec SA negotiation appear in the debug? Let's find out.

ASA01#
%ASA-5-713119: Group = 198.51.100.1, IP = 198.51.100.1, PHASE 1 COMPLETED
%ASA-7-715046: Group = 198.51.100.1, IP = 198.51.100.1, constructing IPSec SA payload
[...]
%ASA-4-752012: IKEv1 was unsuccessful at setting up a tunnel.  Map Tag = DEMOVPN.  Map Sequence Number = 10.
%ASA-3-752015: Tunnel Manager has failed to establish an L2L SA.  All configured IKE versions failed to establish the tunnel. Map Tag= DEMOVPN.  Map Sequence Number = 10.

Again, the output on the Initiator (ASA01) is not very intuitive. However, it does state that Phase 1 completed successfully, which is a clear indication that the problem occurs later in the process. What says ASA02?

ASA02#
%ASA-7-715047: Group = 192.0.2.1, IP = 192.0.2.1, processing IPSec SA payload
%ASA-5-713904: Group = 192.0.2.1, IP = 192.0.2.1, All IPSec SA proposals found unacceptable!

Ha! ASA02 complains that it does not like the IPsec SA proposal(s) from ASA01. Let's compare the configuration.

ASA01# show run crypto map
crypto map DEMOVPN 10 match address L2L
crypto map DEMOVPN 10 set peer 198.51.100.1
crypto map DEMOVPN 10 set ikev1 transform-set X
crypto map DEMOVPN interface outside

ASA01# show run | include transform-set
crypto ipsec ikev1 transform-set X esp-aes-256 esp-md5-hmac

 
ASA02# show run crypto map
crypto map VPN-CRYPTOMAP 10 match address VPN-TRAFFIC
crypto map VPN-CRYPTOMAP 10 set peer 192.0.2.1
crypto map VPN-CRYPTOMAP 10 set transform-set TRANSET
crypto map VPN-CRYPTOMAP interface outside

ASA02# show run | include transform-set
crypto ipsec transform-set TRANSET esp-aes-192 esp-md5-hmac

So ASA01 wants to use AES-256 for encryption, but ASA01 will only accept AES-192.

Mismatched Pre-Shared Keys

Unlike the other problems demonstrated so far, troubleshooting mismatched pre-shared keys is straightforward. Even without issuing the debugs, the syslog messages notify the administrator that a wrong password is a likely cause.

ASA01# ping inside 192.168.1.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.1.1, timeout is 2 seconds:
Oct 31 12:32:22 [IKEv1]Group = 198.51.100.1, IP = 198.51.100.1, Error, peer has indicated that something is wrong with our message.  This could indicate a pre-shared key mismatch.
.....


ASA02# show crypto isakmp sa

   Active SA: 1
    Rekey SA: 0 (A tunnel will report 1 Active and 1 Rekey SA during rekey)
Total IKE SA: 1

1   IKE Peer: 192.0.2.1
    Type    : L2L             Role    : responder
    Rekey   : no              State   : MM_WAIT_MSG5

Oct 31 15:59:35 [IKEv1]: Group = 192.0.2.1, IP = 192.0.2.1, ERROR, had problems decrypting packet, probably due to mismatched pre-shared key.  Aborting

The ISAKMP state indicates that the peers are exchanging pre-shared key hashes, and if they don't match, the Receiver will remain in the MM_WAIT_MSG5 state.

Verifying the pre-shared keys from the command line on the ASA is challenging because they are stored in a hidden format. The only way to ensure they match is to retype them on both ends.

ASA01# show run tunnel-group
tunnel-group 198.51.100.1 type ipsec-l2l
tunnel-group 198.51.100.1 ipsec-attributes
 ikev1 pre-shared-key *****

ASA02# show run tunnel-group
tunnel-group 192.0.2.1 type ipsec-l2l
tunnel-group 192.0.2.1 ipsec-attributes
 pre-shared-key *****

NOTE: The pre-shared key is not five asterisks.

Wrong proxy IDs

As discussed in my previous posts, the proxy IDs define the local and remote networks between which traffic is to be encrypted. The proxy IDs must be mirrored between the peers. So what happens when they are not?

ASA01#
%ASA-5-713119: Group = 198.51.100.1, IP = 198.51.100.1, PHASE 1 COMPLETED
%ASA-7-715009: Group = 198.51.100.1, IP = 198.51.100.1, IKE Deleting SA: Remote Proxy 192.168.1.0, Local Proxy 10.1.1.0
%ASA-4-752012: IKEv1 was unsuccessful at setting up a tunnel.  Map Tag = DEMOVPN.  Map Sequence Number = 10.
%ASA-3-752015: Tunnel Manager has failed to establish an L2L SA.  All configured IKE versions failed to establish the tunnel. Map Tag= DEMOVPN.  Map Sequence Number = 10.

Again, the debug on the Initiator merely points out that Phase 1 completed successfully, but Phase 2 negotiations failed.

ASA02#
%ASA-5-713119: Group = 192.0.2.1, IP = 192.0.2.1, PHASE 1 COMPLETED
%ASA-7-713222: Group = 192.0.2.1, IP = 192.0.2.1, Static Crypto Map check, map = VPN-CRYPTOMAP, seq = 10, ACL does not match proxy IDs src:10.1.1.0 dst:192.168.1.0
%ASA-3-713061: Group = 192.0.2.1, IP = 192.0.2.1, Rejecting IPSec tunnel: no matching crypto map entry for remote proxy 10.1.1.0/255.255.255.0/0/0 local proxy 192.168.1.0/255.255.255.0/0/0 on interface outside

On the contrary, ASA02 distinctly says that there is a problem with the proxy IDs. Let's look at the access-lists.

ASA01# show run crypto map
crypto map DEMOVPN 10 match address L2L
crypto map DEMOVPN 10 set peer 198.51.100.1
crypto map DEMOVPN 10 set ikev1 transform-set ESP-AES192-MD5
crypto map DEMOVPN interface outside

ASA01# show run access-list
access-list L2L extended permit ip 10.1.1.0 255.255.255.0 192.168.1.0 255.255.255.0


ASA02# show run crypto map
crypto map VPN-CRYPTOMAP 10 match address L2L
crypto map VPN-CRYPTOMAP 10 set peer 192.0.2.1
crypto map VPN-CRYPTOMAP 10 set transform-set TRANSET
crypto map VPN-CRYPTOMAP interface outside

ASA02# show run access-list
access-list L2L extended permit ip 10.1.1.0 255.255.255.0 192.168.1.0 255.255.255.0

So what is the problem? The access-list on ASA02 was directly copied from ASA01 without mirroring it.