Saturday, November 22, 2014

FHRPs: HSRP, VRRP, and GLBP

What is this cryptic letter jargon that I've just typed in the title? Well, allow me to explain. FHRP refers to First Hop Redundancy Protocol. There are three main flavors: Hot Standby Router Protocol (HSRP), Virtual Router Redundancy Protocol (VRRP), and Gateway Load Balancing Protocol (GLBP). This post will provide a general introduction to each protocol.

Basic Operation

In essence, FHRPs are used to provide transparent gateway failover for end hosts. FHRPs enable a set of routers to work together to present the illusion of a single virtual router. This is achieved by multiple gateways sharing the same virtual IP address, and end hosts using this virtual IP address as their default gateway.

Look at the scenario below. The PCs have their default gateways set to the virtual IP address. They are basically unaware of the physical routers operating behind the scenes. The PCs only send packets towards their default gateway (the virtual IP) but they do not know which router is actually forwarding the traffic. If a router goes down, the other one will assume its role. The process is transparent and non-disruptive from the PCs' perspective.



This is the basic operation of FHRPs, but each protocol has a unique implementation.

HSRP

HSRP is a Cisco proprietary protocol, but it is also specified in RFC 2281. This means that HSRP can be implemented by other vendors, but special licensing is required. HSRP operates by electing one active router based on highest priority (100 by default, 0-255 range). The active router is responsible for forwarding packets. The other router remains in standby mode. If there are more than two routers in the HSRP group, the rest will stay in the Listen state (i.e. they know the virtual IP address and listen for Hello messages). If the active router fails, the standby router assumes its duties.

In addition to the virtual IP address, the routers in the same HSRP group share a virtual MAC address. The virtual MAC address is 0000.0C07.ACxx, where xx is the HSRP group ID. The group ID does not have to match the VLAN ID, but they are typically configured as the same value for consistency. If no group ID is specified, group ID 0 is assumed (default). The active router "owns" the virtual IP and MAC addresses. When a PC sends an ARP request for its default gateway IP address (the virtual IP), the active router will reply with the virtual MAC address.


The HSRP routers communicate by exchanging UDP-based multicast Hello packets. The Hello packets are sent in the multicast IP address 224.0.0.2 in UDP port 1985 (specific to version 1) every three seconds by default. If Hellos are not heard in ten seconds (the default HoldTime value), the active router is considered dead, and the standby router will assume its role.

Preemption is disabled by default. This means that a router will not attempt to take control as the active router, when it has a higher priority than the current active router. Note that this behavior effects the order of configuration. If preemption is not used, the active router must be configured first to guarantee that it will, in fact, become the active router. Preemption is required to successfully operate certain features (more in upcoming post about object tracking).

Because of the active/standby behavior, load balancing can only be implemented similar to STP. Load balancing is achieved by assigning different active routers for different HSRP groups (similar to STP having different root switches for different VLANs).

HSRP supports clear-text and MD5 authentication.

Other than basic IP connectivity (the routers must have individual IP addresses), the absolute minimum configuration for HSRP is to define the virtual IP address. The other parameters will then use their default values, namely, priority 100, group ID 0, and no preemption.

Example:

A#
interface vlan 10
  ip address 10.1.1.101 255.255.255.0
  standby ip 10.1.1.1

S#
interface vlan 10
  ip address 10.1.1.100 255.255.255.0
  standby ip 10.1.1.1


VRRP

VRRP has only minor differences to HSRP. The main separating factor is that VRRP is a standards-based alternative to HSRP, so it will run in multi-vendor environments. More differences include:
  • terminology: master/backup vs. active/standby
  • communication: uses its own Transport protocol 112, and multicast IP address 224.0.0.18
  • virtual IP address: can use the Master's interface IP address (HSRP requires unique physical IP addresses)
  • virtual MAC address: 0000.5E00.01xx, where xx is the VRRP group number
  • preemptive by default

Example configuration:

A#
interface vlan 10
  ip address 10.1.1.101 255.255.255.0
  vrrp 10 ip 10.1.1.1

S#
interface vlan 10
  ip address 10.1.1.102 255.255.255.0
  vrrp 10 ip 10.1.1.1

GLBP

GLBP performs similar, but not identical, to HSRP and VRRP. It is a Cisco proprietary protocol, and extends HSRP functionality to natively support load balancing. GLBP provides load balancing over multiple router using a single virtual IP address and multiple virtual MAC addresses. Contrary to HSRP and VRRP where only a single router (the active router) processes packets, all routers in a GLBP group participate in forwarding packets. GLBP members communicate with each others by sending Hello messages every 3 seconds to the multicast address 224.0.0.102, UDP port 3222.

Routers in a GLBP group operate in one of two roles: Active Virtual Gateway (AVG) or Active Virtual Forwarder (AVF). Members of a GLBP group elect one router to act as the AVG for that group. The election is based on the highest priority (range 1-255). The priority also determines the order of ascendancy to becoming an AVG if the current AVG fails. In case of a tie, the higher IP address is preferred. The remaining routers in the GLBP group assume the role of AVF. The AVG assigns a virtual MAC address for each  AVF, and each AVF is responsible for forwarding packets sent to their virtual MAC address. The AVG operates as an AVF itself. The AVG also replies to ARP requests for the virtual IP address. Up to four virtual MAC addresses per GLBP group is allowed. Load sharing is achieved by the AVG replying to the ARP requests with different virtual MAC addresses.

By default, preemption is disabled. A backup AVG can become the active AVG only if the current AVG fails, regardless of a higher priority.


Example configuration:

AVG#
interface vlan 10
  ip address 10.1.1.102 255.255.255.0
  glbp 10 ip 10.1.1.1
  glbp 10 priority 150

AVF#
interface vlan 10
  ip address 10.1.1.101 255.255.255.0
  glbp 10 ip 10.1.1.1







Thursday, November 13, 2014

IP Services: ARP, Proxy ARP, Reverse ARP, BOOTP, and DHCP

All these five protocols have a common denominator, namely, they help a host learn information to successfully send and receive IP packets.

ARP

ARP is a protocol used to map IP addresses to MAC addresses. Before a host can send any data, it must know the Layer 3 (IP) and Layer 2 (MAC) addresses to correctly construct a frame. ARP is used to dynamically learn the MAC address of a receiver, when the IP address is known. The protocol works as demonstrated in the figure below.


Proxy ARP

Proxy ARP uses the exact same process as ARP, but the ARP request is requesting a MAC address that is not on the local subnet. Because an ARP request is a broadcast, it will not be forwarded outside the local subnet, and the target host will never receive it. However, if a router has a route to that destination, it can issue a proxy ARP reply with its own MAC address on behalf of the target host.

Before the advent of DHCP, many networks relied on proxy ARP. Today, the use of proxy ARP is not that common. In fact, it is often a "quick fix" for potential routing issues (I'll get to this in another post).

Note that the sending host must believe that the target host is in its local network. Otherwise, the sending host will send an ARP request for its default gateway (this is normal ARP, not proxy ARP). In both cases, however, the default gateway (the first-hop router) will reply with its MAC address.

Let's alter the previous scenario to illustrate proxy ARP. First, let's modify the subnet mask on PC1 to cover a larger range, so it believes the remote network is directly connected. Say 10.1.1.102/16. Next, let's add a remote target host (PC2: 10.1.100.22/24) that PC1 will try to ping.


PC1 performs the AND operation, and believes that PC2 exists in its local network, so it sends an ARP request for PC2's IP address directly. The router receives the broadcast, and realizes that it can reach PC2, so it responds to PC1's ARP request with its own MAC address.

Reverse ARP

RARP, BOOTP, and DHCP represent the evolution of protocols defined to enable a host to dynamically learn its IP address. All three are client-server protocols. The client sends a broadcast to begin discovery, the server hears the request and supplies an IP address.

RARP also uses the same ARP message, but operates, well, in reverse (as conveniently implied by the protocol name). When sending a RARP request, the host inserts its own MAC address and an IP address of 0.0.0.0 (indicating an unknown IP address). A preconfigured RARP server, which must be on the same subnet, receives the request and performs a table lookup. If the MAC address listed in the RARP request is configured on the RARP server, the RARP server sends a RARP reply with the corresponding IP address.

The following figure illustrates the operation of RARP.


BOOTP

BOOTP improves on the address assignment features of RARP. BOOTP also uses a completely different set of messages, the payload is encapsulated inside an IP and UDP header. With minimal router configuration, BOOTP packets can be forwarded to remote subnets - allowing the deployment of a centrally located BOOTP server. In addition to IP address assignment, BOOTP supports the assignment of other information, including the subnet mask, default gateway, and DNS servers. But, similar to RARP, BOOTP still requires the server to be manually preconfigured with the MAC addresses and IP addresses for each client.

BOOTP operates as demonstrated in the following figure.


DHCP

DHCP represents the next in the evolution of dynamic IP address assignment. DHCP is widely deployed today. DHCP builds on the format of the BOOTP protocol, and provides flexible messaging and information assignment capability without requiring predefinition of MAC addresses for each client. DHCP also allows the reuse of IP addresses, meaning that a fixed 1:1 address configuration is not required. DHCP can lease IP addresses for the duration the device needs.

DHCP servers typically reside in a centralized location, with routers forwarding local broadcast DHCP requests  to remote DHCP servers. These type of routers are called DHCP relay agents. They change the DHCP request's destination IP address (255.255.255.255) to match the DHCP server (defined by the ip helper-address <ip> command). A relay agent would also list its own IP address in the gateway IP address (giaddr) field, notifying the DHCP server of the IP address to which the DHCP reply should be sent.



Bonus: Configuring a router as a DHCP server

Perhaps not that common in the real world, but a Cisco router can also act as a DHCP server. Essentially, the configuration requires three steps.

  1. Enable the DHCP service on the router.
  2. Exclude all statically configured IP addresses from being dynamically assigned.
  3. Configure DHCP pools with appropriate parameters.

Example:

service dhcp
!
ip dhcp excluded-address 10.1.1.1 10.1.1.15
!
ip dhcp pool DHCP-POOL
  network 10.1.1.0 255.255.255.0
  default-router 10.1.1.1
  dns-server 8.8.8.8

The above configuration is very basic; it merely enables the DHCP service, excludes IP addresses within the range 10.1.1.1-15, and defines a single DHCP pool


Tuesday, November 4, 2014

BPDU Format and STP Timers

Coming back to STP.. This post is dedicated to examining the BPDU and STP timers in detail.

BPDU Autopsy

Switches exchange STP information using a special frame type called Bridge Protocol Data Unit, or BPDU. A BPDU contains several fields which are compared in STP calculation. A switch uses its base MAC address as the source address for BPDU frames, and forwards them to the well-known STP multicast address 0180.C200.0000.

 Let's break down the BPDU frame format.


The BPDU frame format varies slightly depending on the type of protocol used. We'll focus on Cisco's implementation (represented in Figure A-13). For the purposes of this discussion, we'll ignore the Frame control, Routing Information field, Logical link control fields, which have specific preset values (no need to memorize them). As mentioned, the Destination address and Source address contain the STP multicast address and the base MAC address of the switch, respectively.

Now, the IEEE BPDU section encompasses the heart of STP.


When computing a loop-free Layer 2 topology, a switch simply examines the contents of the BPDU fields, as represented in Figure A-14. Let's view each field more specifically. The Protocol identifier, and Version fields are always set to 0, so we'll just skip them.

Message Type (1 byte): denotes whether the frame is a Configuration or TCN BPDU (described later)

Flags (1 byte): signals the TC and TCA bits (described later)

Root ID (8 bytes): contains the root BID (2-byte priority + 6-byte MAC address)

Root Path Cost (4 bytes): indicates the sender's cost to the root switch

Bridge ID (8 bytes): represents the BID of the sender (2-byte priority +  6-byte MAC address)

Port ID (2 bytes): includes the sender's port number from which the BPDU was sent

Message Age (2 bytes): indicates the amount of time since the root sent the BPDU

Max Age (2 bytes): indicates when the current BPDU should be deleted

Hello Time (2 bytes): indicates how often BPDUs are sent

Forward Delay (2 bytes): indicates the time a switch should wait before transitioning to a new state after a topology change

There are, in fact, two types of BPDUs:
  • Configuration BPDU: "normal" BPDU, used for STP computation
  • Topology Change Notification (TCN) BPDU: used to announce changes in the topology

These two BPDUs are differentiated by the TC (Topology Change) and TCA (Topology Change Notification Acknowledgement) bits, which are marked in the Flags field. A TCN BPDU has either the TC or TCA bit set.

The Configuration BPDUs are originated by the root switch only. Non-root switches merely relay the BPDU information downstream out their designated ports. On the contrary, non-root switches can generate TCN BPDUs to signify a topology change. In short, when a topology change occurs, a non-root switch sets the TC bit in the Flags field, and forwards the TCN BPDU upstream (out the Root Port) towards the root switch. The upstream switch then acknowledges the receipt of the TCN BPDU by sending back a BPDU with the TCA bit set (the process will be described in detail in a future post).

STP Timers

STP uses three configurable timers to ensure that a network converges properly without creating switching loops.

Hello Timer - The time interval between sending BPDUs. The value is dictated by the root switch, because only he originates Configuration BPDUs. The default is 2 seconds (1-10 seconds). However, all switches have a local Hello interval which is used to time the (re)transmission of TCN BPDUs.

Forward Delay - The time interval that a port spends in the Listening and Learning states. The default value is 15 seconds (4-30 seconds).

Max Age - The time interval that a switch stores a BPDU before discarding it. Each port keeps a copy of the "best" BPDU it has learned. If the port no longer receives the BPDUs after the Max Age time has elapsed, the switch assumes that a topology change must have occurred, and the BPDU is aged out. The default Max Age value is 20 seconds (6-40 seconds).

The timers should never be changed from their default values without careful consideration. The default STP timer values are derived from a reference model of a network having a diameter of seven switches. It is recommended that the diameter is adjusted to accurately reflect the actual network size, instead of modifying each timer individually. When the diameter is changed, the switch automatically calculates new values for all three timers.

In addition, there is another time-related parameter: Message Age. Unlike the three timers, Message Age is not a fixed value. The Message Age field contains the time that has passed since the root switch initially originated the BPDU. The root switch sends all its BPDUs with a Message Age of 0, and all subsequent switches add 1 to the value. Effectively, the Message Age tells the switch how far he is from the root switch.

Sources

CCNP Exam Prep: Traditional Spanning Tree Protocol
Frame Formats
Understanding and Tuning Spanning Tree Protocol Timers




Monday, November 3, 2014

Troubleshooting IPsec VPN

Now that I have introduced the theory and configuration of IPsec VPNs in my previous posts, it's time to dive into troubleshooting and examine the most common VPN problems. In fact, troubleshooting VPNs is one of the most difficult tasks in network administration. There are so many factors that can cause the IKE negotiations to fail and the tunnel not establishing.

I would strongly advice against looking at the running configuration. It is easy to become blind to your own mistakes, and, as a result, impossible to pinpoint minor configuration errors. Using debugs will provide more intuitive information, but there is a price. The output is noisy and can overwhelm the device as well as the reader.

Now, let's look at some common problems (not an exhaustive list). Again, using the same example topology.


Testing Connectivity

Well, this is not exactly a problem but a common mishap. It's possible that the configuration is correct but the administrator is improperly testing the connection. Ideally, the VPN connection is tested from the end devices inside the local networks. However, sometimes administrators try to verify the tunnel creation by pinging between the peer devices. In this case, the pings will not be encrypted, nor should they be. The key is the proxy IDs, which defines the "interesting traffic" to be encrypted and initially triggers the tunnel creation. If the ping is issued from the peer itself, by default, it will source the packets from the closest interface towards the destination. Referring to the topology, the ping from ASA01 would contain the following information:
  • source: 192.0.2.1
  • destination 198.51.100.1

This does not match the crypto ACL.

access-list L2L extended permit ip 10.1.1.0 255.255.255.0 192.168.1.0 255.255.255.0

Instead, the ping must be manually sourced from the local LAN interface and destined towards the remote LAN.

ASA01# ping inside 192.168.1.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.1.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/10 ms

ASA01# show crypto ipsec sa detail
interface: outside
    Crypto map tag: DEMOVPN, seq num: 10, local addr: 192.0.2.1

      access-list L2L extended permit ip 10.1.1.0 255.255.255.0 192.168.1.0 255.255.255.0
      local ident (addr/mask/prot/port): (10.1.1.0/255.255.255.0/0/0)
      remote ident (addr/mask/prot/port): (192.168.1.0/255.255.255.0/0/0)
      current_peer: 198.51.100.1

      #pkts encaps: 34, #pkts encrypt: 34, #pkts digest: 34
      #pkts decaps: 34, #pkts decrypt: 34, #pkts verify: 34

[...]

IKE Phase 1 Fail

If a VPN tunnel does not come up, verifying Phase 1 is typically the first troubleshooting step. The show crypto isakmp sa is a handy command for this purpose.

ASA01# show crypto isakmp sa

IKEv1 SAs:

   Active SA: 1
    Rekey SA: 0 (A tunnel will report 1 Active and 1 Rekey SA during rekey)
Total IKE SA: 1

1   IKE Peer: 198.51.100.1
    Type    : user            Role    : initiator
    Rekey   : no              State   : MM_WAIT_MSG2

The output indicates that the Initiator is stuck in MM_WAIT_MSG2 state. This only means that the Initiator has sent its IKE policies to the peer, but has not received a response. Therefore, there are several potential causes, for example the Receiver having no return route/link down, or simply ISAKMP not enabled on the correct interface. It's impossible to tell. Even the debugs (debug crypto isakmp sa and debug crypto ipsec sa) do not provide much further assistance, but merely state that the tunnel establishment failed.

ASA01#
%ASA-4-752012: IKEv1 was unsuccessful at setting up a tunnel.  Map Tag = DEMOVPN.  Map Sequence Number = 10.
%ASA-3-752015: Tunnel Manager has failed to establish an L2L SA.  All configured IKE versions failed to establish the tunnel. Map Tag= DEMOVPN.  Map Sequence Number = 10.

Let's see, if ASA02 offers more clues.

ASA02#
%ASA-7-713906: IP = 192.0.2.1, All SA proposals found unacceptable

Indeed, it does. 'All SA proposals found unacceptable' is the line to look for in the noisy debug output. It clearly indicates that the peers could not agree on the IKE policy to negotiate the IKE Phase 1 SA. ASA02 did not accept the proposal(s) sent by ASA01.

So let's verify the IKE policies.

ASA01# show run crypto ikev1
crypto ikev1 policy 10
 authentication pre-share
 encryption des
 hash sha
 group 2
 lifetime 86400


ASA02# show run crypto isakmp
crypto isakmp policy 15
 authentication pre-share
 encryption aes
 hash sha
 group 2
 lifetime 86400


The encryption methods do not match (des != aes).


IKE Phase 2 Fail

How does a failure in IPsec SA negotiation appear in the debug? Let's find out.

ASA01#
%ASA-5-713119: Group = 198.51.100.1, IP = 198.51.100.1, PHASE 1 COMPLETED
%ASA-7-715046: Group = 198.51.100.1, IP = 198.51.100.1, constructing IPSec SA payload
[...]
%ASA-4-752012: IKEv1 was unsuccessful at setting up a tunnel.  Map Tag = DEMOVPN.  Map Sequence Number = 10.
%ASA-3-752015: Tunnel Manager has failed to establish an L2L SA.  All configured IKE versions failed to establish the tunnel. Map Tag= DEMOVPN.  Map Sequence Number = 10.

Again, the output on the Initiator (ASA01) is not very intuitive. However, it does state that Phase 1 completed successfully, which is a clear indication that the problem occurs later in the process. What says ASA02?

ASA02#
%ASA-7-715047: Group = 192.0.2.1, IP = 192.0.2.1, processing IPSec SA payload
%ASA-5-713904: Group = 192.0.2.1, IP = 192.0.2.1, All IPSec SA proposals found unacceptable!

Ha! ASA02 complains that it does not like the IPsec SA proposal(s) from ASA01. Let's compare the configuration.

ASA01# show run crypto map
crypto map DEMOVPN 10 match address L2L
crypto map DEMOVPN 10 set peer 198.51.100.1
crypto map DEMOVPN 10 set ikev1 transform-set X
crypto map DEMOVPN interface outside


ASA01# show run | include transform-set
crypto ipsec ikev1 transform-set X esp-aes-256 esp-md5-hmac


 
ASA02# show run crypto map
crypto map VPN-CRYPTOMAP 10 match address VPN-TRAFFIC
crypto map VPN-CRYPTOMAP 10 set peer 192.0.2.1
crypto map VPN-CRYPTOMAP 10 set transform-set TRANSET
crypto map VPN-CRYPTOMAP interface outside


ASA02# show run | include transform-set
crypto ipsec transform-set TRANSET esp-aes-192 esp-md5-hmac


So ASA01 wants to use AES-256 for encryption, but ASA01 will only accept AES-192.

Mismatched Pre-Shared Keys

Unlike the other problems demonstrated so far, troubleshooting mismatched pre-shared keys is straightforward. Even without issuing the debugs, the syslog messages notify the administrator that a wrong password is a likely cause.

ASA01# ping inside 192.168.1.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.1.1, timeout is 2 seconds:
Oct 31 12:32:22 [IKEv1]Group = 198.51.100.1, IP = 198.51.100.1, Error, peer has indicated that something is wrong with our message.  This could indicate a pre-shared key mismatch.
.....


ASA02# show crypto isakmp sa

   Active SA: 1
    Rekey SA: 0 (A tunnel will report 1 Active and 1 Rekey SA during rekey)
Total IKE SA: 1

1   IKE Peer: 192.0.2.1
    Type    : L2L             Role    : responder
    Rekey   : no              State   : MM_WAIT_MSG5

Oct 31 15:59:35 [IKEv1]: Group = 192.0.2.1, IP = 192.0.2.1, ERROR, had problems decrypting packet, probably due to mismatched pre-shared key.  Aborting

The ISAKMP state indicates that the peers are exchanging pre-shared key hashes, and if they don't match, the Receiver will remain in the MM_WAIT_MSG5 state.

Verifying the pre-shared keys from the command line on the ASA is challenging because they are stored in a hidden format. The only way to ensure they match is to retype them on both ends.

ASA01# show run tunnel-group
tunnel-group 198.51.100.1 type ipsec-l2l
tunnel-group 198.51.100.1 ipsec-attributes
 ikev1 pre-shared-key *****


ASA02# show run tunnel-group
tunnel-group 192.0.2.1 type ipsec-l2l
tunnel-group 192.0.2.1 ipsec-attributes
 pre-shared-key *****


NOTE: The pre-shared key is not five asterisks.

Wrong proxy IDs

As discussed in my previous posts, the proxy IDs define the local and remote networks between which traffic is to be encrypted. The proxy IDs must be mirrored between the peers. So what happens when they are not?

ASA01#
%ASA-5-713119: Group = 198.51.100.1, IP = 198.51.100.1, PHASE 1 COMPLETED
%ASA-7-715009: Group = 198.51.100.1, IP = 198.51.100.1, IKE Deleting SA: Remote Proxy 192.168.1.0, Local Proxy 10.1.1.0
%ASA-4-752012: IKEv1 was unsuccessful at setting up a tunnel.  Map Tag = DEMOVPN.  Map Sequence Number = 10.
%ASA-3-752015: Tunnel Manager has failed to establish an L2L SA.  All configured IKE versions failed to establish the tunnel. Map Tag= DEMOVPN.  Map Sequence Number = 10.

Again, the debug on the Initiator merely points out that Phase 1 completed successfully, but Phase 2 negotiations failed.

ASA02#
%ASA-5-713119: Group = 192.0.2.1, IP = 192.0.2.1, PHASE 1 COMPLETED
%ASA-7-713222: Group = 192.0.2.1, IP = 192.0.2.1, Static Crypto Map check, map = VPN-CRYPTOMAP, seq = 10, ACL does not match proxy IDs src:10.1.1.0 dst:192.168.1.0
%ASA-3-713061: Group = 192.0.2.1, IP = 192.0.2.1, Rejecting IPSec tunnel: no matching crypto map entry for remote proxy 10.1.1.0/255.255.255.0/0/0 local proxy 192.168.1.0/255.255.255.0/0/0 on interface outside

On the contrary, ASA02 distinctly says that there is a problem with the proxy IDs. Let's look at the access-lists.

ASA01# show run crypto map
crypto map DEMOVPN 10 match address L2L
crypto map DEMOVPN 10 set peer 198.51.100.1
crypto map DEMOVPN 10 set ikev1 transform-set ESP-AES192-MD5
crypto map DEMOVPN interface outside


ASA01# show run access-list
access-list L2L extended permit ip 10.1.1.0 255.255.255.0 192.168.1.0 255.255.255.0


ASA02# show run crypto map
crypto map VPN-CRYPTOMAP 10 match address L2L
crypto map VPN-CRYPTOMAP 10 set peer 192.0.2.1
crypto map VPN-CRYPTOMAP 10 set transform-set TRANSET
crypto map VPN-CRYPTOMAP interface outside


ASA02# show run access-list
access-list L2L extended permit ip 10.1.1.0 255.255.255.0 192.168.1.0 255.255.255.0


So what is the problem? The access-list on ASA02 was directly copied from ASA01 without mirroring it.