Connecting the world…


Cisco ASA & ESX: strange ARP behavior

Last week I had a very strange problem with a Cisco ASA firewall. The firewall is configured with multiple interfaces, including a DMZ interface. There are multiple servers in the DMZ. These servers are physical and virtual servers. The virtual servers are VMware servers in a blade environment.

I configured the feature

ip verify reverse-path interface DMZ

to prevent spoofing to occur. I also configured a transparent static NAT rule between the Inside network and the DMZ network and multiple static NAT rules between the DMZ network and the Outside network. I left the proxy ARP feature configured with its default settings.

The customer was complaining about log in problems and connectivity problems on the DMZ servers, especially between different DMZ servers. I have done some research and noticed that all problems were related to DMZ servers in the blade environment.

I started some connectivity test and noticed some strange ICMP behavior on the specific servers. When I started a ping from one DMZ VMware server to an other DMZ server on the same ESX host, the first ping responded with an echo-reply, but consequent pings failed. Looking at the ARP table of the server, I noticed that the firewall responded with its own MAC address for every ARP broadcast.

Looking at different forums on the Internet, everybody is speaking about the proxy ARP feature and that you should disable this feature. By default proxy ARP is enabled and I always leave it enabled. Till now I never had this problem. After disabling the proxy ARP feature for the DMZ interface

sysopt noproxyarp DMZ

the problem was solved, because the firewall doesn’t respond to the ARP queries, except for its own interface. Digging a bit deeper on forums, I never found one thread who explains why the proxy ARP feature should be disabled to solve this particular problem.

In my opinion this problem is related to the VMware environment, because I don’t have these problems with physical DMZ servers. So it is strange why the DMZ servers on the same ESX hosts cannot see each other and why does the firewall respond to the ARP queries?

In the near future the blade environment (ESX hosts, network configuration and SAN configuration) is changed, so I hope to find the exact cause and solution of the problem. Does anybody else have some suggestions??

Secure HSRP configuration

A friend of mine works for a well known auditing and penetration testing company in the Netherlands. Recently we were talking about how he starts looking for flaws in network infrastructures. My friend told me that the first thing he does is simply starting WireShark and start looking at all the packets he receives.

By default packets like DTP (Dynamic Trunking Protocol), CDP (Cisco Discovery Protocol) and HSRP (Hot Standby Routing Protocol) are broadcasted through all the different edge ports of a switch. Tools like Yersinia can be used by hackers to exploit these packets.

Normally when I configure a switch I always stop the broadcasting of DTP and CDP on normal edge ports, at least if possible. CDP is often used in conjunction with IP phones. I prevent broadcasting DTP and CDP with the following commands:

no cdp enable

switchport nonegotiate

To be honest, I never thought about the broadcasting of HSRP packets. I created a simple test environment with one Cisco Catalyst 3750G switch and configured VLAN 1 with HSRP, like shown below.

interface Vlan1
ip address
standby 1 ip
standby 1 priority 150
standby 1 preempt

This is the most default way of configuring HSRP. By using a tool like Yersinia, somebody could take over the role of active HSRP router by spoofing HSRP packets with a higher priority then the current active HSRP router. So I added a simple authentication text string to the configuration with the following command:

standby 1 authentication HSRP@ICT

This is no success, because when I start WireShark the authentication string is sent in clear text. The picture below shows an example:


In most recent software version you can protect HSRP by using MD5 Authentication. MD5 authentication provides greater security than plain text authentication. This feature allows each HSRP group member to use a secret key to generate a keyed MD5 hash of the packet that is part of the outgoing packet. A keyed hash of an incoming packet is generated and if the generated hash does not match the hash within the incoming packet, the packet is ignored.

To configure MD5 authentication in the previous example, I added the following configuration to interface VLAN 1:

standby 1 authentication md5 key-string hsrp@ictivity=secure,Ihope timeout 60

Now, when looking at the WireShark output, the key-string is composed of a hash and cannot be easily read by an hacker.


The timeout option is important when configuring a new key-string amongst all the members in an HSRP group. The timeout value is the period of time that the old key string will be accepted to allow configuration of all routers in a group with a new key.

So HSRP MD5 Authentication is another way of making our network components and network infrastructure more secure against “evil” attacks and hackers.

Layer 2 security

I attended the session layer 2 security, because I had some discussions about layer 2 security with one of my colleagues. We were discussing about using layer 2 security and especially implementing it in the environments from our customers.

Looking at my/our customers, I don’t see environments where layer 2 threats would be immediate. But in my opinion, you will never know when it will be immediate. I prefer to implement as much layer 2 security (and security in general) as possible. Maybe I am a little paranoia ;-).

I was very interested in the countermeasures used in active networks and the caveats to these countermeasures. Looking at layer 2 networks, the following topics these are the most common attacks.


Connections between two switches are mostly configured as trunk ports (IEEE 802.1Q). All VLAN travel these trunk connections. Cisco switches have the capability to dynamically negotiate a trunk port. This means that all edge ports can become trunk ports. When an attacker can spoof to be switch, he could configure a trunk connection with the switch. By configuring a trunk connections with the switch, the attacker has access to all VLAN configured on the trunk. You have to disable the DTP (Dynamic Trunking Protocol) on the edge ports to mitigate these attacks. DTP is disabled with the command switchport nonegotiate. When tagging VLAN’s to a trunk it is preferred to use VLAN allowed lists on the trunks. This prevents that all VLAN’s have access to the trunk, only the specified VLAN’s will have access to the trunk.


Flooding the CAM table with bogus entries makes a hub out of a switch. When the CAM table is full, all traffic without an entry in the CAM table will be flooded out every port on the switch. This means that an attacker can intercept all traffic from the switch. Flooding the CAM table of a switch can also resultant in full CAM tables on adjacent switches. Countermeasures against CAM flooding attacks is the use of port security. To fill the CAM table an attacker will send a lot of different bogus MAC address to the switch. By limiting the allowed MAC address on a switch port, this attack can be prevented. When limiting the number of allowed MAC addresses on a port, you should pay attention to the number of MAC address you configure. Think of features like CDP, LLDP (Link Layer Discovery Protocol), IP Phones, VMware and so one. These features could generate additional MAC entries on the specific switch port. An example configuration for port security is shown below:

switchport port-security
switchport port-security maximum 1 vlan voice
switchport port-security maximum 1 vlan access
switchport port-security violation restrict
switchport port-security aging time 2
switchport port-security aging type inactivity


DHCP is used in networks to dynamically assign IP addresses and other options to clients. A DHCP scope contains a range of IP addresses and options, which can be assigned to clients. A known attack against DHCP is DHCP starvation. During a DHCP starvation attack an attacker tries to lease all different IP addresses. Most of the tools on the Internet for DHCP starvation use one MAC address for every lease. This means that an attacker uses multiple MAC addresses on the switch port where he connects to the network. Knowing this, you already know the solution to this attack. Just like the MAC address attacks, the solution lies in only allowing a predefined number of MAC addresses on the switch port. There are tools which use the same MAC address, but change Client Hardware Address (CHADDR) in an IPv4 DHCP Packet. This kind of attacks are mitigated by using DHCP Snooping.

Another DHCP attack is adding an untrusted or rogue DHCP server to the network. This way an attacker can assign the “wrong” IP addresses and options, like gateway and DNS servers. This kind of attacks is also mitigated by using DHCP snooping. When turning on DHCP snooping all switch ports will become untrusted. It is important to trust the port to the real DHCP server. The next snippets show examples for configuring DHCP snooping.


ip dhcp snooping vlan 1,10,100

no ip dhcp snooping information option

ip dhcp snooping


no ip dhcp snooping trust

ip dhcp snooping limit rate 10


ip dhcp snooping trust

By enabling DHCP snooping the switch starts building a DHCP snooping binding table. The DHCP snooping binding table is crucial when using DHCP snooping. This table contains the mappings of MAC, IP address, lease time, type of packet, VLAN and snooped interface, like the example below.

DHCP Snooping

The most clients do another DHCP request in the event of a link down and link up event, but not all clients to. For example some Linux systems won’t re-DHCP in the event of link down and link up. When a switch reboots, the DHCP snooping binding table will be lost. If a client doesn’t re-DHCP, the client is denied access to the network, because there is no entry in the DHCP snooping binding table. This means that it is important to backup the DHCP snooping binding table in the event of a switch failure. The DHCP snooping binding table can be written to bootflash, ftp, rcp, slot0 and tftp, like the example shows.

ip dhcp snooping database tftp://

ip dhcp snooping database write-delay 60

Entries in the DHCP snooping binding table stay there until the lease timer expires. When you can a real mobile network it is advised to tune the DHCP lease timers.


An ARP entry maps a MAC address to an IP address. An attacker can claim, by poisoning the ARP table, to be for example the default gateway of the subnet. This is achieved by replying and poisoning the network by “telling” that the attackers MAC address should be mapped to the default gateway IP address. This way the attacker receives all the traffic designated to the default gateway, which gives the attacker the possibility to perform a man-in-the-middle attack.

An ARP attack can be mitigated by the use of Dynamic ARP Inspection. By using Dynamic ARP Inspection (DIA) the switch checks the IP/MAC mappings in the DHCP snooping binding table. This implies that DHCP snooping is needed for DIA. Another method of mitigating ARP attacks is by checking the source and/or destination MAC addresses and/or IP addresses.


ip dhcp snooping vlan 4,104
no ip dhcp snooping information option
ip dhcp snooping
ip arp inspection vlan 4,104
ip arp inspection log-buffer entries 1024
ip arp inspection log-buffer logs 1024 interval 10


no ip dhcp snooping trust

ip dhcp snooping limit rate 10

no ip arp inspection trust


ip dhcp snooping trust
ip arp inspection trust


ip arp inspection validate src-mac dst-mac ip


In an spoofing attack the attacker uses the MAC address or IP address of a “real” networking component. Spoofing attacks are performed for the following reasons:


  • when using MAC authentication on the network the attacker can gain access to the network;
  • the attacker takes over the identity of someone already known on the network;


  • Ping of death attack;
  • ICMP unreachable storm;
  • SYN floods on the network;
  • the attacker takes over the identity of someone already known on the network;

The countermeasure for spoofing attacks is the use of IP Source Guard. IP Source Guard can be compared with Dynamic ARP Inspection, the difference lies in the fact that IP Source Guard checks every packet and not only ARP packets.

These are the most common layer 2 attacks known today. There are some more, like attacks on the Spanning Tree Protocol. These attacks can be mitigated with techniques like BPDUGuard and RootGuard.

I will definitely start using more of the techniques mentioned above in customer environments. It will be a real challenge to implement some of the techniques without disrupting the daily work on the network.