Connecting the world…

ARP

McAfee Firewall – NAT mapping

While testing a McAfee Enterprise Firewall running software 8.2.0, I had some problems with the creation of a NAT mapping. The firewall is configured as standalone firewall. All (NAT / access rule) configuration on the firewall is done using Access Control Rules. McAfee uses two types of NAT mapping:

  1. NAT: mostly used to translate a private IP address to a public IP address;
  2. Redirect: redirect traffic to a public IP address to a private IP address;

I tried to publish an internal network component to the internet. I created a simple rule with the following parameters. These parameters are very straightforward and the configuration is similar to firewalls from different vendors:

Application: SSH Source Zone:
external
Destination Zone:
external
Source Endpoint:
Any
Destination Endpoint:
Public IP address
NAT address:
None
Redirect:
Private IP address

 

I tested the NAT mapping, but couldn’t connect to the internal component using the public IP address. The first step in troubleshooting is looking at the logging, but I couldn’t find any logging on the firewall. It looked like the traffic didn’t even reach the firewall.

We have a shared internet segment with multiple firewalls. So I started doubting the configuration of the different firewalls.

  • Was somebody already using the public IP address in a NAT configuration?
  • Has the default gateway of the internet segment already an ARP entry for the public IP address?

I looked at the configuration of the firewalls, but nobody was using the public IP address. With this in mind, I ruled out the ARP entry “problems” on the ISP router.

When using NAT on a public IP address, which isn’t the same as the interface IP address, the firewall has to proxy ARP the public IP address. So does the firewall proxy ARP for the public IP address?

I started looking at the rest of the configuration with emphasis on the network configuration. I noticed that I had the option to add an alias IP address to the external interface. This can be found under Network – Interfaces – external interface. I added the public IP address as alias.

You guessed it. The NAT mapping is working……

Cisco ASA & ESX: strange ARP behavior

Last week I had a very strange problem with a Cisco ASA firewall. The firewall is configured with multiple interfaces, including a DMZ interface. There are multiple servers in the DMZ. These servers are physical and virtual servers. The virtual servers are VMware servers in a blade environment.

I configured the feature

ip verify reverse-path interface DMZ

to prevent spoofing to occur. I also configured a transparent static NAT rule between the Inside network and the DMZ network and multiple static NAT rules between the DMZ network and the Outside network. I left the proxy ARP feature configured with its default settings.

The customer was complaining about log in problems and connectivity problems on the DMZ servers, especially between different DMZ servers. I have done some research and noticed that all problems were related to DMZ servers in the blade environment.

I started some connectivity test and noticed some strange ICMP behavior on the specific servers. When I started a ping from one DMZ VMware server to an other DMZ server on the same ESX host, the first ping responded with an echo-reply, but consequent pings failed. Looking at the ARP table of the server, I noticed that the firewall responded with its own MAC address for every ARP broadcast.

Looking at different forums on the Internet, everybody is speaking about the proxy ARP feature and that you should disable this feature. By default proxy ARP is enabled and I always leave it enabled. Till now I never had this problem. After disabling the proxy ARP feature for the DMZ interface

sysopt noproxyarp DMZ

the problem was solved, because the firewall doesn’t respond to the ARP queries, except for its own interface. Digging a bit deeper on forums, I never found one thread who explains why the proxy ARP feature should be disabled to solve this particular problem.

In my opinion this problem is related to the VMware environment, because I don’t have these problems with physical DMZ servers. So it is strange why the DMZ servers on the same ESX hosts cannot see each other and why does the firewall respond to the ARP queries?

In the near future the blade environment (ESX hosts, network configuration and SAN configuration) is changed, so I hope to find the exact cause and solution of the problem. Does anybody else have some suggestions??

Layer 2 security

I attended the session layer 2 security, because I had some discussions about layer 2 security with one of my colleagues. We were discussing about using layer 2 security and especially implementing it in the environments from our customers.

Looking at my/our customers, I don’t see environments where layer 2 threats would be immediate. But in my opinion, you will never know when it will be immediate. I prefer to implement as much layer 2 security (and security in general) as possible. Maybe I am a little paranoia ;-).

I was very interested in the countermeasures used in active networks and the caveats to these countermeasures. Looking at layer 2 networks, the following topics these are the most common attacks.

VLAN HOPPING

Connections between two switches are mostly configured as trunk ports (IEEE 802.1Q). All VLAN travel these trunk connections. Cisco switches have the capability to dynamically negotiate a trunk port. This means that all edge ports can become trunk ports. When an attacker can spoof to be switch, he could configure a trunk connection with the switch. By configuring a trunk connections with the switch, the attacker has access to all VLAN configured on the trunk. You have to disable the DTP (Dynamic Trunking Protocol) on the edge ports to mitigate these attacks. DTP is disabled with the command switchport nonegotiate. When tagging VLAN’s to a trunk it is preferred to use VLAN allowed lists on the trunks. This prevents that all VLAN’s have access to the trunk, only the specified VLAN’s will have access to the trunk.

MAC ADDRESS ATTACKS

Flooding the CAM table with bogus entries makes a hub out of a switch. When the CAM table is full, all traffic without an entry in the CAM table will be flooded out every port on the switch. This means that an attacker can intercept all traffic from the switch. Flooding the CAM table of a switch can also resultant in full CAM tables on adjacent switches. Countermeasures against CAM flooding attacks is the use of port security. To fill the CAM table an attacker will send a lot of different bogus MAC address to the switch. By limiting the allowed MAC address on a switch port, this attack can be prevented. When limiting the number of allowed MAC addresses on a port, you should pay attention to the number of MAC address you configure. Think of features like CDP, LLDP (Link Layer Discovery Protocol), IP Phones, VMware and so one. These features could generate additional MAC entries on the specific switch port. An example configuration for port security is shown below:

switchport port-security
switchport port-security maximum 1 vlan voice
switchport port-security maximum 1 vlan access
switchport port-security violation restrict
switchport port-security aging time 2
switchport port-security aging type inactivity

DHCP ATTACK

DHCP is used in networks to dynamically assign IP addresses and other options to clients. A DHCP scope contains a range of IP addresses and options, which can be assigned to clients. A known attack against DHCP is DHCP starvation. During a DHCP starvation attack an attacker tries to lease all different IP addresses. Most of the tools on the Internet for DHCP starvation use one MAC address for every lease. This means that an attacker uses multiple MAC addresses on the switch port where he connects to the network. Knowing this, you already know the solution to this attack. Just like the MAC address attacks, the solution lies in only allowing a predefined number of MAC addresses on the switch port. There are tools which use the same MAC address, but change Client Hardware Address (CHADDR) in an IPv4 DHCP Packet. This kind of attacks are mitigated by using DHCP Snooping.

Another DHCP attack is adding an untrusted or rogue DHCP server to the network. This way an attacker can assign the “wrong” IP addresses and options, like gateway and DNS servers. This kind of attacks is also mitigated by using DHCP snooping. When turning on DHCP snooping all switch ports will become untrusted. It is important to trust the port to the real DHCP server. The next snippets show examples for configuring DHCP snooping.

GLOBAL CONFIG

ip dhcp snooping vlan 1,10,100

no ip dhcp snooping information option

ip dhcp snooping

UNTRUSTED CLIENT INTERFACE CONFIG

no ip dhcp snooping trust

ip dhcp snooping limit rate 10

TRUSTED SERVER OR UPLINK INTERFACE CONFIG

ip dhcp snooping trust

By enabling DHCP snooping the switch starts building a DHCP snooping binding table. The DHCP snooping binding table is crucial when using DHCP snooping. This table contains the mappings of MAC, IP address, lease time, type of packet, VLAN and snooped interface, like the example below.

DHCP Snooping

The most clients do another DHCP request in the event of a link down and link up event, but not all clients to. For example some Linux systems won’t re-DHCP in the event of link down and link up. When a switch reboots, the DHCP snooping binding table will be lost. If a client doesn’t re-DHCP, the client is denied access to the network, because there is no entry in the DHCP snooping binding table. This means that it is important to backup the DHCP snooping binding table in the event of a switch failure. The DHCP snooping binding table can be written to bootflash, ftp, rcp, slot0 and tftp, like the example shows.

ip dhcp snooping database tftp://10.10.1.1/tftpboot/dhcp-snooping-table

ip dhcp snooping database write-delay 60

Entries in the DHCP snooping binding table stay there until the lease timer expires. When you can a real mobile network it is advised to tune the DHCP lease timers.

ARP ATTACKS

An ARP entry maps a MAC address to an IP address. An attacker can claim, by poisoning the ARP table, to be for example the default gateway of the subnet. This is achieved by replying and poisoning the network by “telling” that the attackers MAC address should be mapped to the default gateway IP address. This way the attacker receives all the traffic designated to the default gateway, which gives the attacker the possibility to perform a man-in-the-middle attack.

An ARP attack can be mitigated by the use of Dynamic ARP Inspection. By using Dynamic ARP Inspection (DIA) the switch checks the IP/MAC mappings in the DHCP snooping binding table. This implies that DHCP snooping is needed for DIA. Another method of mitigating ARP attacks is by checking the source and/or destination MAC addresses and/or IP addresses.

GLOBAL CONFIG

ip dhcp snooping vlan 4,104
no ip dhcp snooping information option
ip dhcp snooping
ip arp inspection vlan 4,104
ip arp inspection log-buffer entries 1024
ip arp inspection log-buffer logs 1024 interval 10

UNTRUSTED CLIENT INTERFACE CONFIG

no ip dhcp snooping trust

ip dhcp snooping limit rate 10

no ip arp inspection trust

TRUSTED SERVER OR UPLINK INTERFACE CONFIG

ip dhcp snooping trust
ip arp inspection trust

ENABLE ALL VALIDATION COMMANDS

ip arp inspection validate src-mac dst-mac ip

SPOOFING ATTACKS

In an spoofing attack the attacker uses the MAC address or IP address of a “real” networking component. Spoofing attacks are performed for the following reasons:

MAC SPOOFING

  • when using MAC authentication on the network the attacker can gain access to the network;
  • the attacker takes over the identity of someone already known on the network;

IP SPOOFING

  • Ping of death attack;
  • ICMP unreachable storm;
  • SYN floods on the network;
  • the attacker takes over the identity of someone already known on the network;

The countermeasure for spoofing attacks is the use of IP Source Guard. IP Source Guard can be compared with Dynamic ARP Inspection, the difference lies in the fact that IP Source Guard checks every packet and not only ARP packets.

These are the most common layer 2 attacks known today. There are some more, like attacks on the Spanning Tree Protocol. These attacks can be mitigated with techniques like BPDUGuard and RootGuard.

I will definitely start using more of the techniques mentioned above in customer environments. It will be a real challenge to implement some of the techniques without disrupting the daily work on the network.

WebMarshal performance problems

One of our customers is using WebMarshal for HTTP/HTTPS URL filtering and content scanning. The WebMarshall software is installed on two Microsoft ISA 2003 servers. These ISA servers are behind a Cisco Content Switch for load-balancing and redundancy purposes.

The problem with the WebMarshal is the PERFORMANCE. Internet browsing with the WebMarshal as proxy just doesn’t perform. I tried to troubleshoot the WebMarshal to check where the performance problems are coming from, but you cannot troubleshoot the software on a decent way. I disabled the Access Policies, and guess what, the performance is great. I added a allow all rule on top of every Access Policy subcategory, but no success.

I know the customer is running an old version (3.0.x), and of course if you contact the supplier, the first thing they say is: “Upgrade to the last version!!”. It seems the solution is always upgrading the last version. The second thing the supplier told us, was using Microsoft Network Load Balancing and not the Content Switches. Sadly the customer is using HP ProCurve switches, which don’t support static ARP entries. So NLB is no option.

But again, I give them the benefit of the doubt, so we will install two new servers, which are dedicated for WebMarshal software. Still the servers will be behind the Content Switch, because I believe that the Content Switches are the reason for the bad performance.

I will tell you more about the outcome of the latest version of WebMarshal on dedicated hardware. My opinion so far: “Feed the WebMarshal software to the dogs and buy something else!!!!!!!!!!”