Last week I had a very strange problem with a Cisco ASA firewall. The firewall is configured with multiple interfaces, including a DMZ interface. There are multiple servers in the DMZ. These servers are physical and virtual servers. The virtual servers are VMware servers in a blade environment.
I configured the feature
ip verify reverse-path interface DMZ
to prevent spoofing to occur. I also configured a transparent static NAT rule between the Inside network and the DMZ network and multiple static NAT rules between the DMZ network and the Outside network. I left the proxy ARP feature configured with its default settings.
The customer was complaining about log in problems and connectivity problems on the DMZ servers, especially between different DMZ servers. I have done some research and noticed that all problems were related to DMZ servers in the blade environment.
I started some connectivity test and noticed some strange ICMP behavior on the specific servers. When I started a ping from one DMZ VMware server to an other DMZ server on the same ESX host, the first ping responded with an echo-reply, but consequent pings failed. Looking at the ARP table of the server, I noticed that the firewall responded with its own MAC address for every ARP broadcast.
Looking at different forums on the Internet, everybody is speaking about the proxy ARP feature and that you should disable this feature. By default proxy ARP is enabled and I always leave it enabled. Till now I never had this problem. After disabling the proxy ARP feature for the DMZ interface
sysopt noproxyarp DMZ
the problem was solved, because the firewall doesn’t respond to the ARP queries, except for its own interface. Digging a bit deeper on forums, I never found one thread who explains why the proxy ARP feature should be disabled to solve this particular problem.
In my opinion this problem is related to the VMware environment, because I don’t have these problems with physical DMZ servers. So it is strange why the DMZ servers on the same ESX hosts cannot see each other and why does the firewall respond to the ARP queries?
In the near future the blade environment (ESX hosts, network configuration and SAN configuration) is changed, so I hope to find the exact cause and solution of the problem. Does anybody else have some suggestions??
When configuring a Remote Access VPN or a Site to Site VPN connection you have the ability to filter traffic entering and leaving the VPN connection. You have the ability to enable inbound IPsec sessions to bypass interface access lists. Group policy and per-user authorization access lists still apply to the traffic.
The sysopt connection permit-ipsec command allows all the traffic that enters the security appliance through a VPN tunnel to bypass interface access lists. Group policy and per-user authorization access lists still apply to the traffic. In PIX 7.1 and later, the sysopt connection permit-ipsec command is changed to sysopt connection permit-vpn.
Mostly I use this option and configure some extra ACL’s to filter trafifc. Some customers don’t want to use this option and want to specify all traffic with ACL’s. This is more secure, but is a bigger burden on the management of the firewall.
From IOS 7.1 and later you have the ability to configure VPN filtering through Group Policies. In short you configure an extended ACL, link this ACL to a Group Policy and link the Group Policy to the specific Tunnel Group. The syntax (source and destination) needs to be correct for the ACL to work.
For Site to Site VPN’s the remote network is the source and the local network is the destination. For Remote Access VPN’s the VPN IP pool is the source and the local network the destination, as specified below.
An ACL that is used for a vpn-filter must not also be used for an interface access-group. When a vpn-filter is applied to a group-policy/user name mode that governs Remote Access VPN Client connections, the ACL must be configured with the client assigned IP addresses in the src_ip position of the ACL and the local network in the dest_ip position of the ACL. When a vpn-filter is applied to a group-policy that governs an L2L VPN connection, the ACL must be configured with the remote network in the src_ip position of the ACL and the local network in the dest_ip position of the ACL.
Exercise caution when you construct the ACLs for use with the vpn-filter feature. The ACLs are constructed with the post-decrypted traffic (inbound VPN traffic) in mind. However, they are also applied to the traffic originated in the opposite direction.
More about this matter and examples configurations can be found here.
In different scenario’s it is required to configure some kind of routing protocol between two offices, but the routers should be configured to look directly connected to each other. Normally I always configure an IPsec VPN between the two offices and configure an additional GRE tunnel over the IPsec VPN tunnel. In that way the routers look directly connected and adding a routing protocol is no problem.
In the past I noticed several times that the GRE tunnel doesn’t come up, when using a Cisco PIX firewall or a Cisco ASA firewall. When using IOS 6.x on the PIX or 7.x on both hardware platforms, there is a workaround by using the following command:
clear local-host <remote peer>
Cisco has reported this bug in BugID CSCse36327:
The IPSEC tunnel was previously working and either one of the following events occured:
1. the crypto map and/or isakmp has been removed and reapplied to the interface
2. the PIX/ASA is upgraded from version 6.x to version 7.x
3. the PIX/ASA is rebooted
4. The remote IPSEC peer/s is rebooted
All events except 1 occur when a dynamic crypto map is used without a match address statement.
This typically affects only GRE traffic.
In PIX/ASA 7.x, GRE encryption may stop working (GRE packets are sent in clear) after removing and reapplying the encryption. This behaviour is by design in 7.x. If encryption is disabled but GRE packets are coming to the PIX in this time, GRE session is created on the PIX and marked as clear-text one (“do not encrypt”). When encryption is applied back, non-encrypted GRE session still exists on PIX and GRE packets that should be encrypted still bypass crypto map until old session is timed out or deleted. If there is a dynamic routing (OSPF/EIGRP/etc) running over GRE, this GRE session may never timeout and should be cleared manually.
In PIX/ASA 8.0.2, new functionality was introduced with new CLI command: “sysopt connection reclassify-vpn”. Default state is disabled. If this command is enabled, then enabling encryption causes non-encryption sessions to be dropped and reestablished with encryption.
Looks like there is a new command introduced in IOS 8.0.2 as mentioned above, by using sysopt connection reclassify-vpn.
There is also an entry on the Cisco SupportWiki about this problem. So the next time I will try this new command.