Connecting the world…

pix

Failed to establish VPN through PIX

We migrated our Internet connection lately and reconfigured our PIX firewall. We added some memory to install the latest firmware version (8.0(4)). After putting the PIX firewall in production some of the employees were complaining they couldn’t establish any PPTP VPN Tunnels anymore to customers.

Every time when some one called me, I tried it myself and I was always able to connect using a PPTP VPN Tunnel, but every time I was working remote and not at the office. So I always thought that something was wrong with there laptops, but today I encountered the problem myself.

Looking at the logging of the PIX firewall, I saw the following error message:

%ASA-3-305006: regular translation creation failed for protocol 47 src inside:<IP address> dst outside:<IP address>

The error message indicates that there is no NAT mapping for the specified traffic, which could direct you in the wrong direction. I checked the NAT mappings to be sure, but as I already thought, this couldn’t be the cause of the problem.

PPTP uses a TCP connection that uses port 1723 and an extension of generic routing encapsulation (GRE) [protocol 47] to carry the actual data (PPP frame). The TCP connection is initiated by the client, followed by the GRE connection that is initiated by the server. Because the PPTP connection is initiated as TCP on one port and the response is GRE protocol, the PIX Adaptive Security Algorithm (ASA) does not know that the traffic flows are related.

The PPTP fixup feature in version 6.3 allows the PPTP traffic to traverse the PIX when configured for PAT. Stateful PPTP packet inspection is also performed in the process. The fixup protocol pptp command inspects PPTP packets and dynamically creates the GRE connections and translations necessary to permit PPTP traffic. Specifically, the firewall inspects the PPTP version announcements and the outgoing call request/response sequence. Only PPTP Version 1, as defined in RFC 2637, is inspected. Further inspection on the TCP control channel is disabled if the version announced by either side is not Version 1. In addition, the outgoing call request and reply sequence is tracked. Connections and/or translations are dynamically allocated as necessary to permit subsequent secondary GRE data traffic. The PPTP fixup feature must be enabled for PPTP traffic to be translated by PAT.

So I had to configure the fixup protocol pptp feature with the following command:

fw01(config)# fixup protocol pptp 1723

As stated before, we are using fireware version 8.0(4). This version doesn’t support the fixup protocol pptp command and the converts the command an inspect pptp command as shown below.

fw01(config)# fixup protocol pptp 1723
INFO: converting ‘fixup protocol pptp 1723’ to MPF commands

!

!

policy-map global_policy
class inspection_default
  inspect dns preset_dns_map
  inspect ftp
  inspect pptp

PIX / ASA – Threat Detection

From software release 8.0 and later the Cisco PIX and Cisco ASA firewalls support the feature called Threat Detection. In the default configuration Basic Threat Detection is enabled on the security appliance.

Using Threat Detection the appliance monitors the rate of dropped packets and security events due to these reasons (Source):

  • Denial by access lists;
  • Bad packet format (such as invalid-ip-header or invalid tcp-hdr-length);
  • Connection limits exceeded (both system-wide resource limits, and limits set in the configuration);
  • DoS attack detected (such as an invalid SPI, Stateful Firewall check failure);
  • Basic firewall checks failed (This option is a combined rate that includes all firewall-related packet drops in this bulleted list. It does not include non-firewall-related drops such as interface overload, packets failed at application inspection, and scanning attack detected.);
  • Suspicious ICMP packets detected;
  • Packets failed application inspection;
  • Interface overload;
  • Scanning attack detected;
  • Incomplete session detection such as TCP SYN attack detected or no data UDP session attack detected;

When the security appliance detects a threat a syslog message is send. These syslog messages have the following format:

%ASA-4-733100: Object drop rate rate_ID exceeded. Current burst rate is rate_val per second, max configured rate is rate_val; Current average rate is rate_val per second, max configured rate is rate_val; Cumulative total count is total_cnt

Basic Threat Detection affects the performance of the security appliance only when there are drops or potentials threats. I have monitored the CPU with Basic Threat Detection enabled and disabled in an environment with many deny hits on the outside interface, resulting from port scans and (D)DoS attacks. The performance impact on the security appliance is insignificant.

The security appliance has also the option to actively scan all traffic and shun connections if threats rates are exceeded. The security appliance tracks two types of rates:

  1. the average event rate over an interval;
  2. the burst event rate over a shorter burst interval;

More information about configuring Threat Detection can be found in the Cisco Security Appliance Command Line Configuration Guide, 8.0 and more specific the chapter Preventing Network Attacks.

In my own experience with Cisco PIX and Cisco ASA firewalls running software release 8.0 and later, I normally disable Basic Threat Detection. Often I receive questions from customers about the syslog messages generated by Basic Threat Detection. Customers always think that something is terribly wrong with the security appliance. For some customers I enabled Basic Threat Detection in conjunction with the Scanning Threat Statistics. Enabling the statistics give you more detailed information about the discovered threat rates. The statistics can be viewed via the Firewall Dashboard when using ASDM or with various show commands using the CLI. Below the output of the command show threat-detection rate.

Average(eps)    Current(eps) Trigger    Total events
10-min ACL  drop:                  1               0       0            119
21-hour ACL  drop:                  2               1       0          7556
10-min SYN attck:                  0               0       0            436
1-hour SYN attck:                  0               0       0           2863
10-min  Scanning:                 12               9   31963            721
31-hour  Scanning:                 20              11   21622         74264
10-min Bad  pkts:                  0               0       0            107
1-hour Bad  pkts:                  0               0       0            682
10-min  Firewall:                  2               1       0           1299
1-hour  Firewall:                  2               1       0           8238
10-min Interface:                 10               0       0           6314
1-hour Interface:                 10              10       0          37220

Enabling the use of statistics could have a bad influence on the performance from the PIX / ASA. Especially the memory usage can increase enormously. With Basic Thread Detection, there is also an option for actively scanning all traffic and shun the traffic when certain threshold are reached. Shunning the traffic is accomplished by adding a policy rule to the configuration. This rule is added to the configuration automatically and stays even after a reboot.

I talked with a Cisco engineer about this feature and he advized me not to use Basic Thread Detection with the scanning feature. The feature is rather new and needs a lot of tweaking, because this functionality can basically be compared with Intrusion Detection and Intrusion Prevention System. The engineer also stated that the gathering of statistics can have an influence on the memory usage of the box. Therefore he also advized to only use the statistics feature in certain environments and circumstances.

PIX Failover not working

Today I received the question why a PIX failover configuration wasn’t working. The customer accidentally disconnected the power cable from the primary PIX firewall. The secondary PIX firewall became the active one, but multiple DMZ segments weren’t working anymore. After rebooting the PIX firewall and making that the primary one again, the DMZ segments were reachable again.

I started looking at the failover state of the firewall. Entering the commando show failover on the primary PIX resulted in the following output:

This host: Primary – Active
Active time: 5430735 (sec)
Interface outside (1.1.1.1): Normal
Interface inside (2.2.2.1): Normal
Interface DMZ01 (3.3.3.1): Normal
Interface DMZ02 (4.4.4.1): Normal (Waiting)
Interface DMZ03 (5.5.5.1): Normal (Waiting)
Other host: Secondary – Standby Ready
Active time: 5235 (sec)
Interface outside (1.1.1.2): Normal
Interface inside (2.2.2.2): Normal
Interface DMZ01 (3.3.3.2): Normal
Interface DMZ02 (4.4.4.2): Normal (Waiting)
Interface DMZ03 (5.5.5.2): Normal (Waiting)

Interface DMZ02 and DMZ03 were the problem interfaces.

After doing some simple ping test, I noticed the problem interfaces couldn’t ping each other. Looking at the output above the interfaces are in Normal (Waiting) state. Searching the internet resulted in the following:

Failover does not start to monitor the network interfaces until it has heard the second “hello” packet from the other unit on that interface. This takes about 30 seconds. If the unit is attached to a network switch that runs Spanning Tree Protocol (STP), this takes twice the “forward delay” time configured in the switch (typically configured as 15 seconds), plus this 30 second delay. This is because at PIX bootup and immediately following a failover event, the network switch detects a temporary bridge loop. Upon detection of this loop, it stops forwarding packets on these interfaces for the “forward delay” time. It then enters the “listen” mode for an additional “forward delay” time, during which time the switch listens for bridge loops but not forwarding traffic (and thus not forwarding failover “hello” packets). After twice the forward delay time (30 seconds), traffic resumes flowing. Each PIX remains in “waiting” mode until it hears 30 seconds worth of “hello” packets from the other unit. During the time the PIX is passing traffic, it does not fail the other unit based on not hearing the “hello” packets. All other failover monitoring still occurs (that is, Power, Interface Loss of Link, and Failover Cable “hello”).
Source

Here they are talking about the STP configuration. I couldn’t imagine that the problem is lying in the STP configuration. I started tracking the mac-address on the switches to check the physical patching.

The problem was found directly. The patching of the secondary PIX wasn’t correct. The customer had swapped the two interfaces resulting in patching them in the wrong VLAN. After swapping the patches the failover state functions normally, looking at the output below:

This host: Primary – Active
Active time: 5430735 (sec)
Interface outside (1.1.1.1): Normal
Interface inside (2.2.2.1): Normal
Interface DMZ01 (3.3.3.1): Normal
Interface DMZ02 (4.4.4.1): Normal
Interface DMZ03 (5.5.5.1): Normal
Other host: Secondary – Standby Ready
Active time: 5235 (sec)
Interface outside (1.1.1.2): Normal
Interface inside (2.2.2.2): Normal
Interface DMZ01 (3.3.3.2): Normal
Interface DMZ02 (4.4.4.2): Normal
Interface DMZ03 (5.5.5.2): Normal

This proves again how import correct patching is!!!!!