June 18th, 2008 |
Today I received the question why a PIX failover configuration wasn’t working. The customer accidentally disconnected the power cable from the primary PIX firewall. The secondary PIX firewall became the active one, but multiple DMZ segments weren’t working anymore. After rebooting the PIX firewall and making that the primary one again, the DMZ segments were reachable again.
I started looking at the failover state of the firewall. Entering the commando show failover on the primary PIX resulted in the following output:
This host: Primary - Active
Active time: 5430735 (sec)
Interface outside (1.1.1.1): Normal
Interface inside (2.2.2.1): Normal
Interface DMZ01 (3.3.3.1): Normal
Interface DMZ02 (4.4.4.1): Normal (Waiting)
Interface DMZ03 (5.5.5.1): Normal (Waiting)
Other host: Secondary - Standby Ready
Active time: 5235 (sec)
Interface outside (1.1.1.2): Normal
Interface inside (2.2.2.2): Normal
Interface DMZ01 (3.3.3.2): Normal
Interface DMZ02 (4.4.4.2): Normal (Waiting)
Interface DMZ03 (5.5.5.2): Normal (Waiting)
Interface DMZ02 and DMZ03 were the problem interfaces.
After doing some simple ping test, I noticed the problem interfaces couldn’t ping each other. Looking at the output above the interfaces are in Normal (Waiting) state. Searching the internet resulted in the following:
Failover does not start to monitor the network interfaces until it has heard the second “hello” packet from the other unit on that interface. This takes about 30 seconds. If the unit is attached to a network switch that runs Spanning Tree Protocol (STP), this takes twice the “forward delay” time configured in the switch (typically configured as 15 seconds), plus this 30 second delay. This is because at PIX bootup and immediately following a failover event, the network switch detects a temporary bridge loop. Upon detection of this loop, it stops forwarding packets on these interfaces for the “forward delay” time. It then enters the “listen” mode for an additional “forward delay” time, during which time the switch listens for bridge loops but not forwarding traffic (and thus not forwarding failover “hello” packets). After twice the forward delay time (30 seconds), traffic resumes flowing. Each PIX remains in “waiting” mode until it hears 30 seconds worth of “hello” packets from the other unit. During the time the PIX is passing traffic, it does not fail the other unit based on not hearing the “hello” packets. All other failover monitoring still occurs (that is, Power, Interface Loss of Link, and Failover Cable “hello”).
Source
Here they are talking about the STP configuration. I couldn’t imagine that the problem is lying in the STP configuration. I started tracking the mac-address on the switches to check the physical patching.
The problem was found directly. The patching of the secondary PIX wasn’t correct. The customer had swapped the two interfaces resulting in patching them in the wrong VLAN. After swapping the patches the failover state functions normally, looking at the output below:
This host: Primary - Active
Active time: 5430735 (sec)
Interface outside (1.1.1.1): Normal
Interface inside (2.2.2.1): Normal
Interface DMZ01 (3.3.3.1): Normal
Interface DMZ02 (4.4.4.1): Normal
Interface DMZ03 (5.5.5.1): Normal
Other host: Secondary - Standby Ready
Active time: 5235 (sec)
Interface outside (1.1.1.2): Normal
Interface inside (2.2.2.2): Normal
Interface DMZ01 (3.3.3.2): Normal
Interface DMZ02 (4.4.4.2): Normal
Interface DMZ03 (5.5.5.2): Normal
This proves again how import correct patching is!!!!!
