Connecting the world…


Upgrade Juniper SA cluster

Add On: This procedure also works for the new Juniper MAG appliances. But keep in mind during the upgrade of the second host (and also the first): BE PATIENT!!

A Juniper SA cluster can be configured as active/active or active/standby cluster. An active/active cluster uses an external load balancer or DNS round-robin to enable load-sharing across multiple appliances. Today I had to upgrade an active/standby cluster and found an KB article on the Juniper website (restricted access) about the preferred upgrade method.

Juniper uses the following steps to upgrade a cluster:

  1. 1. Login directly to a member in the cluster as administrator;
  2. 2. Disable the member from the cluster;
  3. 3. Upgrade the service package on the disabled member;
  4. 4. After the upgrade is completed login back to the IVE and enable the disabled member in the cluster configuration;

The following notes are mentioned by Juniper:

  • In active/standby cluster mode, it is recommended to start the upgrade process with the passive members and after completing the upgrade on the passive IVE and moving to the upgrade of the active IVE please note all connections are dropped when the active IVE is disabled. However after disabling the active node the passive IVE becomes active;
  • Once the upgraded member is enabled back in the cluster, it shows the other nodes as Unreachable. This is expected behavior as the cluster members are running different versions and hence cannot sync with each other;
  • Once the second IVE is being upgraded all user connections are dropped and not migrated due to the mismatch of software versions. This limitation is addressed in 4.0 with the Minimal downtime cluster upgrade available in the licensable Central Manager feature set;

I followed the steps mentioned above and the upgrade of the IVE cluster went smoothly. I disabled the passive node and upgraded the firmware with the new package. After the upgrade (and a reboot) the passive node was reachable in standalone mode. Next I logged in to the active IVE and enabled the passive node back into the cluster. When you hit Enable you receive the warning message that the configuration of the new cluster node will be erased and overwritten with the configuration of the active node. Just choose Yes.

After enabling the passive node, you will loose your web session with the active node. The VIP address is taken over by the new node in the cluster and the “old active” node starts updating automatically. This is a little tricky, because you don’t notice anything from the update process taken place. Just have patience and ping the node to check when it is online again. When the node is back online, login to the IVE and check the Cluster Status. Both IVE are now updated and members of the cluster. You could decide to do a manual Fail-Over IP to the “old active” node so everything is back to the original state before the upgrade.

Troubleshooting EIGRP

To troubleshoot EIGRP you should obvious have a grasp understanding of the specific routing protocol. Of course this doesn’t only apply to the EIGRP routing protocol.

Troubleshooting the EIGRP routing protocol on a Cisco devices is mainly about logging the correct information to a syslog server, the buffer or the console and know what the output of several show commands mean.

Neighbor instability on a router has mostly one of the following reasons:

  • Holding time expired;
  • Retry limit exceeded;
  • Manual changes like route filter changes;
  • Unidirectional links;
  • Primary / secondary IP address mismatch (EIGRP uses the primary IP address);
  • Routes get Stuck-in-Active aka SIA (“Active” means recovering from a change in the network, for example flapping links). Routes can get SIA by the following reasons: bad or congested links, query range is “too long”, excessive redundancy, overloaded router, router memory shortage and so one;

Many problems with EIGRP appear when using external routes, like redistributing a routing protocol like OSPF into EIGRP. The most common cause of this problem is the absence of setting the metrics. Important when using the redistribute command is specifying the metrics with the default-metric <metric> or redistribute <protocol> metric <metric> command.

Black Hole Summary Routing is also a common problem when using EIGRP. Black Hole Summary Routing is caused when manual summary routes are configured.


The picture shows a case of Black Hole Routing. Routers A and B summary the different /24 networks as one /16 network to router X. Suddenly the link between router A and router C gets lost. Because we used summarization between router A and router X, router X isn’t aware of the lost link, so router X keeps sending traffic for network to router A and router B ((un)equal cost load-balancing). All traffic send to router A would get lost in the process.

A solution to this problem is connecting routers A and B by using a physical link or creating a GRE tunnel between both, if the physical links isn’t possible.

As mentioned before, troubleshooting the routing protocol can be done by using the correct show, logging and debug commands. Important commands for troubleshooting EIGRP are:

  • show ip eigrp neighbors: for checking neighbors, hold timers, uptime, Smooth Round Trip Timer (SRTT) and Retransmit Time Out (RTO);
  • show ip eigrp topology active: used for noticing if route is SIA;
  • show ip eigrp events: check what EIGRP events are going on;
  • RTR(config-router)#eigrp-log-neighbor-changes;
  • RTR(config-router)#logging buffered <size>;
  • debug eigrp packet hello: showing the processing of hello packets;
  • debug eigrp packet terse: shows all EIGRP packet debugging except hellos;
  • debug ip eigrp notifications: troubleshooting redistribution by showing what happens between EIGRP and the routing table as routes are redistributed;

Troubleshooting a routing protocol can only be done if you know what the protocol is actually doing. When troubleshooting it is necessary to know the correct way to troubleshoot and start exempting possibilities for the routing problems. Exempting possibilities narrows the scope, which can result in finding the actual problem.

Cisco Firewall Design and Deployment

The session about firewall design and deployment didn’t reveal a lot of new things about the Cisco ASA appliance or FWSM module. The only new thing for me was the possibility to configure a redundant interface for a Cisco ASA appliance. The screen shot below shows the cabling scheme for an implementation with and without interface redundancy.

HA redundancy

This interface redundancy makes it possible to connect a ASA to two different physical switches. When the active switch would crash, the second switch would become the active switch.

Important here is to notice that this configuration doesn’t provide load-balancing across two links. The configuration is only for link redundancy.

To configure interface redundancy you can use the configuration snippet shown below.

interface Redundant1
  member-interface GigabitEthernet0/2
  member-interface GigabitEthernet0/1
  no nameif
  no security-level

  no ip address
interface Redundant1.4
  vlan 4
  nameif inside
  security-level 100
  ip address
interface Redundant1.10

  vlan 10
  nameif outside
  security-level 0
  ip address

The configuration of interface redundancy has some caveats as listed below:

  • Firewalls have to be configured in Active/Standby mode. No load-balancing or link aggregation is supported;
  • Interface redundancy is available on Cisco ASA 5510 and above. The ASA 5505 already has a build in switch and FWSM doesn’t have any physical interfaces;
  • Subinterfaces (IEEE 802.1Q) need to be configured on top of the logic redundant interface;

During the session the different modes for the firewalls have been discussed. Normally we only use the Routed Mode, but there are more modes like described below:

  • Routed mode: traditional mode of the firewall. Two or more interfaces that separate two or more layer 3 domains;
  • Transparent mode: the firewall acts as a bridge and functions mostly at layer 2 of the OSI model (this functions is often used for filtering traffic between two routers who, for example, exchange routing information through a dynamic routing protocol);
  • Multi-context: one physical firewall is divided in more virtual firewalls;
  • Mixed mode: using routed and transparent firewalls in a virtual environment (NOTE: mixed mode is only supported in FWSM today);

Firewall virtualization using multiple context has some caveats. We, Ictivity consultants, already noticed these caveats during firewall implementations. Firewall virtualization has the following caveats:

  • No support for VPN services;
  • No support for dynamic routing protocols;
  • No way to configure the sharing of CPU usage between contexts;
  • No support for multicast routing (multicast bridging is supported);

Especially not supporting VPN services (site-to-site VPN, remote access VPN and SSL VPN) is mostly the most used reason for not using multiple context implementation for the firewall.

PDA Active Sync – Invalid Certificate

The usage of Pocket PCs (PDAs) becomes more and more a default feature for business. The last months I have installed quit some Windows ISA 2006 servers for Reverse Proxy purposes. I have installed them normally for webmail only, but lately I have added the Microsoft Active Sync feature.

The Pocket PCs connect to the organization via UMTS, GPRS, USB with laptop or whatever with an internet connection. Today I had the same job on the schedule: Enable Active Sync for Pocket PCs.

I thought by myself: EASY JOB, but NOT. After configuring the ISA reverse proxy I used a Pocket PC emulator to test the Active Sync features. I received the following error message when synchronizing:


I found this a strange message, because clients use the same URL as the Pocket PC for accessing their webmail and they never receive an error message for an untrusted certificate.

The used certificated is issued by Equifax Secure Global eBusiness CA-1. This is a common and one of the better CA’s.

I had to dig deeper into the problem. I tried to install the certificate on the Pocket PC, but no luck. I searched the internet and found a tool called Microsoft Exchange Server Disable Certificate Verification. You can find an executable here, which can be used when using the Pocket PC in conjunction with a PC through USB. I also found a similar tool to install on the pocket PC, this is called The tool wasn’t the solution to the problem, so I had to dig deeper.

I was thinking way to complex, the problem was fixed by requesting a new certificate. The used certificate didn’t support Pocket PC. Comparing the different SSL certificates on I noticed I had to use a QuickSSL Premium certificate. This certificate supports popular mobile devices and smartphones.

After generating a CSR, requesting the certificate and installing the certificate on the ISA server, the connection and synchronization works like a charm. At least for the most PDA’s. Some PDA’s received the following error 80072f7d. After searching some forums, I found the solution in adding a registry key. I added the following registry key:

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows CE Services]

After adding the key to the registry, all Pocket PC’s synchronized perfectly.