Monday I had to migrate an existing network. I added more VLAN’s to the network for segmentation and breaking the broadcast domain. I introduced a regular VLAN, a VoIP VLAN and a management VLAN. So far no problem. The customer is using Cisco Catalyst 3750G and Cisco Catalyst 3560 switches with PoE.
I configured the VTP domain, so the new VLAN got distributed without any problem. I added all switchports to the correct VLAN. The management VLAN is for all management connections from routers, switches, firewall and ESX hosts. The management VLAN is a new VLAN with a new IP subnet. I had to change the IP addresses and routing on the switches, routers and firewall. I also needed to change the IP addresses of the ESX hosts, because they are also placed in the management VLAN.
Unfortunately my colleague Duncan couldn’t come that day, so I started to change the IP addresses (/etc/sysconfig/network-scripts/ifcfg-vswitch0), default gateway settings (/etc/sysconfig/network), host file (/etc/hosts) and NTP configuration. After starting the Infrastructure Client we noticed an error in the HA configuration. The HA feature couldn’t start.
After change the IP addresses I didn’t restart the ESX servers, I only restarted the networking service. I restarted all the ESX hosts, but still no luck. I called Duncan and he advised to try to remove the HA cluster and rebuild it again. This also didn’t do the trick. At the end I disabled HA and waited for Duncan to investigate the problem further.
Duncan visited the customer yesterday and he found the problem. Somewhere in the configuration files, the old IP addresses where still present and hadn’t been changed when rebuilding the cluster.
A detailed solution can be found on Duncan’s blog, right here.