Cisco Catalyst 2960X keeps crashing

Yesterday evening I had to troubleshoot a Cisco Catalyst 2960X switch stack, which didn’t return to normal after a reboot. The following error message was visible on the console:

Error: ASIC/PHY POST failed. Cannot continue.

%Software-forced reload

This error message is listed as a bug (CSCut90593) at Cisco.com. Cisco describes a very “good” workaround: the switch can boot up normally after this crash. I tried to reboot the switch several times, but it didn’t boot normally. The stack is running Cisco IOS version 15.2(2)E6 and the bug should be fixed in 15.2(2)E3.

It took me till midnight to recover the stack, so I am done troubleshooting and I will start the RMA procedure. I have to say that I am not really happy with the Cisco Catalyst 2960X. I have several customers running this type of switch and I had a lot more “troubles” with this switch, then I have/had with other Cisco switches. I hope stability will improve!!!!

Cisco WLC – HA SSO upgrade

“Is the upgrade procedure for a high-availability pair of Cisco Wireless LAN Controllers the same as the procedure for a single Cisco WLC?” Several customers asked me this questions and the answer is YES.

First you check the current and backup firmware image.

(Cisco Controller) >show boot
Primary Boot Image…………………………. 8.2.111.0 (default) (active)
Backup Boot Image………………………….. 8.1.102.0

Next you check if your SSO configuration is working as expected.

(Cisco Controller) >show redundancy summary
Redundancy Mode = SSO ENABLED
Local State = ACTIVE
Peer State = STANDBY HOT
Unit = Primary
Unit ID = 00:81:C4:87:3B:C9
Redundancy State = SSO
Mobility MAC = 00:81:C4:87:3B:C9
BulkSync Status = Complete
Average Redundancy Peer Reachability Latency = 177 Micro Seconds
Average Management Gateway Reachability Latency = 935 Micro Seconds

Upload the new firmware to the controller by using an TFTP or FTP server. I am using an TFTP server in this example.

(Cisco Controller) >transfer download datatype code
(Cisco Controller) >transfer download filename AIR-CT5520-K9-8-2-141-0.aes
(Cisco Controller) >transfer download path .
(Cisco Controller) >transfer download serverip 10.200.8.83
(Cisco Controller) >transfer download mode tftp
(Cisco Controller) >transfer download start

After the TFTP session is finished you’ll notice that the the software is automatically transferred from the active to the standby unit.

TFTP Code transfer starting.

TFTP receive complete… extracting components.

Checking Version Built.

Image version check passed.

Informing the standby to start the transfer download process

Waiting for the Transfer & Validation result from Standby.

Standby – Standby receive complete… extracting components.

Standby – Image version check passed.

Transfer & validation on Standby success, proceed to Flash write on Active.

Writing new AP Image Bundle to flash disk.

Executing fini script.

File transfer is successful.
Reboot the controller for update to complete.
Optionally, pre-download the image to APs before rebooting to reduce network downtime.

Transfer Download complete on Active & Standby

The last step is to reload both controllers to activate the firmware. After you reboot the active controller, you are able to access the standby controller and reboot that controller too. You have the option to reboot both controllers with one command.

(Cisco Controller) >reset system both in 00:05:00 image no-swap reset-aps

The controller also has the option to pre download the firmware from the controller to the access-points. This speeds up the upgrade process for the access-points, because the access-point don’t need to download the firmware after the controllers are online again. The access-point only need to reboot when the loose the connection with the controller. I will describe this process in a separate post.

After the controllers are back online, you should check the primary and backup boot firmware to see if the upgrade was successful.

(Cisco Controller) >show boot
Primary Boot Image…………………………. 8.2.141.0 (default)
Backup Boot Image………………………….. 8.2.111.0 (active)

Cisco FMC – Dashboard Widgets

Some widgets on the dashboard don’t generate graphs after deploying a default configuration of Cisco FireSight Management Center.

The first two widgets, Top Server Applications Seen and Top Operating Systems Seen, are generated after the configuration of a Network Discovery Profile. The configuration of the Network Discover Profile is done via Policies – Network Discovery – Networks. I always configure a Network Discovery Profile to profile all Hosts, Users and Application for the RFC1918 IP address space.

To generate graphs for the URL widgets, you need to make sure that the correct options for the URL filtering service are enable. The URL filtering service configuration is done via System – Integration – Cisco CSI. I use the following settings for URL filtering.

After this you should wait a while (about one hour) to check if the graphs are generated. If you don’t want to wait, you can check the Analysis tab to see if information is gathered and displayed by the Cisco FireSight Management Center appliance.

Aruba ClearPass – Cisco Prime – TACACS+

When using Cisco Prime you have the option to configure authentication to a remote AAA server via RADIUS or TACACS+. Today I configured Cisco Prime to use HPE Aruba ClearPass as remote AAA server based on the TACACS+ protocol. The configuration of an AAA server in Cisco Prime is very straightforward. Configure the AAA Mode Setting under Administration / Users / Users, Role & AAA / AAA Mode Settings. The next step involves adding HPE Aruba ClearPass as TACACS+ servers via the option menu Administration / Users / Users, Role & AAA / TACACS+ Servers.

I configured a TACACS+ service in ClearPass with a very basic Enforcement Profile. In the beginning I used the same Enforcement Profile, which I also use to enforce privilege level 15 to switches and routers. Authentication will fail at this point. In the Access Tracker I get the following error message:

Tacacs service=NCS:HTTP not enabled

And the login screen from Cisco Prime shows me the following error message.

I created a new Enforcement Profile and added the TACACS+ service NCS:HTTP to the Enforcement Profile. Now I see an access granted in the Access Tracker, but I still get the same error message on the Cisco Prime website. After some digging in Cisco Prime I noticed that Cisco Prime needs to receive TACACS+ attributes from the AAA server to grant access and assign privileges and tasks to the user.

First you need to get the TACACS+ attributes from the Virtual Domain configuration. In the menu options navigate to Administration \ Users \ Virtual Domains. At the upper right corner you have the option to “Export Custom Attributes”.

These attributes need to be configured in ClearPass. As you notice you also need to configure these attributes if you would like to use RADIUS as authentication protocol. You also need to add the attributes from the user group. Navigate to Administration / Users / Users, Role & AAA / User Groups. Click the “Task List” option next to the User Group you would like to use. I use Root in this example.

The User Group Root contains 194 tasks, which need to be added to the Enforcement Profile in ClearPass. Below you see a snippet from the Enforcement Profile configuration.

To make it easy for you, I exported the Enforcement Profile including all the 194 tasks for the Root User Group. You can download the Enforcement Profile in XML format below. Just import the profile into ClearPass and you are good to go!!!

Download here: Cisco Prime Enforcement Policy

Cisco ASA: multiple context and capture

Packet captures are very useful for troubleshooting purposes. The Cisco ASA supports packet captures even in multiple context mode. I normally configure packet captures on CLI level. This can be done by configuring an access-list to match the specific traffic you would like to capture. Add the access-list and the specific interface in a capture command. Mostly I download the capture in raw format for further analysis with a tool like WireShark. The capture can be downloaded via TFTP or via a secure connection (HTTPS) to the Cisco ASA firewall.

When running a Cisco ASA in multiple context mode, I always disable the ability to connect directly to a context for management purposes. That way you have to access the admin context for management access, but this also denies the option to download the capture via a secure connection directly from the Cisco ASA traffic context.

The easiest way to download the capture in multiple context mode is via a TFTP transfer from the system context. Check the example command below. The capture is made within the context named contextA and the capture has the name captureA. The following command can be used to download the capture in raw (pcap) format.

copy /pcap capture:contextA/captureA tftp://10.10.10.10/captureA.pcap

You can now analyse the capture with WireShark