Connecting the world…


Downloadable User-Roles and NTP sync

The HPE Aruba switches have this cool feature called downloadable user-roles (DUR). DUR enables the switch to use a central ClearPass server to download user-roles to the switch for authenticated users.

More and more customers want to implement wired authentication to strengthen the security level of their network. Via DUR the switches perform an HTTPS API request against ClearPass to download the user-role configuration. This makes the configuration of multiple switches easier, because you don’t need to configure the user-roles locally on the switches anymore, but you push them from a central server. The communication between switch and ClearPass is illustrated in the picture below.

Source: ClearPass Solution Guide: Wired Policy Enforcement

I won’t describe the whole DUR configuration step-by-step, but below you can find the most important configuration for the switch.

radius-server host “” key “radius-secret”
radius-server host “” dyn-authorization
radius-server host “” time-window plus-or-minus-time-window
radius-server host “” time-window 30
radius-server host “” clearpass
radius-server cppm identity “admdur” key “admdur-key”
ip client-tracker trusted
aaa server-group radius “GRP-CPPM” host “”
aaa authentication port-access eap-radius server-group “GRP-CPPM”
aaa authentication mac-based chap-radius server-group “GRP-CPPM”
aaa accounting network start-stop radius server-group “GRP-CPPM”
aaa authentication captive-portal enable
aaa authorization user-role enable download
aaa port-access authenticator 1/1
aaa port-access authenticator 1/1 tx-period 10
aaa port-access authenticator 1/1 supplicant-timeout 10
aaa port-access authenticator 1/1 client-limit 10
aaa port-access mac-based 1/1
aaa port-access mac-based 1/1 addr-limit 10
aaa port-access 1/1 controlled-direction in

For the HTTP GET to work the switch needs to trust the certificate chain from ClearPass. In ArubaOS 16.08 and later the certificate is automatically downloaded when specifying the option “clearpass” when configuring the RADIUS client. Another very important step for DUR to work is NTP time sync. The time on the switches needs to be in sync and here a “problem” arises.

After a switch power outage, the switch has to sync its time with an NTP server. And the time needs to be in sync before the first wired clients start authenticating. Even when I use the “iburst option with the NTP server for aggressive polling, I see that the time isn’t always synced in time.

Below you see the output from “show log -r” when the client authenticates, but the switch hasn’t synced its time yet.

I 02/12/19 10:55:46 04908 ntp: ST1-CMDR: The system clock time was changed by 918813141 sec 661757827 nsec. The new time is Tue Feb 12 10:55:46 2019
I 01/01/90 01:03:11 04911 ntp: ST1-CMDR: The NTP Server is unreachable.
I 01/01/90 01:02:55 00584 WebMacAuth: ST1-CMDR: Port 1/1, re-auth timeout 10 too short.
I 01/01/90 01:02:55 05747 DFP: ST1-CMDR: device_fingerPrinting: Hardware Rules updated successfully for port:1/1, protocol:80, client:08:00:0F:9D:45:BF
W 01/01/90 01:02:55 05204 dca: ST1-CMDR: Failed to apply user role VOIP___DUR-3005-1_7Z4q to macAuth client 08000F9D45BF on port 1/1: user role is invalid.
W 01/01/90 01:02:55 05620 dca: ST1-CMDR: macAuth client 08000F9D45BF on port 1/1 assigned to initial role as downloading failed for user role VOIP___DUR-3005-1.
I 01/01/90 01:02:51 00076 ports: ST1-CMDR: port 1/1 is now on-line
I 01/01/90 01:02:51 00435 ports: ST1-CMDR: port 1/1 is Blocked by STP

The port is placed in the initial-role which is by default the role denyall. “Problem” with the default role is the missing option “reauthentication period”, so the connected clients will not automatically reauthenticate after an X-period of time.

User Role Information
Name : denyall
Type : predefined
Reauthentication Period (seconds) : 0
Cached Reauth Period (seconds) : 0
Logoff Period (seconds) : 300

To “fix” this issue I added a new local user-role to the switch and configured this user-role as initial-role. I added the reauthentication period to the user-role, so the clients reauthenticate when time isn’t synced yet and they receive this initial-role from the switch. The configuration of the role is displayed below.

class ipv4 “IP_ANY_ANY”
10 match ip
policy user “DENYALL”
10 class ipv4 “IP_ANY_ANY” action deny
aaa authorization user-role name “reauth-role”
policy “DENYALL”
reauth-period 30
vlan-id 1

To use this role as initial-role you need to execute the following command.

aaa authorization user-role initial-role reauth-role

Next I tested the role by rebooting the switch. After rebooting I noticed that the switch port is placed in the “reauth-role“, because I receive the error message “assigned to initial role as downloading failed for user role” in the logs. In ClearPass I see another authentication request from the client after X seconds. At that moment the time on the switch is in sync and the switch port is configured with the correct user-role.

Edited: February 13th 2019
I created a topic on the AirHeads community on this matter and HPE Aruba responded with:

A software fix for the clock reset on cold boot/power loss issue on the 2930F and 2540 is in the works, and is expected to be released by the end of February.

Upgrading Cisco switch stack

I always upgrade a switch stack with one single command. Last week I received a call from a customer with the question about the upgrade procedure for a switch stack. The customer wanted to upload the image separately to every single switch. I told him that he could upgrade all switches at once.

Since I am “playing” with a Cisco switch stack of 9 Catalyst 3750X switches today I will describe the upgrade procedure.

  1. 1. You need to download the correct .tar image file;
  2. 2. Copy it to the root of your FTP or TFTP server;
  3. 3. Upload, extract and install the .tar file to the switches (I always use the /imageonly option, because I don’t need the html files for management);
  4. 4. Reload the switch stack;

The command to upload and extract the .tar file can be found below:

sw-stack#archive download-sw /imageonly /overwrite /allow-feature-upgrade ftp://user:password@<IP address FTP server>/image-file.tar

Loading c3750e-universalk9-tar.122-55.SE1.tar !!!!!!!
[OK – 17745920/4096 bytes]

Loading c3750e-universalk9-tar.122-55.SE1.tar !!!!!!!
examining image…
extracting info (110 bytes)
extracting c3750e-universalk9-mz.122-55.SE1/info (444 bytes)
extracting info (110 bytes)

Stacking Version Number: 1.45

System Type:             0x00000002
Ios Image File Size:   0x00DE8200
Total Image File Size: 0x010ECA00
Minimum Dram required: 0x08000000
Image Suffix:          universalk9-122-55.SE1
Image Directory:       c3750e-universalk9-mz.122-55.SE1
Image Name:            c3750e-universalk9-mz.122-55.SE1.bin
Image Feature:         IP|LAYER_3|PLUS|SSH|3DES|MIN_DRAM_MEG=128

Old image for switch 1: flash:/c3750e-universalk9-mz.122-53.SE2
Old image will be deleted after download.
Old image for switch 2: flash2:/c3750e-universalk9-mz.122-53.SE2
Old image will be deleted after download.
Old image for switch 3: flash3:/c3750e-universalk9-mz.122-53.SE2
Old image will be deleted after download.
Old image for switch 4: flash4:/c3750e-universalk9-mz.122-53.SE2
Old image will be deleted after download.
Old image for switch 5: flash5:/c3750e-universalk9-mz.122-53.SE2
Old image will be deleted after download.
Old image for switch 6: flash6:/c3750e-universalk9-mz.122-53.SE2
Old image will be deleted after download.
Old image for switch 7: flash7:/c3750e-universalk9-mz.122-53.SE2
Old image will be deleted after download.
Old image for switch 8: flash8:/c3750e-universalk9-mz.122-53.SE2
Old image will be deleted after download.
Old image for switch 9: flash9:/c3750e-universalk9-mz.122-53.SE2
Old image will be deleted after download.

Extracting images from archive into flash…
Extracting images from archive into flash on switch 2…
Extracting images from archive into flash on switch 3…
Extracting images from archive into flash on switch 4…
Extracting images from archive into flash on switch 5…
Extracting images from archive into flash on switch 6…
Extracting images from archive into flash on switch 7…
Extracting images from archive into flash on switch 8…
Extracting images from archive into flash on switch 9…

extracting c3750e-universalk9-mz.122-55.SE1/c3750e-universalk9-mz.122-55.SE1.bin (14570585 bytes)
extracting c3750e-universalk9-mz.122-55.SE1/info (444 bytes)
extracting info (110 bytes)


Installing (renaming): `flash:update/c3750e-universalk9-mz.122-55.SE1′ ->
New software image installed in flash:/c3750e-universalk9-mz.122-55.SE1


Removing old image: flash:/c3750e-universalk9-mz.122-53.SE2
Removing old image: flash2:/c3750e-universalk9-mz.122-53.SE2
Removing old image: flash3:/c3750e-universalk9-mz.122-53.SE2
Removing old image: flash4:/c3750e-universalk9-mz.122-53.SE2


All software images installed.

The boot parameters are automatically changed to the new IOS firmware. You can check the boot parameters with the show boot command.

WebMarshal performance problems

One of our customers is using WebMarshal for HTTP/HTTPS URL filtering and content scanning. The WebMarshall software is installed on two Microsoft ISA 2003 servers. These ISA servers are behind a Cisco Content Switch for load-balancing and redundancy purposes.

The problem with the WebMarshal is the PERFORMANCE. Internet browsing with the WebMarshal as proxy just doesn’t perform. I tried to troubleshoot the WebMarshal to check where the performance problems are coming from, but you cannot troubleshoot the software on a decent way. I disabled the Access Policies, and guess what, the performance is great. I added a allow all rule on top of every Access Policy subcategory, but no success.

I know the customer is running an old version (3.0.x), and of course if you contact the supplier, the first thing they say is: “Upgrade to the last version!!”. It seems the solution is always upgrading the last version. The second thing the supplier told us, was using Microsoft Network Load Balancing and not the Content Switches. Sadly the customer is using HP ProCurve switches, which don’t support static ARP entries. So NLB is no option.

But again, I give them the benefit of the doubt, so we will install two new servers, which are dedicated for WebMarshal software. Still the servers will be behind the Content Switch, because I believe that the Content Switches are the reason for the bad performance.

I will tell you more about the outcome of the latest version of WebMarshal on dedicated hardware. My opinion so far: “Feed the WebMarshal software to the dogs and buy something else!!!!!!!!!!”