Connecting the world…

Migrate RAP from AOS 6.x to AOS 8.x

I guess something that many HPE Aruba wireless engineers have to do these days is migrating the “old” AOS 6.x environment to the new AOS 8.x with Mobility Masters. I am not going to explain what the differences between both are and what a Mobility Master does, but I have a tip when you need to migrate remote access points (RAPs).

Yesterday evening I had to perform a migration from 3 7030 controllers to a environment with redundant Mobility Masters. The solution contains 35 campus APs and 20 remote APs. The migration should be relatively simple…….

I installed the Mobility Masters, added the licences, create the node hierarchy, add all the configuration regarding WiFi, AAA, RAP ports, user-roles, VLANs and so on. To make live easier I exported the whitelist db on the old controller (localuserdb export) and imported the whitelist db into the AOS 8.4 (or 8.5) environment (localuserdb import).

I was able to add one controller to the Mobility Masters by distributing the APs in the 6.5 environment over the remaining 2 controllers. So the old environment had 2 controllers left, one for the campus APs and one for the RAPs. I wanted to migrate the campus APs first. Can’t be that difficult……. just provision the campus AP with the IP address of the AOS 8.4 controller and wait for the magic to happen. All campus AP came online, so I added the second controller to the Mobility Master.

Since I now have two controllers I started with the lc-cluster configuration. I configured the lc-cluster group-profile, added the two controllers, including the vrrp-ip and rap-public-ip, like shown below.

lc-cluster group-profile “lc-booches”
controller priority 128 mcast-vlan 0 vrrp-ip vrrp-vlan 1 group 0 rap-public-ip
controller priority 128 mcast-vlan 0 vrrp-ip vrrp-vlan 1 group 0 rap-public-ip

Only two more steps left to add the RAPs:

  1. Add the correct NAT and firewall configuration to the firewall environment
  2. Add a LC-RAP-pool to the Mobility Master >> /mm level

BAM, ready to migrate the RAP. I converted the first by reprovision the AP in the old controller and configure the new public IP. Here we go!! You only need to wait and wait and wait and wait……. but the RAP didn’t come online!!! What can go wrong??

  • Created a whitelist entry? – yes, export / import and I see the entries via show whitelist-db rap
  • Created a LC-RAP-pool? – yes, show lc-rap-pool on /mm level
  • Correct LC-cluster config? – yes, rap-public-ip is configured correctly
  • Firewall config? – Config is correct. Firewall policy is matched and packet sniffer shows that packets are forwarded to controller
  • Controller hardware limit reached? No, there are currently 35 campus APs and the limit of a 7030 is 64 APs

Time for some deeper troubleshooting. The RAPs gets stuck at the logon role in the user-table. Time for some crypto debugging. Lets get started with logging security process crypto level debugging to check if a VPN tunnel establishment.

Hhhhmmmm, there is a strange error message!!!

|ike| HMAC_SHA1_96 ESN_0 <– R Notify: INTERNAL_ADDRESS_FAILURE (ESP spi=ea87a200)#SEND 80

Seems like the RAP IP pool isn’t configured correctly. So double check, but the pool is definitely configured correctly. Okay, another test. One RAP is onsite, so factory default the RAP and convert it from scratch. SAME RESULT AND SAME ERROR message. Hhhhmmm, I am AMFX certificated, but can’t even get a RAP connected?!?!?! So it is time to ask another AMFX’er, my colleague Peter (AMFX#36). Luckily he was willing to power up his RAP and provision it to the customer’s environment.

Okay, let’s add the RAP to the whitelist, let Peter do the conversion magic from IAP to RAP and check what happens. BAM, RAP connects, updates software and is online……. Difference, other hardware. Our RAP is RAP303-H, the customer is using RAP155P. RAP155P is supported in AOS 8.4, so that can’t be the problem.

So what is another difference…. THE WHITELIST. Our RAP was added manually after the configuration was completely done and wasn’t imported at the beginning. So let’s check the whitelist again and there it is!!!! I didn’t really check all columns from the show whitelist-db rap, but one of the columns on the right is called Cluster-InnerIP. The output below shows the column, normally there are even more columns, so you have to scroll to view the Cluster-InnerIP.

You notice that the manually added entry gets an IP address assigned from the LC-RAP-pool and all imported RAPs don’t get an IP address. This definitely reflexes the error INTERNAL_ADDRESS_FAILURE error message. The cert-type is changed to factory-cert after the RAP is converted and online.

So I removed one of the imported entries and added it again (manually) to the whitelist db and guess what. The Cluster-InnerIP is assigned to the RAP and the RAP connects directly and without any problem.

So in my opinion there could be two reasons why this issue arrises:

  1. I imported the whitelist-db to early, because I hadn’t configured the LC-RAP-pool yet;
  2. The import from 6.5 to 8.4 doesn’t work for RAP entries;

To be sure, I removed all entries from the whitelist db and imported the whitelist again. Again the same result, the RAPs don’t get a Cluster-InnerIP assigned. Conclusion:

TIP of the WEEK: when migrating from 6.5 to 8.4 MANUALLY add the whitelist-db rap entries to the 8.4 environment!!!

Or does someone have a better way to import the whitelist for remote APs?!?!?!

Downloadable User-Roles and NTP sync

The HPE Aruba switches have this cool feature called downloadable user-roles (DUR). DUR enables the switch to use a central ClearPass server to download user-roles to the switch for authenticated users.

More and more customers want to implement wired authentication to strengthen the security level of their network. Via DUR the switches perform an HTTPS API request against ClearPass to download the user-role configuration. This makes the configuration of multiple switches easier, because you don’t need to configure the user-roles locally on the switches anymore, but you push them from a central server. The communication between switch and ClearPass is illustrated in the picture below.

Source: ClearPass Solution Guide: Wired Policy Enforcement

I won’t describe the whole DUR configuration step-by-step, but below you can find the most important configuration for the switch.

radius-server host “” key “radius-secret”
radius-server host “” dyn-authorization
radius-server host “” time-window plus-or-minus-time-window
radius-server host “” time-window 30
radius-server host “” clearpass
radius-server cppm identity “admdur” key “admdur-key”
ip client-tracker trusted
aaa server-group radius “GRP-CPPM” host “”
aaa authentication port-access eap-radius server-group “GRP-CPPM”
aaa authentication mac-based chap-radius server-group “GRP-CPPM”
aaa accounting network start-stop radius server-group “GRP-CPPM”
aaa authentication captive-portal enable
aaa authorization user-role enable download
aaa port-access authenticator 1/1
aaa port-access authenticator 1/1 tx-period 10
aaa port-access authenticator 1/1 supplicant-timeout 10
aaa port-access authenticator 1/1 client-limit 10
aaa port-access mac-based 1/1
aaa port-access mac-based 1/1 addr-limit 10
aaa port-access 1/1 controlled-direction in

For the HTTP GET to work the switch needs to trust the certificate chain from ClearPass. In ArubaOS 16.08 and later the certificate is automatically downloaded when specifying the option “clearpass” when configuring the RADIUS client. Another very important step for DUR to work is NTP time sync. The time on the switches needs to be in sync and here a “problem” arises.

After a switch power outage, the switch has to sync its time with an NTP server. And the time needs to be in sync before the first wired clients start authenticating. Even when I use the “iburst option with the NTP server for aggressive polling, I see that the time isn’t always synced in time.

Below you see the output from “show log -r” when the client authenticates, but the switch hasn’t synced its time yet.

I 02/12/19 10:55:46 04908 ntp: ST1-CMDR: The system clock time was changed by 918813141 sec 661757827 nsec. The new time is Tue Feb 12 10:55:46 2019
I 01/01/90 01:03:11 04911 ntp: ST1-CMDR: The NTP Server is unreachable.
I 01/01/90 01:02:55 00584 WebMacAuth: ST1-CMDR: Port 1/1, re-auth timeout 10 too short.
I 01/01/90 01:02:55 05747 DFP: ST1-CMDR: device_fingerPrinting: Hardware Rules updated successfully for port:1/1, protocol:80, client:08:00:0F:9D:45:BF
W 01/01/90 01:02:55 05204 dca: ST1-CMDR: Failed to apply user role VOIP___DUR-3005-1_7Z4q to macAuth client 08000F9D45BF on port 1/1: user role is invalid.
W 01/01/90 01:02:55 05620 dca: ST1-CMDR: macAuth client 08000F9D45BF on port 1/1 assigned to initial role as downloading failed for user role VOIP___DUR-3005-1.
I 01/01/90 01:02:51 00076 ports: ST1-CMDR: port 1/1 is now on-line
I 01/01/90 01:02:51 00435 ports: ST1-CMDR: port 1/1 is Blocked by STP

The port is placed in the initial-role which is by default the role denyall. “Problem” with the default role is the missing option “reauthentication period”, so the connected clients will not automatically reauthenticate after an X-period of time.

User Role Information
Name : denyall
Type : predefined
Reauthentication Period (seconds) : 0
Cached Reauth Period (seconds) : 0
Logoff Period (seconds) : 300

To “fix” this issue I added a new local user-role to the switch and configured this user-role as initial-role. I added the reauthentication period to the user-role, so the clients reauthenticate when time isn’t synced yet and they receive this initial-role from the switch. The configuration of the role is displayed below.

class ipv4 “IP_ANY_ANY”
10 match ip
policy user “DENYALL”
10 class ipv4 “IP_ANY_ANY” action deny
aaa authorization user-role name “reauth-role”
policy “DENYALL”
reauth-period 30
vlan-id 1

To use this role as initial-role you need to execute the following command.

aaa authorization user-role initial-role reauth-role

Next I tested the role by rebooting the switch. After rebooting I noticed that the switch port is placed in the “reauth-role“, because I receive the error message “assigned to initial role as downloading failed for user role” in the logs. In ClearPass I see another authentication request from the client after X seconds. At that moment the time on the switch is in sync and the switch port is configured with the correct user-role.

Edited: February 13th 2019
I created a topic on the AirHeads community on this matter and HPE Aruba responded with:

A software fix for the clock reset on cold boot/power loss issue on the 2930F and 2540 is in the works, and is expected to be released by the end of February.


I used MacOS X already in the past on an “old” MacBook and I have an iMac at home, but recently I am using a MacBook Pro for work. This blog is just a wrap up for “things” that I use often, but for some reason I always forget.

Add Static route

Add a static route via de command.

sudo route -n add -net <network> <gateway>
Example: sudo route -n add -net

Import intermediate / root certificate

Export a certificate from a website, so I am able to import the certificate into the MBP certificate store or import the certificate to other appliances, like ClearPass

openssl s_client -connect -showcerts

ClearPass – REST API


I created some Python scripts for ClearPass. The scripts can be found on Github. There are several directories:

  • config: contains the parameters to authenticate against ClearPass and acquire an access token;
  • general_scripts: some general configuration scripts, like a Password Generator script or Date/Time script;
  • guests: scripts for adding or deleting guest accounts. I created a script to add guest accounts via a CSV file and print the most important information to a Guest Pass in Word format;
  • localusers: scripts for adding or deleting local user accounts;

First of all, I would like to thank Tim Cappalli for the ClearPass Authentication scripts!!

GuestPass Example



A special thanks to Tim Cappalli for the ClearPass Authentication scripts!!

FortiGate – OnDemand Token Timeout

Today’s customer is having a problem with OnDemand tokens on a FortiGate firewall. The FortiGate firewall uses RADIUS authentication for SSL VPN user authentication. FortiAuthenticator is used as RADIUS server. To strengthen the security levels, FortiAuthenticator is configured to demand two-factor authentication (2FA) for successful authentication. FortiAuthenticator has multiple options to demand 2FA from a user, like hardware FortiTokens, FortiToken Mobile or mail or SMS services.

Problem with the latter two could be timeouts. By default, FortiAuthenticator expects the token code after 60 seconds. This value is customizable.

However, only changing the timeout in FortiAuthenticator isn’t enough, because FortiGate has its own timeout value too. So you need to change this value if you would like to increase the time between entering username/password and token code. The timers are configurable via the CLI in “system global”

two-factor-email-expiry: Email-based two-factor authentication session timeout (30 – 300 seconds (5 minutes), default = 60).
two-factor-fac-expiry: FortiAuthenticator token authentication session timeout (10 – 3600 seconds (1 hour), default = 60).
two-factor-ftk-expiry: FortiToken authentication session timeout (60 – 600 sec (10 minutes), default = 60).
two-factor-ftm-expiry: FortiToken Mobile session timeout (1 – 168 hours (7 days), default = 72).
two-factor-sms-expiry: SMS-based two-factor authentication session timeout (30 – 300 sec, default = 60).

In this particular case, I changed the two-factor-fac-expiry setting to match the setting on FortiAuthenticator.

1 2 3 53