Booches.nl

MAB and MDA in an IP Phone environment

I blogged before about the MAC Authentication Bypass (MAB) feature in network environments. MAC Authentication Bypass can be used to secure the wired network by verifying MAC addresses to a central database. By using a radius server, like Microsoft IAS or FreeRadius, you can also redirect verified MAC addresses to a specific VLAN.

Lately I had a new challenge with configuring MAB. These time a single switch port is shared by an IP phone and a workstation. The IP phone is used as a kind of switch. The backend switching network is build on Cisco Catalyst switches. All IP phone traffic is handled by the voice VLAN and all data traffic is handled by  the an access VLAN. The IP phones used in this situation are Mitel 5330 phones. These phones support CDP and also LLDP, which is perfect when using a voice VLAN.

The customer would like the MAC addresses of both devices verified against a central database. In this situation I used Microsoft IAS, because the customer is using Microsoft Active Directory as central database. In Active Directory I created an OU structure with an unique OU and security group for every logical group. So I created an OU voice and a security group voice, and I created a group data and an OU data. The MAC addresses of the components need to be added to Active Directory as users. The account name and the password are exactly the same and equal to the MAC address, like 001f22d712ef. I made the account for the IP phone member of the voice group and the account of the workstation member of the data group.

I started with just connecting a single workstation to the switch and configured IAS to verify the MAC address and automatically redirect the workstation to the correct access VLAN. The configuration of IAS is straightforward. First I installed IAS and registered the service in Active Directory. I added the switch as radius client and configured a radius policy for the data connections. The radius policy checks if the MAC address is member of the data group and returns the access VLAN if the MAC address is positively verified. This works without any problems. The screenshots below show the most important configuration of this policy.

Next you see the switch configuration so far.

aaa new-model
!
aaa authentication dot1x default group radius
aaa authorization network default group radius
!
dot1x system-auth-control
!
interface FastEthernet0/35
switchport access vlan 102
switchport mode access
switchport nonegotiate
switchport voice vlan 150
authentication control-direction in
authentication port-control auto
authentication periodic
authentication timer restart 900
authentication timer reauthenticate 5400
mab
spanning-tree portfast
spanning-tree bpduguard enable
end

I configured another policy, exactly the same, for the voice components. I disconnected the workstation and connected the IP phone to the network. This also works without any problems. The IP phone is authenticated and allowed access to the network. Next I connected the workstation to the IP phone and booted the workstation. I noticed that the IP phone lost his power and checked the switch port status. The switch port went in err-disable state with the following message:

Feb  5 08:54:50.095 GMT+1: %AUTHMGR-5-SECURITY_VIOLATION: Security violation on the interface FastEthernet0/35, new MAC address (0080.647f.c590) is seen.
Feb  5 08:54:50.095 GMT+1: %AUTHMGR-5-SECURITY_VIOLATION: Security violation on the interface FastEthernet0/35, new MAC address (0080.647f.c590) is seen.
Feb  5 08:54:50.095 GMT+1: %PM-4-ERR_DISABLE: security-violation error detected on Fa0/35, putting Fa0/35 in err-disable state

This is a big problem, because both network components aren’t able to communicate with the network. I did some research and found the Multiple Domain Authentication (MDA) feature. Multiple Domain Authentication (MDA) allows both a data device and a voice device, such as an IP phone (Cisco or non-Cisco), to authenticate on the same switch port, which is divided into a data domain and a voice domain. This feature is configured with the authentication host-mode commands and is very useful when combining IEEE 802.1x and/or MAB in an IP phone environment. The following host-modes can be used:

Single-host mode should be configured if only one data host is connected. Do not connect a voice device to authenticate on a single-host port. Voice device authorization fails if no voice VLAN is configured on the port.

Multi-domain mode should be configured if data host is connected through an IP Phone to the port. Multi-domain mode should be configured if the voice device needs to be authenticated.

Multi-auth mode should be configured to allow up to eight devices behind a hub to obtain secured port access through individual authentication. Only one voice device can be authenticated in this mode if a voice VLAN is configured.

Multi-host mode also offers port access for multiple hosts behind a hub, but multi-host mode gives unrestricted port access to the devices after the first user gets authenticated.

I tested the multi-host configuration and it did exactly as explained above. Only one device is authenticated and all next device are allowed without authentication. In my situation I have to use multi-domain. I added the configuration line authentication host-mode multi-domain to the interface configuration above. After this I had a new problem. Both devices are authenticated correctly, but the Mitel IP phone got stuck at DHCP Discovery, while the workstation is working correctly.

After some sniffing I saw the Mitel phone sending its DHCP Discovery to the data VLAN, but the phone didn’t receive any DHCP Offer from a DHCP server. Back to the drawing table and I found the solution in the radius configuration. I configured the radius attribute cisco-av-pair in order to tell the switch that the IP phone is allowed on the voice VLAN, see the picture.

The following steps are taken during the process:

1. 1. The IP Phones learns the voice VLAN ID from CDP;
2. 2. The switch learns the MAC address of the phone and sends an Accept-Request for the phones MAC address to the radius server;
3. 3. The radius server responds with an Access-Accept and adds the Vendor-Specific Attribute (VSA) Cisco-AV-pair with the value device-traffic-class=voice;
4. 4. All traffic from the IP Phone is allowed in the voice VLAN and the DHCP process works flawlessly;
5. 5. The workstation is also authenticated by the radius server and all data traffic is allowed in the data VLAN;

The radius policy for the voice VLAN is almost equal to the radius policy for the data/access VLAN. The only difference is in the radius attributes. Below you see the attributes for the voice radius policy.

I did some testing and the environment is working perfectly. Both devices are authenticated separately from each other. Below you see some output from the show authentication sessions command. You can clearly see the domain where the device is authenticated in.

ONLY IP PHONE IS AUTHENTICATED SUCCESSFULLY

switch#show authentication session interface fa 0/35
Interface:  FastEthernet0/35
MAC Address:  0800.0f46.874a
IP Address:  Unknown
User-Name:  08000f46874a
Status:  Authz Success
Domain:  VOICE
Oper host mode:  multi-domain
Oper control dir:  in
Authorized By:  Authentication Server
Session timeout:  5400s (local), Remaining: 5397s
Timeout action:  Reauthenticate
Idle timeout:  N/A
Common Session ID:  0A0A421B00000065C2FF71B0
Acct Session ID:  0×0000014A
Handle:  0×04000065

Runnable methods list:
Method   State
mab      Authc Success

IP PHONE AND WORKSTATION ARE AUTHENTICATED SUCCESSFULLY

switch#show authentication session interface fa 0/35
Interface:  FastEthernet0/35
MAC Address:  0080.647f.c590
IP Address:  Unknown
User-Name:  0080647fc590
Status:  Authz Success
Domain:  DATA
Oper host mode:  multi-domain
Oper control dir:  in
Authorized By:  Authentication Server
Vlan Policy:  102
Session timeout:  5400s (local), Remaining: 5364s
Timeout action:  Reauthenticate
Idle timeout:  N/A
Common Session ID:  0A0A421B00000068C304A7C5
Acct Session ID:  0×0000014D
Handle:  0×56000068

Runnable methods list:
Method   State
mab      Authc Success

—————————————-
Interface:  FastEthernet0/35
MAC Address:  0800.0f46.874a
IP Address:  Unknown
User-Name:  08000f46874a
Status:  Authz Success
Domain:  VOICE
Oper host mode:  multi-domain
Oper control dir:  in
Authorized By:  Authentication Server
Session timeout:  5400s (local), Remaining: 5340s
Timeout action:  Reauthenticate
Idle timeout:  N/A
Common Session ID:  0A0A421B00000067C3043675
Acct Session ID:  0×0000014C
Handle:  0xE2000067

Runnable methods list:
Method   State
mab      Authc Success

IP PHONE IS AUTHENTICATED SUCCESSFULLY, WORKSTATION ISN’T

switch#show authentication session interface fa 0/35
Interface:  FastEthernet0/35
MAC Address:  0080.647f.c590
IP Address:  Unknown
User-Name:  UNRESPONSIVE
Status:  Authz Failed
Domain:  DATA
Oper host mode:  multi-domain
Oper control dir:  in
Session timeout:  N/A
Idle timeout:  N/A
Common Session ID:  0A0A421B00000066C300CB6C
Acct Session ID:  0×0000014B
Handle:  0xEB000066

Runnable methods list:
Method   State
mab      Failed over

—————————————-
Interface:  FastEthernet0/35
MAC Address:  0800.0f46.874a
IP Address:  Unknown
User-Name:  08000f46874a
Status:  Authz Success
Domain:  VOICE
Oper host mode:  multi-domain
Oper control dir:  in
Authorized By:  Authentication Server
Session timeout:  5400s (local), Remaining: 5261s
Timeout action:  Reauthenticate
Idle timeout:  N/A
Common Session ID:  0A0A421B00000065C2FF71B0
Acct Session ID:  0×0000014A
Handle:  0×04000065

Runnable methods list:
Method   State
mab      Authc Success

Cisco error message: %SYS-2-MALLOCFAIL

While looking through some logging on a switch (Cisco Catalyst 3550), I noticed the following messages popping up multiple times in the buffer logging.

-Process= "Pool Manager", ipl= 0, pid= 5
-Traceback= 1A57D0 1A6DF4 161B3C 1B2BF0 1B2E38 1C6440
Jan 26 14:45:48.970 CET: %SYS-2-MALLOCFAIL: Memory allocation of 1680 bytes failed from 0×161B38, alignment 0
Pool: I/O  Free: 7412  Cause: Memory fragmentation
Alternate Pool: None  Free: 0  Cause: No Alternate pool

That doesn’t look good, but the customer didn’t receive any complaints about troubles or performance issues on the network. I did some research on the memory of the switch, but couldn’t find any strange behavior. The memory allocation looks normal and buffers look normal too. I found some memory allocation failures with the command show memory failures allow, but I already knew that looking at the error message. I found an article on the Cisco website concerning this error message, but that didn’t help much either.

The switch is running IOS 12.1(13)EA1a, which is marked as deferred. The last deferral notice I can find on the Cisco website is about IOS 12.1(19)EA1. The notice displays bugs with memory leakage problems. The next step I took was checking the Bug Toolkit for the running IOS.

I searched for all bugs of the running IOS and the bug toolkit reports 391 bugs. Narrowing the search with the string “%SYS-2-MALLOCFAIL” resulted in three bugs. One bug concerns a possible problem with spanning-tree and the creation of a loop in the network. Looking at the logging of other switch I noticed multiple MAC flap messages and BPDUGuard messages at the same time as the memory message. This indicates a possible loop in the network.

The bug concerns the following behavior:

Spanning-tree BPDUs (802.1d and 802.1w/802.1s) are sent to the incorrect destination MAC address. Consequently, other switches in the network will not process the BPDUs. If the network has been designed with a physical loop, spanning-tree will not correctly block the loop, causing traffic levels to increase and users to not be able to send data. In most cases, switch management will only be possible via the console port due to looping packets. The log might also contain %SYS-2-MALLOCFAIL messages, which indicate that the switch is running out of I/O memory. Spanning-tree loops are just one cause, but not the only one, of this message. Additional testing will help to confirm that the log messages are generated due to a spanning-tree loop that occurs as a result of this specific issue.

The switch is running Per-VLAN Spanning Tree, which can be compared with the default Spanning Tree Protocol (IEEE 802.1d). This bug could be the problem of the failed memory allocation, I recommended the customer to upgrade to the latest IOS. He will do so as soon as possible and informs me if the problem reoccurs.

Upgrade Juniper SA cluster

A Juniper SA cluster can be configured as active/active or active/standby cluster. An active/active cluster uses an external load balancer or DNS round-robin to enable load-sharing across multiple appliances. Today I had to upgrade an active/standby cluster and found an KB article on the Juniper website (restricted access) about the preferred upgrade method.

Juniper uses the following steps to upgrade a cluster:

1. 1. Login directly to a member in the cluster as administrator;
2. 2. Disable the member from the cluster;
3. 3. Upgrade the service package on the disabled member;
4. 4. After the upgrade is completed login back to the IVE and enable the disabled member in the cluster configuration;

The following notes are mentioned by Juniper:

• In active/standby cluster mode, it is recommended to start the upgrade process with the passive members and after completing the upgrade on the passive IVE and moving to the upgrade of the active IVE please note all connections are dropped when the active IVE is disabled. However after disabling the active node the passive IVE becomes active;
• Once the upgraded member is enabled back in the cluster, it shows the other nodes as Unreachable. This is expected behavior as the cluster members are running different versions and hence cannot sync with each other;
• Once the second IVE is being upgraded all user connections are dropped and not migrated due to the mismatch of software versions. This limitation is addressed in 4.0 with the Minimal downtime cluster upgrade available in the licensable Central Manager feature set;

I followed the steps mentioned above and the upgrade of the IVE cluster went smoothly. I disabled the passive node and upgraded the firmware with the new package. After the upgrade (and a reboot) the passive node was reachable in standalone mode. Next I logged in to the active IVE and enabled the passive node back into the cluster. When you hit Enable you receive the warning message that the configuration of the new cluster node will be erased and overwritten with the configuration of the active node. Just choose Yes.

After enabling the passive node, you will loose your web session with the active node. The VIP address is taken over by the new node in the cluster and the “old active” node starts updating automatically. This is a little tricky, because you don’t notice anything from the update process taken place. Just have patience and ping the node to check when it is online again. When the node is back online, login to the IVE and check the Cluster Status. Both IVE are now updated and members of the cluster. You could decide to do a manual Fail-Over IP to the “old active” node so everything is back to the original state before the upgrade.

Cisco router: determine amount of memory/flash

Somebody asked me how he could determine the amount of DRAM and flash memory on a Cisco router. I always thought that everybody would know how to determine this information, but since this isn’t matter, I will tell you how you can determine the values.

You use the show version command to retrieve the requested information. Below you see an example output of the command on a Cisco 876 router.

Router#show version
Cisco IOS Software, C870 Software (C870-ADVIPSERVICESK9-M), Version 12.4(15)T6, RELEASE SOFTWARE (fc2)
If you require further assistance please contact us by sending email to
export@cisco.com.

Cisco 876 (MPC8272) processor (revision 0×200) with 118784K/12288K bytes of memory.
Processor board ID FCZ121160T5
MPC8272 CPU Rev: Part Number 0xC, Mask Number 0×10
4 FastEthernet interfaces
1 ISDN Basic Rate interface
1 ATM interface
128K bytes of non-volatile configuration memory.
24576K bytes of processor board System flash (Intel Strataflash)

Configuration register is 0×2102

The first highlighted line tells you how much Dynamic RAM (DRAM) and Packet memory are installed in your router. Some platforms use a fraction of their DRAM as Packet memory. The memory requirements take this into account, so you have to add both numbers to find the amount of DRAM available on your router (from a memory requirement point of view).

Some types of routers have separate DRAM and Packet memory, so you only need to look at the first number. Other routers use a fraction of DRAM as Packet memory, so you need to add both numbers to find out the real amount of DRAM.

The second highlighted line tells you how much flash memory is installed in your router. This amount can also be determined by using the command show flash:.

Router#show flash:
24576K bytes of processor board System flash (Intel Strataflash)

Directory of flash:/

2  -rwx    18934284   Mar 1 2002 01:33:35 +01:00  c870-advipservicesk9-mz.124-15.T6.bin

23482368 bytes total (4542464 bytes free)

Maybe I can imagine why somebody doesn’t know where to look, because a show version actually gives you a lot of information. So I hope this posts helps all of you who don’t know where to look.

Automated eSafe backup

After configuring an eSafe appliance you have the option to export the configuration through the management interface, but you have to do this manually. eSafe has also a build in command line option to create a backup of the required files.

The command line allows backing up and restoring files using standard backup/restore commands. The command line option creates a tar.gz file; the same file that is created when backing up via the eSafe Appliance Manager.

I did some simple scripting to create a backup file, which is copied to a FTP server daily at 05:00 AM. When using the build in backup feature, the tar.gz file is created in the folder /var/esafe. I created two additional files (backup.sh and ftp_file) to automate the backup proces.

Below you see the content of both files:

backup.sh

#/bin/bash

cd /var/esafe
# Remove old backups
rm -rf *.tar.gz

# Create the backup with build-in eSafe backup
/opt/eSafe/esgapi –createbackup

# FTP files to Management server
ftp -inv </var/esafe/ftp_file &

ftp_file

# FTP files to Management server
open 10.10.1.10
user username password
lcd /var/esafe
cd /backup/esafe
put *.tar.gz
bye
quit

These commands create the necessary tar.gz backup file and copies this file to the FTP server. The last step is configuring the crontab to execute the command daily at 05:00 AM.

crontab

# Backup eSafe configuration
# Backup is copied via FTP to Management server
0 5 * * * bash /var/esafe/backup.sh

I guess the script couldn’t be more easy, but it works perfectly (for me).

When running the build in backup command (/opt/eSafe/esgapi –-createbackup) eSafe looks in the file /opt/eSafe/backup.list to determine the files to backup. You could decide to extend this list with the location of the Anti-Spam & URL filtering database (/opt/eSafe/eSafeCR/ConfigFilter/ofdb/*.fdb). This saves some downloading time when restoring an eSafe appliance.