Booches.nl

Another NVRAM broken?

On Monday I visited another customer who had problems saving the running configuration of a Cisco devices. The devices involved were a Cisco 2620 and a Cisco 2610XM router. Both routers weren’t able to save their running configuration.

Both routers show the following error-message:

startup-config file open failed (Bad file number)

By both routers I was able to look at the contents of the flash memory, but I wasn’t able to look at the contents of the NVRAM. When trying to look at the contents of NVRAM, you receive the following error-message:

Directory of nvram:/

%Error opening nvram:/ (Bad file number)

No space information available

22w0d: %SYS-4-NV_BLOCK_INITFAIL: Unable to initialize the geometry of nvram

The following information can be found on the error-message on the Cisco website:

%SYS-4-NV_BLOCK_INITFAIL : Unable to initialize the geometry of nvram

 

Explanation    The software has detected that it failed to initialize the NVRAM block geometry (a part of the NVRAM that hosts non-configuration data files). Typically, these files are used by SNMP to store and retrieve non-configuration persistent data across a system reload. This condition may occur when the entire NVRAM is packed with the configuration, and the newer version of software that supports this feature could not find enough room in the NVRAM to initialize the block file system.

Recommended Action    Reduce the configurations in the NVRAM by at least 2Kb.

Source

Luckily one of the two routers was the old production router, which was swapped and the customer thought the NVRAM was broken. So I could use that router for testing purposes.

I started looking at the some physical aspects of both routers. They both had the following hardware specifications:

• 16 MB Flash memory
• 64 MB RAM memory
• 32K NVRAM
• IOS version 12.4(5)

While looking at the running configuration of the active router, I noticed that the running configuration was almost 2900 bytes (29K), which is stored in NVRAM. I believed that the error-messages were generated because NVRAM was full. I started deleting some configuration from the broken router, until the running configuration was only 20K. But I still wasn’t able to save the running configuration.

The last change I had was updating the IOS. I downloaded the last Main Release, which is 12.4(23). I formatted the flash memory and uploaded the image to the spare router. And fortunately, after a reload, I was able to save the running configuration.

The running configuration was still almost 2900 bytes. I issues the command to compress the configuration:

service compress-config

Now the running configuration is compressed from almost 2900 bytes to 1000 bytes.

NVRAM corrupt or broken?

Today some of my colleagues and I “rebuild” an existing ESX with NetApp network. We change multiple VLAN’s and did a lot of reconfiguring. Unfortunately some other people were working on the power, so sometimes all equipment had to power down.

After we did our job, we started testing the environment. All DMZ VM’s weren’t able to connect to their default gateway, which is a Cisco ASA 5550 active/standby firewall. I did some research and noticed that the DMZ switch lost his configuration. At first I thought that I didn’t save the configuration to NVRAM. Luckily I had a backup configuration, which I copied and pasted into the switch. To be sure I issued a write mem, which gave me the following response:

SW01#wr mem
Building configuration…

nv_done: unable to open “flash:/config.text.new”
nv_done: unable to open “flash:/private-config.text.new”[OK]
NVRAM Verification Failed

Hhhuuuummm, I tried a show flash: and even a format flash:, but no success. Cisco’s SupportWiki tells me to try and erase the startup-configuration from NVRAM and issue a new write mem, but again no success. While executing the command erase start, the following message appears on the console:

Jan 24 16:05:31: %SYS-7-NV_BLOCK_INIT: Initialized the geometry of nvram

Looking at Cisco’s website, they have the following description with the error message:

%SYS-7-NV_BLOCK_INIT : Initalized the geometry of nvram

 

Explanation    The NVRAM block geometry has been initialized. NVRAM block geometry is a part of the NVRAM that hosts nonconfiguration data files. Typically these files are used by SNMP to store and retrieve nonconfiguration persistent data across the system reload.

Recommended Action    Unavailable.

Source

Unfortunately I don’t have any spare switch, so I am afraid of reloading the switch. The customer will try to get a spare switch to replace this one. Maybe I try to recover the switch, when I replaced it with a new one.

Policy NAT on Cisco router

A colleague of mine had to implement an IPSec VPN tunnel from a customer to a supplier. The customer has a Cisco router for connecting to the Internet, so nothing special. The router is already setup and in production. Configuring an extra IPSec VPN tunnel isn’t very hard, the most important part is the negotiation of Phase 1 and Phase 2 credentials for the IPSec VPN connection.

This particular situation was different, because the customer has to NAT his local IP addresses into the VPN tunnel. I don’t actual configure this quiet often on routers, but more on firewalls. I setup a testing environment with GNS3 to do my own configuration.

The testing environment is displayed below. When connecting from the LAN from R1 (10.1.1.0/24) to the Internet normal NAT overload in configured. In the picture the network behind R3 represents the Internet. When connecting from R1 to R2, the source IP address (Inside Local) is translated to an address from the pool 10.22.44.0/24 (Inside Global). There is also a static NAT mapping for IP address 10.1.1.222 into the VPN tunnel (10.22.44.222).

At first I configured the environment like showed above. I configured the different interfaces and the corresponding IP addresses. Router R1 and R2 use router R3 as default gateway. The configuration of the specific routers is attached at the end of the post.

The first snippet shows the necessary NAT configuration on R1 for the Internet connection.

interface Loopback0
description INSIDE
ip address 10.1.1.1 255.255.255.0
ip nat inside
!
interface FastEthernet0/0

description OUTSIDE
ip address 212.123.212.9 255.255.255.248
ip nat outside
duplex auto
speed auto

!
ip nat inside source list ACL-NAT interface FastEthernet0/0 overload
ip route 0.0.0.0 0.0.0.0 212.123.212.11
!
ip access-list extended ACL-NAT
permit ip 10.1.1.0 0.0.0.255 any

When trying to ping Lo0 on router R3, I see the following NAT table on router R1.

R1#ping 192.168.3.1 source lo0

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.3.1, timeout is 2 seconds:
Packet sent with a source address of 10.1.1.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 32/46/76 ms
R1#sh ip nat translations
Pro Inside global      Inside local       Outside local      Outside global
icmp 212.123.212.9:3   10.1.1.1:3         192.168.3.1:3      192.168.3.1:3

This proves that regular NAT overloading works perfectly with the current configuration. Next I configured the Policy IPSec VPN between routers R1 and R2. The configuration isn’t that spectacular, you should pay attention to the NAT statements and ACL statements. I even configured reverse route injection for the LAN network of R2. When enabling reverse route injection a route with the remote network (172.16.2.0) pops up in the routing table. This feature can be used when redistributing static routes into a routing protocol like EIGRP.

Next I will show some snippets, with some comments, of the configuration of router R1.

\\ Specifying the Phase I and Phase II properties for the IPSec VPN

crypto isakmp policy 10
encr aes 256
authentication pre-share
group 5
crypto isakmp key VPN@Booches address 212.123.212.10
!
crypto ipsec transform-set VPN-TS esp-aes 256 esp-sha-hmac
!
crypto map CM-VPN-R2 10 ipsec-isakmp
set peer 212.123.212.10
set transform-set VPN-TS
match address VPN-R2
reverse-route

!

\\ Loopback interface with 2 IP addresses for testing purposes

\\ Secondary IP address is used for static policy NAT testing

interface Loopback0
ip address 10.1.1.222 255.255.255.0 secondary
ip address 10.1.1.1 255.255.255.0
ip nat inside

!

\\ Outside interface with the crypto map applied to this interface
interface FastEthernet0/0
description OUTSIDE
ip address 212.123.212.9 255.255.255.248
ip nat outside
duplex auto
speed auto
crypto map CM-VPN-R2

!

ip nat translation timeout 30

\\ NAT pool for dynamic policy NATZ
ip nat pool LAN-R2 10.22.44.1 10.22.44.254 netmask 255.255.255.0

\\ Default NAT for regular Internet related traffic
ip nat inside source list ACL-NAT interface FastEthernet0/0 overload

\\ NAT statement for dynamic policy NAT and static policy NAT
ip nat inside source list ACL-POLICY-NAT pool LAN-R2 overload
ip nat inside source static 10.1.1.222 10.22.44.222 route-map RM-STATIC-NAT extendable

!

\\ ACL to define traffic to be NATted for regular Internet related traffic

ip access-list extended ACL-NAT
deny   ip 10.1.1.0 0.0.0.255 172.16.2.0 0.0.0.255
deny   ip 10.22.44.0 0.0.0.255 172.16.2.0 0.0.0.255
permit ip 10.1.1.0 0.0.0.255 any

\\ ACL to define traffic to be dynamic policy NATted into IPSec VPN tunnel
ip access-list extended ACL-POLICY-NAT

deny   ip host 10.1.1.222 172.16.2.0 0.0.0.255
permit ip 10.1.1.0 0.0.0.255 172.16.2.0 0.0.0.255

\\ ACL to define traffic to be static policy NATted into IPSec VPN tunnel
ip access-list extended ACL-STATIC-POLICY-NAT
permit ip host 10.1.1.222 172.16.2.0 0.0.0.255

\\ ACL to define interesting traffic for the IPSec VPN tunnel
ip access-list extended VPN-R2
permit ip 10.22.44.0 0.0.0.255 172.16.2.0 0.0.0.255
!

\\ Route map static to configure static policy NAT statement
route-map RM-STATIC-NAT permit 10
match ip address ACL-STATIC-POLICY-NAT

I issued some ping commands and looked at the IP NAT Translations table to test the environment. The test results are displayed below.

SOURCE 10.1.1.1 (R1) – DESTINATION 172.16.2.1 (R2) & 212.123.212 (R3)

R1#ping 172.16.2.1 source 10.1.1.1
Packet sent with a source address of 10.1.1.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 28/52/120 ms
R1#ping 212.123.212.11 source 10.1.1.1
Packet sent with a source address of 10.1.1.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 16/48/84 ms
R1#show ip nat translations
Pro Inside global      Inside local       Outside local      Outside global
icmp 212.123.212.9:29  10.1.1.1:29        212.123.212.11:29  212.123.212.11:29
icmp 10.22.44.1:28     10.1.1.1:28        172.16.2.1:28      172.16.2.1:28

SOURCE 10.1.1.222 (R1) – DESTINATION 172.16.2.1 (R2) & 212.123.212 (R3)

R1#ping 172.16.2.1 source 10.1.1.222
Packet sent with a source address of 10.1.1.222
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 8/48/84 ms
R1#ping 192.168.3.1 source 10.1.1.222
Packet sent with a source address of 10.1.1.222
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 32/44/60 ms
R1#sh ip nat translation
Pro Inside global      Inside local       Outside local      Outside global
icmp 212.123.212.9:37  10.1.1.222:37      192.168.3.1:37     192.168.3.1:37
icmp 10.22.44.222:36   10.1.1.222:36      172.16.2.1:36      172.16.2.1:36

 

SOURCE 172.16.2.1 (R2) – DESTINATION 10.22.44.222 (R1)

R2#ping 10.22.44.222 source 172.16.2.1

Packet sent with a source address of 172.16.2.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 4/53/112 ms

All ICMP testing gave positive results and policy-based NAT on the Cisco router is working perfectly.

Configurations:

• Router R1
• Router R2
• Router R3

Cisco ASA & ESX: strange ARP behavior

Last week I had a very strange problem with a Cisco ASA firewall. The firewall is configured with multiple interfaces, including a DMZ interface. There are multiple servers in the DMZ. These servers are physical and virtual servers. The virtual servers are VMware servers in a blade environment.

I configured the feature

ip verify reverse-path interface DMZ

to prevent spoofing to occur. I also configured a transparent static NAT rule between the Inside network and the DMZ network and multiple static NAT rules between the DMZ network and the Outside network. I left the proxy ARP feature configured with its default settings.

The customer was complaining about log in problems and connectivity problems on the DMZ servers, especially between different DMZ servers. I have done some research and noticed that all problems were related to DMZ servers in the blade environment.

I started some connectivity test and noticed some strange ICMP behavior on the specific servers. When I started a ping from one DMZ VMware server to an other DMZ server on the same ESX host, the first ping responded with an echo-reply, but consequent pings failed. Looking at the ARP table of the server, I noticed that the firewall responded with its own MAC address for every ARP broadcast.

Looking at different forums on the Internet, everybody is speaking about the proxy ARP feature and that you should disable this feature. By default proxy ARP is enabled and I always leave it enabled. Till now I never had this problem. After disabling the proxy ARP feature for the DMZ interface

sysopt noproxyarp DMZ

the problem was solved, because the firewall doesn’t respond to the ARP queries, except for its own interface. Digging a bit deeper on forums, I never found one thread who explains why the proxy ARP feature should be disabled to solve this particular problem.

In my opinion this problem is related to the VMware environment, because I don’t have these problems with physical DMZ servers. So it is strange why the DMZ servers on the same ESX hosts cannot see each other and why does the firewall respond to the ARP queries?

In the near future the blade environment (ESX hosts, network configuration and SAN configuration) is changed, so I hope to find the exact cause and solution of the problem. Does anybody else have some suggestions??

VPN Filtering through Group Policy

When configuring a Remote Access VPN or a Site to Site VPN connection you have the ability to filter traffic entering and leaving the VPN connection. You have the ability to enable inbound IPsec sessions to bypass interface access lists. Group policy and per-user authorization access lists still apply to the traffic.

The sysopt connection permit-ipsec command allows all the traffic that enters the security appliance through a VPN tunnel to bypass interface access lists. Group policy and per-user authorization access lists still apply to the traffic. In PIX 7.1 and later, the sysopt connection permit-ipsec command is changed to sysopt connection permit-vpn.

Source

Mostly I use this option and configure some extra ACL’s to filter trafifc. Some customers don’t want to use this option and want to specify all traffic with ACL’s. This is more secure, but is a bigger burden on the management of the firewall.

From IOS 7.1 and later you have the ability to configure VPN filtering through Group Policies. In short you configure an extended ACL, link this ACL to a Group Policy and link the Group Policy to the specific Tunnel Group. The syntax (source and destination) needs to be correct for the ACL to work.

For Site to Site VPN’s the remote network is the source and the local network is the destination. For Remote Access VPN’s the VPN IP pool is the source and the local network the destination, as specified below.

An ACL that is used for a vpn-filter must not also be used for an interface access-group. When a vpn-filter is applied to a group-policy/user name mode that governs Remote Access VPN Client connections, the ACL must be configured with the client assigned IP addresses in the src_ip position of the ACL and the local network in the dest_ip position of the ACL. When a vpn-filter is applied to a group-policy that governs an L2L VPN connection, the ACL must be configured with the remote network in the src_ip position of the ACL and the local network in the dest_ip position of the ACL.

 

Exercise caution when you construct the ACLs for use with the vpn-filter feature. The ACLs are constructed with the post-decrypted traffic (inbound VPN traffic) in mind. However, they are also applied to the traffic originated in the opposite direction.

More about this matter and examples configurations can be found here.