Connecting the world…


Another NVRAM broken?

On Monday I visited another customer who had problems saving the running configuration of a Cisco devices. The devices involved were a Cisco 2620 and a Cisco 2610XM router. Both routers weren’t able to save their running configuration.

Both routers show the following error-message:

startup-config file open failed (Bad file number)

By both routers I was able to look at the contents of the flash memory, but I wasn’t able to look at the contents of the NVRAM. When trying to look at the contents of NVRAM, you receive the following error-message:

Directory of nvram:/

%Error opening nvram:/ (Bad file number)

No space information available

22w0d: %SYS-4-NV_BLOCK_INITFAIL: Unable to initialize the geometry of nvram

The following information can be found on the error-message on the Cisco website:

%SYS-4-NV_BLOCK_INITFAIL : Unable to initialize the geometry of nvram


Explanation    The software has detected that it failed to initialize the NVRAM block geometry (a part of the NVRAM that hosts non-configuration data files). Typically, these files are used by SNMP to store and retrieve non-configuration persistent data across a system reload. This condition may occur when the entire NVRAM is packed with the configuration, and the newer version of software that supports this feature could not find enough room in the NVRAM to initialize the block file system.

Recommended Action    Reduce the configurations in the NVRAM by at least 2Kb.


Luckily one of the two routers was the old production router, which was swapped and the customer thought the NVRAM was broken. So I could use that router for testing purposes.

I started looking at the some physical aspects of both routers. They both had the following hardware specifications:

  • 16 MB Flash memory
  • 64 MB RAM memory
  • 32K NVRAM
  • IOS version 12.4(5)

While looking at the running configuration of the active router, I noticed that the running configuration was almost 2900 bytes (29K), which is stored in NVRAM. I believed that the error-messages were generated because NVRAM was full. I started deleting some configuration from the broken router, until the running configuration was only 20K. But I still wasn’t able to save the running configuration.

The last change I had was updating the IOS. I downloaded the last Main Release, which is 12.4(23). I formatted the flash memory and uploaded the image to the spare router. And fortunately, after a reload, I was able to save the running configuration.

The running configuration was still almost 2900 bytes. I issues the command to compress the configuration:

service compress-config

Now the running configuration is compressed from almost 2900 bytes to 1000 bytes.

RSA Authentication Manager 7.1 on VMware

I had to install and configure RSA Authentication Manager 7.1. Looking at the Supported Platforms I couldn’t find VMware ESX as supported platform. VMware ESX was supported for RSA AU6.1. So I thought by myself, let’s give it a try. What I noticed first was the size of the installer. The installation file for RSA AM 7.1 is about 2.5Gb, which I think is a lot compared to the 300Mb for RSA AM 6.1.

I installed a server with the following specs:

  • 2 x Intel Xeon 2.0 Ghz processor
  • 2Gb of RAM
  • 60 Gb partition, solely for RSA
  • 2Gb Paging file

The installation of RSA Authentication Manager 7.1 took 1,5 hours to install, so I really started doubting the installation under VMware. After the installation I wasn’t able to open the management console, which runs webbased in this new version. To be sure, I restarted the server after the installation. Now it took 45 minutes to pass the Applying computer settings and Applying personal settings.

I called RSA and the engineer told me that there are no known issues for running RSA Authentication Manager 7.1 under VMware. The only important thing he told me was the usage of 4Gb RAM and a 4GB Paging file, when running under VMware. I upgraded the memory from 2Gb RAM to 4GB RAM and I configured two 4Gb paging files.

You maybe already guess the following lines of text, but the upgrade didn’t work out. The boot process still took approximately 45 minutes. After booting the server, the performance was really bad. The memory usage was steadily running on 4.2 Gb!!!!

I called RSA a second time and the next engineer took my doubts away. The told that RSA Authentication Manager 7.1 is NOT OFFICIALE supported by RSA. The performance problems are probably caused by the new Oracle database and the different Java instances, which are running on the server. Because RSA had to run in a virtual environment, I downloaded RSA AM 6.1. The installation AND configuration of the complete environment took about 2 hours.

So at the time of writing this blog post:


ADD ON August 15th 2009

RSA 7.1 is now supported under ESX 3.5. Check the updated article on this matter.

Maybe you also want to check this article about configuring On-Demand with RSA 7.1.

WebMarshal performance problems

One of our customers is using WebMarshal for HTTP/HTTPS URL filtering and content scanning. The WebMarshall software is installed on two Microsoft ISA 2003 servers. These ISA servers are behind a Cisco Content Switch for load-balancing and redundancy purposes.

The problem with the WebMarshal is the PERFORMANCE. Internet browsing with the WebMarshal as proxy just doesn’t perform. I tried to troubleshoot the WebMarshal to check where the performance problems are coming from, but you cannot troubleshoot the software on a decent way. I disabled the Access Policies, and guess what, the performance is great. I added a allow all rule on top of every Access Policy subcategory, but no success.

I know the customer is running an old version (3.0.x), and of course if you contact the supplier, the first thing they say is: “Upgrade to the last version!!”. It seems the solution is always upgrading the last version. The second thing the supplier told us, was using Microsoft Network Load Balancing and not the Content Switches. Sadly the customer is using HP ProCurve switches, which don’t support static ARP entries. So NLB is no option.

But again, I give them the benefit of the doubt, so we will install two new servers, which are dedicated for WebMarshal software. Still the servers will be behind the Content Switch, because I believe that the Content Switches are the reason for the bad performance.

I will tell you more about the outcome of the latest version of WebMarshal on dedicated hardware. My opinion so far: “Feed the WebMarshal software to the dogs and buy something else!!!!!!!!!!”