Connecting the world…

open

Another NVRAM broken?

On Monday I visited another customer who had problems saving the running configuration of a Cisco devices. The devices involved were a Cisco 2620 and a Cisco 2610XM router. Both routers weren’t able to save their running configuration.

Both routers show the following error-message:

startup-config file open failed (Bad file number)

By both routers I was able to look at the contents of the flash memory, but I wasn’t able to look at the contents of the NVRAM. When trying to look at the contents of NVRAM, you receive the following error-message:

Directory of nvram:/

%Error opening nvram:/ (Bad file number)

No space information available

22w0d: %SYS-4-NV_BLOCK_INITFAIL: Unable to initialize the geometry of nvram

The following information can be found on the error-message on the Cisco website:

%SYS-4-NV_BLOCK_INITFAIL : Unable to initialize the geometry of nvram

 

Explanation    The software has detected that it failed to initialize the NVRAM block geometry (a part of the NVRAM that hosts non-configuration data files). Typically, these files are used by SNMP to store and retrieve non-configuration persistent data across a system reload. This condition may occur when the entire NVRAM is packed with the configuration, and the newer version of software that supports this feature could not find enough room in the NVRAM to initialize the block file system.

Recommended Action    Reduce the configurations in the NVRAM by at least 2Kb.

Source

Luckily one of the two routers was the old production router, which was swapped and the customer thought the NVRAM was broken. So I could use that router for testing purposes.

I started looking at the some physical aspects of both routers. They both had the following hardware specifications:

  • 16 MB Flash memory
  • 64 MB RAM memory
  • 32K NVRAM
  • IOS version 12.4(5)

While looking at the running configuration of the active router, I noticed that the running configuration was almost 2900 bytes (29K), which is stored in NVRAM. I believed that the error-messages were generated because NVRAM was full. I started deleting some configuration from the broken router, until the running configuration was only 20K. But I still wasn’t able to save the running configuration.

The last change I had was updating the IOS. I downloaded the last Main Release, which is 12.4(23). I formatted the flash memory and uploaded the image to the spare router. And fortunately, after a reload, I was able to save the running configuration.

The running configuration was still almost 2900 bytes. I issues the command to compress the configuration:

service compress-config

Now the running configuration is compressed from almost 2900 bytes to 1000 bytes.

NVRAM corrupt or broken?

Today some of my colleagues and I “rebuild” an existing ESX with NetApp network. We change multiple VLAN’s and did a lot of reconfiguring. Unfortunately some other people were working on the power, so sometimes all equipment had to power down.

After we did our job, we started testing the environment. All DMZ VM’s weren’t able to connect to their default gateway, which is a Cisco ASA 5550 active/standby firewall. I did some research and noticed that the DMZ switch lost his configuration. At first I thought that I didn’t save the configuration to NVRAM. Luckily I had a backup configuration, which I copied and pasted into the switch. To be sure I issued a write mem, which gave me the following response:

SW01#wr mem
Building configuration…

nv_done: unable to open “flash:/config.text.new”
nv_done: unable to open “flash:/private-config.text.new”[OK]
NVRAM Verification Failed

Hhhuuuummm, I tried a show flash: and even a format flash:, but no success. Cisco’s SupportWiki tells me to try and erase the startup-configuration from NVRAM and issue a new write mem, but again no success. While executing the command erase start, the following message appears on the console:

Jan 24 16:05:31: %SYS-7-NV_BLOCK_INIT: Initialized the geometry of nvram

Looking at Cisco’s website, they have the following description with the error message:

%SYS-7-NV_BLOCK_INIT : Initalized the geometry of nvram

 

Explanation    The NVRAM block geometry has been initialized. NVRAM block geometry is a part of the NVRAM that hosts nonconfiguration data files. Typically these files are used by SNMP to store and retrieve nonconfiguration persistent data across the system reload.

Recommended Action    Unavailable.

Source

Unfortunately I don’t have any spare switch, so I am afraid of reloading the switch. The customer will try to get a spare switch to replace this one. Maybe I try to recover the switch, when I replaced it with a new one.