Connecting the world…

failed

Cacti: corrupt database

After rebooting a Cacti server, the customer complained that no new graphs were drawn by the server. I tried to run the poller.php script with the –-force option and noticed the following output:

06/16/2011 10:34:48 AM – SPINE: Poller[0] ERROR: SQL Failed! Error:’145′, Message:’Table ‘./cacti/poller_output’ is marked as crashed and should be repaired’, SQL Fragment:’INSERT INTO poller_output (local_data_id, rrd_name, time, output) VALUES (514,’traffic_in’,’2011-06-16 10:34:48′,’3446319166′),(357,’traffic_in’,’2011-06-16

This log entry was shown multiple time and it looks like the database got corrupted after the reboot. Mysql has an option to check and repair the database. So I gave that a try via the following command:

mysqlcheck –-auto-repair –-databases cacti

The command gives the following output:

cacti.cdef                                         OK
cacti.cdef_items                                   OK
cacti.colors                                       OK
cacti.data_input                                   OK
cacti.data_input_data                              OK
cacti.data_input_fields                            OK
cacti.data_local                                   OK
cacti.data_template                                OK
cacti.data_template_data
warning  : 1 client is using or hasn’t closed the table properly
status   : OK
cacti.data_template_data_rra                       OK
cacti.data_template_rrd
warning  : 1 client is using or hasn’t closed the table properly
status   : OK
cacti.graph_local                                  OK
cacti.graph_template_input                         OK
cacti.graph_template_input_defs                    OK
cacti.graph_templates                              OK
cacti.graph_templates_gprint                       OK
cacti.graph_templates_graph                        OK
cacti.graph_templates_item                         OK
cacti.graph_tree                                   OK
cacti.graph_tree_items                             OK
cacti.host
warning  : 2 clients are using or haven’t closed the table properly
status   : OK
cacti.host_graph                                   OK
cacti.host_snmp_cache
warning  : 1 client is using or hasn’t closed the table properly
status   : OK
cacti.host_snmp_query                              OK
cacti.host_template                                OK
cacti.host_template_graph                          OK
cacti.host_template_snmp_query                     OK
cacti.mac_track_approved_macs                      OK
cacti.mac_track_device_types                       OK
cacti.mac_track_devices                            OK
cacti.mac_track_ip_ranges                          OK
cacti.mac_track_ips
note     : The storage engine for the table doesn’t support check
cacti.mac_track_macauth                            OK
cacti.mac_track_macwatch                           OK
cacti.mac_track_oui_database                       OK
cacti.mac_track_ports                              OK
cacti.mac_track_processes                          OK
cacti.mac_track_scan_dates                         OK
cacti.mac_track_scanning_functions                 OK
cacti.mac_track_sites                              OK
cacti.mac_track_temp_ports
note     : The storage engine for the table doesn’t support check
cacti.plugin_color_templates                       OK
cacti.plugin_color_templates_item                  OK
cacti.plugin_config                                OK
cacti.plugin_db_changes
warning  : 2 clients are using or haven’t closed the table properly
status   : OK
cacti.plugin_discover_hosts                        OK
cacti.plugin_discover_template                     OK
cacti.plugin_flowview_devices                      OK
cacti.plugin_flowview_dnscache
note     : The storage engine for the table doesn’t support check
cacti.plugin_flowview_queries                      OK
cacti.plugin_flowview_schedules                    OK
cacti.plugin_hooks                                 OK
cacti.plugin_realms                                OK
cacti.plugin_routerconfigs_accounts                OK
cacti.plugin_routerconfigs_backups                 OK
cacti.plugin_routerconfigs_devices                 OK
cacti.plugin_routerconfigs_devicetypes             OK
cacti.plugin_thold_contacts                        OK
cacti.plugin_thold_log                             OK
cacti.plugin_thold_template_contact                OK
cacti.plugin_thold_threshold_contact               OK
cacti.plugin_update_info                           OK
cacti.plugin_wmi_accounts                          OK
cacti.plugin_wmi_queries                           OK
cacti.poller                                       OK
cacti.poller_command                               OK
cacti.poller_item
warning  : 1 client is using or hasn’t closed the table properly
status   : OK
cacti.poller_output
warning  : Table is marked as crashed
warning  : 1 client is using or hasn’t closed the table properly
error    : Invalid key block position: 107523441122544244  key block size: 1024  file_length: 25600
error    : key delete-link-chain corrupted
error    : Corrupt
cacti.poller_output_boost
note     : The storage engine for the table doesn’t support check
cacti.poller_output_boost_processes
note     : The storage engine for the table doesn’t support check
cacti.poller_output_rt                             OK
cacti.poller_reindex
warning  : 1 client is using or hasn’t closed the table properly
status   : OK
cacti.poller_time                                  OK
cacti.quicktree_graphs                             OK
cacti.reportit_cache_measurands                    OK
cacti.reportit_cache_reports                       OK
cacti.reportit_cache_variables                     OK
cacti.reportit_measurands                          OK
cacti.reportit_presets                             OK
cacti.reportit_recipients                          OK
cacti.reportit_reports                             OK
cacti.reportit_rvars                               OK
cacti.reportit_templates                           OK
cacti.reportit_variables                           OK
cacti.rra                                          OK
cacti.rra_cf                                       OK
cacti.settings
warning  : 1 client is using or hasn’t closed the table properly
status   : OK
cacti.settings_graphs                              OK
cacti.settings_tree                                OK
cacti.snmp_query                                   OK
cacti.snmp_query_graph                             OK
cacti.snmp_query_graph_rrd                         OK
cacti.snmp_query_graph_rrd_sv                      OK
cacti.snmp_query_graph_sv                          OK
cacti.superlinks_auth                              OK
cacti.superlinks_pages                             OK
cacti.thold_data                                   OK
cacti.thold_template                               OK
cacti.user_auth                                    OK
cacti.user_auth_perms                              OK
cacti.user_auth_realm                              OK
cacti.user_log                                     OK
cacti.version                                      OK
cacti.weathermap_auth                              OK
cacti.weathermap_data                              OK
cacti.weathermap_maps                              OK

Repairing tables
cacti.poller_output                                OK

After the repair I ran the poller.php script again with the –-force option and this time I didn’t receive any errors and the graphs were updated again.

Afterwards I noticed that Cacti has a script of its own to repair the database. This script is called repair_database.php and can be found in the directory /var/www/html/cli/.

Another NVRAM broken?

On Monday I visited another customer who had problems saving the running configuration of a Cisco devices. The devices involved were a Cisco 2620 and a Cisco 2610XM router. Both routers weren’t able to save their running configuration.

Both routers show the following error-message:

startup-config file open failed (Bad file number)

By both routers I was able to look at the contents of the flash memory, but I wasn’t able to look at the contents of the NVRAM. When trying to look at the contents of NVRAM, you receive the following error-message:

Directory of nvram:/

%Error opening nvram:/ (Bad file number)

No space information available

22w0d: %SYS-4-NV_BLOCK_INITFAIL: Unable to initialize the geometry of nvram

The following information can be found on the error-message on the Cisco website:

%SYS-4-NV_BLOCK_INITFAIL : Unable to initialize the geometry of nvram

 

Explanation    The software has detected that it failed to initialize the NVRAM block geometry (a part of the NVRAM that hosts non-configuration data files). Typically, these files are used by SNMP to store and retrieve non-configuration persistent data across a system reload. This condition may occur when the entire NVRAM is packed with the configuration, and the newer version of software that supports this feature could not find enough room in the NVRAM to initialize the block file system.

Recommended Action    Reduce the configurations in the NVRAM by at least 2Kb.

Source

Luckily one of the two routers was the old production router, which was swapped and the customer thought the NVRAM was broken. So I could use that router for testing purposes.

I started looking at the some physical aspects of both routers. They both had the following hardware specifications:

  • 16 MB Flash memory
  • 64 MB RAM memory
  • 32K NVRAM
  • IOS version 12.4(5)

While looking at the running configuration of the active router, I noticed that the running configuration was almost 2900 bytes (29K), which is stored in NVRAM. I believed that the error-messages were generated because NVRAM was full. I started deleting some configuration from the broken router, until the running configuration was only 20K. But I still wasn’t able to save the running configuration.

The last change I had was updating the IOS. I downloaded the last Main Release, which is 12.4(23). I formatted the flash memory and uploaded the image to the spare router. And fortunately, after a reload, I was able to save the running configuration.

The running configuration was still almost 2900 bytes. I issues the command to compress the configuration:

service compress-config

Now the running configuration is compressed from almost 2900 bytes to 1000 bytes.

NVRAM corrupt or broken?

Today some of my colleagues and I “rebuild” an existing ESX with NetApp network. We change multiple VLAN’s and did a lot of reconfiguring. Unfortunately some other people were working on the power, so sometimes all equipment had to power down.

After we did our job, we started testing the environment. All DMZ VM’s weren’t able to connect to their default gateway, which is a Cisco ASA 5550 active/standby firewall. I did some research and noticed that the DMZ switch lost his configuration. At first I thought that I didn’t save the configuration to NVRAM. Luckily I had a backup configuration, which I copied and pasted into the switch. To be sure I issued a write mem, which gave me the following response:

SW01#wr mem
Building configuration…

nv_done: unable to open “flash:/config.text.new”
nv_done: unable to open “flash:/private-config.text.new”[OK]
NVRAM Verification Failed

Hhhuuuummm, I tried a show flash: and even a format flash:, but no success. Cisco’s SupportWiki tells me to try and erase the startup-configuration from NVRAM and issue a new write mem, but again no success. While executing the command erase start, the following message appears on the console:

Jan 24 16:05:31: %SYS-7-NV_BLOCK_INIT: Initialized the geometry of nvram

Looking at Cisco’s website, they have the following description with the error message:

%SYS-7-NV_BLOCK_INIT : Initalized the geometry of nvram

 

Explanation    The NVRAM block geometry has been initialized. NVRAM block geometry is a part of the NVRAM that hosts nonconfiguration data files. Typically these files are used by SNMP to store and retrieve nonconfiguration persistent data across the system reload.

Recommended Action    Unavailable.

Source

Unfortunately I don’t have any spare switch, so I am afraid of reloading the switch. The customer will try to get a spare switch to replace this one. Maybe I try to recover the switch, when I replaced it with a new one.

Cisco 877W wireless authentication failed

At home I have a Cisco 877W router. I use the wireless functionality to connect the different laptops to the networks. After upgrading the software from the router I have problems with the wireless authentication. The router is working perfectly, but after some time the laptops are able to connect to the wireless network. Vista tells me to enter the correct pre-shared key, so this doesn’t help much.

In the buffer logging I see the following error messages:

Jan 6 2009  22:48:05.666 CET: %DOT11-7-CCKM_AUTH_FAILED: Station <mac address> CCKM authentication failed

Looking at different forums more people experience the same problem. They offer different solutions like:

  • Changing the broadcast key change interval
  • Enable AES encryption

Both solutions didn’t work for me. Because I noticed the problems after upgrading the software, I decided to downgrade the software. I downgraded from ADVSECURITY Version 12.4(22)T to ADVSECURITY 12.4(15)T8.

I searched the Cisco website and Bug Toolkit, but I couldn’t find any possible bug information about my problem. But I am sure this problem is related to the IOS image previously used. After downgrading I didn’t have any more problems with the wireless environment.

Failed to establish VPN through PIX

We migrated our Internet connection lately and reconfigured our PIX firewall. We added some memory to install the latest firmware version (8.0(4)). After putting the PIX firewall in production some of the employees were complaining they couldn’t establish any PPTP VPN Tunnels anymore to customers.

Every time when some one called me, I tried it myself and I was always able to connect using a PPTP VPN Tunnel, but every time I was working remote and not at the office. So I always thought that something was wrong with there laptops, but today I encountered the problem myself.

Looking at the logging of the PIX firewall, I saw the following error message:

%ASA-3-305006: regular translation creation failed for protocol 47 src inside:<IP address> dst outside:<IP address>

The error message indicates that there is no NAT mapping for the specified traffic, which could direct you in the wrong direction. I checked the NAT mappings to be sure, but as I already thought, this couldn’t be the cause of the problem.

PPTP uses a TCP connection that uses port 1723 and an extension of generic routing encapsulation (GRE) [protocol 47] to carry the actual data (PPP frame). The TCP connection is initiated by the client, followed by the GRE connection that is initiated by the server. Because the PPTP connection is initiated as TCP on one port and the response is GRE protocol, the PIX Adaptive Security Algorithm (ASA) does not know that the traffic flows are related.

The PPTP fixup feature in version 6.3 allows the PPTP traffic to traverse the PIX when configured for PAT. Stateful PPTP packet inspection is also performed in the process. The fixup protocol pptp command inspects PPTP packets and dynamically creates the GRE connections and translations necessary to permit PPTP traffic. Specifically, the firewall inspects the PPTP version announcements and the outgoing call request/response sequence. Only PPTP Version 1, as defined in RFC 2637, is inspected. Further inspection on the TCP control channel is disabled if the version announced by either side is not Version 1. In addition, the outgoing call request and reply sequence is tracked. Connections and/or translations are dynamically allocated as necessary to permit subsequent secondary GRE data traffic. The PPTP fixup feature must be enabled for PPTP traffic to be translated by PAT.

So I had to configure the fixup protocol pptp feature with the following command:

fw01(config)# fixup protocol pptp 1723

As stated before, we are using fireware version 8.0(4). This version doesn’t support the fixup protocol pptp command and the converts the command an inspect pptp command as shown below.

fw01(config)# fixup protocol pptp 1723
INFO: converting ‘fixup protocol pptp 1723’ to MPF commands

!

!

policy-map global_policy
class inspection_default
  inspect dns preset_dns_map
  inspect ftp
  inspect pptp