Connecting the world…
Cacti: corrupt database
After rebooting a Cacti server, the customer complained that no new graphs were drawn by the server. I tried to run the poller.php script with the –-force option and noticed the following output:
06/16/2011 10:34:48 AM – SPINE: Poller[0] ERROR: SQL Failed! Error:’145′, Message:’Table ‘./cacti/poller_output’ is marked as crashed and should be repaired’, SQL Fragment:’INSERT INTO poller_output (local_data_id, rrd_name, time, output) VALUES (514,’traffic_in’,’2011-06-16 10:34:48′,’3446319166′),(357,’traffic_in’,’2011-06-16
This log entry was shown multiple time and it looks like the database got corrupted after the reboot. Mysql has an option to check and repair the database. So I gave that a try via the following command:
mysqlcheck –-auto-repair –-databases cacti
The command gives the following output:
cacti.cdef OK
cacti.cdef_items OK
cacti.colors OK
cacti.data_input OK
cacti.data_input_data OK
cacti.data_input_fields OK
cacti.data_local OK
cacti.data_template OK
cacti.data_template_data
warning : 1 client is using or hasn’t closed the table properly
status : OK
cacti.data_template_data_rra OK
cacti.data_template_rrd
warning : 1 client is using or hasn’t closed the table properly
status : OK
cacti.graph_local OK
cacti.graph_template_input OK
cacti.graph_template_input_defs OK
cacti.graph_templates OK
cacti.graph_templates_gprint OK
cacti.graph_templates_graph OK
cacti.graph_templates_item OK
cacti.graph_tree OK
cacti.graph_tree_items OK
cacti.host
warning : 2 clients are using or haven’t closed the table properly
status : OK
cacti.host_graph OK
cacti.host_snmp_cache
warning : 1 client is using or hasn’t closed the table properly
status : OK
cacti.host_snmp_query OK
cacti.host_template OK
cacti.host_template_graph OK
cacti.host_template_snmp_query OK
cacti.mac_track_approved_macs OK
cacti.mac_track_device_types OK
cacti.mac_track_devices OK
cacti.mac_track_ip_ranges OK
cacti.mac_track_ips
note : The storage engine for the table doesn’t support check
cacti.mac_track_macauth OK
cacti.mac_track_macwatch OK
cacti.mac_track_oui_database OK
cacti.mac_track_ports OK
cacti.mac_track_processes OK
cacti.mac_track_scan_dates OK
cacti.mac_track_scanning_functions OK
cacti.mac_track_sites OK
cacti.mac_track_temp_ports
note : The storage engine for the table doesn’t support check
cacti.plugin_color_templates OK
cacti.plugin_color_templates_item OK
cacti.plugin_config OK
cacti.plugin_db_changes
warning : 2 clients are using or haven’t closed the table properly
status : OK
cacti.plugin_discover_hosts OK
cacti.plugin_discover_template OK
cacti.plugin_flowview_devices OK
cacti.plugin_flowview_dnscache
note : The storage engine for the table doesn’t support check
cacti.plugin_flowview_queries OK
cacti.plugin_flowview_schedules OK
cacti.plugin_hooks OK
cacti.plugin_realms OK
cacti.plugin_routerconfigs_accounts OK
cacti.plugin_routerconfigs_backups OK
cacti.plugin_routerconfigs_devices OK
cacti.plugin_routerconfigs_devicetypes OK
cacti.plugin_thold_contacts OK
cacti.plugin_thold_log OK
cacti.plugin_thold_template_contact OK
cacti.plugin_thold_threshold_contact OK
cacti.plugin_update_info OK
cacti.plugin_wmi_accounts OK
cacti.plugin_wmi_queries OK
cacti.poller OK
cacti.poller_command OK
cacti.poller_item
warning : 1 client is using or hasn’t closed the table properly
status : OK
cacti.poller_output
warning : Table is marked as crashed
warning : 1 client is using or hasn’t closed the table properly
error : Invalid key block position: 107523441122544244 key block size: 1024 file_length: 25600
error : key delete-link-chain corrupted
error : Corrupt
cacti.poller_output_boost
note : The storage engine for the table doesn’t support check
cacti.poller_output_boost_processes
note : The storage engine for the table doesn’t support check
cacti.poller_output_rt OK
cacti.poller_reindex
warning : 1 client is using or hasn’t closed the table properly
status : OK
cacti.poller_time OK
cacti.quicktree_graphs OK
cacti.reportit_cache_measurands OK
cacti.reportit_cache_reports OK
cacti.reportit_cache_variables OK
cacti.reportit_measurands OK
cacti.reportit_presets OK
cacti.reportit_recipients OK
cacti.reportit_reports OK
cacti.reportit_rvars OK
cacti.reportit_templates OK
cacti.reportit_variables OK
cacti.rra OK
cacti.rra_cf OK
cacti.settings
warning : 1 client is using or hasn’t closed the table properly
status : OK
cacti.settings_graphs OK
cacti.settings_tree OK
cacti.snmp_query OK
cacti.snmp_query_graph OK
cacti.snmp_query_graph_rrd OK
cacti.snmp_query_graph_rrd_sv OK
cacti.snmp_query_graph_sv OK
cacti.superlinks_auth OK
cacti.superlinks_pages OK
cacti.thold_data OK
cacti.thold_template OK
cacti.user_auth OK
cacti.user_auth_perms OK
cacti.user_auth_realm OK
cacti.user_log OK
cacti.version OK
cacti.weathermap_auth OK
cacti.weathermap_data OK
cacti.weathermap_maps OKRepairing tables
cacti.poller_output OK
After the repair I ran the poller.php script again with the –-force option and this time I didn’t receive any errors and the graphs were updated again.
Afterwards I noticed that Cacti has a script of its own to repair the database. This script is called repair_database.php and can be found in the directory /var/www/html/cli/.
Another NVRAM broken?
On Monday I visited another customer who had problems saving the running configuration of a Cisco devices. The devices involved were a Cisco 2620 and a Cisco 2610XM router. Both routers weren’t able to save their running configuration.
Both routers show the following error-message:
startup-config file open failed (Bad file number)
By both routers I was able to look at the contents of the flash memory, but I wasn’t able to look at the contents of the NVRAM. When trying to look at the contents of NVRAM, you receive the following error-message:
Directory of nvram:/
%Error opening nvram:/ (Bad file number)
No space information available
22w0d: %SYS-4-NV_BLOCK_INITFAIL: Unable to initialize the geometry of nvram
The following information can be found on the error-message on the Cisco website:
%SYS-4-NV_BLOCK_INITFAIL : Unable to initialize the geometry of nvram
Explanation The software has detected that it failed to initialize the NVRAM block geometry (a part of the NVRAM that hosts non-configuration data files). Typically, these files are used by SNMP to store and retrieve non-configuration persistent data across a system reload. This condition may occur when the entire NVRAM is packed with the configuration, and the newer version of software that supports this feature could not find enough room in the NVRAM to initialize the block file system.
Recommended Action Reduce the configurations in the NVRAM by at least 2Kb.
Luckily one of the two routers was the old production router, which was swapped and the customer thought the NVRAM was broken. So I could use that router for testing purposes.
I started looking at the some physical aspects of both routers. They both had the following hardware specifications:
- 16 MB Flash memory
- 64 MB RAM memory
- 32K NVRAM
- IOS version 12.4(5)
While looking at the running configuration of the active router, I noticed that the running configuration was almost 2900 bytes (29K), which is stored in NVRAM. I believed that the error-messages were generated because NVRAM was full. I started deleting some configuration from the broken router, until the running configuration was only 20K. But I still wasn’t able to save the running configuration.
The last change I had was updating the IOS. I downloaded the last Main Release, which is 12.4(23). I formatted the flash memory and uploaded the image to the spare router. And fortunately, after a reload, I was able to save the running configuration.
The running configuration was still almost 2900 bytes. I issues the command to compress the configuration:
service compress-config
Now the running configuration is compressed from almost 2900 bytes to 1000 bytes.
NVRAM corrupt or broken?
Today some of my colleagues and I “rebuild” an existing ESX with NetApp network. We change multiple VLAN’s and did a lot of reconfiguring. Unfortunately some other people were working on the power, so sometimes all equipment had to power down.
After we did our job, we started testing the environment. All DMZ VM’s weren’t able to connect to their default gateway, which is a Cisco ASA 5550 active/standby firewall. I did some research and noticed that the DMZ switch lost his configuration. At first I thought that I didn’t save the configuration to NVRAM. Luckily I had a backup configuration, which I copied and pasted into the switch. To be sure I issued a write mem, which gave me the following response:
SW01#wr mem
Building configuration…nv_done: unable to open “flash:/config.text.new”
nv_done: unable to open “flash:/private-config.text.new”[OK]
NVRAM Verification Failed
Hhhuuuummm, I tried a show flash: and even a format flash:, but no success. Cisco’s SupportWiki tells me to try and erase the startup-configuration from NVRAM and issue a new write mem, but again no success. While executing the command erase start, the following message appears on the console:
Jan 24 16:05:31: %SYS-7-NV_BLOCK_INIT: Initialized the geometry of nvram
Looking at Cisco’s website, they have the following description with the error message:
%SYS-7-NV_BLOCK_INIT : Initalized the geometry of nvram
Explanation The NVRAM block geometry has been initialized. NVRAM block geometry is a part of the NVRAM that hosts nonconfiguration data files. Typically these files are used by SNMP to store and retrieve nonconfiguration persistent data across the system reload.
Recommended Action Unavailable.
Unfortunately I don’t have any spare switch, so I am afraid of reloading the switch. The customer will try to get a spare switch to replace this one. Maybe I try to recover the switch, when I replaced it with a new one.
Cisco 877W wireless authentication failed
At home I have a Cisco 877W router. I use the wireless functionality to connect the different laptops to the networks. After upgrading the software from the router I have problems with the wireless authentication. The router is working perfectly, but after some time the laptops are able to connect to the wireless network. Vista tells me to enter the correct pre-shared key, so this doesn’t help much.
In the buffer logging I see the following error messages:
Jan 6 2009 22:48:05.666 CET: %DOT11-7-CCKM_AUTH_FAILED: Station <mac address> CCKM authentication failed
Looking at different forums more people experience the same problem. They offer different solutions like:
- Changing the broadcast key change interval
- Enable AES encryption
Both solutions didn’t work for me. Because I noticed the problems after upgrading the software, I decided to downgrade the software. I downgraded from ADVSECURITY Version 12.4(22)T to ADVSECURITY 12.4(15)T8.
I searched the Cisco website and Bug Toolkit, but I couldn’t find any possible bug information about my problem. But I am sure this problem is related to the IOS image previously used. After downgrading I didn’t have any more problems with the wireless environment.
Failed to establish VPN through PIX
We migrated our Internet connection lately and reconfigured our PIX firewall. We added some memory to install the latest firmware version (8.0(4)). After putting the PIX firewall in production some of the employees were complaining they couldn’t establish any PPTP VPN Tunnels anymore to customers.
Every time when some one called me, I tried it myself and I was always able to connect using a PPTP VPN Tunnel, but every time I was working remote and not at the office. So I always thought that something was wrong with there laptops, but today I encountered the problem myself.
Looking at the logging of the PIX firewall, I saw the following error message:
%ASA-3-305006: regular translation creation failed for protocol 47 src inside:<IP address> dst outside:<IP address>
The error message indicates that there is no NAT mapping for the specified traffic, which could direct you in the wrong direction. I checked the NAT mappings to be sure, but as I already thought, this couldn’t be the cause of the problem.
PPTP uses a TCP connection that uses port 1723 and an extension of generic routing encapsulation (GRE) [protocol 47] to carry the actual data (PPP frame). The TCP connection is initiated by the client, followed by the GRE connection that is initiated by the server. Because the PPTP connection is initiated as TCP on one port and the response is GRE protocol, the PIX Adaptive Security Algorithm (ASA) does not know that the traffic flows are related.
The PPTP fixup feature in version 6.3 allows the PPTP traffic to traverse the PIX when configured for PAT. Stateful PPTP packet inspection is also performed in the process. The fixup protocol pptp command inspects PPTP packets and dynamically creates the GRE connections and translations necessary to permit PPTP traffic. Specifically, the firewall inspects the PPTP version announcements and the outgoing call request/response sequence. Only PPTP Version 1, as defined in RFC 2637, is inspected. Further inspection on the TCP control channel is disabled if the version announced by either side is not Version 1. In addition, the outgoing call request and reply sequence is tracked. Connections and/or translations are dynamically allocated as necessary to permit subsequent secondary GRE data traffic. The PPTP fixup feature must be enabled for PPTP traffic to be translated by PAT.
So I had to configure the fixup protocol pptp feature with the following command:
fw01(config)# fixup protocol pptp 1723
As stated before, we are using fireware version 8.0(4). This version doesn’t support the fixup protocol pptp command and the converts the command an inspect pptp command as shown below.
fw01(config)# fixup protocol pptp 1723
INFO: converting ‘fixup protocol pptp 1723’ to MPF commands!
!
policy-map global_policy
class inspection_default
inspect dns preset_dns_map
inspect ftp
inspect pptp
Follow Me
