Troubleshooting Availability Reports
This article explains how to troubleshoot missing or incorrect availability data, lingering downtimes, and outdated BI downtime states in Checkmk.
LAST TESTED ON CHECKMK 2.5.0P1
Overview
Availability reports rely on historical host and service state information. When troubleshooting incorrect availability data, it is important to collect the relevant history files together with the current agent output or SNMP walk. This information can be used to identify the exact timestamps of state changes and reproduce the reported behavior.
Common Problems
Missing availability history or N/A states
In distributed monitoring environments, newly added hosts may initially show availability correctly, but after restarting the Checkmk server or activating changes, the availability timeline may show N/A for an extended period.
In some cases, monitoring only begins displaying the expected states again after manually restarting the site.
Constant downtime without an end time
A host or service may appear to remain in downtime even though the downtime is no longer active. This can result in incorrect availability calculations or outdated downtime states in BI aggregates.
Collect troubleshooting data
Before investigating the cause, collect the history data and current monitoring output for the affected host.
Extract the host history
Replace <myhost> with the affected host.
Translated history
Run the following command as the site user to extract the history and convert the Unix timestamps into readable dates:
for history in $(find ~/var/check_mk/core -name "history*"); do cat "$history" | grep -ai "\<myhost>" | awk -F '[][]' '{$2=strftime("%Y-%m-%d %H:%M:%S", $2); print $0}'; done > translated_history.txtThe translated output makes it easier to identify when specific host or service state changes occurred.
Raw history
Also collect the original history data. This can be imported into another Checkmk site if the issue needs to be reproduced.
for history in $(find ~/var/check_mk/core -name "history*"); do cat "$history" | grep -ia "<myhost>"; done > history-$(date +%s)
Collect the agent output or SNMP walk
Collect the current monitoring data for the affected host:
For an agent-based host, provide the agent output.
For an SNMP device, provide the SNMP walk.
This provides the current service and monitoring information needed when comparing the host against its historical data.
Collect availability or report details
If the issue occurs in the Availability view, provide screenshots showing the affected values and the relevant display and computation settings.
If the issue affects a customized report, export the report as an MKP using the green puzzle icon under Customize > Reports.
More information about this in our official User Guide article MKPs for GUI extensions.
Troubleshoot lingering downtimes
Check the active downtimes
First, verify whether the affected host or service still has an active downtime configured.
If no matching downtime can be found in the GUI, query all existing downtimes using Livestatus:
lq "GET downtimes Columns: downtime_author downtime_comment downtime_duration downtime_end_time downtime_entry_time downtime_fixed downtime_id downtime_is_service downtime_origin downtime_recurring downtime_start_time host_has_been_checked host_labels host_name host_scheduled_downtime_depth host_state service_description service_has_been_checked service_state"This allows you to determine whether Checkmk still considers the downtime active even when it is not visible where expected.
Search the history
If the downtime cannot be found through Livestatus, search the Checkmk history files as the site user using the downtime comment or summary.
For example:
OMD[mysite]:~$ grep -rl "<DOWNTIMESUMMARY>" ~/var/check_mk/core/history
If the downtime summary is DT2:
OMD[mysite]:~$ grep -r "DT2" ~/var/check_mk/core/historyThe output can show when the downtime was scheduled and when the corresponding downtime event started.
If the event is not present in the current history file, also search the archive:
OMD[mysite]:~$ grep -rl "<DOWNTIMESUMMARY>" ~/var/check_mk/core/archive/*The archived history files contain the older state and downtime events required for investigating availability calculations.
Track specific state changes
To determine exactly when a host or service changed state, filter the history files for the affected object and translate the timestamps.
For example, to search for DOWN states for a service:
for history in $(find ~/var/check_mk/core -name "history*"); do cat "$history" | grep -ai "<SERVICE_DESCRIPTION>" | awk -F '[][]' '{$2=strftime("%Y-%m-%d %H:%M:%S", $2); print $0}'; done | sort | grep -ai downThis can help identify the state transition responsible for an unexpected availability calculation.
Troubleshoot outdated BI downtime states
BI aggregates calculate their state from the underlying hosts and services. If a BI availability report continues to show an outdated downtime, verify the individual objects that make up the aggregate first.
Check the following:
Verify that the underlying hosts and services contain the expected downtime start and end events in their history.
Check whether states such as Downtime or N/A are influencing the resulting BI availability calculation.
Verify that the BI state reflects the current state of its underlying nodes.
Remove a lingering downtime from the history
Warning
The following procedure is unsupported. Create a backup of the Checkmk site before making any changes to the history file.
If the downtime exists in the history file but cannot be found anywhere else and continues to affect the availability calculation, stop the site as the site user:
OMD[mysite]:~$ omd stop
Open the history file:
OMD[mysite]:~$ vi ~/var/check_mk/core/historyLocate and remove the relevant downtime entry.
Then start the site again:
OMD[mysite]:~$ omd startThis removes the lingering downtime entry from the active history.
Related articles