How-to troubleshoot missing RRD files and graph errors in Checkmk
This article explains how to identify and resolve missing Round Robin Database (RRD) files that prevent Checkmk from displaying service graphs.
LAST TESTED ON CHECKMK 2.4.0P37
For additional information about performance data storage and graphing, refer to the Checkmk 2.4 documentation on measured values and graphing.
Problem
When attempting to view service graphs in Checkmk, you may encounter the following error:
Cannot create graph
Cannot get RRD data
Checkmk uses Round Robin Database (RRD) files to store performance data and generate graphs for individual hosts and their services.. These files are located in the following directory:
~/var/check_mk/rrd/<HOSTNAME>/
For each service metric, the directory must contain two associated files:
.rrd: Stores the collected performance data used to generate graphs..info: Contains information describing the associated service metrics.
For example:
-rw------- 1 mysite mysite 65 Oct 9 13:34 CPU_utilization.info
-rw-r--r-- 1 mysite mysite 1534768 Oct 30 09:00 CPU_utilization.rrdIf an .info file exists but its corresponding .rrd file is missing, Checkmk may be unable to display the affected graph.
The Checkmk Micro Core (CMC) log may contain errors similar to the following:
OMD[mysite]:~$ tail -f ~/var/log/cmc.log
Example output:
2024-10-30 09:02:55 [4] [client 0] Error accessing RRD: opening
'/opt/omd/sites/mysite/var/check_mk/rrd/linux/CPU_utilization.rrd': No such file or directory
2024-10-30 09:02:55 [4] [client 0] Error accessing RRD: opening
'/opt/omd/sites/mysite/var/check_mk/rrd/linux/TCP_Connections.rrd': No such file or directory
2024-10-30 09:02:55 [4] [client 0] Error accessing RRD: opening
'/opt/omd/sites/mysite/var/check_mk/rrd/linux/Systemd_Timesyncd_Time.rrd': No such file or directoryThese messages indicate that Checkmk attempted to access RRD files that do not exist at the specified locations.
Solution
The following procedure identifies .info files without corresponding .rrd files and removes the affected metadata files so Checkmk can recreate them when new performance data is processed.
Warning
This procedure requires stopping the Checkmk site, which temporarily interrupts monitoring. Removing an .info file does not restore historical performance data from a missing .rrd file. If historical data is required, investigate whether the missing RRD files can be restored from a backup before proceeding.
Step 1: Stop the Checkmk site
All commands must be executed as the Checkmk site user.
Stop the Checkmk site before modifying the RRD directory:
OMD[mysite]:~$ omd stop
Step 2: Identify orphaned INFO files
Run the following command to identify .info files that do not have a corresponding .rrd file:
Note
This command does not modify or delete any files. It only lists .info files for which the corresponding .rrd file is missing.
OMD[mysite]:~$ for i in $(find ~/var/check_mk/rrd -name "*.info"); do test -e "${i%info}rrd" || echo "$i"; done
Example output:
/omd/sites/mysite/var/check_mk/rrd/linux/CPU_utilization.info
/omd/sites/mysite/var/check_mk/rrd/linux/TCP_Connections.info
/omd/sites/mysite/var/check_mk/rrd/linux/Systemd_Timesyncd_Time.infoIf no files are listed, the problem may have another cause. Continue troubleshooting rather than removing files unnecessarily.
Step 3: Remove the affected INFO files
After reviewing the output from Step 2, remove the orphaned .info files.
You can remove individual files manually:
OMD[mysite]:~$ rm ~/var/check_mk/rrd/linux/CPU_utilization.info
Alternatively, use the following command to remove all .info files that do not have a corresponding .rrd file:
Warning
The command permanently removes the affected .info files. Verify that the files identified in Step 2 are the intended targets before running it.
OMD[mysite]:~$ for i in $(find ~/var/check_mk/rrd -name "*.info"); do test -e "${i%info}rrd" || rm "$i"; done
Step 4: start the Checkmk site
After removing the affected files, restart the Checkmk site:
OMD[mysite]:~$ omd startAs Checkmk processes new performance data for the affected services, the required files should be recreated.
Step 5: Verify that the graphs are working
After restarting the site, allow Checkmk to collect and process new monitoring data.
Verify that the problem has been resolved:
Open the affected service in the Checkmk web interface.
Check whether the service graph is displayed correctly.
Confirm that new performance data is being recorded.
Review the CMC log to determine whether RRD-related errors are still occurring.
To inspect the CMC log:
OMD[mysite]:~$ tail -f ~/var/log/cmc.logIf the files are recreated and the graph is displayed correctly, the recovery procedure was successful.