How-to troubleshoot missing RRD files and graph errors in Checkmk

How-to troubleshoot missing RRD files and graph errors in Checkmk

This article explains how to identify and resolve missing Round Robin Database (RRD) files that prevent Checkmk from displaying service graphs.

LAST TESTED ON CHECKMK 2.4.0P37

For additional information about performance data storage and graphing, refer to the Checkmk 2.4 documentation on measured values and graphing.

Problem

When attempting to view service graphs in Checkmk, you may encounter the following error:

Cannot create graph Cannot get RRD data

 

Checkmk uses Round Robin Database (RRD) files to store performance data and generate graphs for individual hosts and their services.. These files are located in the following directory:

~/var/check_mk/rrd/<HOSTNAME>/

 

For each service metric, the directory must contain two associated files:

  • .rrd: Stores the collected performance data used to generate graphs.

  • .info: Contains information describing the associated service metrics.

 

For example:

-rw------- 1 mysite mysite 65 Oct 9 13:34 CPU_utilization.info -rw-r--r-- 1 mysite mysite 1534768 Oct 30 09:00 CPU_utilization.rrd

If an .info file exists but its corresponding .rrd file is missing, Checkmk may be unable to display the affected graph.

 

The Checkmk Micro Core (CMC) log may contain errors similar to the following:

OMD[mysite]:~$ tail -f ~/var/log/cmc.log

 

Example output:

2024-10-30 09:02:55 [4] [client 0] Error accessing RRD: opening '/opt/omd/sites/mysite/var/check_mk/rrd/linux/CPU_utilization.rrd': No such file or directory 2024-10-30 09:02:55 [4] [client 0] Error accessing RRD: opening '/opt/omd/sites/mysite/var/check_mk/rrd/linux/TCP_Connections.rrd': No such file or directory 2024-10-30 09:02:55 [4] [client 0] Error accessing RRD: opening '/opt/omd/sites/mysite/var/check_mk/rrd/linux/Systemd_Timesyncd_Time.rrd': No such file or directory

These messages indicate that Checkmk attempted to access RRD files that do not exist at the specified locations.

 

Solution

The following procedure identifies .info files without corresponding .rrd files and removes the affected metadata files so Checkmk can recreate them when new performance data is processed.

Warning

This procedure requires stopping the Checkmk site, which temporarily interrupts monitoring. Removing an .info file does not restore historical performance data from a missing .rrd file. If historical data is required, investigate whether the missing RRD files can be restored from a backup before proceeding.

 

Step 1: Stop the Checkmk site

All commands must be executed as the Checkmk site user.

Stop the Checkmk site before modifying the RRD directory:

OMD[mysite]:~$ omd stop

 

Step 2: Identify orphaned INFO files

Run the following command to identify .info files that do not have a corresponding .rrd file:

Note

This command does not modify or delete any files. It only lists .info files for which the corresponding .rrd file is missing.

OMD[mysite]:~$ for i in $(find ~/var/check_mk/rrd -name "*.info"); do test -e "${i%info}rrd" || echo "$i"; done

 

Example output:

/omd/sites/mysite/var/check_mk/rrd/linux/CPU_utilization.info /omd/sites/mysite/var/check_mk/rrd/linux/TCP_Connections.info /omd/sites/mysite/var/check_mk/rrd/linux/Systemd_Timesyncd_Time.info

If no files are listed, the problem may have another cause. Continue troubleshooting rather than removing files unnecessarily.

 

Step 3: Remove the affected INFO files

After reviewing the output from Step 2, remove the orphaned .info files.

You can remove individual files manually:

OMD[mysite]:~$ rm ~/var/check_mk/rrd/linux/CPU_utilization.info

 

Alternatively, use the following command to remove all .info files that do not have a corresponding .rrd file:

Warning

The command permanently removes the affected .info files. Verify that the files identified in Step 2 are the intended targets before running it.

OMD[mysite]:~$ for i in $(find ~/var/check_mk/rrd -name "*.info"); do test -e "${i%info}rrd" || rm "$i"; done

 

Step 4: start the Checkmk site

After removing the affected files, restart the Checkmk site:

OMD[mysite]:~$ omd start

As Checkmk processes new performance data for the affected services, the required files should be recreated.

 

Step 5: Verify that the graphs are working

After restarting the site, allow Checkmk to collect and process new monitoring data.

Verify that the problem has been resolved:

  1. Open the affected service in the Checkmk web interface.

  2. Check whether the service graph is displayed correctly.

  3. Confirm that new performance data is being recorded.

  4. Review the CMC log to determine whether RRD-related errors are still occurring.

 

To inspect the CMC log:

OMD[mysite]:~$ tail -f ~/var/log/cmc.log

If the files are recreated and the graph is displayed correctly, the recovery procedure was successful.

 

Related articles