Troubleshooting Check trend missing or incorrect

Troubleshooting Check trend missing or incorrect

This article explains how Checkmk trends work and how to troubleshoot missing or inaccurate trend data.

LAST TESTED ON CHECKMK 2.4.0P21

Table of Contents

Overview

A number of Checkmk checks support trend computation, including:

Trend computation allows Checkmk to estimate the future state of a service based on recent performance data using an exponential moving average.

In some situations, trend values may appear inaccurate, unrealistic, or may not appear at all. This article explains how trend computation works and outlines the most common causes of missing or incorrect trend data.

 

How Trend Computation Works

Checkmk does not use RRD data to calculate trends.

Instead, when trends are enabled for a service, Checkmk stores trend information in:

OMD[mysite]:~$ ~/tmp/check_mk/counters

The stored data is based on the delta between two check intervals. Checkmk then calculates a trend over the configured trend computation window, which is 24 hours by default.

 

Initial Trend Computation Behavior

When trends are enabled for the first time:

  • In Checkmk 2.4 and older, the displayed trend may initially be inaccurate.

  • In Checkmk 2.5 and newer, no trend may be displayed initially.

This is expected behavior. 

Trend computation requires at least one full configured trend interval to pass before accurate trend data becomes available. With the default configuration, this means trends may not appear correctly for up to 24 hours after being enabled.

Once sufficient data has been collected, trend information should remain accurate for the lifetime of the service unless the stored trend data is lost or corrupted.

 

Troubleshooting

If trend data is still missing or incorrect after the initial computation interval has passed, review the following areas.

Ensure the Checkmk Site Was Shut Down Cleanly

On Linux systems, the site tmp directory is typically stored on a tmpfs filesystem.

Because tmpfs is memory-based, its contents do not survive a system reboot.

Trend data stored in:

OMD[mysite]:~$ ~/tmp/check_mk/counters

Trend data will be retained across restarts when the site is shut down cleanly. If the shutdown is interrupted or unclean, the trend data may not be preserved.


During a proper shutdown, Checkmk archives the tmpfs contents to:

OMD[mysite]:~$ ~/var/omd/tmpfs-dump.tar

The archive is then restored automatically when the site starts again.

If the system or site is stopped unexpectedly, the trend data may not be preserved.



Verify tmpfs Restoration

When a site is stopped cleanly, Checkmk creates the following archive:

OMD[mysite]:~$ ls -lh ~/var/omd/tmpfs-dump.tar

This file is typically only present while the site is stopped. After the site starts, the archive is restored back into the tmpfs.

You can verify that trend data exists after startup by checking the counters directory:

OMD[mysite]:~$ ls -lh ~/tmp/check_mk/counters

 


Ensure the Site tmpfs Is Configured Properly

In containerized or custom deployments, the site tmpfs must be configured correctly.

If the tmpfs is missing, mounted incorrectly, or assigned the wrong permissions, Checkmk may be unable to store or restore trend data.

This is especially common in containerized deployments where the tmpfs may be missing, mounted incorrectly, or assigned improper permissions.

 

Ensure the tmpfs:

  • Is mounted during container startup

  • Uses the correct UID and GID for the Checkmk site user

  • Persists according to the Checkmk container documentation

Please refer to the official Checkmk Docker container documentation for the recommended tmpfs configuration.



Disable Third-Party Extensions or Security Software

Third-party software may interfere with Checkmk's ability to write or restore trend data.

For troubleshooting purposes:

  • Temporarily disable non-essential MKPs

  • Exclude the Checkmk site directories from EDR/XDR monitoring

  • Review security software logs for blocked file operations

After disabling the software, allow enough time for a new trend interval to complete and verify whether trend data is generated correctly.

 

Related articles