Messages missing in Event Console

Problem

There are log lines, that are missing from the event console but present in the monitored log file.
The configured patterns and filters should allow them to be forwarded to the event console.
Additionally, this behavior is not easily reproducible and happens at random times.

Reason

The reason for this is a certain constellation of configuration options.
Per default the periodic service discovery uses cached agent output stored on the Checkmk server for 120 seconds, after that amount of time it will fetch new data from the host.
If the default check interval for the agent is equal or smaller than that 120 seconds (which it is by default: 60 seconds) then this error can not occur.
If the normal check interval for the Checkmk agent is larger than 120 seconds, it can come to the following situation:

1. The agent is queried normally, and the agent output is cached on the Checkmk server
2. The periodic service discovery runs at their configured interval and 120 seconds or more after the agent was queried
3. The periodic service discovery considers the cached data too old and fetches a fresh agent output
4. This output contains new log lines but is never processed by the logwatch check plugin and therefore not forwarded to the event console
5. When the agent is queried on the normal interval again it can not realize, that log messages have been fetched but not been processed

Solution

There are two main solutions:

1. Decrease the normal check interval for the Checkmk agent below 120 seconds

2. Manually change the affected site's main.mk and add the following lines. Adapt the $SECONDS to be at least the normal check interval of the Checkmk agent!

   This is not supported by Checkmk! Do this at your own risk!

   max_cachefile_age = $SECONDS
   cluster_max_cachefile_age = $SECONDS

Related articles