Messages missing in Event Console

Problem

There are log lines, that are missing from the event console but present in the monitored log file.

Specifically this behavior is not easily reproducible and happens at random times.

Reason

The reason for this is a certain constellation of configuration options.
Per default the periodic service discovery uses cached agent output stored on the Checkmk server for 120 seconds, after that amount of time it will fetch new data from the host.
If the default check interval for the agent is equal or smaller than that 120 seconds (which it is by default: 60 seconds) then this error can not occur.
If the normal check interval for the Checkmk agent is larger than 120 seconds, it can come to the following situation:

1. The agent is queried normally, and the agent output is cached on the Checkmk server
2. The periodic service discovery runs at their configured interval and 120 seconds after the agent was queried
3. The periodic service discovery considers the cached output too old and fetches a new agent output
4. This output contains new log lines but is never processed by the logwatch check plugin and therefore not forwarded to the event console
5. When the agent is queried on the normal interval again it can not realize, that log messages have been fetched but not been processed

Solution

There are two main solutions:

1. Decrease the normal check interval for the Checkmk agent below 120 seconds

2. Manually change the affected site's main.mk and add the following line.

   This is not supported by tribe29! Do this at your own risk!

   TBD

Related articles