Troubleshooting Checkmk agent systemd service repeatedly failing on CentOS 7

Owned by Former user (Deleted)
Last updated: Dec 11, 2023 by Matthew Hierholzer
1 min read

This article helps troubleshoot issues with the systemd agent on CentOS7

LAST TESTED ON CHECKMK 2.0.0P1

Table of Contents

Problem

If you experience that the Checkmk agent is failing with systemd with some of the following messages:

root@mylinuxhost~# systemctl list-units | grep check-mk
...
check-mk-agent@99995-<$IP_OF_HOST>:6556-<$IP_OF_CHECKMK_SERVER>56520.service loaded failed failed Checkmk agent (<$IP_OF_CHECKMK_SERVER>56520)
check-mk-agent@99997-<$IP_OF_HOST>:6556-<$IP_OF_CHECKMK_SERVER>57602.service loaded failed failed Checkmk agent (<$IP_OF_CHECKMK_SERVER>57602)
check-mk-agent@99999-<$IP_OF_HOST>:6556-<$IP_OF_CHECKMK_SERVER>58616.service loaded failed failed Checkmk agent (<$IP_OF_CHECKMK_SERVER>58616)
check-mk-agent.socket
root@mylinuxhost~# systemctl status check-mk-agent@99999-<$IP_OF_HOST>:6556-<$IP_OF_CHECKMK_SERVER>58616.service
check-mk-agent@99999-<$IP_OF_HOST>:6556-<$IP_OF_CHECKMK_SERVER>58616.service - Checkmk agent (<$IP_OF_CHECKMK_SERVER>58616)
Loaded: loaded (/etc/systemd/system/check-mk-agent@.service; static; vendor preset: disabled)
Active: failed (Result: resources) since Fri 2021-11-19 23:24:49 UTC; 3 weeks 3 days ago
Process: 23951 ExecStart=/usr/bin/check_mk_agent (code=exited, status=0/SUCCESS)
Main PID: 23951 (code=exited, status=0/SUCCESS)


Nov 19 23:24:48 <hostname> systemd[1]: Starting Checkmk agent (<$IP_OF_CHECKMK_SERVER>58616)...
Nov 19 23:24:49 <hostname> systemd[1]: check-mk-agent@99999-<$IP_OF_HOST>:6556-<$IP_OF_CHECKMK_SERVER>58616.service failed to run 'stop-post' task: Transport endpoint is not connected
Nov 19 23:24:49 <hostname> systemd[1]: Failed to start Checkmk agent (<$IP_OF_CHECKMK_SERVER>58616).
Nov 19 23:24:49 <hostname> systemd[1]: Unit check-mk-agent@99999-<$IP_OF_HOST>:6556-<$IP_OF_CHECKMK_SERVER>58616.service entered failed state.
Nov 19 23:24:49 <hostname>systemd[1]: check-mk-agent@99999-<$IP_OF_HOST>:6556-<$IP_OF_CHECKMK_SERVER>58616.service failed.

Reason

The root cause is that CentOS 7 is shipping an old systemd version. Checkmk requires systemd >=235.

You can check your systemd version with systemctl --version



This is caused by Werk #12153

Solution

Then there are two options to resolve this issue.

  • Contact the agent through xinetd or SSH. For xinetd, have a look at the rule set: Checkmk agent network service (Linux)

  • Upgrade to a suitable (preferably binary-compatible) alternative (e.g., Rocky Linux or AlmaLinux)