Troubleshooting the agent controller
- Anastasios Thomaidis
- Matthew Hierholzer
- Robin Gierse
With Checkmk 2.1, we released the new agent controller with a TLS connection. Detailed docs can be found here:
- German: Linux überwachen / Windows überwachen
- English: Monitoring Linux / Monitoring Windows
LAST TESTED ON CHECKMK 2.3.0P1
Agent controller - Connection refused
Problem
In this manual, we will show you how to debug the below error:
.png?version=1&modificationDate=1653568942260&cacheVersion=1&api=v2&width=793&height=247 "Screenshot showing an error of Communication failed. Connection refused.")
Please run the following commands to check the state of the agent controller and the port states:
For Linux
root@mylinuxhost~# ss -tulpn | grep 6556 root@mylinuxhost~# ps waux | grep cmk-agent-ctl root@mylinuxhost~# cmk-agent-ctl status root@mylinuxhost~# systemctl status check-mk-agent.socket root@mylinuxhost~# systemctl status cmk-agent-ctl-daemon.service
For Windows
netsh interface ipv4 show excludedportrange protocol=tcp
Netstat for Windows (check the open/listening ports):
netstat -anb > output.txt
Solution
For Linux
One possible solution is to follow the steps in the article Troubleshooting Checkmk agent systemd service repeatedly failing on CentOS 7
Registration with cmk-agent-ctl is not working
Problem
After installing the agent, the registration is not working and showing the following issue:
root@mylinuxhost~# cmk-agent-ctl register ERROR [cmk_agent_ctl] Something seems wrong with the agent socket (/run/check-mk-agent.socket), aborting
Solution
Please check and verify that the agent controller is running:
root@mylinuxhost~# ss -tulpn | grep 6556 root@mylinuxhost~# ps waux | grep cmk-agent-ctl root@mylinuxhost~# cmk-agent-ctl status root@mylinuxhost~# systemctl status check-mk-agent.socket root@mylinuxhost~# systemctl status cmk-agent-ctl-daemon.service
In that case, the agent controller is not running because xinetd is running. For the agent controller, systemd is necessary: Werk #13865: Agent controller on Unix: Deny registration if agent socket is not operational
If you're using the bakery, you need to create the following rule:
Without the bakery, please follow these steps: Agent Linux Legacy
ERROR [cmk_agent_ctl] Failed to discover agent receiver port from Checkmk REST API, both with http and https.
Problem
You encounter this error when registering the agent controller:
root@mylinuxhost:~# cmk-agent-ctl register -H <host> -s <checkmk-server> -i <site> -U <username> ERROR [cmk_agent_ctl] Failed to discover agent receiver port from Checkmk REST API, both with http and https. Error with http: Failed to discover agent receiver port from http://<checkmk-server>/<site>/check_mk/api/1.0/domain-types/internal/actions/discover-receiver/invoke error sending request for url (http://<checkmk-server>/<site>/check_mk/api/1.0/domain-types/internal/actions/discover-receiver/invoke): error trying to connect: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1914: (unable to get local issuer certificate) error trying to connect: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1914: (unable to get local issuer certificate) error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1914: (unable to get local issuer certificate) error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1914: Error with https: Failed to discover agent receiver port from https://<checkmk-server>/<site>/check_mk/api/1.0/domain-types/internal/actions/discover-receiver/invoke error sending request for url (https://<checkmk-server>/<site>/check_mk/api/1.0/domain-types/internal/actions/discover-receiver/invoke): error trying to connect: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1914: (unable to get local issuer certificate) error trying to connect: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1914: (unable to get local issuer certificate) error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1914: (unable to get local issuer certificate) error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1914:
Solution #1
Register the agent with the receiver port (by default 8000, counting up similar to site Apache 5000 → 5001 ...) like so:
root@mylinuxhost:~# cmk-agent-ctl register -H <host> -s <checkmk-server:8000> -i <site> -U <username>
Solution #2
Add the self-signed certificate to the OS's cert store. Example for Ubuntu: Install a root CA certificate in the trust store
ERROR [cmk_agent_ctl] Error while loading registered connections.
Detailed error message:
ERROR [cmk_agent_ctl] Error while loading registered connections. Caused by: Failed to split into server and port at ':' at line 4 column 24
If you see this error message when trying to work with any subcommand of cmk-agent-ctl, there is probably something wrong with the file /var/lib/cmk-agent/registered_connections.json.
Solution
First, move the file registered_connections.json to registered_connections.json.bak and re-run the command. If that works, you can start checking the content of the file. If you still need the registration data stored in the file, check the line and column in the error message and try to repair it. If you don't need it anymore, delete the file.
Request failed with code 403 Forbidden: Unauthorized - Details: Unauthorized to read the global settings.
Detailed error message:
cmk-agent-ctl.exe register --server myserver --site mysite --user cmkadmin --password mypwd --hostname myhost Attempting to register at myserver, port 8000. Server certificate details: PEM-encoded certificate: ----BEGIN CERTIFICATE---- … ----END CERTIFICATE---- Issued by: Site 'mysite' local CA Issued to: mysite Validity: From Mon, 12 Jun 2023 16:20:18 +0000 To Wed, 12 Jun 3022 16:20:18 +0000 Do you want to establish this connection? [Y/n] Y [2023-07-21 15:04:06.714043 +02:00] ERROR [cmk_agent_ctl] src [main.rs:29|http://main.rs:29]: Error registering existing host at https://myserver:8000/mysite Caused by: Request failed with code 403 Forbidden: Unauthorized - Details: Unauthorized to read the global settings
If you see this error message when trying to work with any subcommand of cmk-agent-ctl, there is probably something wrong with the file /var/lib/cmk-agent/registered_connections.json.
Solution
This error occurs when the automation user, which is used internally by the agent controller to gather more information, doesn't have the admin role.
Request failed with code 401 Unauthorized
Detailed error message:
ERROR [cmk_agent_ctl] Error registering existing host at https://myserver:8000/mysite Caused by: Request failed with code 401 Unauthorized: Unauthorized - Details: Wrong credentials (Bearer header)
If you see this error message, when trying to register the agent controller, there is probably something wrong with the user.
Possible solutions
- It is obvious, but it needs to be said: Check whether the username and the password or the secret are correct.
- This error message can also be displayed when the option disable the login to this account is activated for this user. So please go to Setup > Users > Users in Checkmk and check if you see the following icon in the row for the user you are trying to authenticate with:
In this case, you have to edit the user and deactivate the checkbox in front of disable the login to this account.
Related articles