Info

With Checkmk 2.1, we released the new agent controller with a TLS connection. Detailed docs can be found here:

Status

colour	Green
title	LAST TESTED ON CHECKMK 2.2.0P1

Table of Contents

Agent controller - Connection refused

Problem

In this manual, we will show you how to debug the below error:

Please run the following commands to check the state of the agent controller and the port states:

For Linux

Code Block

language	bash
theme	RDark

root@mylinuxhost~# ss -tulpn | grep 6556  

root@mylinuxhost~# ps waux | grep cmk-agent-ctl  

root@mylinuxhost~# cmk-agent-ctl status  

root@mylinuxhost~# systemctl status check-mk-agent.socket

root@mylinuxhost~# systemctl status cmk-agent-ctl-daemon.service

For Windows

Code Block

language	bash
theme	RDark

netsh interface ipv4 show excludedportrange protocol=tcp

Netstat for Windows (check the open/listening ports):

Code Block

language	bash
theme	RDark

netstat -anb > output.txt

Solution

For Linux

One possible solution is to follow the steps in the article Checkmk agent systemd service repeatedly failing on CentOS 7

Registration with cmk-agent-ctl is not working

Problem

After installing the agent, the registration is not working and showing the following issue:

Solution

Please check and verify that the agent controller is running:

Code Block

language	bash
theme	RDark

root@mylinuxhost~# ss -tulpn | grep 6556   

root@mylinuxhost~# ps waux | grep cmk-agent-ctl   

root@mylinuxhost~# cmk-agent-ctl status   

root@mylinuxhost~# systemctl status check-mk-agent.socket

root@mylinuxhost~# systemctl status cmk-agent-ctl-daemon.service

In that case, the agent controller is not running because xinetd is running. For the agent controller, systemd is necessary: https://checkmk.com/werk/13865

If you're using the bakery, you need to create the following rule:

Without the bakery, please follow these steps: https://docs.checkmk.com/latest/en/agent_linux_legacy.html#_systemd

ERROR [cmk_agent_ctl] Failed to discover agent receiver port from Checkmk REST API, both with http and https.

Problem

You encounter this error when registering the agent controller:

Code Block

theme	RDark

root@mylinuxhost:~# cmk-agent-ctl register -H <host> -s <checkmk-server> -i <site> -U <username>
ERROR [cmk_agent_ctl] Failed to discover agent receiver port from Checkmk REST API, both with http and https.

Error with http:
Failed to discover agent receiver port from http://<checkmk-server>/<site>/check_mk/api/1.0/domain-types/internal/actions/discover-receiver/invoke
error sending request for url (http://<checkmk-server>/<site>/check_mk/api/1.0/domain-types/internal/actions/discover-receiver/invoke): error trying to connect: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1914: (unable to get local issuer certificate)
error trying to connect: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1914: (unable to get local issuer certificate)
error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1914: (unable to get local issuer certificate)
error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1914:

Error with https:
Failed to discover agent receiver port from https://<checkmk-server>/<site>/check_mk/api/1.0/domain-types/internal/actions/discover-receiver/invoke
error sending request for url (https://<checkmk-server>/<site>/check_mk/api/1.0/domain-types/internal/actions/discover-receiver/invoke): error trying to connect: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1914: (unable to get local issuer certificate)
error trying to connect: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1914: (unable to get local issuer certificate)
error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1914: (unable to get local issuer certificate)
error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1914:

Solution #1

Register the agent with the receiver port (by default 8000, counting up similar to site Apache 5000 → 5001 ...) like so:

Code Block

theme	RDark

root@mylinuxhost:~# cmk-agent-ctl register -H <host> -s <checkmk-server> -i <site> -U <username> -p:8000

Solution #2

Add the self-signed certificate to the OS's cert store. Example for Ubuntu: https://ubuntu.com/server/docs/security-trust-store

ERROR [cmk_agent_ctl] Error while loading registered connections.

Detailed error message:

Code Block

theme	RDark

ERROR [cmk_agent_ctl] Error while loading registered connections.

Caused by:
    Failed to split into server and port at ':' at line 4 column 24

If you see this error message when trying to work with any subcommand of cmk-agent-ctl, there is probably something wrong with the file /var/lib/cmk-agent/registered_connections.json.

Solution

First, move the file registered_connections.json to registered_connections.json.bak and re-run the command. If that works, you can start checking the content of the file. If you still need the registration data stored in the file, check the line and column in the error message and try to repair it. If you don't need it anymore, delete the file.

Request failed with code 403 Forbidden: Unauthorized - Details: Unauthorized to read the global settings.

Detailed error message:

Code Block

theme	RDark

cmk-agent-ctl.exe register --server myserver --site mysite --user cmkadmin --password mypwd --hostname myhost
Attempting to register at myserver, port 8000. Server certificate details:

PEM-encoded certificate:
----BEGIN CERTIFICATE----
…
----END CERTIFICATE----

Issued by:
Site 'mysite' local CA
Issued to:
mysite
Validity:
From Mon, 12 Jun 2023 16:20:18 +0000
To Wed, 12 Jun 3022 16:20:18 +0000

Do you want to establish this connection? [Y/n]

Y
[2023-07-21 15:04:06.714043 +02:00] ERROR [cmk_agent_ctl] src
[main.rs:29|http://main.rs:29]: Error registering existing host at https://myserver:8000/mysite

Caused by:
Request failed with code 403 Forbidden: Unauthorized - Details: Unauthorized to read the global settings

If you see this error message when trying to work with any subcommand of cmk-agent-ctl, there is probably something wrong with the file /var/lib/cmk-agent/registered_connections.json.

Solution

This error occurs when the automation user, which is used internally by the agent controller to gather more information, doesn't have the admin role.

Version	Old Version 42	New Version 43
Changes made by	Matthew Hierholzer	Matthew Hierholzer
Saved on	Sept 26, 2023	Sept 26, 2023

Versions Compared

Key

Agent controller - Connection refused

Problem

Netstat for Windows (check the open/listening ports):

Solution

For Linux

Registration with cmk-agent-ctl is not working

Problem

Solution

ERROR [cmk_agent_ctl] Failed to discover agent receiver port from Checkmk REST API, both with http and https.

Problem

Solution #1

Solution #2

ERROR [cmk_agent_ctl] Error while loading registered connections.

Solution

Request failed with code 403 Forbidden: Unauthorized - Details: Unauthorized to read the global settings.

Solution

Related articles

Page Comparison

Versions Compared

Key

Agent controller - Connection refused

Problem

Netstat for Windows (check the open/listening ports):

Solution

For Linux

Registration with cmk-agent-ctl is not working

Problem

Solution

ERROR [cmk_agent_ctl] Failed to discover agent receiver port from Checkmk REST API, both with http and https.

Problem

Solution #1

Solution #2

ERROR [cmk_agent_ctl] Error while loading registered connections.

Solution

Request failed with code 403 Forbidden: Unauthorized - Details: Unauthorized to read the global settings.

Solution

Related articles