Working with Distributed Agent Bakery

Deploying baked agents via remote sites within a distributed monitoring configuration is now possible.

LAST TESTED ON CHECKMK 2.0.0P1

Table of Contents

Getting Started

Background information regarding this subject is available on our:

Prerequisites

Prerequisites for this feature are a monitoring setup using distributed WATO and the possibility of establishing a connection from remote sites to the central site via HTTP/HTTPS.

Also, you need to configure the distributed Agent Bakery.

With Checkmk 2.0, there is a new option for the Agent Updater:

Usage: In the case of "Registration and fallback," the agent updater will use this information only on registration and rely on the update URL provided by the Agent Bakery afterward. If the connection fails, the agent updater will additionally use this information as a fallback. In the case of "Use for all connections," the agent updater will ignore the update URL provided by the Agent Bakery.

If you want to register and update all remote hosts from a remote site, now you can add the remote site here:

Screenshot of Agent updater for Linux and Windows

Now you can bake and sign the agent and register the agent updater:

root@linux:~# cmk-update-agent register -v

+-------------------------------------------------------------------+
|                                                                   |
|  Check_MK Agent Updater v2.0.0b5 - Registration                   |
|                                                                   |
|  Activation of automatic agent updates. Your first step is to     |
|  register this host at your deployment server for agent updates.  |
|  For this step you need an administration account on WATO for     |
|  that server.                                                     |
|                                                                   |
+-------------------------------------------------------------------+
Our host name in the monitoring: 
myremotesite

WATO user with admin permissions: 
cmkadmin     

Password: 


Going to register agent at deployment server
Applying new update URL http://myhost/myremotesite/check_mk/ from deployment server
Successfully registered agent of host "myremotesite" for deployment.
You can now update your agent by running 'cmk-update-agent -v'
Saved your registration settings to /etc/cmk-update-agent.state.

If you want to run the agent updater, this will not work.

root@linux:/home/ubuntu# cmk-update-agent -v 

+-------------------------------------------------------------------+
|                                                                   |
|  Check_MK Agent Updater v2.0.0b5 - Update                         |
|                                                                   |
+-------------------------------------------------------------------+
Getting target agent configuration for host 'myremotesite' from deployment server
Agent Bakery: Registering/updating at remote site but found no URL to central site. Please provide it in "automatic agent updates" section at global settings.
See syslog or Logfile at /var/lib/check_mk_agent/cmk-update-agent.log for details.

To get the updater working, please use the following step-by-step guide!

Step-by-step guide

Configure the connection to the central Agent Bakery

Setup → General → Global settings → AUTOMATIC AGENT UPDATES → Connection to the central agent bakery

Place checks on both URL to central site & Automation User for connection to central site. Fill in the information for the appropriate URL.

Now you can update the host.

root@linux:~# cmk-update-agent -v

+-------------------------------------------------------------------+
|                                                                   |
|  Check_MK Agent Updater v2.0.0b5 - Update                         |
|                                                                   |
+-------------------------------------------------------------------+
Getting target agent configuration for host 'myremotesite' from deployment server
Target state (from deployment server):
  Agent Available:     True
  Signatures:          1
  Target Hash:         1ef2302ca00a9b89
Agent 1ef2302ca00a9b89 already installed.

.

In 'Setup → General → Global settings → AUTOMATIC AGENT UPDATES' there is now a new second option for the distributed Agent Bakery:
Configure an alternative URL for the connection from the host to remote Agent Bakery. If not configured here, the URL from distributed monitoring configuration is used instead. You might often want to make this a site-specific global setting at the distributed monitoring configuration because this URL is likely different for every remote site.

Functionality

This feature is realized as follows: Update requests to the remote sites get forwarded to the central site – The entire configuration and the agent baking process are done on the central site. Agent packages that got requested once at a remote site will get cached (as long as they are valid) there to save unnecessary traffic. Additionally, the requested data will be checked for consistency on the remote site, again to save unnecessary traffic to the central site.

General Communication Settings

Depending on the setting Update server information in the rule "Agent updater (Linux, Windows, Solaris)" and on the command line options during registration, the communication between the agent updater and the Checkmk servers can be configured granular. That way, you can restrict the communication to either the central Bakery or to the remote site server, if needed.

CLI options during registration	Settings in Bakery Rule	Remote Site server reachable	Registration Server used	Package Download Server used
-s and -i	not used	yes	The server/site provided with -s	The dedicated remote server with the URL configured in the „Distributed Monitoring“ settings
none	„use for all connections“	yes	The server/site provided in the Bakery rule	The server provided in the Bakery rule
none	„use for registration and fallback“	yes	The server/site provided in the Bakery rule	The dedicated remote server with the URL configured in the „Distributed Monitoring“ settings
none	„use for registration and fallback“	no	The server/site provided in the Bakery rule	The server provided in the Bakery rule
none	not configured	yes	Interactively asks for server	The dedicated remote server with the URL configured in the „Distributed Monitoring“ settings

Example:

If you want to restrict the communication of the agent updater to the particular remote server,

you can skip the Update server information in the Bakery rule.
That way, you have a single software package per OS to install all your hosts.
You just have to provide the remote server and site during the registration on the CLI.
With https, you have to add the certificates of all your servers to the Bakery rule (when using Self signed certificates) or at least the root and intermediate ones (when using a CA).

Registration

Going to register agent at deployment server
2021-02-04 16:18:08,715 DEBUG: Authenticating at Check_MK Server (using requests): http://myhost/myremotesite/check_mk/login.py
2021-02-04 16:18:09,034 DEBUG: Fetching content (using requests): http://myhost/myremotesite/check_mk/register_agent.py
2021-02-04 16:18:09,070 DEBUG: Response from agent bakery:
{"result_code": 0, "result": {"host_secret": "rlkaihuoazcdcnseimvfpzaocsjitoabpieyzubuictninlfdtgciztydmkmfrfj", "update_url": "http:\/\/myhost\/myremotesite\/check_mk\/", "monitored": true}}
2021-02-04 16:18:09,070 INFO: Applying new update URL http://myhost/myremotesite/check_mk/ from deployment server
2021-02-04 16:18:09,071 DEBUG: No state file found yet. New state data will be saved to /etc/cmk-update-agent.state
2021-02-04 16:18:09,073 DEBUG: Saved deployment status to /etc/cmk-update-agent.state.
2021-02-04 16:18:09,073 INFO: Successfully registered agent of host "myremotesite" for deployment.
2021-02-04 16:18:09,073 DEBUG: Secret is rlkaihuoazcdcnseimvfpzaocsjitoabpieyzubuictninlfdtgciztydmkmfrfj.
2021-02-04 16:18:09,073 DEBUG: Successfully read /etc/cmk-update-agent.state.
2021-02-04 16:18:09,075 DEBUG: Saved deployment status to /etc/cmk-update-agent.state.
2021-02-04 16:18:09,076 INFO: Saved your registration settings to /etc/cmk-update-agent.state.
2021-02-04 16:18:09,076 DEBUG: Successfully scheduled an automatic update with next Check_MK Agent execution.
2021-02-04 16:18:09,076 DEBUG: Done.

The host is using the central site for registration!

Update

2021-02-04 16:18:19,630 DEBUG: Starting Check_MK Agent Updater v2.0.0b5
2021-02-04 16:18:19,630 DEBUG: Successfully read /etc/cmk-update-agent.state.
2021-02-04 16:18:19,630 DEBUG: Successfully read /etc/check_mk/cmk-update-agent.cfg.
2021-02-04 16:18:19,630 INFO: Starting Update mode as plugin.
2021-02-04 16:18:19,631 INFO: Getting target agent configuration for host 'myremotesite' from deployment server
2021-02-04 16:18:19,657 DEBUG: Fetching content (using requests): http://myhost/myremotesite/check_mk/deploy_agent.py
2021-02-04 16:18:19,717 DEBUG: Response from agent bakery:
{"result_code": 0, "result": {"update_url": "http:\/\/myhost\/myremotesite\/check_mk\/", "AgentAvailable": true, "TargetHash": "1ef2302ca00a9b89", "Signatures": [{"certificate": "-----BEGIN CERTIFICATE-----\nMIIC2jCCAcICAQEwDQYJKoZIhvcNAQEFBQAwMjEdMBsGA1UECgwUQ2hlY2tfTUsg\nU2l0ZSBtYXN0ZXIxETAPBgNVBAMMCGNta2FkbWluMCAXDTIxMDIwNDExNTExNVoY\nDzIwNTEwMTI4MTE1MTE1WjAyMR0wGwYDVQQKDBRDaGVja19NSyBTaXRlIG1hc3Rl\ncjERMA8GA1UEAwwIY21rYWRtaW4wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK\nAoIBAQDgFptynmaSVcnwqj69SDiJmHnSQAcb\/iMMXclptBo+7tcOOpU36COA0\/eL\nX7kp5V7Mu1zEfQq9wPmuQy+cVyWkMPj03tKZjly3Y+342WpFn\/hk8g0H72f+zTre\nebpM4d+UmkRWzo6\/aACgPTy0oPr4pDEiaPkzoGSrWtE+14jIIL5Cv+t8bIIs4uI0\n0VJ1HqR09FmuZhoYfV9hVoAEFYMRzo6BBweMIOz5cYEhJXa30VaGt9Fvk6FPwf9u\naKx25KxPHrz2oXgTnnP6afffx\/rl7oKQPGx6uT3LlNhkOJW\/R0RwVIKqQcjB66DB\njaESRa27L2vy7A5gO7muktMyIl73AgMBAAEwDQYJKoZIhvcNAQEFBQADggEBALSt\nz22vVZcBZ8RSy9ZX5nu4E+hBrZz\/84+U52UnL+ATbSKRA+VxltACezrBAXGB1pMO\nuCZ6YlKT6yMGqP0vyqhjAQnNJTlEr2V9MAh0IBuPUxCKTUbr6gzauAqT8FrVKgrY\nD2ZbCGKlls26tgbyyaH+Qqec9WsJQSDgEnIDoabR5Ip1hPjmWK0xG9kwQwFFZo+I\n3VMIOr52vys99TVBt4682560vO\/vhP1OQMmiyax\/\/M2fKuONo0zq7rWFiDbqvuL6\nTeuSSpANXEhfN\/42HCAqziWS85NYebUGLDbMYC6jJMfvBzOT\/uHrjbkop9dqOLfL\nnsIcFFkBV4e07TOLz6c=\n-----END CERTIFICATE-----\n", "signature": "vWcDferK6N7eVq7iVc7Ni++YS0GD9N25PWtT7dHbsykFGQ7QZJQ0kFEuAQz4sn3qy3T6feKgISOocXzSxZzpfQsWWlaWhPCpB4FXQ4pcDjcXuXsCN2Jd14dPKhEifTjLQyQ\/IuewwXVgezz63c5g+iyItQ2gA0GvxL6\/YRGTBV62r81ggafvmEnhswprS4WvykcO2+lS0pLfnkkVFGejFKKTIK0IfHoVNfnalyd1n2FD4\/ghRpwJlxafwPoZ1egG2xgtmzvTXBngHN7F+wG4oqpGImevDaDXu+pkJYqRu4IyFCK3OZFV3jlvn+5AX\/tP8LgU8D\/qNSHL36TMKEPAZw=="}]}}
2021-02-04 16:18:19,717 DEBUG: Successfully read /etc/cmk-update-agent.state.
2021-02-04 16:18:19,721 DEBUG: Saved deployment status to /etc/cmk-update-agent.state.
2021-02-04 16:18:19,721 DEBUG: Successfully read /etc/cmk-update-agent.state.
2021-02-04 16:18:19,723 DEBUG: Saved deployment status to /etc/cmk-update-agent.state.
2021-02-04 16:18:19,723 INFO: Target state (from deployment server):
2021-02-04 16:18:19,723 INFO:   Agent Available:     True
2021-02-04 16:18:19,723 INFO:   Signatures:          1
2021-02-04 16:18:19,723 INFO:   Target Hash:         1ef2302ca00a9b89
2021-02-04 16:18:19,749 DEBUG: Fetching content (using requests): http://myhost/myremotesite/check_mk/deploy_agent.py
2021-02-04 16:18:20,411 INFO: Downloaded agent has size 13116004 bytes.
2021-02-04 16:18:20,427 INFO: Signature check OK.
2021-02-04 16:18:20,436 INFO: Invoking package manager: dpkg -i /tmp/check-mk-agent-a8apxscl
2021-02-04 16:18:21,504 INFO: Output from dpkg:
  (Reading database ... 158728 files and directories currently installed.)
  Preparing to unpack /tmp/check-mk-agent-a8apxscl ...
  Disable Checkmk agent in systemd (if active)...
  Unpacking check-mk-agent (2.0.0b5-17.1ef2302ca00a9b89) over (2.0.0b5-17.1ef2302ca00a9b89) ...
  Reloading xinetd...
  Setting up check-mk-agent (2.0.0b5-17.1ef2302ca00a9b89) ...
  Reloading xinetd...
  Enable Checkmk agent in systemd...
  Created symlink /etc/systemd/system/sockets.target.wants/check-mk-agent.socket → /etc/systemd/system/check-mk-agent.socket.

The host fetches all updates from the myremotesite

Config

The agent updater creates a config and a state file. Please don't modify this file! You can take a look inside to see if the config is correct!

root@linux:/home/ubuntu# cat /etc/cmk-update-agent.state 
{'update_url': 'http://myhost/myremotesite02/check_mk/', 'host_secret': 'zewqptluwgrjzamdvwjplosjhrwqpboxqjtxogreibeqkadfhswhotzifjqqramw', 'server': 'myserverip', 'site': 'mycentralsite', 'host_name': 'mycentralsite02', 'protocol': 'http', 'user': 'cmkadmin', 'last_check': 1611063314.5359132, 'installed_aghash': '67f84ee7e41b73aa', 'last_update': 1611063316.2098463}

root@linux:/home/ubuntu# cat /etc/check_mk/cmk-update-agent.cfg 
# Created by Check_MK Agent Bakery.
# This file is managed via WATO, do not edit manually or you
# lose your changes next time when you update the agent.

{'activated': True,
 'certificates': [],
 'ignore_update_url': False,
 'interval': 3600,
 'protocol': 'http',
 'server': 'myserverip',
 'signature_keys': ['-----BEGIN CERTIFICATE-----\n'
                    'MIIC2jCCAcICAQEwDQYJKoZIhvcNAQEFBQAwMjEdMBsGA1UECgwUQ2hlY2tfTUsg\n'
                    'U2l0ZSBtYXN0ZXIxETAPBgNVBAMMCGNta2FkbWluMCAXDTIxMDExOTExNTMxMVoY\n'
                    'DzIwNTEwMTEyMTE1MzExWjAyMR0wGwYDVQQKDBRDaGVja19NSyBTaXRlIG1hc3Rl\n'
                    'cjERMA8GA1UEAwwIY21rYWRtaW4wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK\n'
                    'AoIBAQDWbh45st1xnzhsVzMjBBXzO7+5qz3/HPezvaOeUpSSIYHns2DnuDlwpLCO\n'
                    'XTdyfRUamgRtiOrzbUsSsARmMF9AqExao1+yM1rbEfcXP6mtBqMkqShlDw3n0L7F\n'
                    'HGGYG2CNqcRSmgxFe2Eck0ydv/p4cJONHIw4UFwqDnHfIC1L3tCnTQJagCig/Jvk\n'
                    'kZSZZqgkrjl4TuwRkFxdrnwURjm2hQGg+TkJcxpPHqSutbdnQrXnuuAEdx/ZIpFL\n'
                    'ivmjLDATjGzCy0LPZsVtjHZu5Lz4WWOZsA1BCZLecT81PI5edWs0RZ9mRH7QC+E8\n'
                    'CkIdlLgMUbfB093SGx3YEP8u+RtRAgMBAAEwDQYJKoZIhvcNAQEFBQADggEBAAU/\n'
                    'DtXHbs3cUPBGDs/Go0PgfqhwiZjn1aprCSh8O/RBLWA/jep65eU2pzsuoiKWT5CE\n'
                    'P/xc7pKXWmxQpyA4KA1Fz9BS80QHq5heXkG2FSLMFunDLxzDcbQI7QXH4oskIDkV\n'
                    'zzYio3VDXXDyhMBHd25XKKCQT3L9ou3bD7Esh/Cf/8Y2GQHkRvIWOpwLEknq0ovy\n'
                    'jkJ0o7Gx8GrqJYG5gUW44TJU/IhYFCxfhTS09RW842Z10iDSYiBPKoZLonM6F3SY\n'
                    'SLVRDn3vhnCpbkiO0Dc1op+0d9b5Wc+UCVW02G+l+G+aRw89L7v2xERXNPczCZgV\n'
                    'g+9DtIl5cu1fl4qVqmc=\n'
                    '-----END CERTIFICATE-----\n'],
 'site': 'mycentralsite'}

Common Issues

HTTPSConnectionPool(host='checkmk.test.intern', port=443): Max retries exceeded with url: /check_mk/check_mk/login.py (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)')))

Possible Solution

Please make you follow the steps to import the SSL Certificates:

Distributed monitoring

Agent updates in distributed monitoring

Checkmk Knowledge Base