Working with Distributed Agent Bakery
- Anastasios Thomaidis
- Matthew Hierholzer
- Walter Fisch
LAST TESTED ON CHECKMK 2.3.0P1
Getting Started
Background information regarding this subject is available on our:
Prerequisites
Prerequisites for this feature are a distributed monitoring setup using distributed WATO and the possibility of establishing a connection from remote sites to the central site via HTTP/HTTPS.
With Checkmk 2.0, there is a new option for the Agent Updater:
Usage: In the case of "Registration and fallback," the agent updater will use this information only on registration and rely on the update URL provided by the Agent Bakery afterward. If the connection fails, the agent updater will additionally use this information as a fallback. In the case of "Use for all connections," the agent updater will ignore the update URL provided by the Agent Bakery.
If you want to register and update all remote hosts from a remote site, now you can add the remote site here:
Now you can bake and sign the agent and register the agent updater:
root@linux:~# cmk-update-agent register -v +-------------------------------------------------------------------+ | | | Check_MK Agent Updater v2.3.0p21 - Registration | | | | Activation of automatic agent updates. Your first step is to | | register this host at your deployment server for agent updates. | | For this step you need an administration account on WATO for | | that server. | | | +-------------------------------------------------------------------+ Our host name in the monitoring: myremotesite WATO user with admin permissions: cmkadmin Password: Going to register agent at deployment server Applying new update URL http://myhost/myremotesite/check_mk/ from deployment server Successfully registered agent of host "myremotesite" for deployment. You can now update your agent by running 'cmk-update-agent -v' Saved your registration settings to /etc/cmk-update-agent.state.
If you want to run the agent updater, this will not work.
root@linux:~# cmk-update-agent -v +-------------------------------------------------------------------+ | | | Check_MK Agent Updater v2.3.0p21 - Update | | | +-------------------------------------------------------------------+ Getting target agent configuration for host 'myremotesite' from deployment server Agent Bakery: Registering/updating at remote site but found no URL to central site. Please provide it in "automatic agent updates" section at global settings. See syslog or Logfile at /var/lib/check_mk_agent/cmk-update-agent.log for details.
To get the updater working, please use the following step-by-step guide!
Step-by-step guide
Configure the connection to the central Agent Bakery
Setup → General → Global settings → Automatic agent updates → Connection to the central agent bakery
Place checks on both URL to central site & Automation User for connection to central site. Fill in the information for the appropriate URL.
Now you can update the host.
root@linux:~# cmk-update-agent -v +-------------------------------------------------------------------+ | | | Check_MK Agent Updater v2.3.0p21 - Update | | | +-------------------------------------------------------------------+ Getting target agent configuration for host 'myremotesite' from deployment server Target state (from deployment server): Agent Available: True Signatures: 1 Target Hash: 1ef2302ca00a9b89 Agent 1ef2302ca00a9b89 already installed.
.
In Setup → General → Global settings → Automatic agent updates there is now a new second option for the distributed Agent Bakery:
Configure an alternative URL for the connection from the host to remote Agent Bakery. If not configured here, the URL from distributed monitoring configuration is used instead. You might often want to make this a site-specific global setting at the distributed monitoring configuration because this URL is likely different for every remote site.
Functionality
This feature is realized as follows: Update requests to the remote sites get forwarded to the central site – The entire configuration and the agent baking process are done on the central site. Agent packages that got requested once at a remote site will get cached (as long as they are valid) there to save unnecessary traffic. Additionally, the requested data will be checked for consistency on the remote site, again to avoid unnecessary traffic to the central site.
An agent is considered valid as long as a new agent hasn't been baked and as long as the first Interval for update check after an agent is baked hasn't passed. By default, the update check interval is 1 hour but can be adjusted in the "Agent Updater" rule.
Example:
When an agent is baked and signed, all hosts the newly baked agent applies to will pull the update at the next update interval. After all the hosts have pulled the new agent, there isn't a need to retain the agent past the following update interval.
If new hosts are added within the update interval it will pull the cached agent, but if added after, the remote server will pull the agent again for the new hosts.
General Communication Settings
Depending on the setting Update server information in the rule "Agent updater (Linux, Windows, Solaris)" and on the command line options during registration, the communication between the agent updater and the Checkmk servers can be configured granular. That way, you can restrict the communication to either the central Bakery or to the remote site server, if needed.
| CLI options during registration | Settings in Bakery Rule | Remote Site server reachable | Registration Server used | Package Download Server used |
|---|---|---|---|---|
| -s and -i | not used | yes | The server/site provided with -s | The dedicated remote server with the URL configured in the „Distributed Monitoring“ settings |
| none | „use for all connections“ | yes | The server/site provided in the Bakery rule | The server provided in the Bakery rule |
| none | „use for registration and fallback“ | yes | The server/site provided in the Bakery rule | The dedicated remote server with the URL configured in the „Distributed Monitoring“ settings |
| none | „use for registration and fallback“ | no | The server/site provided in the Bakery rule | The server provided in the Bakery rule |
| none | not configured | yes | Interactively asks for server | The dedicated remote server with the URL configured in the „Distributed Monitoring“ settings |
Example:
If you want to restrict the communication of the agent updater to the particular remote server,
- you can skip the Update server information in the Bakery rule.
- That way, you have a single software package per OS to install all your hosts.
- You just have to provide the remote server and site during the registration on the CLI.
- With https, you have to add the certificates of all your servers to the Bakery rule (when using Self signed certificates) or at least the root and intermediate ones (when using a CA).
Registration
Going to register agent at deployment server
2021-02-04 16:18:08,715 DEBUG: Authenticating at Check_MK Server (using requests): http://myhost/myremotesite/check_mk/login.py
2021-02-04 16:18:09,034 DEBUG: Fetching content (using requests): http://myhost/myremotesite/check_mk/register_agent.py
2021-02-04 16:18:09,070 DEBUG: Response from agent bakery:
{"result_code": 0, "result": {"host_secret": "rlkaihuoazcdcnseimvfpzaocsjitoabpieyzubuictninlfdtgciztydmkmfrfj", "update_url": "http:\/\/myhost\/myremotesite\/check_mk\/", "monitored": true}}
2021-02-04 16:18:09,070 INFO: Applying new update URL http://myhost/myremotesite/check_mk/ from deployment server
2021-02-04 16:18:09,071 DEBUG: No state file found yet. New state data will be saved to /etc/cmk-update-agent.state
2021-02-04 16:18:09,073 DEBUG: Saved deployment status to /etc/cmk-update-agent.state.
2021-02-04 16:18:09,073 INFO: Successfully registered agent of host "myremotesite" for deployment.
2021-02-04 16:18:09,073 DEBUG: Secret is rlkaihuoazcdcnseimvfpzaocsjitoabpieyzubuictninlfdtgciztydmkmfrfj.
2021-02-04 16:18:09,073 DEBUG: Successfully read /etc/cmk-update-agent.state.
2021-02-04 16:18:09,075 DEBUG: Saved deployment status to /etc/cmk-update-agent.state.
2021-02-04 16:18:09,076 INFO: Saved your registration settings to /etc/cmk-update-agent.state.
2021-02-04 16:18:09,076 DEBUG: Successfully scheduled an automatic update with next Check_MK Agent execution.
2021-02-04 16:18:09,076 DEBUG: Done.
The host is using the central site for registration!
Update
2021-02-04 16:18:19,630 DEBUG: Starting Check_MK Agent Updater v2.0.0b5
2021-02-04 16:18:19,630 DEBUG: Successfully read /etc/cmk-update-agent.state.
2021-02-04 16:18:19,630 DEBUG: Successfully read /etc/check_mk/cmk-update-agent.cfg.
2021-02-04 16:18:19,630 INFO: Starting Update mode as plugin.
2021-02-04 16:18:19,631 INFO: Getting target agent configuration for host 'myremotesite' from deployment server
2021-02-04 16:18:19,657 DEBUG: Fetching content (using requests): http://myhost/myremotesite/check_mk/deploy_agent.py
2021-02-04 16:18:19,717 DEBUG: Response from agent bakery:
{"result_code": 0, "result": {"update_url": "http:\/\/myhost\/myremotesite\/check_mk\/", "AgentAvailable": true, "TargetHash": "1ef2302ca00a9b89", "Signatures": [{"certificate": "-----BEGIN CERTIFICATE-----\nMIIC2jCCAcICAQEwDQYJKoZIhvcNAQEFBQAwMjEdMBsGA1UECgwUQ2hlY2tfTUsg\nU2l0ZSBtYXN0ZXIxETAPBgNVBAMMCGNta2FkbWluMCAXDTIxMDIwNDExNTExNVoY\nDzIwNTEwMTI4MTE1MTE1WjAyMR0wGwYDVQQKDBRDaGVja19NSyBTaXRlIG1hc3Rl\ncjERMA8GA1UEAwwIY21rYWRtaW4wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK\nAoIBAQDgFptynmaSVcnwqj69SDiJmHnSQAcb\/iMMXclptBo+7tcOOpU36COA0\/eL\nX7kp5V7Mu1zEfQq9wPmuQy+cVyWkMPj03tKZjly3Y+342WpFn\/hk8g0H72f+zTre\nebpM4d+UmkRWzo6\/aACgPTy0oPr4pDEiaPkzoGSrWtE+14jIIL5Cv+t8bIIs4uI0\n0VJ1HqR09FmuZhoYfV9hVoAEFYMRzo6BBweMIOz5cYEhJXa30VaGt9Fvk6FPwf9u\naKx25KxPHrz2oXgTnnP6afffx\/rl7oKQPGx6uT3LlNhkOJW\/R0RwVIKqQcjB66DB\njaESRa27L2vy7A5gO7muktMyIl73AgMBAAEwDQYJKoZIhvcNAQEFBQADggEBALSt\nz22vVZcBZ8RSy9ZX5nu4E+hBrZz\/84+U52UnL+ATbSKRA+VxltACezrBAXGB1pMO\nuCZ6YlKT6yMGqP0vyqhjAQnNJTlEr2V9MAh0IBuPUxCKTUbr6gzauAqT8FrVKgrY\nD2ZbCGKlls26tgbyyaH+Qqec9WsJQSDgEnIDoabR5Ip1hPjmWK0xG9kwQwFFZo+I\n3VMIOr52vys99TVBt4682560vO\/vhP1OQMmiyax\/\/M2fKuONo0zq7rWFiDbqvuL6\nTeuSSpANXEhfN\/42HCAqziWS85NYebUGLDbMYC6jJMfvBzOT\/uHrjbkop9dqOLfL\nnsIcFFkBV4e07TOLz6c=\n-----END CERTIFICATE-----\n", "signature": "vWcDferK6N7eVq7iVc7Ni++YS0GD9N25PWtT7dHbsykFGQ7QZJQ0kFEuAQz4sn3qy3T6feKgISOocXzSxZzpfQsWWlaWhPCpB4FXQ4pcDjcXuXsCN2Jd14dPKhEifTjLQyQ\/IuewwXVgezz63c5g+iyItQ2gA0GvxL6\/YRGTBV62r81ggafvmEnhswprS4WvykcO2+lS0pLfnkkVFGejFKKTIK0IfHoVNfnalyd1n2FD4\/ghRpwJlxafwPoZ1egG2xgtmzvTXBngHN7F+wG4oqpGImevDaDXu+pkJYqRu4IyFCK3OZFV3jlvn+5AX\/tP8LgU8D\/qNSHL36TMKEPAZw=="}]}}
2021-02-04 16:18:19,717 DEBUG: Successfully read /etc/cmk-update-agent.state.
2021-02-04 16:18:19,721 DEBUG: Saved deployment status to /etc/cmk-update-agent.state.
2021-02-04 16:18:19,721 DEBUG: Successfully read /etc/cmk-update-agent.state.
2021-02-04 16:18:19,723 DEBUG: Saved deployment status to /etc/cmk-update-agent.state.
2021-02-04 16:18:19,723 INFO: Target state (from deployment server):
2021-02-04 16:18:19,723 INFO: Agent Available: True
2021-02-04 16:18:19,723 INFO: Signatures: 1
2021-02-04 16:18:19,723 INFO: Target Hash: 1ef2302ca00a9b89
2021-02-04 16:18:19,749 DEBUG: Fetching content (using requests): http://myhost/myremotesite/check_mk/deploy_agent.py
2021-02-04 16:18:20,411 INFO: Downloaded agent has size 13116004 bytes.
2021-02-04 16:18:20,427 INFO: Signature check OK.
2021-02-04 16:18:20,436 INFO: Invoking package manager: dpkg -i /tmp/check-mk-agent-a8apxscl
2021-02-04 16:18:21,504 INFO: Output from dpkg:
(Reading database ... 158728 files and directories currently installed.)
Preparing to unpack /tmp/check-mk-agent-a8apxscl ...
Disable Checkmk agent in systemd (if active)...
Unpacking check-mk-agent (2.0.0b5-17.1ef2302ca00a9b89) over (2.0.0b5-17.1ef2302ca00a9b89) ...
Reloading xinetd...
Setting up check-mk-agent (2.0.0b5-17.1ef2302ca00a9b89) ...
Reloading xinetd...
Enable Checkmk agent in systemd...
Created symlink /etc/systemd/system/sockets.target.wants/check-mk-agent.socket → /etc/systemd/system/check-mk-agent.socket.
The host fetches all updates from the myremotesite.
Config
The agent updater creates a config and a state file. Please don't modify this file! You can take a look inside to see if the config is correct!
root@linux:/home/ubuntu# cat /etc/cmk-update-agent.state
{'update_url': 'http://myhost/myremotesite02/check_mk/', 'host_secret': 'zewqptluwgrjzamdvwjplosjhrwqpboxqjtxogreibeqkadfhswhotzifjqqramw', 'server': 'myserverip', 'site': 'mycentralsite', 'host_name': 'mycentralsite02', 'protocol': 'http', 'user': 'cmkadmin', 'last_check': 1611063314.5359132, 'installed_aghash': '67f84ee7e41b73aa', 'last_update': 1611063316.2098463}
root@linux:/home/ubuntu# cat /etc/check_mk/cmk-update-agent.cfg
# Created by Check_MK Agent Bakery.
# This file is managed via WATO, do not edit manually or you
# lose your changes next time when you update the agent.
{'activated': True,
'certificates': [],
'ignore_update_url': False,
'interval': 3600,
'protocol': 'http',
'server': 'myserverip',
'signature_keys': ['-----BEGIN CERTIFICATE-----\n'
'MIIC2jCCAcICAQEwDQYJKoZIhvcNAQEFBQAwMjEdMBsGA1UECgwUQ2hlY2tfTUsg\n'
'U2l0ZSBtYXN0ZXIxETAPBgNVBAMMCGNta2FkbWluMCAXDTIxMDExOTExNTMxMVoY\n'
'DzIwNTEwMTEyMTE1MzExWjAyMR0wGwYDVQQKDBRDaGVja19NSyBTaXRlIG1hc3Rl\n'
'cjERMA8GA1UEAwwIY21rYWRtaW4wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK\n'
'AoIBAQDWbh45st1xnzhsVzMjBBXzO7+5qz3/HPezvaOeUpSSIYHns2DnuDlwpLCO\n'
'XTdyfRUamgRtiOrzbUsSsARmMF9AqExao1+yM1rbEfcXP6mtBqMkqShlDw3n0L7F\n'
'HGGYG2CNqcRSmgxFe2Eck0ydv/p4cJONHIw4UFwqDnHfIC1L3tCnTQJagCig/Jvk\n'
'kZSZZqgkrjl4TuwRkFxdrnwURjm2hQGg+TkJcxpPHqSutbdnQrXnuuAEdx/ZIpFL\n'
'ivmjLDATjGzCy0LPZsVtjHZu5Lz4WWOZsA1BCZLecT81PI5edWs0RZ9mRH7QC+E8\n'
'CkIdlLgMUbfB093SGx3YEP8u+RtRAgMBAAEwDQYJKoZIhvcNAQEFBQADggEBAAU/\n'
'DtXHbs3cUPBGDs/Go0PgfqhwiZjn1aprCSh8O/RBLWA/jep65eU2pzsuoiKWT5CE\n'
'P/xc7pKXWmxQpyA4KA1Fz9BS80QHq5heXkG2FSLMFunDLxzDcbQI7QXH4oskIDkV\n'
'zzYio3VDXXDyhMBHd25XKKCQT3L9ou3bD7Esh/Cf/8Y2GQHkRvIWOpwLEknq0ovy\n'
'jkJ0o7Gx8GrqJYG5gUW44TJU/IhYFCxfhTS09RW842Z10iDSYiBPKoZLonM6F3SY\n'
'SLVRDn3vhnCpbkiO0Dc1op+0d9b5Wc+UCVW02G+l+G+aRw89L7v2xERXNPczCZgV\n'
'g+9DtIl5cu1fl4qVqmc=\n'
'-----END CERTIFICATE-----\n'],
'site': 'mycentralsite'}
Common Issues
HTTPSConnectionPool(host='checkmk.test.intern', port=443): Max retries exceeded with url: /check_mk/check_mk/login.py (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)')))
Possible Solution
Please make you follow the steps to import the SSL Certificates:
Related articles