This article explains how to manage Apache SSL certificate changes in Checkmk’s Agent Bakery without breaking agent trust.

APPLICABLE TO ALL CHECKMK VERSIONS

Table of Contents

Getting Started

Background information regarding this subject is available on our:

The Core Problem: Broken Certificate Trust

When the Agent Bakery creates an agent, it embeds the Certificate Authority (CA) that signed your server's Apache certificate.
The Agent Updater uses this CA to verify the server’s identity.

Two outcomes are possible when replacing the Apache certificate:

Renewed by the same CA
If you renew with the same CA, the trust root remains intact, and agents will continue to function.
Signed by a new CA
If you switch to a different CA (e.g., from self-signed to a corporate or public CA), existing agents will reject the new certificate, breaking trust and halting communication.

The Reactive Approach: Recovering After Trust is Broken

If the certificate is replaced before agents are updated, the updater stops working, forcing manual recovery.

Steps for Recovery

Update the CA in Checkmk
Upload the new CA in the rule under Agent updater → Certificates for HTTPS verification.
Re-bake agents
Bake and sign agents so they include the new CA.
Re-deploy manually
Deploy the new agents to every host. In large environments, automation tools like Ansible, Puppet, or SCCM may be required.

Automating Recovery

Instead of redeploying full packages, automation tools can run commands directly on each host:

cmk-update-agent --trust-cert
Permanently trusts the new certificate.
cmk-update-agent -x
Temporarily disables validation for one connection, allowing the agent to fetch the updated configuration (which includes the new CA).

The Proactive Solution: Zero-Downtime Transition (Best Practice)

The recommended method is to prepare agents before replacing the Apache certificate. This ensures continuous communication with no downtime.

Step-by-Step Guide

Get the new certificate
Obtain the new certificate, including CA and chain.
Check if the certificate chain changed
- Yes: Add the new chain to the Agent Updater (do not replace the old one), then bake and sign agents. Wait until all agents are updated.
- No: Proceed to step 4.
Verify deployment on test hosts
Restrict agent updates to 2–3 hosts and confirm that communication works as expected.
Replace the Apache certificate
- For Appliances: go to Device Settings → Web Access → Upload Certificate.
  
  For other systems, deploy the new certificate to Apache.
  Ensure both test hosts and general agents communicate successfully.
Finalize rollout
- If the chain changed, remove the old chain from the updater rule and bake again.
- Remove host restrictions and allow all agents to update.

Automating the Proactive Workflow with the REST API

This workflow can be fully automated with the Checkmk REST API:

Script retrieves the new certificate.
REST API updates the agent_config:cmk_update_agent ruleset with the new CA.
Script triggers a bake and sign.
After confirming deployment, the Apache certificate is replaced.

This approach is especially useful when certificates rotate automatically on a schedule.

Verification

To confirm that the agent and server certificates match:

On the agent host, list installed certificates:

root@mylinuxhost:~$ echo "Agent updater certificate store:"
openssl crl2pkcs7 -nocrl -certfile <(cat /etc/check_mk/cmk-update-agent.cfg | egrep "\\n'" | cut)

Check the server certificate:

root@mylinuxhost:~$ s=$(cat /etc/check_mk/cmk-update-agent.cfg | grep "'server'" | cut -f4 -d"'")
echo "CMK Server HTTPS Certificate for $s"
openssl s_client -connect $s:443 </dev/null 2>/dev/null | openssl x509 -noout -text | grep -e Issuer -e Subject -e "Subject Alternative Name"

Compare the Issuer, Subject, and SANs against what is baked into the agent

For more information about Appliances and SSL certificates, please refer to our Official Documentation.

Managing Apache Certificate Changes with the Agent Bakery