Overview

Checkmk special agents are standalone programs used to collect monitoring data from systems that cannot be monitored with the standard Checkmk agent, such as virtualization environments, storage systems, cloud and container platforms.

When a special agent fails, the Checkmk GUI usually provides only limited error information. Proper troubleshooting requires identifying the exact agent command generated by Checkmk, running it manually as the site user, and enabling debug and trace options to capture detailed output.

AWS special agent

The Amazon Web Service (AWS) special agent collects metrics and inventory data from AWS using Application Programming Interface (API) credentials.

Checkmk Versions 2.2 and above
If you want to execute the special agent from the command line, please run the following command.

Viewing Effective Configuratio

To view more AWS options, use the cmk -D aws combined with grep.

Debugging

Enable debug, verbose output, and API tracing by running the agent with the --debug, --verbose, and --vcrtrace options.

Checkmk Versions 2.2.0p42 and Above

Two files must be generated:

• debug.log containing agent execution details

• trace.txt containing recorded AWS API calls

Azure special agent

The Azure special agent retrieves metrics and inventory data via Azure Resource Manager and Microsoft Graph Application Programming Interface (API)s. Errors often stem from missing permissions or blocked outbound connectivity.

Checkmk Versions 2.2.0p42 and Above

Service examples that can be used in the command above:

• usage_details

• Microsoft.Compute/virtualMachines

• Microsoft.Network/virtualNetworkGateways

• Microsoft.Sql/servers/databases

• Microsoft.Storage/storageAccounts

• Microsoft.Web/sites

• Microsoft.DBforMySQL/servers

• Microsoft.DBforMySQL/flexibleServers

• Microsoft.DBforPostgreSQL/servers

• Microsoft.DBforPostgreSQL/flexibleServers

• Microsoft.Network/trafficmanagerprofiles

• Microsoft.Network/loadBalancers

• Microsoft.RecoveryServices/vaults

• Microsoft.Network/applicationGateways

Viewing Effective Configuration

To view more Azure options, use the cmk -D azure combined with grep.

Common Error: Insufficient Graph Privileges

Error message:
Graph client: Insufficient privileges to complete the operation

Possible Solution:

1. Open the Azure Portal
   
   

2. Navigate to Azure Active Directory

   Open

   

    

3. Open App registrations on the left side

   Open

   

    

4. Select the Checkmk application
   
   

5. Open API permissions on the left side

   Open

   

    

6. Click Add Permissions and add a permissions for Microsoft Graph

   Open

   

Full list of access rights needed:

These are the metrics we get via the Azure agents

Common Error: SSL "bad handshake"

When trying to monitor Microsoft Azure environment, you see the following error message in the Checkmk service Azure Agent Info:

Possible Solution

You need to make sure that your Checkmk server can connect to the following two addresses:

• management.azure.com 

• login.microsoft.com

If your Checkmk server cannot establish a connection or the connection times out, Azure monitoring will not function. You can quickly verify connectivity by logging in as the Checkmk site user and testing the connection with either Telnet or Netcat.

The output of these commands should look like this:

Firewalls blocking these endpoints are a frequent cause of agent failure.

BI Special Agent

A BI Special Agent is a monitoring agent that collects health, status, and performance data directly from Business Intelligence (BI) systems using BI-specific APIs or interfaces, providing visibility beyond basic infrastructure monitoring. With Werk #6679, a new BI Special Agent was introduced that securely receives the required key directly via stdin, rather than through command-line arguments or configuration files.

If you use special agents installed from a Feature Pack, you can find the special agents in:

The special agents that are already shipped with Checkmk can be found here:

If you want to execute the special agent from the command line, use cmk -D bi which can be piped (|) to head.

With this command, you will get the special agent call. Now you can continue the debugging.

How to execute the command manually:

Jenkins Special Agent

For the Jenkins Special Agent to function correctly, the Jenkins user account used for authentication must have sufficient permissions to query job, node, and build information. At a minimum, the user must be granted the following access rights within Jenkins:

• General: Read
  To allow basic access to the Jenkins instance.
  
  

• Agent: Connect
  To enable communication with build agents.
  
  

• Element: Read & Workspace
  To retrieve job details and workspace information.
  
  

• Views: Read
  To access and enumerate configured views.

Without these permissions, the agent may fail to collect monitoring data or return incomplete results.

Kubernetes Special Agent

Getting Started

Background information regarding this subject is available on our:

• Official documentation

• YouTube Video

Supported Kubernetes Versions

Support is limited to the three most recent Kubernetes versions. When a newly released Kubernetes version is in use and that release is less than one month old, a support ticket should be opened to verify whether compatibility and official support for that version are already in place.

Installation of Checkmk Cluster Collectors (a.k.a. Checkmk Kubernetes agent)

We strongly recommend using our helm charts for installing the Checkmk Cluster Collectors unless you are well-experienced with Kubernetes and want to install the agent using the manifests in YAML format provided by us.

The Helm chart installs and configures all necessary components to run the agent and exposes several helpful configuration options that will help you automatically set up complex resources. The prerequisites have to be fulfilled before you proceed with the installation.

Below is an example of deploying the helm charts using a LoadBalancer (requires the ability of the cluster to create a LoadBalancer): 

As an example, you can further set some configuration options on the command line to the above helm command (these are some examples, but depending on your requirement, you can specify multiple or separate values) :

We recommend using these values.yaml to configure your Helm chart.

For this, you need to then run the command:

After the chart has been successfully deployed, you will be presented with a set of commands to access the cluster-collector from the command line.  In case you want to see those commands, you can do the following:

At the same time, you can also verify if all the essential resources in the namespace have been deployed successfully. The below command in the code-block lists some important resources:

Exposing the Checkmk Cluster Collector

By default, the API of Checkmk Cluster Collector is not exposed to the outside (not to be mistaken with Kubernetes API itself). This is required to gather usage metrics and enrich your monitoring.

Checkmk pulls data from this API, which can be exposed via the service checkmk-cluster-collector. To do so, you must run it with one of the following flags or set them in a values.yaml.

Debugging

To begin debugging, identify the full command used to execute the Kubernetes special agent.

The command is shown under Type of agent → Program and includes all parameters defined by the datasource program rule.

As a shortcut, the following command can be used to extract and execute the Kubernetes special agent command automatically:

If multiple Type of agent and Program blocks exist in the output, this command may produce incorrect results.

The special agent provides the following options for debugging.

You can now modify the Kubernetes special agent command as shown below.

Here, you can also reduce the number of '--monitored-objects' to a few resources to get less output. 

Run the special agent without debug options to generate a clean agent output, or download the agent output from the cluster host via the Checkmk web interface.

Please upload the following files to the support ticket.

• Agent output
  ~/tmp/k8s_agent_output.txt

• Debug output
  ~/tmp/k8s_with_debug.txt

• Trace file
  ~/tmp/vcrtrace.txt

Common Error: Cluster Collector Errors

Cluster Collector connection issues are reported by the Cluster Collector service and do not cause the Kubernetes special agent to terminate. The Kubernetes special agent can collect data from multiple datasources, and connections to the Cluster Collector datasources are optional and controlled by the configured datasource rule.

In production mode, failures to connect to the Cluster Collector are not raised as agent errors. Detailed error information is available only when the agent is executed with the --debug flag.

Common Causes

Cluster Collector connection problems are typically caused by configuration or infrastructure issues. The Cluster Collector must be properly exposed so that it can be reached, either via a NodePort or an Ingress. If it is not exposed, the agent will be unable to connect.

Another frequent cause is unhealthy or unstable Kubernetes resources. Core components such as pods, deployments, daemon sets, and replica sets must be running correctly. If these resources are not running or are restarting frequently, the Cluster Collector may be unavailable.

Network restrictions can also prevent connectivity. Firewalls or security groups may block access to the Cluster Collector’s IP address or port, or the configured IP and port may simply be incorrect.

In environments where a proxy is required to reach the Cluster Collector, the agent will be unable to establish a connection if the proxy is not configured in the datasource rule.

Common Error: API Connection & Processesing

The connection to the Kubernetes API server is mandatory for the Kubernetes special agent. If the agent fails to connect to the API, for example by returning a 401 Unauthorized error when querying api/v1/core/pods, the failure is reported by the corresponding Checkmk service and the agent terminates.

The debug output generated using the --debug flag is usually sufficient to diagnose Kubernetes API connection problems. In cases where the agent reports an API processing error similar to “value … was not set”, the debug output alone may not be sufficient. In these situations, the user should be asked to provide the vcrtrace.txt file to allow deeper analysis of API request and response handling.

Common Causes

Kubernetes API connection issues are most often related to authentication or connectivity problems. A common cause is an incorrectly configured service account, which may be missing the required roles or permissions to access the Kubernetes API.

Authentication failures can also occur if the token is wrong or has expired, preventing the agent from successfully querying the API.

TLS configuration problems are another frequent source of errors. If the --verify-cert-api option is enabled but the corresponding ca.crt file has not been uploaded under Global settings → Trusted certificate authorities for SSL, the API connection will fail due to certificate validation errors.

Connectivity issues should also be checked. An incorrect API server IP address or port will prevent the agent from reaching the Kubernetes API endpoint.

In environments where a proxy is required, the connection will fail if the proxy is missing or incorrectly configured in the datasource rule.

Linux Agent Over SSH

When running the Checkmk Linux agent via SSH, you may encounter error messages if the SSH setup is incorrect. In most cases, the Check_MK service will report connection-related issues.

Below are common error messages, their typical causes, and steps to resolve them.

Common Error: Permission denied

Error Message

Possible Cause

The public SSH key stored in the authorized_keys file on the monitored host is invalid or corrupted. This often happens when:

• A line break is accidentally inserted into the key

• One or more characters are missing

• The key was incorrectly copied 

Possible Solution

Carefully verify that the public key on the monitored host matches exactly the public key generated on your Checkmk server.

• Ensure there are no extra spaces or line breaks

• Compare the keys character by character if necessary

Even a single incorrect character will cause SSH authentication to fail.

Common Error: Host key verification failed

Error Message

Possible Cause

The error message here is clear. The "host key verification failed". But what does this mean? It might just mean that you never introduced your Checkmk server and the host to one another, and hence the key fingerprint is not available in the file "~/.ssh/known_hosts" on your Checkmk server.

Possible Solution

This issue can be resolved easily. Log in to your site and initiate an SSH connection to the target host. When prompted to confirm the host’s authenticity, type yes. This action will add the host’s key to the list of known hosts, allowing future connections to proceed without errors.

NetApp Special Agent

The NetApp special agent in Checkmk runs on the monitoring server and uses the REST API over HTTPS to collect health, storage, and performance data from a NetApp cluster. It retrieves status information as well as capacity and performance metrics, which Checkmk turns into services for alerting and graphing.

Debugging

1. Login to the Checkmk server and become siteuser.

   root@linux:$ su mysite OMD[mysite]:~$ cmk -D <netapp_host> | head -n 15

   This should display the whole special agent query, including all arguments (similar to vSphere debugging)
   
   

2. Copy the whole output.
   .

3. Paste it and add the debug option to it like so:

   /omd/sites/yoursitename/share/check_mk/agents/special/agent_netapp --hostname {yourhostname} --username {yourusername} --password {yourpassword} --vcrtrace /tmp/TRACEFILE '-no_counters' --debug --xml > /tmp/debug.txt 2>&1

   .

4. Add the agent_netapp command line (password stripped) and the dump.txt to your support ticket.

   c0::security login rest-role> show -role checkmk Role Access Vserver Name API Level ---------- ------------- ------------------- ------ c0 checkmk /api/cluster/counter/tables readonly /api/cluster/nodes readonly /api/cluster/sensors readonly /api/network/ethernet/ports readonly /api/network/fc/ports readonly /api/network/ip/interfaces readonly /api/private/support/alerts readonly /api/snapmirror/relationships readonly /api/storage/disks readonly /api/storage/luns readonly /api/storage/quota/reports readonly /api/storage/shelves readonly /api/storage/volumes readonly /api/svm/svms readonly 14 entries were displayed.

Special Agents Using stdin Parameters

Some Checkmk special agents receive their configuration exclusively via standard input (stdin) rather than command-line arguments or config files. This approach is used by agents such as the Prometheus and AWS special agents, allowing Checkmk to securely and centrally pass structured configuration data at runtime, simplify parameter handling, and avoid exposing sensitive information in process arguments.

Debugging

To debug the Prometheus special agent, first ensure that a corresponding rule is configured and pinned to the host myprometheushost. Next, log in to the host as the site user and execute the following command:

After running the command, you should see output similar to the following example:

Now copy the block that appears after Program stdin:, wrap it in double quotes, and prepend it with echo. Then add a pipe (|) followed by the path to the special agent shown on the line that begins with Program:. Combined, the command looks like this:

In most cases, special agents support verbose or debug output from Python. To enable this, simply append -vvv and/or --debug to the very end of the command shown above. 

StoreOnce 4x Special Agent

The StoreOnce 4x Special Agent is a dedicated special agent in Checkmk designed to monitor HPE StoreOnce 4x backup systems via their REST API. Unlike classic Checkmk agents that run directly on the monitored host, this special agent is executed by the Checkmk site and communicates remotely with the StoreOnce appliance.

Common Error: Crash due to expired OAuth token

The StoreOnce 4x special agent crashes during execution with an OAuth-related error. The agent output shows that the stored access token has expired and the token refresh fails because no new access token is returned.

Error Message:

This indicates that the OAuth token stored locally is no longer valid and cannot be refreshed successfully.

Possible Solution
The issue can be diagnosed and resolved by enabling debugging on the special agent and resetting the stored OAuth token.

1. Determine the exact special agent command
   First, find out how the StoreOnce 4x special agent is executed for the affected host. You can check this in the Type of agent section.

   OMD[mysite]:~$ cmk -D hostname

   
   An easier way to extract the command is:

   /bin/sh -c "$(cmk -D hostname | grep -A1 "^Type of agent:" | grep "Program:" | cut -f2 -d':')"

   Note: If the output contains multiple blocks with Type of agent: followed by Program:, the extracted command may be incorrect.

   
   
   

2. Check available debugging options
   The StoreOnce 4x special agent provides several debugging and tracing options.

   OMD[mysite]:~$ ~/share/check_mk/agents/special/agent_storeonce4x -h

    

   Relevant options are:

   --debug, -d Enable debug mode (keep some exceptions unhandled) --verbose, -v --vcrtrace TRACEFILE, --tracefile TRACEFILE If this flag is set to a TRACEFILE that does not exist yet, it will be created and all requests the program sends and their corresponding answers will be recorded in said file. If the file already exists, no requests are sent to the server, but the responses will be replayed from the tracefile.

    
   
   

3. Run the special agent with debugging enabled
   Extend the original special agent command by adding the debugging options:

   OMD[mysite]:~$ ~/share/check_mk/agents/special/agent_storeonce4x <OTHER ARGUMENTS> --debug -v --vcrtrace ~/tmp/vcrtrace.txt 2>1 ~/tmp/storeonce4x_with_debug.txt

   This will create a detailed debug log and a trace file containing the REST API communication.
   
   
   

4. Create a clean agent output (without debugging)
   For reproducibility, also run the agent once without debug options and store the raw output:

   OMD[mysite]:~$ /omd/sites/mysite/share/check_mk/agents/special/agent_kube <OTHER ARGUMENTS> > ~/tmp/k8s_agent_output.txt

   With this file, we can reproduce your issue.

   
   
   

5. Reset the stored OAuth token
   The StoreOnce 4x special agent authenticates using username and password. After a successful login, an OAuth access token is stored locally and reused for future REST requests.
   
   

   The token file is stored under:

   ~/tmp/check_mk/special_agents/agent_storeonce4x/<hostname>_oAuthToken.json

    

   Rename the token file to force the agent to perform a fresh login:

   OMD[mysite]:~$ mv \ ~/tmp/check_mk/special_agents/agent_storeonce4x/<hostname>_oAuthToken.json \ ~/tmp/check_mk/special_agents/agent_storeonce4x/<hostname>_oAuthToken.json.back

   
   
   

6. Run the special agent again
   After renaming the token file, run the StoreOnce 4x special agent again. A new OAuth token will be requested and stored automatically, which usually resolves the crash caused by expired or invalid tokens.

For further details about authentication and available endpoints, refer to the StoreOnce REST API documentation.

VMware vSphere Special Agent

VMware vSphere remains a cornerstone of many on-premises and hybrid IT environments. While containers and Kubernetes have gained significant traction, classic virtualization continues to be essential wherever containerization is not practical or supported.

This following describes the vSphere extension to VMware ESXi monitoring in Checkmk, explains how datastore provisioning and piggyback monitoring work, and provides debugging procedures for troubleshooting the vSphere Special Agent.

Datastore provisioning in vSphere

When a vCenter Server is added to Checkmk, datastore provisioning is monitored automatically and with full accuracy. This allows you to detect and alert on excessive thin-provisioning, where virtual machines collectively claim more logical storage than the datastore can physically deliver.

If only an ESXi host is monitored, provisioning metrics may still appear in the filesystem checks, but they simply mirror the used filesystem values. In this setup, true over-provisioning cannot be identified, as only vCenter has complete knowledge of the actual logical provisioning across all virtual machines.

Piggyback-only with ESXI hosts

While using a read-only account on both ESXi hosts and vCenter is considered best practice for continuous monitoring, direct access to the hosts is not always permitted.

With piggyback monitoring configured correctly, a read-only user on vCenter alone is sufficient. Using a local vSphere account, for example monitoring@vsphere.local, is recommended over Active Directory accounts, as AD authentication may time out during inventory queries. Access should be validated by logging in to the vSphere Web Client with this user.

When vCenter is added to Checkmk with all available data and the ESXi hosts are added afterward, piggyback automatically assigns vCenter-derived resources such as CPU, memory, and datastores to the corresponding ESXi hosts.

When virtual machines are added as hosts in Checkmk, additional checks are enabled automatically without further configuration. These checks typically start with “ESX” and focus on VM resource consumption. One example is ESX Snapshots, which monitors all snapshots assigned to a VM and can alert when snapshots exceed a defined age. This helps ensure that manually created snapshots are removed in a timely manner.

Limitation: If ESXi hosts are not monitored directly via the Special Agent or SNMP, local host partitions are not visible.

Debugging

1. Identify the Special Agent Command

   Use the following command to inspect how Checkmk runs the vSphere Special Agent:

   OMD[mysite]:~$ cmk -D <vcenter-host> | more vcenter Addresses: x.x.x.x Tags: [add_ip_addresses:add_ip_addresses_1], [address_family:ip-v4-only], [agent:special-agents], [criticality:prod], [ip-v4:ip-v4], [networking:lan], [piggyback:auto-piggyback], [site:nagnis_master], [snmp_ds:no-snmp], [tcp:tcp] Labels: [cmk/vsphere_object:vm] Host groups: check_mk Contact groups: all Agent mode: No Checkmk agent, all configured special agents Type of agent: Program: /omd/sites/mysite/share/check_mk/agents/special/agent_vsphere -u 'user' -s 'password' -i hos tsystem,virtualmachine,datastore,counters,licenses -P --spaces cut --snapshot_display vCenter --no-cert-check 'x.x.x.x' Process piggyback data from /omd/sites/mysite/tmp/check_mk/piggyback/vcenter Services: checktype item params

   Look for the Program: line under Type of agent.This shows the full command Checkmk executes internally.