Checkmk 2.2 and Grafana

Checkmk 2.1 uses Web API, while version 2.2+ uses REST API. Upgrading to version 2.2+ and using data source v2.0.3 may cause issues.

LAST TESTED ON CHECKMK 2.2.0P1

Table of Contents



If you're using Checkmk version 2.1, use data source release v2.0.3.

To avoid this, upgrade the data source to v3.0.0.

Data sources can be downloaded from: https://github.com/tribe29/grafana-checkmk-datasource/releases


Getting Started

Background information regarding this subject is available in our Official documentation

Web vs. Rest API

Checkmk version 2.1 and below use the Web API while 2.2 and greater uses REST API.


Checkmk VersionData Source VersionWeb or Rest API
2.0.0p*2.0. beta3Web API
2.1.0p*2.0.3Web API
2.2.0p*3.0.0REST API


You can verify which API Checkmk uses by going to Setup → Global Setting -> and searching for API at the top. 

Screenshot of location of REST API option


Web API Testing (checkmk 2.0 & 2.1)

If testing the Web API via Grafana is needed, you first want to open the Network Analyze with your internet browser. We can test the Web API using Curl, but we must form the API request first.

Start the Network Monitor


After you open the Network Analyzer tool, you will want to:

  1. Click Network within the top bar
    .
  2. Click Save & Test on the Grafana Data Source page
    .
  3. Select webapi.py response on the left sidebar of the Inspector page
    .
  4. Click Headers on the response bar
    .
  5. Copy the Request URL line to your text editor so adjustments can be made. Copy everything from HTTP:// through webapi.py?


Screenshot of page inspector showing the location of API request.


To form your API request, you will need several pieces of information that will be required.
Grafana Username & Password

  • URL Copied from Internet Inspector
  • Checkmk Automation User & Password
  • API commands for the information that you are trying to query


.

In the following example, we will request an API to list all known hosts currently added to our Checkmk instance.

bash-5.1# curl "http://grafanaUsername:grafanaPassword@localhost:3000/api/datasources/proxy/uid/BZ7iqAa4z/cmk/check_mk/webapi.py?action=get_host_names&_username=checkmkAutomationUser&_secret=checkmkAutomationSecret&output_format=pythyon" 
{"result_code": 0, "result": ["host001.domain.com", "host002.domain.com", "host003.domain.com", "host004.domain.com"]}


To break this command down, we used the following:

  • Grafana Username & Password
    username:password
    .
  • URL Copied from Internet Inspector
    @localhost:3000/api/datasources/proxy/uid/BZ7iqAa4z/cmk/check_mk/webapi.py?
    .
  • API Action or Command
    ?action=get_host_names 
    .
  • Checkmk Username & Password
    &_username=checkmkAutomationUser&_secret=checkmkAutomationSecret
    .
  • API Output (Python or JSON) 
    &output_format=pythyon




Visual understanding of the above API command:

Screenshot to visually represent the sections of api string that need to be modified



REST API Testing (checkmk 2.2 & above)

If testing the Web API via Grafana is needed, you first want to open the Network Analyze with your internet browser. We can test the Web API using Curl, but we must form the API request first.

Start the Network Monitor


After you open the Network Analyzer tool, you will want to:

  1. Click Network within the top bar
    .
  2. Click Save & Test on the Grafana Data Source page
    .
  3. Select webapi.py response on the left sidebar of the Inspector page
    .
  4. Click Headers on the response bar

  5. Copy the Request URL line to your text editor so adjustments can be made. Copy everything from HTTP:// through /1.0/


Screenshot of page inspector showing the location of API request.


To form your API request, you will need several pieces of information that will be required.

  • Grafana Username & Password
  • URL Copied from Internet Inspector
  • Checkmk Automation User & Password
  • API commands for the information that you are trying to query


In the following example, we will request an API to list all known hosts currently added to our Checkmk instance.

bash-5.1# curl -X 'GET' 'http://grafanaUsername:grafanaPassword@localhost:3000/api/datasources/proxy/uid/BZ7iqAa4z/rest/check_mk/api/1.0/domain-types/host_config/collections/all?effective_attributes=false' -H 'accept: application/json'

{
  "links": [
    {
      "domainType": "link",
      "rel": "self",
      "href": "http://localhost/sitename/check_mk/api/1.0/domain-types/host_config/collections/all",
      "method": "GET",
      "type": "application/json"
    }
  ],
  "id": "host",
  "domainType": "host_config",
  "value": [
    {
      "links": [
        {
          "domainType": "link",
          "rel": "self",
          "href": "http://localhost/sitename/check_mk/api/1.0/objects/host_config/host001_domain",
          "method": "GET",
          "type": "application/json"
        },
...full response truncated


To break this command down, we used the following:

  • Grafana Username & Password
    username:password
    .
  • URL Copied from Internet Inspector
    @localhost:3000/api/datasources/proxy/uid/BZ7iqAa4z/rest/check_mk/api/1.0/
    .
  • API Action or Command
    domain-types/host_config/collections/all?effective_attributes=false' 
    .
  • API Output
    -H 'accept: application/json'

A handy resource can be found within the help documentation on your Checkmk 2.2+ site

Screenshot of REST Api interactive gui location

Checkmk REST API Reference Guide


Visual understanding of the above API command:

Screenshot to visually represent the sections of api string that need to be modified


Checkmk 2.1 using Data Source v2.0.3

With this example, you can see that the Data Source is saved and is working.

Checkmk 2.1 using Data Source 2.0.3

Checkmk 2.1 using Data Source v3.0.0

Fortunately, data source v3.0.0 is backward compatible with Checkmk 2.1. To use it, when configuring Checkmk in Grafana, change the version from ">= 2.2" to "< 2.2". See the example below:

Checkmk 2.1 using Data Source 3.0


Checkmk 2.2 using Data Source v2.0.3 (not supported)

Using this combination produces a "Could not read API response" error.

Checkmk 2.2 using Data Source 2.0.3


Checkmk 2.2 using Data Source v3.0.0 (recommended)

Recommended configuration for Checkmk 2.2.

Checkmk 2.2 using Data Source 3.0


SSL Configuration and Testing

During the configuration process, you may encounter errors related to a Bad Gateway. SSL issues between Checkmk and Grafana typically cause this. To prevent this, verifying that SSL communication is functioning correctly between the two sites is recommended.


Screenshot of bad gateway error that happens during bad SSL configs


Curl can be used to help troubleshoot SSL issues. If SSL works correctly, Curl should respond like the one below. This should be run on the Grafana server that you're using. 

bash-5.1# curl https://localhost/checkmksite/

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>302 Found</title>
</head><body>
<h1>Found</h1>
<p>The document has moved <a href="http://127.0.0.1:5008/checkmksite/check_mk/">here</a>.</p>
</body></html>


An unsuccessful attempt may look like the following:

bash-5.1# curl https://localhost/checkmksite/ 
curl: (60) SSL certificate problem: self signed certificate in certificate chain
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.


If you are using a Self Signed Certificate for your Checkmk server and receive an error of "Could not read  API response" like the below: 

Screenshot of could not read api response error


You may need to copy your self-signed .crt file to /usr/local/share/ca-certificates/ and run update-ca-certificates on the Grafana server.


If you have no SSL certificates installed, you will want to review the Securing the Web interface with HTTPS document to create and install your certificates.