How-to query historical metrics

Owned by Anastasios Thomaidis
Last updated: Dec 11, 2023 by Matthew Hierholzer
7 min read
This manual will show you the different options for fetching metrics from Checkmk to push this data to a 3rd party tool.

LAST TESTED ON CHECKMK 2.0.0P1

Table of Contents

Query via Livestatus

Syntax

Columns: rrddata:var1:metric1.max:start:end:step rrddata:var2:metric2.max:start:end:step

or

Columns: rrddata:m1:fs_used.max,1,*:1614839543:1614929543:1

With "1,*" you can rescale high values. This is only needed for high values like MB. 
You can rescale such values with
 "1024,*" or "1024,/"

Example

With this example, you will receive all perfdata for a specific service/host. You can use livestatus for that.

OMD[mysite]:~$ lq "GET services\nFilter: host_name = mysite\nFilter: service_description = Filesystem /\nColumns: host_name\nColumns: service_description\nColumns: rrddata:m1:fs_used.max,1024,/:1614839543:1614929543:30\nOutputFormat: python"

[[u"mysite",u"Filesystem /",[1614839400,1614931200,1800,None,None,None,None,None,None,None,None,None,None,None,None,None,None,None,None,333.982,334.178,334.465,334.638,334.773,334.923,335.082,335.249,335.361,None,None,None,None,None,None,None,None,None,None,None,None,None,None,None,None,None,None,None,None,None,None,None,None,336.493,336.673]]]

Query via REST-API

With Checkmk 2.1, announced by Werk #13640, the Web API was deprecated. This release now (Checkmk 2.2) removes the Web API.

We recommend migrating all existing scripts that use the Web API to the REST API.

Checkmk VersionWeb or Rest API
2.0.0p*Web-API
2.1.0p*Web-API
2.2.0p*REST API


  1. To access the REST-API Documentation, click on Help → REST API Documentation
    Screenshot showing the location of the REST API documentation link.
    .
  2. For the REST API queries, you will need to know what your metric IDs are.

    To find this information, first, go to Monitoring → All Host → and select your desired host. From here, click on the Check_MK service.
    Screenshot displaying the Check_mk service in an OK state.
    .
  3. Next, click on Display → Modify display options
    Screenshot displaying the location of the Modify Display Options link.
    .
  4. Now enable Show internal graph and metric IDs and click submit
    Screenshot detailing the location of the Show Internal Graph and Metric IDs option.
    .
  5. Scroll down this page and look for Service Metrics. You will see the Metric IDs on the right-hand side like the example below:


Query via Web-API

Checkmk VersionWeb or Rest API
2.0.0p*Web-API
2.1.0p*Web-API
2.2.0p*REST-API

Custom graph (Web-API)

With this example, you can query the metrics of a custom graph in Checkmk

root@linux:~# curl -k "https://mylinuxhost/mysite/check_mk/webapi.py?action=get_graph&_username=automation&_secret=mysecret&request_format=python&output_format=python" -d 'request={"specification":["custom","custom_graph_1"],"data_range":{"time_range":[1615627389,1616062956]}}'

{'result_code': 0, 'result': {'start_time': 1615626000, 'end_time': 1616063400, 'step': 1800, 'curves': [{'line_type': 'area', 'color': '#00b2ff', 'title': 'Service Costs per Day', 'rrddata': [None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None]}]}}


Forecast graph (Web-API)

With this example, you can query the metrics of a forecast graph in Checkmk.

root@linux:~# curl -k "https://mylinuxhost/mysite/check_mk/webapi.py?action=get_graph&_username=automation&_secret=mysecret&request_format=python&output_format=python" -d 'request={"specification":["forecast","forecast_graph_4"],"data_range":{"time_range":[1616062956,1616513102]}}'

{'result_code': 0, 'result': {'start_time': 1616504400, 'end_time': 1616527800, 'step': 1800, 'curves': [{'line_type': 'area', 'color': '#FFFFFF', 'title': 'CPU utilization - lower edge', 'rrddata': [33.345051190851265, 37.67683226938787, 44.50125333428577, 53.2724501059588, 60.44537606761632, 66.75387999685792, 72.64393851155472, 78.29311232060269, 83.75780489204897, 89.06739515972093, 94.1829539622409, 99.15563480780335, 104.09381440605844]}, {'line_type': 'stacked', 'color': '#F4E750', 'title': 'CPU utilization - confidence interval', 'rrddata': [11.100249458703228, 10.146064241405181, 10.102262795192633, 11.3949147901306, 16.27542796258677, 22.884785199874802, 30.331033266252426, 38.25905074392767, 46.55603069680629, 55.163215257233624, 64.15846274796492, 73.43946615261119, 82.78947205187224]}, {'line_type': 'line', 'color': '#0000FF', 'title': 'CPU utilization - original', 'rrddata': [42.2174, 40.4999, 47.7736, 60.8701, None, None, None, None, None, None, None, None, None]}, {'line_type': 'line', 'color': '#FF0000', 'title': 'CPU utilization - fit & forecast', 'rrddata': [38.89517592020288, 42.749864390090465, 49.552384731882086, 58.969907501024096, 68.58309004890971, 78.19627259679532, 87.80945514468094, 97.42263769256652, 107.03582024045211, 116.64900278833774, 126.26218533622335, 135.87536788410895, 145.48855043199455]}]}}


Builtin graph (Web-API)

With this example, you can query the metrics of a built-in graph in Checkmk. You will receive the data for all lines of a specific line.

root@linux:~# curl -k "https://mylinuxhost/mysite/check_mk/webapi.py?action=get_graph&_username=automation&_secret=mysecret&request_format=python&output_format=python" -d 'request={"specification":["template",{"service_description":"Filesystem /","site":"mysite","graph_index":0,"host_name":"mysite"}],"data_range":{"time_range":[1615627389, 1616062956]}}'
{'result_code': 0, 'result': {'start_time': 1615626000, 'end_time': 1616063400, 'step': 1800, 'curves': [{'line_type': 'area', 'color': '#00ffc6', 'title': 'Used filesystem space', 'rrddata': [None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None]}, {'line_type': 'stack', 'color': '#e3fff9', 'title': 'Free space', 'rrddata': [None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None]}, {'line_type': 'line', 'color': '#006040', 'title': 'Filesystem size', 'rrddata': [None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None,


RRDcached

If you want the values unfiltered, you can access these from the Round Robin Database (RRD) journal.log directly:

OMD[mysite]:~/var/rrdcached$ ls -ltr
total 8744
-rw-r----- 1 mysite mysite 2435087 Mar 23 15:45 rrd.journal.1616503549.911643
-rw-r----- 1 mysite mysite 1720320 Mar 23 17:07 rrd.journal.1616510749.911407
-rw-r--r-- 1 mysite mysite 2456557 Mar 24 09:13 rrd.journal.1616566388.199299
-rw-r--r-- 1 mysite mysite 2322432 Mar 24 10:49 rrd.journal.1616573588.199726


OMD[mysite]:~/var/rrdcached$ tail -f rrd.journal.1616573588.199726 
update /opt/omd/sites/mysite/var/check_mk/rrd/mysite/OMD_mysite_apache.rrd 1616579503:0:0:0:0:0:0:0:0:0.08098:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0.002154:19.176001:0:0:0:0:0:0:0:0:0:0:0
update /opt/omd/sites/mysite/var/check_mk/rrd/mysite/OMD_mysite_performance.rrd 1616579503:559:25:0:0.000009:0:364608.943527:0:0:0.583053:0:0.000008:0.000038:0:0.259135:3.09749:3.611691:5.571399:0.015021
update /opt/omd/sites/mysite/var/check_mk/rrd/mysite/OMD_dashboard_Notification_Spooler.rrd 1616579503:0:2560278
update /opt/omd/sites/mysite/var/check_mk/rrd/mysite/OMD_nagnis_master_Notification_Spooler.rrd 1616579503:0:63841:U:U
update /opt/omd/sites/mysite/var/check_mk/rrd/mysite/OMD_nagnis_master_backup_test.rrd 1616579503:3377776640:128932341.39868:49.073688
update /opt/omd/sites/mysite/var/check_mk/rrd/mysite/OMD_nagvis_master_backup_test.rrd 1616579503:3377776640:128932341.39868:49.073688
update /opt/omd/sites/mysite/var/check_mk/rrd/mysite/Process_chrome.rrd 1616579503:3264916:543481580:7.288174:23
update /opt/omd/sites/mysite/var/check_mk/rrd/mysite/Site_mysite_statistics.rrd 1616579503:13:42:18:1:119:366:3:0:1:21
update /opt/omd/sites/mysite/var/check_mk/rrd/mysite/Systemd_Timesyncd_Time.rrd 1616579503:0.000024:0.000463


Sources

https://docs.checkmk.com/latest/en/livestatus.html

https://docs.checkmk.com/1.6.0/de/web_api_references.html

https://docs.checkmk.com/1.6.0/de/graphing.html