How-to query historical metrics

This manual will show you the different options for fetching metrics from Checkmk to push this data to a 3rd party tool.

LAST TESTED ON CHECKMK 2.3.0P1

Table of Contents

Query via Livestatus

Syntax

Columns: rrddata:var1:metric1.max:start:end:step rrddata:var2:metric2.max:start:end:step

or

Columns: rrddata:m1:fs_used.max,1,*:1614839543:1614929543:1

With "1,*" you can rescale high values. This is only needed for high values like MB. 
You can rescale such values with
 "1024,*" or "1024,/"

Example

With this example, you will receive all perfdata for a specific service/host. You can use livestatus for that.

OMD[mysite]:~$ lq "GET services\nFilter: host_name = mysite\nFilter: service_description = Filesystem /\nColumns: host_name\nColumns: service_description\nColumns: rrddata:m1:fs_used.max,1024,/:1614839543:1614929543:30\nOutputFormat: python"

[[u"mysite",u"Filesystem /",[1614839400,1614931200,1800,None,None,None,None,None,None,None,None,None,None,None,None,None,None,None,None,333.982,334.178,334.465,334.638,334.773,334.923,335.082,335.249,335.361,None,None,None,None,None,None,None,None,None,None,None,None,None,None,None,None,None,None,None,None,None,None,None,None,336.493,336.673]]]

Query via REST-API

With Checkmk 2.1, announced by Werk #13640, the Web API was deprecated. This release now (Checkmk 2.2) removes the Web API.

We recommend migrating all existing scripts that use the Web API to the REST API.

Checkmk Version	Web or Rest API
2.0.0p*	Web-API
2.1.0p*	Web-API
2.2.0p*	REST API

To access the REST-API Documentation, click on Help → REST API Documentation

.
For the REST API queries, you will need to know what your metric IDs are.

To find this information, first, go to Monitoring → All Host → and select your desired host. From here, click on the Check_MK service.

.
Next, click on Display → Modify display options

.
Now enable Show internal graph and metric IDs and click submit

.
Scroll down this page and look for Service Metrics. You will see the Metric IDs on the right-hand side like the example below:

Query via Web-API

Checkmk Version	Web or Rest API
2.0.0p*	Web-API
2.1.0p*	Web-API
2.2.0p*	REST-API

Custom graph (Web-API)

With this example, you can query the metrics of a custom graph in Checkmk

root@linux:~# curl -k "https://mylinuxhost/mysite/check_mk/webapi.py?action=get_graph&_username=automation&_secret=mysecret&request_format=python&output_format=python" -d 'request={"specification":["custom","custom_graph_1"],"data_range":{"time_range":[1615627389,1616062956]}}'

{'result_code': 0, 'result': {'start_time': 1615626000, 'end_time': 1616063400, 'step': 1800, 'curves': [{'line_type': 'area', 'color': '#00b2ff', 'title': 'Service Costs per Day', 'rrddata': [None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None]}]}}

Forecast graph (Web-API)

With this example, you can query the metrics of a forecast graph in Checkmk.

root@linux:~# curl -k "https://mylinuxhost/mysite/check_mk/webapi.py?action=get_graph&_username=automation&_secret=mysecret&request_format=python&output_format=python" -d 'request={"specification":["forecast","forecast_graph_4"],"data_range":{"time_range":[1616062956,1616513102]}}'

{'result_code': 0, 'result': {'start_time': 1616504400, 'end_time': 1616527800, 'step': 1800, 'curves': [{'line_type': 'area', 'color': '#FFFFFF', 'title': 'CPU utilization - lower edge', 'rrddata': [33.345051190851265, 37.67683226938787, 44.50125333428577, 53.2724501059588, 60.44537606761632, 66.75387999685792, 72.64393851155472, 78.29311232060269, 83.75780489204897, 89.06739515972093, 94.1829539622409, 99.15563480780335, 104.09381440605844]}, {'line_type': 'stacked', 'color': '#F4E750', 'title': 'CPU utilization - confidence interval', 'rrddata': [11.100249458703228, 10.146064241405181, 10.102262795192633, 11.3949147901306, 16.27542796258677, 22.884785199874802, 30.331033266252426, 38.25905074392767, 46.55603069680629, 55.163215257233624, 64.15846274796492, 73.43946615261119, 82.78947205187224]}, {'line_type': 'line', 'color': '#0000FF', 'title': 'CPU utilization - original', 'rrddata': [42.2174, 40.4999, 47.7736, 60.8701, None, None, None, None, None, None, None, None, None]}, {'line_type': 'line', 'color': '#FF0000', 'title': 'CPU utilization - fit & forecast', 'rrddata': [38.89517592020288, 42.749864390090465, 49.552384731882086, 58.969907501024096, 68.58309004890971, 78.19627259679532, 87.80945514468094, 97.42263769256652, 107.03582024045211, 116.64900278833774, 126.26218533622335, 135.87536788410895, 145.48855043199455]}]}}

Builtin graph (Web-API)

With this example, you can query the metrics of a built-in graph in Checkmk. You will receive the data for all lines of a specific line.

root@linux:~# curl -k "https://mylinuxhost/mysite/check_mk/webapi.py?action=get_graph&_username=automation&_secret=mysecret&request_format=python&output_format=python" -d 'request={"specification":["template",{"service_description":"Filesystem /","site":"mysite","graph_index":0,"host_name":"mysite"}],"data_range":{"time_range":[1615627389, 1616062956]}}'
{'result_code': 0, 'result': {'start_time': 1615626000, 'end_time': 1616063400, 'step': 1800, 'curves': [{'line_type': 'area', 'color': '#00ffc6', 'title': 'Used filesystem space', 'rrddata': [None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None]}, {'line_type': 'stack', 'color': '#e3fff9', 'title': 'Free space', 'rrddata': [None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None]}, {'line_type': 'line', 'color': '#006040', 'title': 'Filesystem size', 'rrddata': [None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None,

RRDcached

If you want the values unfiltered, you can access these from the Round Robin Database (RRD) journal.log directly:

OMD[mysite]:~/var/rrdcached$ ls -ltr
total 8744
-rw-r----- 1 mysite mysite 2435087 Mar 23 15:45 rrd.journal.1616503549.911643
-rw-r----- 1 mysite mysite 1720320 Mar 23 17:07 rrd.journal.1616510749.911407
-rw-r--r-- 1 mysite mysite 2456557 Mar 24 09:13 rrd.journal.1616566388.199299
-rw-r--r-- 1 mysite mysite 2322432 Mar 24 10:49 rrd.journal.1616573588.199726


OMD[mysite]:~/var/rrdcached$ tail -f rrd.journal.1616573588.199726 
update /opt/omd/sites/mysite/var/check_mk/rrd/mysite/OMD_mysite_apache.rrd 1616579503:0:0:0:0:0:0:0:0:0.08098:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0.002154:19.176001:0:0:0:0:0:0:0:0:0:0:0
update /opt/omd/sites/mysite/var/check_mk/rrd/mysite/OMD_mysite_performance.rrd 1616579503:559:25:0:0.000009:0:364608.943527:0:0:0.583053:0:0.000008:0.000038:0:0.259135:3.09749:3.611691:5.571399:0.015021
update /opt/omd/sites/mysite/var/check_mk/rrd/mysite/OMD_dashboard_Notification_Spooler.rrd 1616579503:0:2560278
update /opt/omd/sites/mysite/var/check_mk/rrd/mysite/OMD_nagnis_master_Notification_Spooler.rrd 1616579503:0:63841:U:U
update /opt/omd/sites/mysite/var/check_mk/rrd/mysite/OMD_nagnis_master_backup_test.rrd 1616579503:3377776640:128932341.39868:49.073688
update /opt/omd/sites/mysite/var/check_mk/rrd/mysite/OMD_nagvis_master_backup_test.rrd 1616579503:3377776640:128932341.39868:49.073688
update /opt/omd/sites/mysite/var/check_mk/rrd/mysite/Process_chrome.rrd 1616579503:3264916:543481580:7.288174:23
update /opt/omd/sites/mysite/var/check_mk/rrd/mysite/Site_mysite_statistics.rrd 1616579503:13:42:18:1:119:366:3:0:1:21
update /opt/omd/sites/mysite/var/check_mk/rrd/mysite/Systemd_Timesyncd_Time.rrd 1616579503:0.000024:0.000463

Sources

Retrieving status data via Livestatus

Command reference for the HTTP-API

Performance data and graphing

Checkmk Knowledge Base