/
How-to query historical metrics

How-to query historical metrics

This manual will show you the different options for fetching metrics from Checkmk to push this data to a 3rd party tool.

LAST TESTED ON CHECKMK 2.3.0P1

Table of Contents

Query via Livestatus

Syntax

Columns: rrddata:var1:metric1.max:start:end:step rrddata:var2:metric2.max:start:end:step or Columns: rrddata:m1:fs_used.max,1,*:1614839543:1614929543:1 With "1,*" you can rescale high values. This is only needed for high values like MB. You can rescale such values with "1024,*" or "1024,/"

Example

With this example, you will receive all perfdata for a specific service/host. You can use livestatus for that.

OMD[mysite]:~$ lq "GET services\nFilter: host_name = mysite\nFilter: service_description = Filesystem /\nColumns: host_name\nColumns: service_description\nColumns: rrddata:m1:fs_used.max,1024,/:1614839543:1614929543:30\nOutputFormat: python" [[u"mysite",u"Filesystem /",[1614839400,1614931200,1800,None,None,None,None,None,None,None,None,None,None,None,None,None,None,None,None,333.982,334.178,334.465,334.638,334.773,334.923,335.082,335.249,335.361,None,None,None,None,None,None,None,None,None,None,None,None,None,None,None,None,None,None,None,None,None,None,None,None,336.493,336.673]]]

Query via REST-API

With Checkmk 2.1, announced by Werk #13640, the Web API was deprecated. Werk #14298 (Checkmk 2.2) removes the Web API.

We recommend migrating all existing scripts that use the Web API to the REST API.

Checkmk Version

Web or Rest API

Checkmk Version

Web or Rest API

2.0.0p*

Web-API

2.1.0p*

Web-API

2.2.0p*

REST API

 

  1. To access the REST-API Documentation, click on Help → REST API Documentation

    Screenshot showing the location of the REST API documentation link.


    .

  2. For the REST API queries, you will need to know what your metric IDs are.

    To find this information, first, go to Monitoring → All Host → and select your desired host. From here, click on the Check_MK service.

    Screenshot displaying the Check_mk service in an OK state.


    .

  3. Next, click on Display → Modify display options

    Screenshot displaying the location of the Modify Display Options link.


    .

  4. Now enable Show internal graph and metric IDs and click submit

    Screenshot detailing the location of the Show Internal Graph and Metric IDs option.


    .

  5. Scroll down this page and look for Service Metrics. You will see the Metric IDs on the right-hand side like the example below:

 

Query via Web-API

Checkmk Version

Web or Rest API

Checkmk Version

Web or Rest API

2.0.0p*

Web-API

2.1.0p*

Web-API

2.2.0p*

REST-API

Custom graph (Web-API)

With this example, you can query the metrics of a custom graph in Checkmk

root@linux:~$ curl -k "https://mylinuxhost/mysite/check_mk/webapi.py?action=get_graph&_username=automation&_secret=mysecret&request_format=python&output_format=python" -d 'request={"specification":["custom","custom_graph_1"],"data_range":{"time_range":[1615627389,1616062956]}}' {'result_code': 0, 'result': {'start_time': 1615626000, 'end_time': 1616063400, 'step': 1800, 'curves': [{'line_type': 'area', 'color': '#00b2ff', 'title': 'Service Costs per Day', 'rrddata': [None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None]}]}}

 

Forecast graph (Web-API)

With this example, you can query the metrics of a forecast graph in Checkmk.

root@linux:~$ curl -k "https://mylinuxhost/mysite/check_mk/webapi.py?action=get_graph&_username=automation&_secret=mysecret&request_format=python&output_format=python" -d 'request={"specification":["forecast","forecast_graph_4"],"data_range":{"time_range":[1616062956,1616513102]}}' {'result_code': 0, 'result': {'start_time': 1616504400, 'end_time': 1616527800, 'step': 1800, 'curves': [{'line_type': 'area', 'color': '#FFFFFF', 'title': 'CPU utilization - lower edge', 'rrddata': [33.345051190851265, 37.67683226938787, 44.50125333428577, 53.2724501059588, 60.44537606761632, 66.75387999685792, 72.64393851155472, 78.29311232060269, 83.75780489204897, 89.06739515972093, 94.1829539622409, 99.15563480780335, 104.09381440605844]}, {'line_type': 'stacked', 'color': '#F4E750', 'title': 'CPU utilization - confidence interval', 'rrddata': [11.100249458703228, 10.146064241405181, 10.102262795192633, 11.3949147901306, 16.27542796258677, 22.884785199874802, 30.331033266252426, 38.25905074392767, 46.55603069680629, 55.163215257233624, 64.15846274796492, 73.43946615261119, 82.78947205187224]}, {'line_type': 'line', 'color': '#0000FF', 'title': 'CPU utilization - original', 'rrddata': [42.2174, 40.4999, 47.7736, 60.8701, None, None, None, None, None, None, None, None, None]}, {'line_type': 'line', 'color': '#FF0000', 'title': 'CPU utilization - fit & forecast', 'rrddata': [38.89517592020288, 42.749864390090465, 49.552384731882086, 58.969907501024096, 68.58309004890971, 78.19627259679532, 87.80945514468094, 97.42263769256652, 107.03582024045211, 116.64900278833774, 126.26218533622335, 135.87536788410895, 145.48855043199455]}]}}

 

Builtin graph (Web-API)

With this example, you can query the metrics of a built-in graph in Checkmk. You will receive the data for all lines of a specific line.

root@linux:~$ curl -k "https://mylinuxhost/mysite/check_mk/webapi.py?action=get_graph&_username=automation&_secret=mysecret&request_format=python&output_format=python" -d 'request={"specification":["template",{"service_description":"Filesystem /","site":"mysite","graph_index":0,"host_name":"mysite"}],"data_range":{"time_range":[1615627389, 1616062956]}}' {'result_code': 0, 'result': {'start_time': 1615626000, 'end_time': 1616063400, 'step': 1800, 'curves': [{'line_type': 'area', 'color': '#00ffc6', 'title': 'Used filesystem space', 'rrddata': [None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None]}, {'line_type': 'stack', 'color': '#e3fff9', 'title': 'Free space', 'rrddata': [None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None]}, {'line_type': 'line', 'color': '#006040', 'title': 'Filesystem size', 'rrddata': [None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None,

 

RRDcached

If you want the values unfiltered, you can access these from the Round Robin Database (RRD) journal.log directly:

OMD[mysite]:~/var/rrdcached$ ls -ltr total 8744 -rw-r----- 1 mysite mysite 2435087 Mar 23 15:45 rrd.journal.1616503549.911643 -rw-r----- 1 mysite mysite 1720320 Mar 23 17:07 rrd.journal.1616510749.911407 -rw-r--r-- 1 mysite mysite 2456557 Mar 24 09:13 rrd.journal.1616566388.199299 -rw-r--r-- 1 mysite mysite 2322432 Mar 24 10:49 rrd.journal.1616573588.199726 OMD[mysite]:~/var/rrdcached$ tail -f rrd.journal.1616573588.199726 update /opt/omd/sites/mysite/var/check_mk/rrd/mysite/OMD_mysite_apache.rrd 1616579503:0:0:0:0:0:0:0:0:0.08098:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0.002154:19.176001:0:0:0:0:0:0:0:0:0:0:0 update /opt/omd/sites/mysite/var/check_mk/rrd/mysite/OMD_mysite_performance.rrd 1616579503:559:25:0:0.000009:0:364608.943527:0:0:0.583053:0:0.000008:0.000038:0:0.259135:3.09749:3.611691:5.571399:0.015021 update /opt/omd/sites/mysite/var/check_mk/rrd/mysite/OMD_dashboard_Notification_Spooler.rrd 1616579503:0:2560278 update /opt/omd/sites/mysite/var/check_mk/rrd/mysite/OMD_nagnis_master_Notification_Spooler.rrd 1616579503:0:63841:U:U update /opt/omd/sites/mysite/var/check_mk/rrd/mysite/OMD_nagnis_master_backup_test.rrd 1616579503:3377776640:128932341.39868:49.073688 update /opt/omd/sites/mysite/var/check_mk/rrd/mysite/OMD_nagvis_master_backup_test.rrd 1616579503:3377776640:128932341.39868:49.073688 update /opt/omd/sites/mysite/var/check_mk/rrd/mysite/Process_chrome.rrd 1616579503:3264916:543481580:7.288174:23 update /opt/omd/sites/mysite/var/check_mk/rrd/mysite/Site_mysite_statistics.rrd 1616579503:13:42:18:1:119:366:3:0:1:21 update /opt/omd/sites/mysite/var/check_mk/rrd/mysite/Systemd_Timesyncd_Time.rrd 1616579503:0.000024:0.000463

 

Sources

 

Related articles