Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.


Info

This article details how to debug performance issues involving Livestatus.

...

This command will show you some statistics (run it in /omd/sites/<SITENAME>/var/log):

Code Block
languagebash
themeRDark
OMD[mysite]~# grep "processed request in" $OMD_ROOT/var/log/cmc.log | cut -d" " -f9 | sed 's,$, 100 / 100 * p,' | dc | sort -n | uniq -c

...

Code Block
languagebash
themeRDark
OMD[mysite]~# cat $OMD_ROOT/var/log/cmc.log |grep -Po "client.*processed request in [\d]* ms"  |sort -n |uniq -c

...

Code Block
languagebash
themeRDark
OMD[mysite]~# grep -Eo "request: GET \w+" $OMD_ROOT/var/log/cmc.log | uniq |sort -u
request: GET eventconsoleevents
request: GET hosts
request: GET log
request: GET services
request: GET status
request: GET timeperiods

...

Code Block
languagebash
themeRDark
OMD[mysite]~# cat $OMD_ROOT/var/log/cmc.log |grep "client 22" |grep "request"

...

In Setup → General → Global settings we have several settings to improve the log parsing.

Screenshot of global settings with max concurrent livestatus connections set to 20. Max number of cached log messages set to 500000. History log rotation size limit set to 50 MiB.  Max number of parsed lines per log file set to 1000000.




Maximum concurrent Livestatus connections UsuallyTypically, the default value should be acceptable. If you have a larger number of users, views, or distributed monitoring, you can increase this value step-by-step (50 - 100 - 150)

Maximum number of cached log messages

To speed up queries for historical data, the core keeps an in-memory cache of log file messages. This number can be configured here. A larger number needs requires more RAM. Note: even if you set this to 0, there might be some cases where messages need to be cached anyway.

You can set this to one million if you have enough memory

History log rotation: Rotate by size (Limit of the size)A log file rotation will be forced whenever its size exceeds that limit. In a large environment, you can increase this value to, e.g., 200mb. Checkmk will now need to parse the same amount of data through fewer files.
Maximum number of parsed lines per log file

To avoid large timeouts in the case of oversized history log files, the core limits the number of lines read from history log files. The limit is on a per-file base basis and can be configured here. Exceeding lines are ignored, and an error is logged in the CMC daemon log file.


E.g.,:

Code Block
languagebash
themeRDark
2021-12-03 09:30:30 [3] [client 1] more than 500000 lines in "/omd/sites/cmk$OMD_SITE/var/check_mk/core/history", ignoring the rest! 
2021-12-03 09:30:37 [3] [client 1] more than 500000 lines in "/omd/sites/cmk$OMD_SITE/var/check_mk/core/history", ignoring the rest! 
2021-12-03 09:30:42 [3] [client 2] more than 500000 lines in "/omd/sites/cmk$OMD_SITE/var/check_mk/core/history", ignoring the rest! 
2021-12-03 09:30:42 [3] [client 1] more than 500000 lines in "/omd/sites/cmk$OMD_SITE/var/check_mk/core/history", ignoring the rest! 


...