Debugging of long running activate changes

Owned by Anastasios Thomaidis
Last updated: Oct 30, 2024 by Matthew Hierholzer

The article explains how to debug long-running activating changes.

LAST TESTED ON CHECKMK 2.3.0P1

Table of Contents

Problem

Activating Changes, specifically, a CMC reload, takes a long time.

Solution

Debug long-running activate changes on a single site

With these commands, you will check the /var/check_mk/web and /local directories. 

OMD[mysite]:~$ find ~/var/check_mk/web -follow -type f | wc -l
OMD[mysite]:~$ find ~/local -follow -type f | wc -l

If, for example, ~/var/check_mk/web has five-figure files, this can be problematic.

A big ~/local directory could also cause a problem, as Checkmk creates a .tar file every time we activate changes.

Debug long-running activate changes in a distributed setup

Please run this command on the central and remote sites to see if the local files are the same:

OMD[mysite]:~$ find ~/local -follow -type f | wc -l


It's also helpful to search for ASCII filenames or files containing ASCII characters:

OMD[mysite]:~/local$ find . -type f | xargs ls -ltr |grep --color='auto' -P -n "[^\x00-\x7F]"^C
OMD[mysite]:~/local$ find . -type f | grep --color='auto' -P -n "[^\x00-\x7F]"


Advanced debugging

If the above suggestions do not point you to the solution, you can use the profiling mechanism to profile Activate changes.

BTW: Every Checkmk command can be profiled. The whole syntax of the profile command is described here: Enable Checkmk profiling#Profilingviacli.


Let's run the following command on the central and remote sites to create a profile file of cmk -O. cmk -O  is doing a configuration reload, which is part of a typical activation process.

OMD[mysite]:~$ cmk -O --profile --debug -vv &>activation_debug.log

Now three files will be created:

  • show_profile.py
  • profile.out
  • activation_debug.log

If you want to debug the profile files alone, please follow the manual Enable Checkmk profiling#GUIProfiling otherwise, please open a support case and include these three files.