Debugging of long running activate changes

LAST TESTED ON CHECKMK 2.2.0P1

Problem

Activating Changes, specifically, a CMC reload, takes a long time.

Solution

Debug long-running activate changes on a single site

With these commands, you will check the /var/check_mk/web and /local directories. 

OMD[mysite]:~$ find ~/var/check_mk/web -follow -type f | wc -l
OMD[mysite]:~$find ~/local -follow -type f | wc -l

Debug long-running activate changes in a distributed setup

Please run this command on the master and remote sites to see if the local files are the same:

OMD[mysite]:~$ find ~/local -follow -type f | wc -l

It's also helpful to search for ASCII filenames or files containing ASCII characters:

OMD[mysite]:~/local$ find . -type f | xargs ls -ltr |grep --color='auto' -P -n "[^\x00-\x7F]"^C
OMD[mysite]:~/local$ find . -type f | grep --color='auto' -P -n "[^\x00-\x7F]"

Advanced debugging

If the above suggestions do not point you to the solution, you can use the profiling mechanism to profile Activate changes.

Let's run the following command on the master and remote sites to create a profile file of cmk -O. cmk -O  is doing a configuration reload, which is part of a typical activation process.

OMD[mysite]:~$ cmk -O --profile --debug -vv &>activation_debug.log

Now three files will be created:

• show_profile.py
• profile.out
• activation_debug.log

If you want to debug the profile files alone, please follow the manual Checkmk profiling otherwise, please open a support case and include these three files.

Related articles