/
Debugging of long-running activate changes

Debugging of long-running activate changes

The article explains how to debug long-running activating changes.

LAST TESTED ON CHECKMK 2.3.0P1

Table of Contents

Problem

Activating Changes, specifically, a CMC reload, takes a long time.

Solution

Debug long-running activate changes on a single site

With these commands, you will check the /var/check_mk/web and /local directories. 

OMD[mysite]:~$ find ~/var/check_mk/web -follow -type f | wc -l OMD[mysite]:~$ find ~/local -follow -type f | wc -l

If, for example, ~/var/check_mk/web has five-figure files, this can be problematic.

A big ~/local directory could also cause a problem, as Checkmk creates a .tar file every time we activate changes.

 

Debug long-running activate changes in a distributed setup

Please run this command on the central and remote sites to see if the local files are the same:

OMD[mysite]:~$ find ~/local -follow -type f | wc -l

 

It's also helpful to search for ASCII filenames or files containing ASCII characters:

OMD[mysite]:~/local$ find . -type f | xargs ls -ltr |grep --color='auto' -P -n "[^\x00-\x7F]"^C OMD[mysite]:~/local$ find . -type f | grep --color='auto' -P -n "[^\x00-\x7F]"

 

Advanced debugging

If the above suggestions do not point you to the solution, you can use the profiling mechanism to profile Activate changes.

BTW: Every Checkmk command can be profiled. The whole syntax of the profile command is described here: Profiling via CLI.

 

Let's run the following command on the central and remote sites to create a profile file of cmk -O. cmk -O is doing a configuration reload, which is part of a typical activation process.

OMD[mysite]:~$ cmk -O --profile --debug -vv &>activation_debug.log

Now three files will be created:

  • show_profile.py

  • profile.out

  • activation_debug.log

If you want to debug the profile files alone, please follow the manual GUI profiling otherwise, please open a support case and include these three files.

Related articles