Problem

Activating changes (in particular a CMC reload) takes an unusually long time, does not complete, or appears to hang.

This article describes advanced troubleshooting and debugging techniques for Activate Changes and is intended for cases where standard checks do not identify the cause.

Solution

Debug long-running activate changes on a single site

Long activation times are often caused by a large number of files that must be processed during activation.

Check the number of files in the relevant directories:

A very large number of files in ~/var/check_mk/web can significantly increase activation time.

A large ~/local directory can also affect performance, as Checkmk creates a .tar archive from this directory during each activation.

Debug long-running activate changes in a distributed setup

In distributed environments, discrepancies between the central site and remote sites can cause long or incomplete activations.

Run the following command on the central site and all remote sites and compare the results:

Additionally, check for non-ASCII filenames or file contents, which may lead to unexpected behavior:

Advanced debugging

If the checks above do not identify the root cause, advanced debugging techniques are required.

This section covers profiling and strace-based debugging, which are useful when Activate Changes:

• Takes a very long time

• Appears to hang

• Is blocked by long-running background processes (for example, CMC or Apache)

Profiling Activate Changes

Before continuing, disable parallel core configuration generation:

1. Go to Setup → General → Global settings

2. Edit the global settings
   
   

3. Remove the checkbox for “Generate core config parallelized”

This step is required to produce a usable and consistent profile.

Open

Create a profiling output

Run the following command on the central site and all remote sites:

The cmk -O command performs a configuration reload, which is part of the Activate Changes process.

The following files are generated:

• show_profile.py

• profile.out

• activation_debug.log

To analyze the profile data manually, follow the GUI profiling documentation. 

Otherwise, open a support case and include all three files.

Low-level debugging with strace

If profiling does not reveal the cause, strace can be used to trace filesystem access during Activate Changes.

This approach is particularly useful for scenarios such as:

• Another activation process is currently in progress or locked

• CMC reloads that do not complete

• Apache-related delays during activation

To improve trace readability, the affected binary is started directly under strace. This method:

• Follows child processes automatically

• Creates separate trace files per process

• Produces clearer output than attaching to a running process

All commands must be executed as the site user.

Tracing the CMC process

Tracing the Apache process

Tracing Apache can help identify web server–related delays during Activate Changes.

The resulting trace files can help identify:

• Files or directories repeatedly accessed

• Lock files or sockets blocking progress

• Unexpected filesystem locations involved in activation

For step-by-step troubleshooting scenarios involving locked or blocked activations, refer to the “Cannot activate changes – another activation process is currently in progress or locked” article.

Related articles