Debugging the Checkmk Micro Core (CMC) old

This manual will show you a few tools for debugging the CMC core if it's crashing.

LAST TESTED ON CHECKMK 2.3.0P1

Table of Contents

Before you delve into low-level debugging of why the CMC is running but not working (without a stack trace), please check the "Master Control" snap-in in the sidebar first!

If the Service Checks and Host Checks are disabled, that might be the reason for your problem.

Image: Screenshot of the Master Control with both service checks and host checks disabled.

Analyze CMC core

strace

You can use strace to track the CMC process when you face any issue:

root@linux~# strace -o cmc-strace.log -p $(cat ~<MYSITE>/tmp/run/cmc.pid)

valgrind

You can use valgrind to start the CMC in the debug mode. Here you will get a full stack trace. If valgrind is unavailable on your system, install it or run the CMC only with the -g option.

root@linux~# su mysite
OMD[mysite]:~$ omd stop cmc
OMD[mysite]:~$ valgrind --num-callers=30 cmc -g

or   

root@linux~# su mysite
OMD[mysite]:~$ omd stop cmc
OMD[mysite]:~$ cmc -g

gdb

With gdb, you can analyze the coredump if checkmk will create one. Note: Checkmk will only create one if you enable it in the global settings.

With the -r option, you can re-run the CMC to analyze inside gdb.

root@linux:~# gdb /omd/sites/mysite/bin/cmc --core=<PATH/TO/COREUMP>
(gdb) r

frozen CMC

When the CMC seems to freeze and nothing happens, please run this command before restarting the CMC:

Or to write that to a file:

Another option to collect more traces would be to run gdb in a loop (5 minutes)and write the output in a file:

Analyze coredump file

By default, there is no coredump creation enabled. You can enable that via Setup → Global settings → Monitoring core → Enable core dumps

After a crash of the CMC, a coredump in ~/var/check_mk/core/ will be written

Image: Enable core dumps disabled.

gdb

With gdb, you can analyze the coredump if checkmk will create one. Note: Checkmk will only create one if you enable it in the global settings.

Enable log within gdb

objdump

With objdump, you can fetch the content of the dump.

file command

With the file command, you can also fetch the content of the dump.

Open a support case

If your investigation is not successful, please open a ticket and provide us with the following data:

Please send us the following data to help us reproduce the issue.

Checkmk Knowledge Base