How to collect troubleshooting data for various issue types

This article details how to collect information and logs for troubleshooting purposes.

LAST TESTED ON CHECKMK 2.1.0P1

Table of Contents

Acquiring Host Reports

Collect the following two PDF reports for affected hosts:

Collecting Agent Data

Collecting Agent Output

Via the command line

As site User:
OMD[workshop]:~$ cmk -d HOSTNAME >/path/to/file.txt

or on the affected system:
root@mylinuxhost:~# check_mk_agent >/path/to/file.txt

use the option -d to see what the agent is doing in detail:
root@mylinuxhost:~# check_mk_agent -d &>/path/to/file.txt

Via the GUI

Click on the Hamburger menu
Click Download agent output

Screenshot of the hamburger menu and the Download agent output location

Collecting a series of Agent Outputs

When counter-based checks are involved, one agent output is insufficient, as the delta or the "per minute" value must be calculated.

You can collect a series of 10 agent outputs using the following code snippet:

# As site user:

# Fill in your hostname here:
host="myhost"

t=60 # Check interval in, default is 60s
s=0
while [ $s -lt 600 ]; do
    echo $s
    cmk -d $host > /tmp/agent_output_series_$s
    let s=$s+$t
    sleep $t
done
tar -cvzf /tmp/agent_output_series.tar.gz /tmp/agent_output_series_off_*

In addition to that, you can also How-to temporarily enable debugging in the Linux agent

Debugging Special Agents

Please see our Knowledge Base manual: Special Agents

SNMP Walk

Via GUI

Click on the Hamburger menu
Click Download SNMP Walk

Screenshot of the hamburger menu and the Download SNMP walk location

Via CLI

If you run the below command as site-user

OMD[mysite]:~$cmk --snmpwalk <your_snmp_host>

a file will be created under the following path:

OMD[mysite]:~$OMD_ROOT/var/check_mk/snmpwalks/<your_snmp_host>

or directly with the SNMP command

for snmp v3:

root@mylinuxhost:~# snmpwalk -v3 -l authPriv -u <SNMP-User> -a SHA -A <SNMP-Password> -x DES -X <privKey> -Onqet <HOSTNAME or IP Address> .1.3.6.1 >/tmp/snmpwalk.txt

for snmpv2 :

root@mylinuxhost:~# snmpwalk -v 2c -c <community> -Onqet <HOSTNAME or IP Address> .1.3.6.1 >/tmp/snmpwalk-v2.txt

If your community has a special character, you need to escape this with a backslash:

snmpwalk  -v2c -c 123\$asddk  -Onqet 192.168.20.5 .1.3.6.1 > /tmp/snmpwalk-v2.txt

Increase Log level

Please don't forget to decrease the log level after you submit the file to us.
Otherwise, the log file will become huge, and you may run out of space.

Core

Setup → General → Global Settings → Monitoring Core → Logging of the core

Alerts
Carbon connections
Core
Downtime
Checkmk helpers
Livestatus
Notification system
RRD processing
Smartping

Setup → General → Global Settings → Monitoring Core → Logging of the notification mechanics

Setup → General → Global Settings → Monitoring Core → Logging of RRD creation

Setup → General → Global Settings → Monitoring Core → Debugging of Checkmk helpers