This article details how to collect information and logs for troubleshooting purposes. 



Acquiring Host Reports

Collect the following two PDF reports for affected hosts:

Screenshot of location for reports of host and performance graphs

Collecting Agent Data

Collecting Agent Output

Via the command line

As site User:
OMD[workshop]:~$ cmk -d HOSTNAME >/path/to/file.txt

or on the affected system:
root@mylinuxhost:~# check_mk_agent >/path/to/file.txt

use the option -d to see what the agent is doing in detail:
root@mylinuxhost:~# check_mk_agent -d &>/path/to/file.txt


Via the GUI

  1. Click on the Hamburger menu

  2. Click Download agent output

Screenshot of the hamburger menu and the Download agent output location

Collecting a series of Agent Outputs

When counter-based checks are involved, one agent output is insufficient, as the delta or the "per minute" value must be calculated.

You can collect a series of 10 agent outputs using the following code snippet:

# As site user:

# Fill in your hostname here:
host="myhost"

t=60 # Check interval in, default is 60s
s=0
while [ $s -lt 600 ]; do
    echo $s
    cmk -d $host > /tmp/agent_output_series_$s
    let s=$s+$t
    sleep $t
done
tar -cvzf /tmp/agent_output_series.tar.gz /tmp/agent_output_series_off_*

In addition to that, you can also How-to temporarily enable debugging in the Linux agent

Debugging Special Agents

Please see our Knowledge Base manual: Special Agents

SNMP Walk

Via GUI

  1. Click on the Hamburger menu

  2. Click Download SNMP Walk

Screenshot of the hamburger menu and the Download SNMP walk location

Via CLI

If you run the below command as site-user 

OMD[mysite]:~$cmk --snmpwalk <your_snmp_host>

a file will be created under the following path:

OMD[mysite]:~$OMD_ROOT/var/check_mk/snmpwalks/<your_snmp_host>


or directly with the SNMP command

for snmp v3:

root@mylinuxhost:~# snmpwalk -v3 -l authPriv -u <SNMP-User> -a SHA -A <SNMP-Password> -x DES -X <privKey> -Onqet <HOSTNAME or IP Address> .1.3.6.1 >/tmp/snmpwalk.txt


for snmpv2 :

root@mylinuxhost:~# snmpwalk -v 2c -c <community> -Onqet <HOSTNAME or IP Address> .1.3.6.1 >/tmp/snmpwalk-v2.txt



If your community has a special character, you need to escape this with a backslash:


snmpwalk  -v2c -c 123\$asddk  -Onqet 192.168.20.5 .1.3.6.1 > /tmp/snmpwalk-v2.txt

Increase Log level

Please don't forget to decrease the log level after you submit the file to us.
Otherwise, the log file will become huge, and you may run out of space.


Core 

Setup → General → Global Settings → Monitoring Core → Logging of the core

  • Alerts
  • Carbon connections
  • Core
  • Downtime
  • Checkmk helpers
  • Livestatus
  • Notification system
  • RRD processing
  • Smartping

Setup → General → Global Settings → Monitoring Core → Logging of the notification mechanics

Setup → General → Global Settings → Monitoring Core → Logging of RRD creation

Setup → General → Global Settings → Monitoring Core → Debugging of Checkmk helpers


The log file will then be available for the site user at:

  • ~/var/log/cmc.log

Livestatus Proxy

Setup → General → Global Settings → Livestatus Proxy → Livestatus Proxy Logging


The log file will then be available for the site user at:

  • ~/var/log/liveproxyd.log
  • ~/var/log/liveproxyd.state

DCD

Setup → General → Global Settings → Dynamic Configuration → Dynamic configuration logging 


The log file will then be available for the site user at:

  • ~/var/log/dcd.log


Alert Handler

Setup → General → Global Settings → Alert Handlers → Alert Handler log level 


The log file will then be available for the site user at:

  • ~/var/log/alerts.log

Notification

Setup → General → Global Settings → Notifications → Notification log level

Setup → General → Global Settings → Notifications → Notification Spooler Configuration → Verbosity of logging


The log file will then be available for the site user at:

  • ~/var/log/notify.log
  • ~/var/log/mknotifyd.log

User Interface

Setup → General → Global Settings → User Interface → Logging 

  • Web
  • Authentication
  • LDAP
  • BI compilation
  • Automation calls
  • Background jobs
  • Slow views
  • Agent Registration


The log file will then be available for the site user at:

  • ~/var/log/web.log


Periodic service discovery rules

Do the following to find active periodic service discovery rules:

  1. Click on Monitor and then enter the host you're looking for within the search field

    Image showing the location of host within Monitoring section of Checkmk


  2. Click on the hostname at the top of the table

    Image showing location of a host within the Services page


  3. Select Effective parameters from the host menu

    Image showing location for the Effective parameters option for a host


  4. Scroll down to the Service Discovery rules section and then click on Periodic Service Discovery

    Image showing location of the Periodic service discovery rule


  5. Here you will find your Periodic service discovery rules



Collect logs from Checkmk Appliance

Log in to the Webconf of the Checkmk Appliance and select "View Log Files" in the Main Menu.

Screenshot of the checkmk appliance and the view log file location


Now you can select the log file and Download the file:

Screenshot of the download button and where to select which log


Get the OMD performance report

Log on to Checkmk UI and navigate to the server Checkmk is running on. If the agent is installed and rulesets are correctly configured, you'll get several services named  "OMD <sitename>."

Locate the service "OMD <sitename> performance" and click on it:

Screenshot of the omd home performance service


On the next page, please scroll down and click on a graph that fits the time best (we usually ask for the last 8 or 35 days)

goto "Export → This view as PDF"

Screenshot of view as pdf location

Related articles

Related articles appear here based on the labels you select. Click to edit the macro and add or change labels.



Related issues