vSphere debugging

Owned by Former user (Deleted)
Last updated: Dec 13, 2023 by Matthew Hierholzer
2 min read

This article explains how to debug special agents used by vSphere.

LAST TESTED ON CHECKMK 2.0.0P1

Table of Contents

Basic Debugging

  1. Example with Special Agent of vSphere
    .

    1. Find out the detailed special agent command

      OMD[mysite]:~$ cmk -D <vcenter-host> | more

vcenter 
Addresses: x.x.x.x
Tags: [add_ip_addresses:add_ip_addresses_1], [address_family:ip-v4-only], [agent:special-agents], [criticality:prod], 
[ip-v4:ip-v4], [networking:lan], [piggyback:auto-piggyback], [site:nagnis_master], [snmp_ds:no-snmp], [tcp:tcp]
Labels: [cmk/vsphere_object:vm]
Host groups: check_mk
Contact groups: all
Agent mode: No Checkmk agent, all configured special agents
Type of agent: 
Program: /omd/sites/mysite/share/check_mk/agents/special/agent_vsphere -u 'user' -s 'password' -i hos
tsystem,virtualmachine,datastore,counters,licenses -P --spaces cut --snapshot_display vCenter --no-cert-check 'x.x.x.x'
Process piggyback data from /omd/sites/mysite/tmp/check_mk/piggyback/vcenter
Services:
checktype item params

      An easier way would be this command: /bin/sh -c "$(cmk -D vcenter | grep -A1 "^Type of agent:" | grep "^ Program:" | cut -f2 -d':')"

      Please note that if a line matching "^Type of agent:" followed by a line matching "^  Program:" exists more than once, the output might be messed up.

      .

    2. Check if there are options for debugging.

      OMD[mysite]:~$ /omd/sites/mysite/share/check_mk/agents/special/agent_vsphere -h

      There are two options for debugging the request.

      --debug                       Debug mode: let Python exceptions come through

--tracefile FILENAME          Log all outgoing and incoming data into the given tracefile

      .

    3. Modify the special agent command by adding these two options

      OMD[mysite]:~$ /omd/sites/mysite/share/check_mk/agents/special/agent_vsphere  -u 'user' -s 'password' --debug --tracefile $OMD_ROOT/tmp/vcenter.out -i hostsystem,virtualmachine,datastore,counters,licenses -P --spaces cut --no-cert-check '$HOST_ADDRESS' > $OMD_ROOT/tmp/vcenter.debug

      In CMK 1.6.0, you might find the option "--snapshot_display vCenter" in your CMK -D output. If that's the case, you can include this parameter.

      .

    4. Run the special agent with no debug options to create an agent output. With this file, we can reproduce your issue.

      root@linux~# /omd/sites/mysite/share/check_mk/agents/special/agent_vsphere -u 'user' -s 'password' -i hostsystem,virtualmachine,datastore,counters,licenses -P --spaces cut --no-cert-check 'x.x.x.x' >/~tmp/agent.output

      .

  2. Please send us all three files. Now we're able to investigate further

    1
    2
    3

    ~/tmp/vcenter.debug      # Debug Output
    ~/tmp/vcenter.out        # Tracefile
    /~tmp/agent.output       # Agent Output

Advanced Debugging Examples

Collect several agent outputs over a period of time:

export t=60; export s=0; while [ $s -le 600 ]; do echo $s; cmk -d $VSPHERE_HOST > /tmp/agent_vsphere_output.$s; let s=$s+$t; sleep $t; done


Collect several trace files over a period of time:

export t=60; export s=0; while [ $s -le 600 ]; do echo $s; ./agent_vsphere --trace /tmp/agent_vsphere_trace.$s $OTHER_COMMAND_PARAMS; let s=$s+$t; sleep $t; done