Versions Compared

Version Old Version 5 New Version 6
Changes made by

Matthew Hierholzer

Matthew Hierholzer

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.


Info

This article helps debug issues with various Checkmk special agents.

...

  1. The first step would be to find the complete command of the Kubernetes special agent.
     

    1. The command can be found under "Type of agent >> Program." It will consist of multiple parameters depending on how the datasource program rule has been configured. 

      Code Block
      languagebash
      themeRDark
      OMD[mysite]:~$ cmk -D k8s | more

k8s 
Addresses: No IP
Tags: [address_family:no-ip], [agent:special-agents], [criticality:prod], [networking:lan],
[piggyback:auto-piggyback], [site:a21], [snmp_ds:no-snmp], [tcp:tcp]
Labels: [cmk/kubernetes/cluster:at], [cmk/kubernetes/object:cluster], [cmk/site:k8s]
Host groups: check_mk
Contact groups: all
Agent mode: No Checkmk agent, all configured special agents
Type of agent: 
Program: /omd/sites/mysite/share/check_mk/agents/special/agent_kube '--cluster' 'k8s' '--token' 'xyz' '--monitored-objects' 'deployments' 'daemonsets' 'statefulsets' 'nodes' 'pods' '--api-server-endpoint' 'https://<YOUR-IP>:6443' '--api-server-proxy' 'FROM_ENVIRONMENT' '--cluster-collector-endpoint' 'https://<YOUR-ENDPOINT>:30035' '--cluster-collector-proxy' 'FROM_ENVIRONMENT'
Process piggyback data from /omd/sites/mysite/tmp/check_mk/piggyback/k8s
Services:
...


      Note

      An easier way would be this command: /bin/sh -c "$(cmk -D k8s | grep -A1 "^Type of agent:" | grep "Program:" | cut -f2- -d':')"

      Please note that if a line matching "^Type of agent:" followed by a line matching "^  Program:" exists more than once, the output might be messed up.

      .

    2. The special agent has the below options available for debugging purposes:

      Code Block
      languagebash
      themeRDark
      OMD[mysite]:~$ /omd/sites/mysite/share/check_mk/agents/special/agent_kube -h
...
--debug                     Debug mode: raise Python exceptions
-v / --verbose 				Verbose mode (for even more output use -vvv)
--vcrtrace FILENAME         Enables VCR tracing for the API calls
...

      .

    3. Now, you can modify the above  command of the Kubernetes special agent like this:

      Code Block
      languagebash
      themeRDark
      OMD[mysite]:~$ /omd/sites/mysite/share/check_mk/agents/special/agent_kube  \
'--cluster' 'at' \
'--token' 'xyz' \
'--monitored-objects' 'deployments' 'daemonsets' 'statefulsets' 'nodes' 'pods' \
'--api-server-endpoint' 'https://<YOUR-IP>:6443' \
'--api-server-proxy' 'FROM_ENVIRONMENT' \
'--cluster-collector-endpoint' 'https://<YOUR-ENDPOINT>:30035' \
'--cluster-collector-proxy' 'FROM_ENVIRONMENT' \
--debug -vvv --vcrtrace ~/tmp/vcrtrace.txt > ~/tmp/k8s_with_debug.txt 2>&1

      Here, you can also reduce the number of '--monitored-objects' to a few resources to get less output. 
      .

    4. Run the special agent with no debug options to create an agent output, or you could download it from the cluster host via the Checkmk web interface. 

      Code Block
      languagebash
      themeRDark
      /omd/sites/mysite/share/check_mk/agents/special/agent_kube '--cluster' 'at' '--token' 'xyz' '--monitored
-objects' 'deployments' 'daemonsets' 'statefulsets' 'nodes' 'pods' '--api-server-endpoint' 'https://<YOUR-IP>:6443' '--api-server-proxy' 'FROM_ENVIRONMENT' '--cluster-collector-endpoint' 'https://<YOUR-ENDPOINT>:30035' '--cluster-collector-proxy' 'FROM_ENVIRONMENT' > ~/tmp/k8s_agent_output.txt 2>&1

      .

  2. Please upload the following files to the support ticket.

...

  1. Example with Special Agent of storeonce4x

    1. Find out the detailed special agent command (Type of agent column)

      Code Block
      languagebash
      themeRDark
      OMD[mysite]:~$ cmk -D hostname


      Note

      an easier way would be this command: /bin/sh -c "$(cmk -D k8s | grep -A1 "^Type of agent:" | grep "Program:" | cut -f2 -d':')"

      Please note that if a line matching "^Type of agent:" followed by a line matching "^  Program:" exists more than once, then the output might be messed up.

      .

    2. Check if there are some options for debugging

      Code Block
      languagebash
      themeRDark
      OMD[mysite]:~$ ~/share/check_mk/agents/special/agent_storeonce4x -h



      There are three options for debugging the request:

      Code Block
      languagebash
      themeRDark
      --debug, -d           Enable debug mode (keep some exceptions unhandled)
--verbose, -v
--vcrtrace TRACEFILE, --tracefile TRACEFILE
                            If this flag is set to a TRACEFILE that does not exist yet, it will be created and
                            all requests the program sends and their corresponding answers will be recorded in said file.
                            If the file already exists, no requests are sent to the server, but the responses will be
                            replayed from the tracefile.

      .

    3. Modify the special agent command by adding these three options

      Code Block
      languagebash
      themeRDark
      OMD[mysite]:~$ ~/share/check_mk/agents/special/agent_storeonce4x <OTHER ARGUMENTS> --debug -v --vcrtrace ~/tmp/vcrtrace.txt 2>1 ~/tmp/storeonce4x_with_debug.txt

      .

    4. Run the special agent with no debug options to create an agent output. With this file, we can reproduce your issue

      Code Block
      languagebash
      themeRDark
      OMD[mysite]:~$ /omd/sites/mysite/share/check_mk/agents/special/agent_kube <OTHER ARGUMENTS> > ~/tmp/k8s_agent_output.txt



  2. Rename the token file

    The storeonce4x special agent is using username/password for authentication. After the successful login, we obtain the access token. The access token is used for future REST requests.

    If you want to read more, you can check this out: https://hewlettpackard.github.io/storeonce-rest/#Authentication

    1. We save the token file inside the site in

      Code Block
      languagebash
      themeRDark
      ~/tmp/check_mk/special_agents/agent_storeonce4x/<hostname>_oAuthToken.json

      .

    2. Rename the file to _oAuthToken.json.back

      Code Block
      languagebash
      themeRDark
      OMD[mysite]~# mv ~/tmp/check_mk/special_agents/agent_storeonce4x/<hostname>_oAuthToken.json ~/tmp/check_mk/special_agents/agent_storeonce4x/<hostname>_oAuthToken.json.back

      .

    3. Run the special agent again

...

Info

Although Containers and their management with Kubernetes took the IT industry by storm, virtualization still has its "right to exist" in on-prem environments and everywhere where containerization would not fit.

This is an extension to https://docs.checkmk.com/latest/en/monitoring_vmware.htmlMonitoring VMware ESXi. 

Getting Started

Background information regarding this subject is available in our Official documentation

...

  1. Example with Special Agent of vSphere
    .

    1. Find out the detailed special agent command

      Code Block
      languagebash
      themeRDark
      OMD[mysite]:~$ cmk -D <vcenter-host> | more

vcenter 
Addresses: x.x.x.x
Tags: [add_ip_addresses:add_ip_addresses_1], [address_family:ip-v4-only], [agent:special-agents], [criticality:prod], 
[ip-v4:ip-v4], [networking:lan], [piggyback:auto-piggyback], [site:nagnis_master], [snmp_ds:no-snmp], [tcp:tcp]
Labels: [cmk/vsphere_object:vm]
Host groups: check_mk
Contact groups: all
Agent mode: No Checkmk agent, all configured special agents
Type of agent: 
Program: /omd/sites/mysite/share/check_mk/agents/special/agent_vsphere -u 'user' -s 'password' -i hos
tsystem,virtualmachine,datastore,counters,licenses -P --spaces cut --snapshot_display vCenter --no-cert-check 'x.x.x.x'
Process piggyback data from /omd/sites/mysite/tmp/check_mk/piggyback/vcenter
Services:
checktype item params



      Note

      An easier way would be this command: /bin/sh -c "$(cmk -D vcenter | grep -A1 "^Type of agent:" | grep "^ Program:" | cut -f2 -d':')"

      Please note that if a line matching "^Type of agent:" followed by a line matching "^  Program:" exists more than once, the output might be messed up.

      .

    2. Check if there are options for debugging.

      Code Block
      languagebash
      themeRDark
      OMD[mysite]:~$ /omd/sites/mysite/share/check_mk/agents/special/agent_vsphere -h


      There are two options for debugging the request.

      Code Block
      languagebash
      themeRDark
      --debug                       Debug mode: let Python exceptions come through

--tracefile FILENAME          Log all outgoing and incoming data into the given tracefile

      .

    3. Modify the special agent command by adding these two options

      Code Block
      languagebash
      themeRDark
      OMD[mysite]:~$ /omd/sites/mysite/share/check_mk/agents/special/agent_vsphere  -u 'user' -s 'password' --debug --tracefile $OMD_ROOT/tmp/vcenter.out -i hostsystem,virtualmachine,datastore,counters,licenses -P --spaces cut --no-cert-check '$HOST_ADDRESS' > $OMD_ROOT/tmp/vcenter.debug

      In CMK 1.6.0, you might find the option "--snapshot_display vCenter" in your CMK -D output. If that's the case, you can include this parameter.

      .

    4. Run the special agent with no debug options to create an agent output. With this file, we can reproduce your issue.

      Code Block
      languagebash
      themeRDark
      root@linux~# /omd/sites/mysite/share/check_mk/agents/special/agent_vsphere -u 'user' -s 'password' -i hostsystem,virtualmachine,datastore,counters,licenses -P --spaces cut --no-cert-check 'x.x.x.x' >/~tmp/agent.output

      .

  2. Please send us all three files. Now we're able to investigate further.

    1
    2
    3

    ~/tmp/vcenter.debug      # Debug Output
    ~/tmp/vcenter.out        # Tracefile
    /~tmp/agent.output       # Agent Output


...