Page Comparison

...

/omd/sites/mysite/share/check_mk/agents/special/agent_aws --access-key-id MYACCESSKEYID --secret-access-key MYSECRETEACCESSKEY --regions MYAWSREGION --global-services ce cloudfront route53 --services cloudwatch_alarms dynamodb ebs ec2 ecs elasticache elb elbv2 glacier lambda rds s3 sns wafv2 --ec2-limits --ebs-limits --s3-limits --glacier-limits --elb-limits --elbv2-limits --rds-limits --cloudwatch_alarms-limits --dynamodb-limits --wafv2-limits --lambda-limits --sns-limits --ecs-limits --elasticache-limits --s3-requests --cloudwatch-alarms --wafv2-cloudfront --cloudfront-host-assignment aws_host --hostname aws --piggyback-naming-convention ip_region_instance

Additional debugging

/omd/sites/mysite/share/check_mk/agents/special/agent_aws --access-key-id MYACCESSKEYID --secret-access-key MYSECRETEACCESSKEY --regions MYAWSREGION --services ecs --ecs-limits --hostname MYHOSTNAME --piggyback-naming-convention ip_region_instance --debug --verbose --vcrtrace trace.txt &> debug.log 

Two files should be created on the current active directory, "trace.txt" and "debug.log". Please verify these files were populated and send them to us.

Viewing AWS Options

To view more AWS options, use the cmk -D aws combined with grep

...

1. The first step would be to find the complete command of the Kubernetes special agent.
    

   1. The command can be found under "Type of agent >> Program." It will consist of multiple parameters depending on how the datasource program rule has been configured. 

      Code Block
      language bash theme RDark
      OMD[mysite]:~$ cmk -D k8s | more k8s Addresses: No IP Tags: [address_family:no-ip], [agent:special-agents], [criticality:prod], [networking:lan], [piggyback:auto-piggyback], [site:a21], [snmp_ds:no-snmp], [tcp:tcp] Labels: [cmk/kubernetes/cluster:at], [cmk/kubernetes/object:cluster], [cmk/site:k8s] Host groups: check_mk Contact groups: all Agent mode: No Checkmk agent, all configured special agents Type of agent: Program: /omd/sites/mysite/share/check_mk/agents/special/agent_kube '--cluster' 'k8s' '--token' 'xyz' '--monitored-objects' 'deployments' 'daemonsets' 'statefulsets' 'nodes' 'pods' '--api-server-endpoint' 'https://<YOUR-IP>:6443' '--api-server-proxy' 'FROM_ENVIRONMENT' '--cluster-collector-endpoint' 'https://<YOUR-ENDPOINT>:30035' '--cluster-collector-proxy' 'FROM_ENVIRONMENT' Process piggyback data from /omd/sites/mysite/tmp/check_mk/piggyback/k8s Services: ...

      
      

      Note
      An easier way would be this command: /bin/sh -c "$(cmk -D k8s | grep -A1 "^Type of agent:" | grep "Program:" | cut -f2- -d':')" Please note that if a line matching "^Type of agent:" followed by a line matching "^  Program:" exists more than once, the output might be messed up.

      .

   2. The special agent has the below options available for debugging purposes:

      Code Block
      language bash theme RDark
      OMD[mysite]:~$ /omd/sites/mysite/share/check_mk/agents/special/agent_kube -h ... --debug Debug mode: raise Python exceptions -v / --verbose Verbose mode (for even more output use -vvv) --vcrtrace FILENAME Enables VCR tracing for the API calls ...

      .

   3. Now, you can modify the above  command of the Kubernetes special agent like this:

      Code Block
      language bash theme RDark
      OMD[mysite]:~$ /omd/sites/mysite/share/check_mk/agents/special/agent_kube \ '--cluster' 'at' \ '--token' 'xyz' \ '--monitored-objects' 'deployments' 'daemonsets' 'statefulsets' 'nodes' 'pods' \ '--api-server-endpoint' 'https://<YOUR-IP>:6443' \ '--api-server-proxy' 'FROM_ENVIRONMENT' \ '--cluster-collector-endpoint' 'https://<YOUR-ENDPOINT>:30035' \ '--cluster-collector-proxy' 'FROM_ENVIRONMENT' \ --debug -vvv --vcrtrace ~/tmp/vcrtrace.txt > ~/tmp/k8s_with_debug.txt 2>&1

      Here, you can also reduce the number of '--monitored-objects' to a few resources to get less output. 
      .

   4. Run the special agent with no debug options to create an agent output, or you could download it from the cluster host via the Checkmk web interface. 

      Code Block
      language bash theme RDark
      /omd/sites/mysite/share/check_mk/agents/special/agent_kube '--cluster' 'at' '--token' 'xyz' '--monitored -objects' 'deployments' 'daemonsets' 'statefulsets' 'nodes' 'pods' '--api-server-endpoint' 'https://<YOUR-IP>:6443' '--api-server-proxy' 'FROM_ENVIRONMENT' '--cluster-collector-endpoint' 'https://<YOUR-ENDPOINT>:30035' '--cluster-collector-proxy' 'FROM_ENVIRONMENT' > ~/tmp/k8s_agent_output.txt 2>&1

      .

2. Please upload the following files to the support ticket.
   
   

...

1. Example with Special Agent of storeonce4x
   
   

   1. Find out the detailed special agent command (Type of agent column)

      Code Block
      language bash theme RDark
      OMD[mysite]:~$ cmk -D hostname

      
      

      Note
      An easier way would be this command: /bin/sh -c "$(cmk -D k8s | grep -A1 "^Type of agent:" | grep "Program:" | cut -f2 -d':')" Please note that if a line matching "^Type of agent:" followed by a line matching "^  Program:" exists more than once, then the output might be messed up.

      .

   2. Check if there are some options for debugging

      Code Block
      language bash theme RDark
      OMD[mysite]:~$ ~/share/check_mk/agents/special/agent_storeonce4x -h

      
      

      
      There are three options for debugging the request:
      
      

      Code Block
      language bash theme RDark
      --debug, -d Enable debug mode (keep some exceptions unhandled) --verbose, -v --vcrtrace TRACEFILE, --tracefile TRACEFILE If this flag is set to a TRACEFILE that does not exist yet, it will be created and all requests the program sends and their corresponding answers will be recorded in said file. If the file already exists, no requests are sent to the server, but the responses will be replayed from the tracefile.

      .

   3. Modify the special agent command by adding these three options

      Code Block
      language bash theme RDark
      OMD[mysite]:~$ ~/share/check_mk/agents/special/agent_storeonce4x <OTHER ARGUMENTS> --debug -v --vcrtrace ~/tmp/vcrtrace.txt 2>1 ~/tmp/storeonce4x_with_debug.txt

      .

   4. Run the special agent with no debug options to create an agent output. With this file, we can reproduce your issue

      Code Block
      language bash theme RDark
      OMD[mysite]:~$ /omd/sites/mysite/share/check_mk/agents/special/agent_kube <OTHER ARGUMENTS> > ~/tmp/k8s_agent_output.txt

      
      

      
      

2. Rename the token file
   
   The storeonce4x special agent is using username/password for authentication. After the successful login, we obtain the access token. The access token is used for future REST requests.
   
   If you want to read more, you can check this out: https://hewlettpackard.github.io/storeonce-rest/#Authentication
   
   

   1. We save the token file inside the site in
      

      Code Block
      language bash theme RDark
      ~/tmp/check_mk/special_agents/agent_storeonce4x/<hostname>_oAuthToken.json

      .

   2. Rename the file to _oAuthToken.json.back
      

      Code Block
      language bash theme RDark
      OMD[mysite]~# mv ~/tmp/check_mk/special_agents/agent_storeonce4x/<hostname>_oAuthToken.json ~/tmp/check_mk/special_agents/agent_storeonce4x/<hostname>_oAuthToken.json.back

      .

   3. Run the special agent again

...

1. Example with Special Agent of vSphere
   .
   

   1. Find out the detailed special agent command

      Code Block
      language bash theme RDark
      OMD[mysite]:~$ cmk -D <vcenter-host> | more vcenter Addresses: x.x.x.x Tags: [add_ip_addresses:add_ip_addresses_1], [address_family:ip-v4-only], [agent:special-agents], [criticality:prod], [ip-v4:ip-v4], [networking:lan], [piggyback:auto-piggyback], [site:nagnis_master], [snmp_ds:no-snmp], [tcp:tcp] Labels: [cmk/vsphere_object:vm] Host groups: check_mk Contact groups: all Agent mode: No Checkmk agent, all configured special agents Type of agent: Program: /omd/sites/mysite/share/check_mk/agents/special/agent_vsphere -u 'user' -s 'password' -i hos tsystem,virtualmachine,datastore,counters,licenses -P --spaces cut --snapshot_display vCenter --no-cert-check 'x.x.x.x' Process piggyback data from /omd/sites/mysite/tmp/check_mk/piggyback/vcenter Services: checktype item params

      
      
      

      Note
      An easier way would be this command: /bin/sh -c "$(cmk -D vcenter | grep -A1 "^Type of agent:" | grep "^ Program:" | cut -f2 -d':')" Please note that if a line matching "^Type of agent:" followed by a line matching "^  Program:" exists more than once, the output might be messed up.

      .

   2. Check if there are options for debugging.

      Code Block
      language bash theme RDark
      OMD[mysite]:~$ /omd/sites/mysite/share/check_mk/agents/special/agent_vsphere -h

      
      

      There are two options for debugging the request.

      Code Block
      language bash theme RDark
      --debug Debug mode: let Python exceptions come through --tracefile FILENAME Log all outgoing and incoming data into the given tracefile

      .

   3. Modify the special agent command by adding these two options

      Code Block
      language bash theme RDark
      OMD[mysite]:~$ /omd/sites/mysite/share/check_mk/agents/special/agent_vsphere -u 'user' -s 'password' --debug --tracefile $OMD_ROOT/tmp/vcenter.out -i hostsystem,virtualmachine,datastore,counters,licenses -P --spaces cut --no-cert-check '$HOST_ADDRESS' > $OMD_ROOT/tmp/vcenter.debug

      In CMK 1.6.0, you might find the option "--snapshot_display vCenter" in your CMK -D output. If that's the case, you can include this parameter.

      .

   4. Run the special agent with no debug options to create an agent output. With this file, we can reproduce your issue.

      Code Block
      language bash theme RDark
      root@linux~# /omd/sites/mysite/share/check_mk/agents/special/agent_vsphere -u 'user' -s 'password' -i hostsystem,virtualmachine,datastore,counters,licenses -P --spaces cut --no-cert-check 'x.x.x.x' >/~tmp/agent.output

      .

2. Please send us all three files. Now we're able to investigate further.

   1 2 3

   ~/tmp/vcenter.debug      # Debug Output ~/tmp/vcenter.out        # Tracefile /~tmp/agent.output       # Agent Output

   
   

...