Page Comparison

...

Troubleshooting Microsoft Azure - "Graph client: Insufficient privileges to complete the operation" error

If you see the error message "Graph client: Insufficient privileges to complete the operation." when connecting to Azure, do the following:

1. Open the Azure Portal
   
   
2. Click Azure Active Directory 
   
   Image Added
   
   
   
3. Click App registrations in the left bar
   
   Image Added
   
   
   
4. Click the app you registered for Checkmk
   
   
   
5. Click API permissions in the left bar
   
   Image Added
   
   
   
6. Click Add Permissions and add a permissions for Microsoft Graph
   
   

Image Added

Full list of access rights needed:

These are the metrics we get via the Azure agents

SSL error - bad handshake

...

1. The first step would be to find the complete command of the Kubernetes special agent.
    

   1. The command can be found under "Type of agent >> Program." It will consist of multiple parameters depending on how the datasource program rule has been configured. 

      Code Block
      language bash theme RDark
      OMD[mysite]:~$ cmk -D k8s | more k8s Addresses: No IP Tags: [address_family:no-ip], [agent:special-agents], [criticality:prod], [networking:lan], [piggyback:auto-piggyback], [site:a21], [snmp_ds:no-snmp], [tcp:tcp] Labels: [cmk/kubernetes/cluster:at], [cmk/kubernetes/object:cluster], [cmk/site:k8s] Host groups: check_mk Contact groups: all Agent mode: No Checkmk agent, all configured special agents Type of agent: Program: /omd/sites/mysite/share/check_mk/agents/special/agent_kube '--cluster' 'k8s' '--token' 'xyz' '--monitored-objects' 'deployments' 'daemonsets' 'statefulsets' 'nodes' 'pods' '--api-server-endpoint' 'https://<YOUR-IP>:6443' '--api-server-proxy' 'FROM_ENVIRONMENT' '--cluster-collector-endpoint' 'https://<YOUR-ENDPOINT>:30035' '--cluster-collector-proxy' 'FROM_ENVIRONMENT' Process piggyback data from /omd/sites/mysite/tmp/check_mk/piggyback/k8s Services: ...

      
      

      Note
      An easier way would be this command: /bin/sh -c "$(cmk -D k8s | grep -A1 "^Type of agent:" | grep "Program:" | cut -f2- -d':')" Please note that if a line matching "^Type of agent:" followed by a line matching "^  Program:" exists more than once, the output might be messed up.

      .

   2. The special agent has the below options available for debugging purposes:

      Code Block
      language bash theme RDark
      OMD[mysite]:~$ /omd/sites/mysite/share/check_mk/agents/special/agent_kube -h ... --debug Debug mode: raise Python exceptions -v / --verbose Verbose mode (for even more output use -vvv) --vcrtrace FILENAME Enables VCR tracing for the API calls ...

      .

   3. Now, you can modify the above  command of the Kubernetes special agent like this:

      Code Block
      language bash theme RDark
      OMD[mysite]:~$ /omd/sites/mysite/share/check_mk/agents/special/agent_kube \ '--cluster' 'at' \ '--token' 'xyz' \ '--monitored-objects' 'deployments' 'daemonsets' 'statefulsets' 'nodes' 'pods' \ '--api-server-endpoint' 'https://<YOUR-IP>:6443' \ '--api-server-proxy' 'FROM_ENVIRONMENT' \ '--cluster-collector-endpoint' 'https://<YOUR-ENDPOINT>:30035' \ '--cluster-collector-proxy' 'FROM_ENVIRONMENT' \ --debug -vvv --vcrtrace ~/tmp/vcrtrace.txt > ~/tmp/k8s_with_debug.txt 2>&1

      Here, you can also reduce the number of '--monitored-objects' to a few resources to get less output. 
      .

   4. Run the special agent with no debug options to create an agent output, or you could download it from the cluster host via the Checkmk web interface. 

      Code Block
      language bash theme RDark
      /omd/sites/mysite/share/check_mk/agents/special/agent_kube '--cluster' 'at' '--token' 'xyz' '--monitored -objects' 'deployments' 'daemonsets' 'statefulsets' 'nodes' 'pods' '--api-server-endpoint' 'https://<YOUR-IP>:6443' '--api-server-proxy' 'FROM_ENVIRONMENT' '--cluster-collector-endpoint' 'https://<YOUR-ENDPOINT>:30035' '--cluster-collector-proxy' 'FROM_ENVIRONMENT' > ~/tmp/k8s_agent_output.txt 2>&1

      .

2. Please upload the following files to the support ticket.
   
   

...

1. Example with Special Agent of storeonce4x
   
   

   1. Find out the detailed special agent command (Type of agent column)

      Code Block
      language bash theme RDark
      OMD[mysite]:~$ cmk -D hostname

      
      

      Note
      an easier way would be this command: /bin/sh -c "$(cmk -D k8s | grep -A1 "^Type of agent:" | grep "Program:" | cut -f2 -d':')" Please note that if a line matching "^Type of agent:" followed by a line matching "^  Program:" exists more than once, then the output might be messed up.

      .

   2. Check if there are some options for debugging

      Code Block
      language bash theme RDark
      OMD[mysite]:~$ ~/share/check_mk/agents/special/agent_storeonce4x -h

      
      

      
      There are three options for debugging the request:
      
      

      Code Block
      language bash theme RDark
      --debug, -d Enable debug mode (keep some exceptions unhandled) --verbose, -v --vcrtrace TRACEFILE, --tracefile TRACEFILE If this flag is set to a TRACEFILE that does not exist yet, it will be created and all requests the program sends and their corresponding answers will be recorded in said file. If the file already exists, no requests are sent to the server, but the responses will be replayed from the tracefile.

      .

   3. Modify the special agent command by adding these three options

      Code Block
      language bash theme RDark
      OMD[mysite]:~$ ~/share/check_mk/agents/special/agent_storeonce4x <OTHER ARGUMENTS> --debug -v --vcrtrace ~/tmp/vcrtrace.txt 2>1 ~/tmp/storeonce4x_with_debug.txt

      .

   4. Run the special agent with no debug options to create an agent output. With this file, we can reproduce your issue

      Code Block
      language bash theme RDark
      OMD[mysite]:~$ /omd/sites/mysite/share/check_mk/agents/special/agent_kube <OTHER ARGUMENTS> > ~/tmp/k8s_agent_output.txt

      
      

      
      

2. Rename the token file
   
   The storeonce4x special agent is using username/password for authentication. After the successful login, we obtain the access token. The access token is used for future REST requests.
   
   If you want to read more, you can check this out: https://hewlettpackard.github.io/storeonce-rest/#Authentication
   
   

   1. We save the token file inside the site in
      

      Code Block
      language bash theme RDark
      ~/tmp/check_mk/special_agents/agent_storeonce4x/<hostname>_oAuthToken.json

      .

   2. Rename the file to _oAuthToken.json.back
      

      Code Block
      language bash theme RDark
      OMD[mysite]~# mv ~/tmp/check_mk/special_agents/agent_storeonce4x/<hostname>_oAuthToken.json ~/tmp/check_mk/special_agents/agent_storeonce4x/<hostname>_oAuthToken.json.back

      .

   3. Run the special agent again

...