Troubleshooting lack of Prometheus data after updating OpenShift 4.16 and above
Troubleshoots missing Prometheus data or “Invalid response” errors after an OpenShift update.
LAST TESTED ON CHECKMK 2.3.0P1
Table of Contents
Please note that this is only an issue at the time this article was written. If at any point in time, OpenShift changes this behavior, this article may become irrelevant.
Problem
One may experience an issue with Prometheus data displaying in OpenShift after upgrading OpenShift 4.16 and onwards.
Invalid response: did not match Prometheus HTTP API
This issue was discussed in detail on this GitHub page. This update to OpenShift 4.16 appears to prevent Checkmk from reaching the Prometheus API.
Solution
Our official documentation includes a preconfigured YAML file published on GitHub; however, to resolve this specific issue, an edit to the file is required.
checkmk-serviceaccount.yaml
---
kind: ServiceAccount
apiVersion: v1
metadata:
name: checkmk
namespace: checkmk-monitoring
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: checkmk-metrics-reader
rules:
- apiGroups:
- "" # core API group
- apps
- batch
- autoscaling
resources:
- nodes
- nodes/proxy
- pods
- deployments
- replicasets
- jobs
- daemonsets
- statefulsets
- cronjobs
- services
- endpoints
- ingresses
- persistentvolumes
- persistentvolumeclaims
- storageclasses
- horizontalpodautoscalers
- replicasets
- namespaces
- resourcequotas
verbs:
- "get"
- "list"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: checkmk-metrics-reader-binding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: checkmk-metrics-reader
subjects:
- kind: ServiceAccount
name: checkmk
namespace: checkmk-monitoring