/
Troubleshooting lack of Prometheus data after updating OpenShift 4.16 and above

Troubleshooting lack of Prometheus data after updating OpenShift 4.16 and above

Owned by Matthew Hierholzer
Last updated: Apr 11, 2025

This article helps to troubleshoot if Prometheus data is missing post OpenShift update.  An error such as “Invalid response: did not match Prometheus HTTP API” may occur.

LAST TESTED ON CHECKMK 2.3.0P1

Table of Contents


Please note that this is only an issue at the time this article was written. If at any point in time, OpenShift changes this behavior, this article may become irrelevant. 

Problem

One may experience an issue with Prometheus data displaying in OpenShift after upgrading OpenShift 4.16 and onwards.

Error

Invalid response: did not match Prometheus HTTP API


screenshot showing details of a openshift service


This issue was discussed in detail on this GitHub page.


This update to OpenShift 4.16 appears to prevent Checkmk from reaching the Prometheus API.

Solution

Our official documentation includes a preconfigured YAML file published on GitHub; however, to resolve this specific issue, an edit to the file is required. This 


checkmk-serviceaccount.yaml
---
kind: ServiceAccount
apiVersion: v1
metadata:
name: checkmk
namespace: checkmk-monitoring
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: checkmk-metrics-reader
rules:
verbs:
- "get"
- "list"
apiGroups:
- "" # core API group
- apps
- batch
- autoscaling
- monitoring.coreos.com 
resources:
- prometheuses/api
- alertmanagers/api 
- nodes
- nodes/proxy
- pods
- deployments
- replicasets
- jobs
- daemonsets
- statefulsets
- cronjobs
- services
- endpoints
- tickets
- persistentvolumes
- persistent volume claims
- storageclasses
- horizontalpodautoscalers
- replicasets
- namespaces
- resourcequotas
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: checkmk-metrics-reader-binding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: checkmk-metrics-reader
subjects:
- kind: ServiceAccount
name: checkmk
namespace: checkmk-monitoring