Health metrics
Health metrics
Abstract
Chapter 1. Health metrics
You can use metrics to monitor the health of your components.
See the following documentation:
1.1. Using the metrics service
You can use metrics to monitor component health across Red Hat Advanced Cluster Management for Kubernetes. Many custom metrics are documented in the Metrics chronicle overview.
1.1.1. Accessing the hub cluster metrics service
To view the collected metrics, you must expose the metrics service on the hub cluster. If your metrics are already exposed in the Grafana dashboard, this procedure is optional.
From the OpenShift Container Platform console, find the metrics service. Click Observe > Metrics.
If you do not see the metrics in the Grafana dashboard, Grafana Explorer, or in the OpenShift Container Platform console, Prometheus might not be configured to scrape metrics. Continue with Scrapping with Prometheus to expose your metrics.
1.1.2. Scraping with Prometheus
You can use Prometheus to expose metrics that are not exposed from the product console. See the procedures for both the hub and managed cluster metrics.
1.1.2.1. Scraping the hub cluster
See the following procedure to expose metrics for the hub cluster. These files are within the openshift-monitoring
namespace:
Create a
ServiceMonitor
for collecting services and exposing metrics. See the following YAML example:apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: hub-subscription-metrics namespace: openshift-monitoring spec: endpoints: - port: metrics namespaceSelector: matchNames: - open-cluster-management selector: matchLabels: app: hub-subscription-metrics
Run the following command to apply the file:
oc apply -f
Create a
Role
for setting the permissions for monitoring. See the following YAML file:apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: prometheus-k8s-monitoring namespace: open-cluster-management rules: - apiGroups: - "" resources: - services - endpoints - pods verbs: - get - list - watch - apiGroups: - extensions resources: - ingresses verbs: - get - list - watch - apiGroups: - networking.k8s.io resources: - ingresses verbs: - get - list - watch
Run the following command to apply the file:
oc apply -f
Create a
RoleBinding
for binding the role to the Prometheus monitoringServiceAccount
, as it is in the following example:apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: prometheus-k8s-monitoring-binding namespace: open-cluster-management roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: prometheus-k8s-monitoring subjects: - kind: ServiceAccount name: prometheus-k8s namespace: monitoring
Run the following command to apply the file:
oc apply -f
To verify, run the following query in the dashboard to find metrics that are reported by the Subscription Operator Metrics Service:
{service="hub-subscription-metrics"}
1.1.2.2. Scraping the managed cluster
See the following procedure to expose metrics for managed clusters. These files are within the openshift-monitoring
namespace:
Create a
ServiceMonitor
for collecting services exposing metrics. See the following YAML file example:apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: mc-subscription-metrics namespace: openshift-monitoring spec: endpoints: - port: metrics namespaceSelector: matchNames: - open-cluster-management-agent-addon selector: matchLabels: app: mc-subscription-metrics
Run the following command to apply your file:
oc apply -f
Create a
Role
for setting the permissions for monitoring. See the following YAML file example:apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: prometheus-k8s-monitoring namespace: open-cluster-management-agent-addon rules: - apiGroups: - "" resources: - services - endpoints - pods verbs: - get - list - watch - apiGroups: - extensions resources: - ingresses verbs: - get - list - watch - apiGroups: - networking.k8s.io resources: - ingresses verbs: - get - list - watch
Run the following command to apply the file:
oc apply -f
Create a
RoleBinding
for binding theRole
to the Prometheus monitoringServiceAccount
:apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: prometheus-k8s-monitoring-binding namespace: open-cluster-management-agent-addon roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: prometheus-k8s-monitoring subjects: - kind: ServiceAccount name: prometheus-k8s namespace: monitoring
Run the following command to apply the file:
oc apply -f
Verify in the
Prometheus
dashboard by running the following query to find metrics that are reported by the Subscription Operator Metrics Service:{service="mc-subscription-metrics"}
1.1.3. Scraping the standalone cluster
Create a
ServiceMonitor
for collecting services exposing metrics:apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: standalone-subscription-metrics namespace: openshift-monitoring spec: endpoints: - port: metrics namespaceSelector: matchNames: - open-cluster-management selector: matchLabels: app: standalone-subscription-metrics
Create a
Role
for setting the permissions for monitoring:oc apply -f
apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: prometheus-k8s-monitoring namespace: open-cluster-management rules: - apiGroups: - "" resources: - services - endpoints - pods verbs: - get - list - watch - apiGroups: - extensions resources: - ingresses verbs: - get - list - watch - apiGroups: - networking.k8s.io resources: - ingresses verbs: - get - list - watch
Create a
RoleBinding
for binding theRole
to the Prometheus monitoringServiceAccount
. See the following YAML file example:apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: prometheus-k8s-monitoring-binding namespace: open-cluster-management roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: prometheus-k8s-monitoring subjects: - kind: ServiceAccount name: prometheus-k8s namespace: monitoring
Run the following command to apply the file:
oc apply -f
Verify in the
Prometheus
dashboard by running the following query to find metrics that are reported by the Subscription Operator Metrics Service:{service="standalone-subscription-metrics"}