Red Hat Summit Red Hat SummitSupportKonsoleEntwicklerDemo startenKontakt

Dieser Inhalt ist in der von Ihnen ausgewählten Sprache nicht verfügbar.

Chapter 10. Enabling Volume Metrics in OpenShift 3.10 and 3.11

Prometheus is a stand-alone, open source systems monitoring and alerting toolkit and shipped with OpenShift. You can use Prometheus to visualize metrics and alerts for OpenShift Container Platform system resources as PVs and services like heketi.

Heketi provides a RESTful management interface which can be used to manage the life cycle of GlusterFS volumes,as well as a metrics endpoint which can be scraped by Prometheus.

The way Prometheus is integrated into OpenShift differs slightly between OCP 3.10 and 3.11.

For more information on how to setup Prometheus in OCP 3.10, see Prometheus on OpenShift Container Platform.

For more information on how to setup Prometheus in OCP 3.11, see Prometheus Cluster Monitoring.

10.1. Available Metrics for File Storage and Block Storage
Link kopieren

The following list provides different metrics of the PVs that can be viewed on Prometheus:

kubelet_volume_stats_available_bytes
Number of available bytes in the volume.
kubelet_volume_stats_capacity_bytes
Capacity in bytes of the volume.
kubelet_volume_stats_inodes
Maximum number of inodes in the volume.
kubelet_volume_stats_inodes_free
Number of free inodes in the volume.
kubelet_volume_stats_inodes_used
Number of used inodes in the volume.
kubelet_volume_stats_used_bytes
Number of used bytes in the volume.

The Heketi service provides the following metrics:

heketi_cluster_count
Number of clusters.
heketi_device_brick_count
Number of bricks on device.
heketi_device_count
Number of devices on host.
heketi_device_free_bytes
Amount of free space available on the device.
heketi_device_size_bytes
Total size of the device.
heketi_device_used_bytes
Amount of space used on the device.
heketi_nodes_count
Number of nodes on the cluster.
heketi_up
Verifies if heketi is running.
heketi_volumes_count
Number of volumes on cluster.
heketi_block_volumes_count
Number of block volumes on cluster.

10.2. Enabling Heketi Metrics in OpenShift 3.10
Link kopieren

To view Heketi metrics on Prometheus in OCP 3.10, execute the following commands:

  1. Add annotations to the heketi-storage service (normally running in the app-storage namespace). 

    # oc project app-storage
# oc annotate svc heketi-storage prometheus.io/scheme=http
# oc annotate svc heketi-storage prometheus.io/scrape=true
    Copy to Clipboard Toggle word wrap 
    # oc describe svc heketi-storage
Name:          	heketi-storage
Namespace:     	app-storage
Labels:        	glusterfs=heketi-storage-service
   	            heketi=storage-service
Annotations:   	description=Exposes Heketi service
   	            prometheus.io/scheme=http
   	             prometheus.io/scrape=true
Selector:      	glusterfs=heketi-storage-pod
Type:          	ClusterIP
IP:            	172.30.90.87
Port:          	heketi  8080/TCP
TargetPort:    	8080/TCP
Endpoints:     	172.18.14.2:8080
Session Affinity:  None
    Copy to Clipboard Toggle word wrap

  2. Add the app-storage namespace for the heketi service in the Prometheus configmap. 

    # oc get cm prometheus -o yaml -n openshift-metrics
....
- job_name: 'kubernetes-service-endpoints'

     tls_config:
     ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
     # TODO: this should be per target
     insecure_skip_verify: true

     kubernetes_sd_configs:
     - role: endpoints

     relabel_configs:
     # only scrape infrastructure components
     - source_labels: [__meta_kubernetes_namespace]
     action: keep
regex: 'default|logging|metrics|kube-.+|openshift|openshift-.+|app-storage'
    Copy to Clipboard Toggle word wrap

    Do the above for all other storage namespaces (for example: infra-storage).

  3. Restart the prometheus-0 pod to query the Heketi metrics in Prometheus.

10.3. Enabling Heketi Metrics in OpenShift 3.11
Link kopieren

In OCP 3.11, Prometheus uses servicemonitors, which are new resources introduced by the Prometheus Operator. The servicemonitors need to be created for every storage namespace and they describe the set of targets to be monitored.

To view Heketi metrics on Prometheus in OCP 3.11, execute the following commands:

  1. Add annotations to the heketi-storage service. 

    # oc project app-storage
# oc annotate svc heketi-storage prometheus.io/scheme=http
# oc annotate svc heketi-storage prometheus.io/scrape=true
    Copy to Clipboard Toggle word wrap

  2. Create a heketi-app servicemonitor in the openshift-monitoring namespace using the below template: 

    # cat heketi-app-sm.yml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: heketi-app
  labels:
    k8s-app: heketi-app
  namespace: openshift-monitoring
spec:
  endpoints:
  - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    interval: 30s
    port: heketi
    scheme: http
    targetPort: 0
  namespaceSelector:
    matchNames:
    - app-storage
  selector:
    matchLabels:
      heketi: storage-service
    Copy to Clipboard Toggle word wrap

    Where, the namespaceSelector and label need to match the values in the heketi-storage service: 

    # oc describe svc heketi-storage -n app-storage
Name:              heketi-storage
Namespace:         app-storage
Labels:            glusterfs=heketi-storage-service
                   heketi=storage-service
Annotations:       description=Exposes Heketi service
                   prometheus.io/scheme=http
                   prometheus.io/scrape=true
Selector:          glusterfs=heketi-storage-pod
Type:              ClusterIP
IP:                172.30.3.92
Port:              heketi  8080/TCP
TargetPort:        8080/TCP
Endpoints:         10.128.4.12:8080
Session Affinity:  None
Events:            <none>
    Copy to Clipboard Toggle word wrap

    With the correct selectors set, create the servicemonitor in the openshift-monitoring namespace with the correct selectors set. 

    # oc create -f heketi-app-sm.yml -n openshift-monitoring
servicemonitor.monitoring.coreos.com "heketi-app" created
    Copy to Clipboard Toggle word wrap 
    # oc get servicemonitor -n openshift-monitoring
NAME                          AGE
alertmanager                  20d
cluster-monitoring-operator   20d
heketi-app                    1m
kube-apiserver                20d
kube-controllers              20d
kube-state-metrics            20d
kubelet                       20d
node-exporter                 20d
prometheus                    20d
prometheus-operator           20d
    Copy to Clipboard Toggle word wrap
  3. In case you have multiple OCS clusters, one servicemonitor needs to be created per OCS cluster using the steps above.

  4. Execute the following command to add cluster-reader rights to prometheus: 

    # oc adm policy add-cluster-role-to-user cluster-reader \
system:serviceaccount:openshift-monitoring:prometheus-k8s -n \
openshift-monitoring
cluster role "cluster-reader" added: "system:serviceaccount:openshift-monitoring:prometheus-k8s"
    Copy to Clipboard Toggle word wrap
  5. After some minutes, Prometheus loads the new servicemonitors.

10.4. Viewing Metrics
Link kopieren

To view any metrics:

  1. Add the metrics name in Prometheus, and click Execute.

  2. In the Graph tab, the value for the metrics for the volume is displayed as a graph.

    For example, in the following image, to check the available bytes, kubelet_volume_stats_available_bytes metric is added to the search bar on Prometheus. On clicking Execute, the available bytes value is depicted as a graph. You can hover the mouse on the line to get more details. (To view the image in detail, right-click and select View Image.)

    Viewing metrics

Nach oben
Red Hat logoGithubredditYoutubeTwitter

Lernen

Testen, kaufen und verkaufen

Communitys

Über Red Hat Dokumentation

Wir helfen Red Hat Benutzern, mit unseren Produkten und Diensten innovativ zu sein und ihre Ziele zu erreichen – mit Inhalten, denen sie vertrauen können. Entdecken Sie unsere neuesten Updates.

Mehr Inklusion in Open Source

Red Hat hat sich verpflichtet, problematische Sprache in unserem Code, unserer Dokumentation und unseren Web-Eigenschaften zu ersetzen. Weitere Einzelheiten finden Sie in Red Hat Blog.

Über Red Hat

Wir liefern gehärtete Lösungen, die es Unternehmen leichter machen, plattform- und umgebungsübergreifend zu arbeiten, vom zentralen Rechenzentrum bis zum Netzwerkrand.

Theme

© 2025 Red Hat