Este contenido no está disponible en el idioma seleccionado.
Chapter 12. Monitoring
12.1. Monitoring overview
You can monitor the health of your cluster and virtual machines (VMs) with the following tools:
- Monitoring OpenShift Virtualization VM health status
-
View the overall health of your OpenShift Virtualization environment in the web console by navigating to the Home
Overview page in the OpenShift Container Platform web console. The Status card displays the overall health of OpenShift Virtualization based on the alerts and conditions. - OpenShift Container Platform cluster checkup framework
Run automated tests on your cluster with the OpenShift Container Platform cluster checkup framework to check the following conditions:
- Network connectivity and latency between two VMs attached to a secondary network interface
- VM running a Data Plane Development Kit (DPDK) workload with zero packet loss
- Cluster storage is optimally configured for OpenShift Virtualization
- Prometheus queries for virtual resources
- Query vCPU, network, storage, and guest memory swapping usage and live migration progress.
- VM custom metrics
-
Configure the
node-exporter
service to expose internal VM metrics and processes. - VM health checks
- Configure readiness, liveness, and guest agent ping probes and a watchdog for VMs.
- Runbooks
- Diagnose and resolve issues that trigger OpenShift Virtualization alerts in the OpenShift Container Platform web console.
12.2. OpenShift Virtualization cluster checkup framework
OpenShift Virtualization includes the following predefined checkups that can be used for cluster maintenance and troubleshooting:
Latency checkup, which verifies network connectivity and measures latency between two virtual machines (VMs) that are attached to a secondary network interface.
ImportantBefore you run a latency checkup, you must first create a bridge interface on the cluster nodes to connect the VM’s secondary interface to any interface on the node. If you do not create a bridge interface, the VMs do not start and the job fails.
- Storage checkup, which verifies if the cluster storage is optimally configured for OpenShift Virtualization.
- DPDK checkup, which verifies that a node can run a VM with a Data Plane Development Kit (DPDK) workload with zero packet loss.
The OpenShift Virtualization cluster checkup framework is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
12.2.1. About the OpenShift Virtualization cluster checkup framework
A checkup is an automated test workload that allows you to verify if a specific cluster functionality works as expected. The cluster checkup framework uses native Kubernetes resources to configure and execute the checkup.
By using predefined checkups, cluster administrators and developers can improve cluster maintainability, troubleshoot unexpected behavior, minimize errors, and save time. They can also review the results of the checkup and share them with experts for further analysis. Vendors can write and publish checkups for features or services that they provide and verify that their customer environments are configured correctly.
Running a predefined checkup in an existing namespace involves setting up a service account for the checkup, creating the Role
and RoleBinding
objects for the service account, enabling permissions for the checkup, and creating the input config map and the checkup job. You can run a checkup multiple times.
You must always:
- Verify that the checkup image is from a trustworthy source before applying it.
-
Review the checkup permissions before creating the
Role
andRoleBinding
objects.
12.2.2. Running checkups by using the web console
Use the following procedures the first time you run checkups by using the web console. For additional checkups, click Run checkup on either checkup tab, and select the appropriate checkup from the drop down menu.
12.2.2.1. Running a latency checkup by using the web console
Run a latency checkup to verify network connectivity and measure the latency between two virtual machines attached to a secondary network interface.
Prerequisites
-
You must add a
NetworkAttachmentDefinition
to the namespace.
Procedure
-
Navigate to Virtualization
Checkups in the web console. - Click the Network latency tab.
- Click Install permissions.
- Click Run checkup.
- Enter a name for the checkup in the Name field.
- Select a NetworkAttachmentDefinition from the drop-down menu.
- Optional: Set a duration for the latency sample in the Sample duration (seconds) field.
- Optional: Define a maximum latency time interval by enabling Set maximum desired latency (milliseconds) and defining the time interval.
- Optional: Target specific nodes by enabling Select nodes and specifying the Source node and Target node.
- Click Run.
You can view the status of the latency checkup in the Checkups list on the Latency checkup tab. Click on the name of the checkup for more details.
12.2.2.2. Running a storage checkup by using the web console
Run a storage checkup to validate that storage is working correctly for virtual machines.
Procedure
-
Navigate to Virtualization
Checkups in the web console. - Click the Storage tab.
- Click Install permissions.
- Click Run checkup.
- Enter a name for the checkup in the Name field.
- Enter a timeout value for the checkup in the Timeout (minutes) fields.
- Click Run.
You can view the status of the storage checkup in the Checkups list on the Storage tab. Click on the name of the checkup for more details.
12.2.3. Running checkups by using the command line
Use the following procedures the first time you run checkups by using the command line.
12.2.3.1. Running a latency checkup by using the command line
You use a predefined checkup to verify network connectivity and measure latency between two virtual machines (VMs) that are attached to a secondary network interface. The latency checkup uses the ping utility.
You run a latency checkup by performing the following steps:
- Create a service account, roles, and rolebindings to provide cluster access permissions to the latency checkup.
- Create a config map to provide the input to run the checkup and to store the results.
- Create a job to run the checkup.
- Review the results in the config map.
- Optional: To rerun the checkup, delete the existing config map and job and then create a new config map and job.
- When you are finished, delete the latency checkup resources.
Prerequisites
-
You installed the OpenShift CLI (
oc
). - The cluster has at least two worker nodes.
- You configured a network attachment definition for a namespace.
Procedure
Create a
ServiceAccount
,Role
, andRoleBinding
manifest for the latency checkup:Example 12.1. Example role manifest file
--- apiVersion: v1 kind: ServiceAccount metadata: name: vm-latency-checkup-sa --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: kubevirt-vm-latency-checker rules: - apiGroups: ["kubevirt.io"] resources: ["virtualmachineinstances"] verbs: ["get", "create", "delete"] - apiGroups: ["subresources.kubevirt.io"] resources: ["virtualmachineinstances/console"] verbs: ["get"] - apiGroups: ["k8s.cni.cncf.io"] resources: ["network-attachment-definitions"] verbs: ["get"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: kubevirt-vm-latency-checker subjects: - kind: ServiceAccount name: vm-latency-checkup-sa roleRef: kind: Role name: kubevirt-vm-latency-checker apiGroup: rbac.authorization.k8s.io --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: kiagnose-configmap-access rules: - apiGroups: [ "" ] resources: [ "configmaps" ] verbs: ["get", "update"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: kiagnose-configmap-access subjects: - kind: ServiceAccount name: vm-latency-checkup-sa roleRef: kind: Role name: kiagnose-configmap-access apiGroup: rbac.authorization.k8s.io
Apply the
ServiceAccount
,Role
, andRoleBinding
manifest:$ oc apply -n <target_namespace> -f <latency_sa_roles_rolebinding>.yaml 1
- 1
<target_namespace>
is the namespace where the checkup is to be run. This must be an existing namespace where theNetworkAttachmentDefinition
object resides.
Create a
ConfigMap
manifest that contains the input parameters for the checkup:Example input config map
apiVersion: v1 kind: ConfigMap metadata: name: kubevirt-vm-latency-checkup-config labels: kiagnose/checkup-type: kubevirt-vm-latency data: spec.timeout: 5m spec.param.networkAttachmentDefinitionNamespace: <target_namespace> spec.param.networkAttachmentDefinitionName: "blue-network" 1 spec.param.maxDesiredLatencyMilliseconds: "10" 2 spec.param.sampleDurationSeconds: "5" 3 spec.param.sourceNode: "worker1" 4 spec.param.targetNode: "worker2" 5
- 1
- The name of the
NetworkAttachmentDefinition
object. - 2
- Optional: The maximum desired latency, in milliseconds, between the virtual machines. If the measured latency exceeds this value, the checkup fails.
- 3
- Optional: The duration of the latency check, in seconds.
- 4
- Optional: When specified, latency is measured from this node to the target node. If the source node is specified, the
spec.param.targetNode
field cannot be empty. - 5
- Optional: When specified, latency is measured from the source node to this node.
Apply the config map manifest in the target namespace:
$ oc apply -n <target_namespace> -f <latency_config_map>.yaml
Create a
Job
manifest to run the checkup:Example job manifest
apiVersion: batch/v1 kind: Job metadata: name: kubevirt-vm-latency-checkup labels: kiagnose/checkup-type: kubevirt-vm-latency spec: backoffLimit: 0 template: spec: serviceAccountName: vm-latency-checkup-sa restartPolicy: Never containers: - name: vm-latency-checkup image: registry.redhat.io/container-native-virtualization/vm-network-latency-checkup-rhel9:v4.16.0 securityContext: allowPrivilegeEscalation: false capabilities: drop: ["ALL"] runAsNonRoot: true seccompProfile: type: "RuntimeDefault" env: - name: CONFIGMAP_NAMESPACE value: <target_namespace> - name: CONFIGMAP_NAME value: kubevirt-vm-latency-checkup-config - name: POD_UID valueFrom: fieldRef: fieldPath: metadata.uid
Apply the
Job
manifest:$ oc apply -n <target_namespace> -f <latency_job>.yaml
Wait for the job to complete:
$ oc wait job kubevirt-vm-latency-checkup -n <target_namespace> --for condition=complete --timeout 6m
Review the results of the latency checkup by running the following command. If the maximum measured latency is greater than the value of the
spec.param.maxDesiredLatencyMilliseconds
attribute, the checkup fails and returns an error.$ oc get configmap kubevirt-vm-latency-checkup-config -n <target_namespace> -o yaml
Example output config map (success)
apiVersion: v1 kind: ConfigMap metadata: name: kubevirt-vm-latency-checkup-config namespace: <target_namespace> labels: kiagnose/checkup-type: kubevirt-vm-latency data: spec.timeout: 5m spec.param.networkAttachmentDefinitionNamespace: <target_namespace> spec.param.networkAttachmentDefinitionName: "blue-network" spec.param.maxDesiredLatencyMilliseconds: "10" spec.param.sampleDurationSeconds: "5" spec.param.sourceNode: "worker1" spec.param.targetNode: "worker2" status.succeeded: "true" status.failureReason: "" status.completionTimestamp: "2022-01-01T09:00:00Z" status.startTimestamp: "2022-01-01T09:00:07Z" status.result.avgLatencyNanoSec: "177000" status.result.maxLatencyNanoSec: "244000" 1 status.result.measurementDurationSec: "5" status.result.minLatencyNanoSec: "135000" status.result.sourceNode: "worker1" status.result.targetNode: "worker2"
- 1
- The maximum measured latency in nanoseconds.
Optional: To view the detailed job log in case of checkup failure, use the following command:
$ oc logs job.batch/kubevirt-vm-latency-checkup -n <target_namespace>
Delete the job and config map that you previously created by running the following commands:
$ oc delete job -n <target_namespace> kubevirt-vm-latency-checkup
$ oc delete config-map -n <target_namespace> kubevirt-vm-latency-checkup-config
Optional: If you do not plan to run another checkup, delete the roles manifest:
$ oc delete -f <latency_sa_roles_rolebinding>.yaml
12.2.3.2. Running a storage checkup by using the command line
Use a predefined checkup to verify that the OpenShift Container Platform cluster storage is configured optimally to run OpenShift Virtualization workloads.
Prerequisites
-
You have installed the OpenShift CLI (
oc
). The cluster administrator has created the required
cluster-reader
permissions for the storage checkup service account and namespace, such as in the following example:apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: kubevirt-storage-checkup-clustereader roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-reader subjects: - kind: ServiceAccount name: storage-checkup-sa namespace: <target_namespace> 1
- 1
- The namespace where the checkup is to be run.
Procedure
Create a
ServiceAccount
,Role
, andRoleBinding
manifest file for the storage checkup:Example 12.2. Example service account, role, and rolebinding manifest
--- apiVersion: v1 kind: ServiceAccount metadata: name: storage-checkup-sa --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: storage-checkup-role rules: - apiGroups: [ "" ] resources: [ "configmaps" ] verbs: ["get", "update"] - apiGroups: [ "kubevirt.io" ] resources: [ "virtualmachines" ] verbs: [ "create", "delete" ] - apiGroups: [ "kubevirt.io" ] resources: [ "virtualmachineinstances" ] verbs: [ "get" ] - apiGroups: [ "subresources.kubevirt.io" ] resources: [ "virtualmachineinstances/addvolume", "virtualmachineinstances/removevolume" ] verbs: [ "update" ] - apiGroups: [ "kubevirt.io" ] resources: [ "virtualmachineinstancemigrations" ] verbs: [ "create" ] - apiGroups: [ "cdi.kubevirt.io" ] resources: [ "datavolumes" ] verbs: [ "create", "delete" ] - apiGroups: [ "" ] resources: [ "persistentvolumeclaims" ] verbs: [ "delete" ] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: storage-checkup-role subjects: - kind: ServiceAccount name: storage-checkup-sa roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: storage-checkup-role
Apply the
ServiceAccount
,Role
, andRoleBinding
manifest in the target namespace:$ oc apply -n <target_namespace> -f <storage_sa_roles_rolebinding>.yaml
Create a
ConfigMap
andJob
manifest file. The config map contains the input parameters for the checkup job.Example input config map and job manifest
--- apiVersion: v1 kind: ConfigMap metadata: name: storage-checkup-config namespace: $CHECKUP_NAMESPACE data: spec.timeout: 10m spec.param.storageClass: ocs-storagecluster-ceph-rbd-virtualization spec.param.vmiTimeout: 3m --- apiVersion: batch/v1 kind: Job metadata: name: storage-checkup namespace: $CHECKUP_NAMESPACE spec: backoffLimit: 0 template: spec: serviceAccount: storage-checkup-sa restartPolicy: Never containers: - name: storage-checkup image: quay.io/kiagnose/kubevirt-storage-checkup:main imagePullPolicy: Always env: - name: CONFIGMAP_NAMESPACE value: $CHECKUP_NAMESPACE - name: CONFIGMAP_NAME value: storage-checkup-config
Apply the
ConfigMap
andJob
manifest file in the target namespace to run the checkup:$ oc apply -n <target_namespace> -f <storage_configmap_job>.yaml
Wait for the job to complete:
$ oc wait job storage-checkup -n <target_namespace> --for condition=complete --timeout 10m
Review the results of the checkup by running the following command:
$ oc get configmap storage-checkup-config -n <target_namespace> -o yaml
Example output config map (success)
apiVersion: v1 kind: ConfigMap metadata: name: storage-checkup-config labels: kiagnose/checkup-type: kubevirt-storage data: spec.timeout: 10m status.succeeded: "true" 1 status.failureReason: "" 2 status.startTimestamp: "2023-07-31T13:14:38Z" 3 status.completionTimestamp: "2023-07-31T13:19:41Z" 4 status.result.cnvVersion: 4.16.2 5 status.result.defaultStorageClass: trident-nfs 6 status.result.goldenImagesNoDataSource: <data_import_cron_list> 7 status.result.goldenImagesNotUpToDate: <data_import_cron_list> 8 status.result.ocpVersion: 4.16.0 9 status.result.pvcBound: "true" 10 status.result.storageProfileMissingVolumeSnapshotClass: <storage_class_list> 11 status.result.storageProfilesWithEmptyClaimPropertySets: <storage_profile_list> 12 status.result.storageProfilesWithSmartClone: <storage_profile_list> 13 status.result.storageProfilesWithSpecClaimPropertySets: <storage_profile_list> 14 status.result.storageProfilesWithRWX: |- ocs-storagecluster-ceph-rbd ocs-storagecluster-ceph-rbd-virtualization ocs-storagecluster-cephfs trident-iscsi trident-minio trident-nfs windows-vms status.result.vmBootFromGoldenImage: VMI "vmi-under-test-dhkb8" successfully booted status.result.vmHotplugVolume: |- VMI "vmi-under-test-dhkb8" hotplug volume ready VMI "vmi-under-test-dhkb8" hotplug volume removed status.result.vmLiveMigration: VMI "vmi-under-test-dhkb8" migration completed status.result.vmVolumeClone: 'DV cloneType: "csi-clone"' status.result.vmsWithNonVirtRbdStorageClass: <vm_list> 15 status.result.vmsWithUnsetEfsStorageClass: <vm_list> 16
- 1
- Specifies if the checkup is successful (
true
) or not (false
). - 2
- The reason for failure if the checkup fails.
- 3
- The time when the checkup started, in RFC 3339 time format.
- 4
- The time when the checkup has completed, in RFC 3339 time format.
- 5
- The OpenShift Virtualization version.
- 6
- Specifies if there is a default storage class.
- 7
- The list of golden images whose data source is not ready.
- 8
- The list of golden images whose data import cron is not up-to-date.
- 9
- The OpenShift Container Platform version.
- 10
- Specifies if a PVC of 10Mi has been created and bound by the provisioner.
- 11
- The list of storage profiles using snapshot-based clone but missing VolumeSnapshotClass.
- 12
- The list of storage profiles with unknown provisioners.
- 13
- The list of storage profiles with smart clone support (CSI/snapshot).
- 14
- The list of storage profiles spec-overriden claimPropertySets.
- 15
- The list of virtual machines that use the Ceph RBD storage class when the virtualization storage class exists.
- 16
- The list of virtual machines that use an Elastic File Store (EFS) storage class where the GID and UID are not set in the storage class.
Delete the job and config map that you previously created by running the following commands:
$ oc delete job -n <target_namespace> storage-checkup
$ oc delete config-map -n <target_namespace> storage-checkup-config
Optional: If you do not plan to run another checkup, delete the
ServiceAccount
,Role
, andRoleBinding
manifest:$ oc delete -f <storage_sa_roles_rolebinding>.yaml
12.2.3.3. Running a DPDK checkup by using the command line
Use a predefined checkup to verify that your OpenShift Container Platform cluster node can run a virtual machine (VM) with a Data Plane Development Kit (DPDK) workload with zero packet loss. The DPDK checkup runs traffic between a traffic generator and a VM running a test DPDK application.
You run a DPDK checkup by performing the following steps:
- Create a service account, role, and role bindings for the DPDK checkup.
- Create a config map to provide the input to run the checkup and to store the results.
- Create a job to run the checkup.
- Review the results in the config map.
- Optional: To rerun the checkup, delete the existing config map and job and then create a new config map and job.
- When you are finished, delete the DPDK checkup resources.
Prerequisites
-
You have installed the OpenShift CLI (
oc
). - The cluster is configured to run DPDK applications.
- The project is configured to run DPDK applications.
Procedure
Create a
ServiceAccount
,Role
, andRoleBinding
manifest for the DPDK checkup:Example 12.3. Example service account, role, and rolebinding manifest file
--- apiVersion: v1 kind: ServiceAccount metadata: name: dpdk-checkup-sa --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: kiagnose-configmap-access rules: - apiGroups: [ "" ] resources: [ "configmaps" ] verbs: [ "get", "update" ] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: kiagnose-configmap-access subjects: - kind: ServiceAccount name: dpdk-checkup-sa roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: kiagnose-configmap-access --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: kubevirt-dpdk-checker rules: - apiGroups: [ "kubevirt.io" ] resources: [ "virtualmachineinstances" ] verbs: [ "create", "get", "delete" ] - apiGroups: [ "subresources.kubevirt.io" ] resources: [ "virtualmachineinstances/console" ] verbs: [ "get" ] - apiGroups: [ "" ] resources: [ "configmaps" ] verbs: [ "create", "delete" ] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: kubevirt-dpdk-checker subjects: - kind: ServiceAccount name: dpdk-checkup-sa roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: kubevirt-dpdk-checker
Apply the
ServiceAccount
,Role
, andRoleBinding
manifest:$ oc apply -n <target_namespace> -f <dpdk_sa_roles_rolebinding>.yaml
Create a
ConfigMap
manifest that contains the input parameters for the checkup:Example input config map
apiVersion: v1 kind: ConfigMap metadata: name: dpdk-checkup-config labels: kiagnose/checkup-type: kubevirt-dpdk data: spec.timeout: 10m spec.param.networkAttachmentDefinitionName: <network_name> 1 spec.param.trafficGenContainerDiskImage: "quay.io/kiagnose/kubevirt-dpdk-checkup-traffic-gen:v0.4.0 2 spec.param.vmUnderTestContainerDiskImage: "quay.io/kiagnose/kubevirt-dpdk-checkup-vm:v0.4.0" 3
- 1
- The name of the
NetworkAttachmentDefinition
object. - 2
- The container disk image for the traffic generator. In this example, the image is pulled from the upstream Project Quay Container Registry.
- 3
- The container disk image for the VM under test. In this example, the image is pulled from the upstream Project Quay Container Registry.
Apply the
ConfigMap
manifest in the target namespace:$ oc apply -n <target_namespace> -f <dpdk_config_map>.yaml
Create a
Job
manifest to run the checkup:Example job manifest
apiVersion: batch/v1 kind: Job metadata: name: dpdk-checkup labels: kiagnose/checkup-type: kubevirt-dpdk spec: backoffLimit: 0 template: spec: serviceAccountName: dpdk-checkup-sa restartPolicy: Never containers: - name: dpdk-checkup image: registry.redhat.io/container-native-virtualization/kubevirt-dpdk-checkup-rhel9:v4.16.0 imagePullPolicy: Always securityContext: allowPrivilegeEscalation: false capabilities: drop: ["ALL"] runAsNonRoot: true seccompProfile: type: "RuntimeDefault" env: - name: CONFIGMAP_NAMESPACE value: <target-namespace> - name: CONFIGMAP_NAME value: dpdk-checkup-config - name: POD_UID valueFrom: fieldRef: fieldPath: metadata.uid
Apply the
Job
manifest:$ oc apply -n <target_namespace> -f <dpdk_job>.yaml
Wait for the job to complete:
$ oc wait job dpdk-checkup -n <target_namespace> --for condition=complete --timeout 10m
Review the results of the checkup by running the following command:
$ oc get configmap dpdk-checkup-config -n <target_namespace> -o yaml
Example output config map (success)
apiVersion: v1 kind: ConfigMap metadata: name: dpdk-checkup-config labels: kiagnose/checkup-type: kubevirt-dpdk data: spec.timeout: 10m spec.param.NetworkAttachmentDefinitionName: "dpdk-network-1" spec.param.trafficGenContainerDiskImage: "quay.io/kiagnose/kubevirt-dpdk-checkup-traffic-gen:v0.4.0" spec.param.vmUnderTestContainerDiskImage: "quay.io/kiagnose/kubevirt-dpdk-checkup-vm:v0.4.0" status.succeeded: "true" 1 status.failureReason: "" 2 status.startTimestamp: "2023-07-31T13:14:38Z" 3 status.completionTimestamp: "2023-07-31T13:19:41Z" 4 status.result.trafficGenSentPackets: "480000000" 5 status.result.trafficGenOutputErrorPackets: "0" 6 status.result.trafficGenInputErrorPackets: "0" 7 status.result.trafficGenActualNodeName: worker-dpdk1 8 status.result.vmUnderTestActualNodeName: worker-dpdk2 9 status.result.vmUnderTestReceivedPackets: "480000000" 10 status.result.vmUnderTestRxDroppedPackets: "0" 11 status.result.vmUnderTestTxDroppedPackets: "0" 12
- 1
- Specifies if the checkup is successful (
true
) or not (false
). - 2
- The reason for failure if the checkup fails.
- 3
- The time when the checkup started, in RFC 3339 time format.
- 4
- The time when the checkup has completed, in RFC 3339 time format.
- 5
- The number of packets sent from the traffic generator.
- 6
- The number of error packets sent from the traffic generator.
- 7
- The number of error packets received by the traffic generator.
- 8
- The node on which the traffic generator VM was scheduled.
- 9
- The node on which the VM under test was scheduled.
- 10
- The number of packets received on the VM under test.
- 11
- The ingress traffic packets that were dropped by the DPDK application.
- 12
- The egress traffic packets that were dropped from the DPDK application.
Delete the job and config map that you previously created by running the following commands:
$ oc delete job -n <target_namespace> dpdk-checkup
$ oc delete config-map -n <target_namespace> dpdk-checkup-config
Optional: If you do not plan to run another checkup, delete the
ServiceAccount
,Role
, andRoleBinding
manifest:$ oc delete -f <dpdk_sa_roles_rolebinding>.yaml
12.2.3.3.1. DPDK checkup config map parameters
The following table shows the mandatory and optional parameters that you can set in the data
stanza of the input ConfigMap
manifest when you run a cluster DPDK readiness checkup:
Parameter | Description | Is Mandatory |
---|---|---|
| The time, in minutes, before the checkup fails. | True |
|
The name of the | True |
| The container disk image for the traffic generator. | True |
| The node on which the traffic generator VM is to be scheduled. The node should be configured to allow DPDK traffic. | False |
| The number of packets per second, in kilo (k) or million(m). The default value is 8m. | False |
| The container disk image for the VM under test. | True |
| The node on which the VM under test is to be scheduled. The node should be configured to allow DPDK traffic. | False |
| The duration, in minutes, for which the traffic generator runs. The default value is 5 minutes. | False |
| The maximum bandwidth of the SR-IOV NIC. The default value is 10Gbps. | False |
|
When set to | False |
12.2.3.3.2. Building a container disk image for RHEL virtual machines
You can build a custom Red Hat Enterprise Linux (RHEL) 8 OS image in qcow2
format and use it to create a container disk image. You can store the container disk image in a registry that is accessible from your cluster and specify the image location in the spec.param.vmContainerDiskImage
attribute of the DPDK checkup config map.
To build a container disk image, you must create an image builder virtual machine (VM). The image builder VM is a RHEL 8 VM that can be used to build custom RHEL images.
Prerequisites
-
The image builder VM must run RHEL 8.7 and must have a minimum of 2 CPU cores, 4 GiB RAM, and 20 GB of free space in the
/var
directory. -
You have installed the image builder tool and its CLI (
composer-cli
) on the VM. You have installed the
virt-customize
tool:# dnf install libguestfs-tools
-
You have installed the Podman CLI tool (
podman
).
Procedure
Verify that you can build a RHEL 8.7 image:
# composer-cli distros list
NoteTo run the
composer-cli
commands as non-root, add your user to theweldr
orroot
groups:# usermod -a -G weldr user
$ newgrp weldr
Enter the following command to create an image blueprint file in TOML format that contains the packages to be installed, kernel customizations, and the services to be disabled during boot time:
$ cat << EOF > dpdk-vm.toml name = "dpdk_image" description = "Image to use with the DPDK checkup" version = "0.0.1" distro = "rhel-87" [[customizations.user]] name = "root" password = "redhat" [[packages]] name = "dpdk" [[packages]] name = "dpdk-tools" [[packages]] name = "driverctl" [[packages]] name = "tuned-profiles-cpu-partitioning" [customizations.kernel] append = "default_hugepagesz=1GB hugepagesz=1G hugepages=1" [customizations.services] disabled = ["NetworkManager-wait-online", "sshd"] EOF
Push the blueprint file to the image builder tool by running the following command:
# composer-cli blueprints push dpdk-vm.toml
Generate the system image by specifying the blueprint name and output file format. The Universally Unique Identifier (UUID) of the image is displayed when you start the compose process.
# composer-cli compose start dpdk_image qcow2
Wait for the compose process to complete. The compose status must show
FINISHED
before you can continue to the next step.# composer-cli compose status
Enter the following command to download the
qcow2
image file by specifying its UUID:# composer-cli compose image <UUID>
Create the customization scripts by running the following commands:
$ cat <<EOF >customize-vm #!/bin/bash # Setup hugepages mount mkdir -p /mnt/huge echo "hugetlbfs /mnt/huge hugetlbfs defaults,pagesize=1GB 0 0" >> /etc/fstab # Create vfio-noiommu.conf echo "options vfio enable_unsafe_noiommu_mode=1" > /etc/modprobe.d/vfio-noiommu.conf # Enable guest-exec,guest-exec-status on the qemu-guest-agent configuration sed -i '/^BLACKLIST_RPC=/ { s/guest-exec-status//; s/guest-exec//g }' /etc/sysconfig/qemu-ga sed -i '/^BLACKLIST_RPC=/ { s/,\+/,/g; s/^,\|,$//g }' /etc/sysconfig/qemu-ga EOF
Use the
virt-customize
tool to customize the image generated by the image builder tool:$ virt-customize -a <UUID>-disk.qcow2 --run=customize-vm --selinux-relabel
To create a Dockerfile that contains all the commands to build the container disk image, enter the following command:
$ cat << EOF > Dockerfile FROM scratch COPY --chown=107:107 <UUID>-disk.qcow2 /disk/ EOF
where:
- <UUID>-disk.qcow2
-
Specifies the name of the custom image in
qcow2
format.
Build and tag the container by running the following command:
$ podman build . -t dpdk-rhel:latest
Push the container disk image to a registry that is accessible from your cluster by running the following command:
$ podman push dpdk-rhel:latest
-
Provide a link to the container disk image in the
spec.param.vmUnderTestContainerDiskImage
attribute in the DPDK checkup config map.
12.2.4. Additional resources
- Attaching a virtual machine to multiple networks
- Using a virtual function in DPDK mode with an Intel NIC
- Using SR-IOV and the Node Tuning Operator to achieve a DPDK line rate
- Installing image builder
- How to register and subscribe a RHEL system to the Red Hat Customer Portal using Red Hat Subscription Manager
12.3. Prometheus queries for virtual resources
OpenShift Virtualization provides metrics that you can use to monitor the consumption of cluster infrastructure resources, including vCPU, network, storage, and guest memory swapping. You can also use metrics to query live migration status.
12.3.1. Prerequisites
-
To use the vCPU metric, the
schedstats=enable
kernel argument must be applied to theMachineConfig
object. This kernel argument enables scheduler statistics used for debugging and performance tuning and adds a minor additional load to the scheduler. For more information, see Adding kernel arguments to nodes. - For guest memory swapping queries to return data, memory swapping must be enabled on the virtual guests.
12.3.2. Querying metrics
The OpenShift Container Platform monitoring dashboard enables you to run Prometheus Query Language (PromQL) queries to examine metrics visualized on a plot. This functionality provides information about the state of a cluster and any user-defined workloads that you are monitoring.
As a cluster administrator, you can query metrics for all core OpenShift Container Platform and user-defined projects.
As a developer, you must specify a project name when querying metrics. You must have the required privileges to view metrics for the selected project.
12.3.2.1. Querying metrics for all projects as a cluster administrator
As a cluster administrator or as a user with view permissions for all projects, you can access metrics for all default OpenShift Container Platform and user-defined projects in the Metrics UI.
Prerequisites
-
You have access to the cluster as a user with the
cluster-admin
cluster role or with view permissions for all projects. -
You have installed the OpenShift CLI (
oc
).
Procedure
-
From the Administrator perspective in the OpenShift Container Platform web console, select Observe
Metrics. To add one or more queries, do any of the following:
Option Description Create a custom query.
Add your Prometheus Query Language (PromQL) query to the Expression field.
As you type a PromQL expression, autocomplete suggestions appear in a drop-down list. These suggestions include functions, metrics, labels, and time tokens. You can use the keyboard arrows to select one of these suggested items and then press Enter to add the item to your expression. You can also move your mouse pointer over a suggested item to view a brief description of that item.
Add multiple queries.
Select Add query.
Duplicate an existing query.
Select the Options menu next to the query, then choose Duplicate query.
Disable a query from being run.
Select the Options menu next to the query and choose Disable query.
To run queries that you created, select Run queries. The metrics from the queries are visualized on the plot. If a query is invalid, the UI shows an error message.
NoteQueries that operate on large amounts of data might time out or overload the browser when drawing time series graphs. To avoid this, select Hide graph and calibrate your query using only the metrics table. Then, after finding a feasible query, enable the plot to draw the graphs.
NoteBy default, the query table shows an expanded view that lists every metric and its current value. You can select ˅ to minimize the expanded view for a query.
- Optional: The page URL now contains the queries you ran. To use this set of queries again in the future, save this URL.
Explore the visualized metrics. Initially, all metrics from all enabled queries are shown on the plot. You can select which metrics are shown by doing any of the following:
Option Description Hide all metrics from a query.
Click the Options menu for the query and click Hide all series.
Hide a specific metric.
Go to the query table and click the colored square near the metric name.
Zoom into the plot and change the time range.
Either:
- Visually select the time range by clicking and dragging on the plot horizontally.
- Use the menu in the left upper corner to select the time range.
Reset the time range.
Select Reset zoom.
Display outputs for all queries at a specific point in time.
Hold the mouse cursor on the plot at that point. The query outputs will appear in a pop-up box.
Hide the plot.
Select Hide graph.
12.3.2.2. Querying metrics for user-defined projects as a developer
You can access metrics for a user-defined project as a developer or as a user with view permissions for the project.
In the Developer perspective, the Metrics UI includes some predefined CPU, memory, bandwidth, and network packet queries for the selected project. You can also run custom Prometheus Query Language (PromQL) queries for CPU, memory, bandwidth, network packet and application metrics for the project.
Developers can only use the Developer perspective and not the Administrator perspective. As a developer, you can only query metrics for one project at a time.
Prerequisites
- You have access to the cluster as a developer or as a user with view permissions for the project that you are viewing metrics for.
- You have enabled monitoring for user-defined projects.
- You have deployed a service in a user-defined project.
-
You have created a
ServiceMonitor
custom resource definition (CRD) for the service to define how the service is monitored.
Procedure
-
From the Developer perspective in the OpenShift Container Platform web console, select Observe
Metrics. - Select the project that you want to view metrics for in the Project: list.
Select a query from the Select query list, or create a custom PromQL query based on the selected query by selecting Show PromQL. The metrics from the queries are visualized on the plot.
NoteIn the Developer perspective, you can only run one query at a time.
Explore the visualized metrics by doing any of the following:
Option Description Zoom into the plot and change the time range.
Either:
- Visually select the time range by clicking and dragging on the plot horizontally.
- Use the menu in the left upper corner to select the time range.
Reset the time range.
Select Reset zoom.
Display outputs for all queries at a specific point in time.
Hold the mouse cursor on the plot at that point. The query outputs appear in a pop-up box.
12.3.3. Virtualization metrics
The following metric descriptions include example Prometheus Query Language (PromQL) queries. These metrics are not an API and might change between versions. For a complete list of virtualization metrics, see KubeVirt components metrics.
The following examples use topk
queries that specify a time period. If virtual machines are deleted during that time period, they can still appear in the query output.
12.3.3.1. vCPU metrics
The following query can identify virtual machines that are waiting for Input/Output (I/O):
kubevirt_vmi_vcpu_wait_seconds_total
- Returns the wait time (in seconds) for a virtual machine’s vCPU. Type: Counter.
A value above '0' means that the vCPU wants to run, but the host scheduler cannot run it yet. This inability to run indicates that there is an issue with I/O.
To query the vCPU metric, the schedstats=enable
kernel argument must first be applied to the MachineConfig
object. This kernel argument enables scheduler statistics used for debugging and performance tuning and adds a minor additional load to the scheduler.
Example vCPU wait time query
topk(3, sum by (name, namespace) (rate(kubevirt_vmi_vcpu_wait_seconds_total[6m]))) > 0 1
- 1
- This query returns the top 3 VMs waiting for I/O at every given moment over a six-minute time period.
12.3.3.2. Network metrics
The following queries can identify virtual machines that are saturating the network:
kubevirt_vmi_network_receive_bytes_total
- Returns the total amount of traffic received (in bytes) on the virtual machine’s network. Type: Counter.
kubevirt_vmi_network_transmit_bytes_total
- Returns the total amount of traffic transmitted (in bytes) on the virtual machine’s network. Type: Counter.
Example network traffic query
topk(3, sum by (name, namespace) (rate(kubevirt_vmi_network_receive_bytes_total[6m])) + sum by (name, namespace) (rate(kubevirt_vmi_network_transmit_bytes_total[6m]))) > 0 1
- 1
- This query returns the top 3 VMs transmitting the most network traffic at every given moment over a six-minute time period.
12.3.3.3. Storage metrics
12.3.3.3.1. Storage-related traffic
The following queries can identify VMs that are writing large amounts of data:
kubevirt_vmi_storage_read_traffic_bytes_total
- Returns the total amount (in bytes) of the virtual machine’s storage-related traffic. Type: Counter.
kubevirt_vmi_storage_write_traffic_bytes_total
- Returns the total amount of storage writes (in bytes) of the virtual machine’s storage-related traffic. Type: Counter.
Example storage-related traffic query
topk(3, sum by (name, namespace) (rate(kubevirt_vmi_storage_read_traffic_bytes_total[6m])) + sum by (name, namespace) (rate(kubevirt_vmi_storage_write_traffic_bytes_total[6m]))) > 0 1
- 1
- This query returns the top 3 VMs performing the most storage traffic at every given moment over a six-minute time period.
12.3.3.3.2. Storage snapshot data
kubevirt_vmsnapshot_disks_restored_from_source
- Returns the total number of virtual machine disks restored from the source virtual machine. Type: Gauge.
kubevirt_vmsnapshot_disks_restored_from_source_bytes
- Returns the amount of space in bytes restored from the source virtual machine. Type: Gauge.
Examples of storage snapshot data queries
kubevirt_vmsnapshot_disks_restored_from_source{vm_name="simple-vm", vm_namespace="default"} 1
- 1
- This query returns the total number of virtual machine disks restored from the source virtual machine.
kubevirt_vmsnapshot_disks_restored_from_source_bytes{vm_name="simple-vm", vm_namespace="default"} 1
- 1
- This query returns the amount of space in bytes restored from the source virtual machine.
12.3.3.3.3. I/O performance
The following queries can determine the I/O performance of storage devices:
kubevirt_vmi_storage_iops_read_total
- Returns the amount of write I/O operations the virtual machine is performing per second. Type: Counter.
kubevirt_vmi_storage_iops_write_total
- Returns the amount of read I/O operations the virtual machine is performing per second. Type: Counter.
Example I/O performance query
topk(3, sum by (name, namespace) (rate(kubevirt_vmi_storage_iops_read_total[6m])) + sum by (name, namespace) (rate(kubevirt_vmi_storage_iops_write_total[6m]))) > 0 1
- 1
- This query returns the top 3 VMs performing the most I/O operations per second at every given moment over a six-minute time period.
12.3.3.4. Guest memory swapping metrics
The following queries can identify which swap-enabled guests are performing the most memory swapping:
kubevirt_vmi_memory_swap_in_traffic_bytes
- Returns the total amount (in bytes) of memory the virtual guest is swapping in. Type: Gauge.
kubevirt_vmi_memory_swap_out_traffic_bytes
- Returns the total amount (in bytes) of memory the virtual guest is swapping out. Type: Gauge.
Example memory swapping query
topk(3, sum by (name, namespace) (rate(kubevirt_vmi_memory_swap_in_traffic_bytes[6m])) + sum by (name, namespace) (rate(kubevirt_vmi_memory_swap_out_traffic_bytes[6m]))) > 0 1
- 1
- This query returns the top 3 VMs where the guest is performing the most memory swapping at every given moment over a six-minute time period.
Memory swapping indicates that the virtual machine is under memory pressure. Increasing the memory allocation of the virtual machine can mitigate this issue.
12.3.3.5. Live migration metrics
The following metrics can be queried to show live migration status:
kubevirt_vmi_migration_data_processed_bytes
- The amount of guest operating system data that has migrated to the new virtual machine (VM). Type: Gauge.
kubevirt_vmi_migration_data_remaining_bytes
- The amount of guest operating system data that remains to be migrated. Type: Gauge.
kubevirt_vmi_migration_memory_transfer_rate_bytes
- The rate at which memory is becoming dirty in the guest operating system. Dirty memory is data that has been changed but not yet written to disk. Type: Gauge.
kubevirt_vmi_migrations_in_pending_phase
- The number of pending migrations. Type: Gauge.
kubevirt_vmi_migrations_in_scheduling_phase
- The number of scheduling migrations. Type: Gauge.
kubevirt_vmi_migrations_in_running_phase
- The number of running migrations. Type: Gauge.
kubevirt_vmi_migration_succeeded
- The number of successfully completed migrations. Type: Gauge.
kubevirt_vmi_migration_failed
- The number of failed migrations. Type: Gauge.
12.3.4. Additional resources
12.4. Exposing custom metrics for virtual machines
OpenShift Container Platform includes a preconfigured, preinstalled, and self-updating monitoring stack that provides monitoring for core platform components. This monitoring stack is based on the Prometheus monitoring system. Prometheus is a time-series database and a rule evaluation engine for metrics.
In addition to using the OpenShift Container Platform monitoring stack, you can enable monitoring for user-defined projects by using the CLI and query custom metrics that are exposed for virtual machines through the node-exporter
service.
12.4.1. Configuring the node exporter service
The node-exporter agent is deployed on every virtual machine in the cluster from which you want to collect metrics. Configure the node-exporter agent as a service to expose internal metrics and processes that are associated with virtual machines.
Prerequisites
-
Install the OpenShift Container Platform CLI
oc
. -
Log in to the cluster as a user with
cluster-admin
privileges. -
Create the
cluster-monitoring-config
ConfigMap
object in theopenshift-monitoring
project. -
Configure the
user-workload-monitoring-config
ConfigMap
object in theopenshift-user-workload-monitoring
project by settingenableUserWorkload
totrue
.
Procedure
Create the
Service
YAML file. In the following example, the file is callednode-exporter-service.yaml
.kind: Service apiVersion: v1 metadata: name: node-exporter-service 1 namespace: dynamation 2 labels: servicetype: metrics 3 spec: ports: - name: exmet 4 protocol: TCP port: 9100 5 targetPort: 9100 6 type: ClusterIP selector: monitor: metrics 7
- 1
- The node-exporter service that exposes the metrics from the virtual machines.
- 2
- The namespace where the service is created.
- 3
- The label for the service. The
ServiceMonitor
uses this label to match this service. - 4
- The name given to the port that exposes metrics on port 9100 for the
ClusterIP
service. - 5
- The target port used by
node-exporter-service
to listen for requests. - 6
- The TCP port number of the virtual machine that is configured with the
monitor
label. - 7
- The label used to match the virtual machine’s pods. In this example, any virtual machine’s pod with the label
monitor
and a value ofmetrics
will be matched.
Create the node-exporter service:
$ oc create -f node-exporter-service.yaml
12.4.2. Configuring a virtual machine with the node exporter service
Download the node-exporter
file on to the virtual machine. Then, create a systemd
service that runs the node-exporter service when the virtual machine boots.
Prerequisites
-
The pods for the component are running in the
openshift-user-workload-monitoring
project. -
Grant the
monitoring-edit
role to users who need to monitor this user-defined project.
Procedure
- Log on to the virtual machine.
Download the
node-exporter
file on to the virtual machine by using the directory path that applies to the version ofnode-exporter
file.$ wget https://github.com/prometheus/node_exporter/releases/download/v1.3.1/node_exporter-1.3.1.linux-amd64.tar.gz
Extract the executable and place it in the
/usr/bin
directory.$ sudo tar xvf node_exporter-1.3.1.linux-amd64.tar.gz \ --directory /usr/bin --strip 1 "*/node_exporter"
Create a
node_exporter.service
file in this directory path:/etc/systemd/system
. Thissystemd
service file runs the node-exporter service when the virtual machine reboots.[Unit] Description=Prometheus Metrics Exporter After=network.target StartLimitIntervalSec=0 [Service] Type=simple Restart=always RestartSec=1 User=root ExecStart=/usr/bin/node_exporter [Install] WantedBy=multi-user.target
Enable and start the
systemd
service.$ sudo systemctl enable node_exporter.service $ sudo systemctl start node_exporter.service
Verification
Verify that the node-exporter agent is reporting metrics from the virtual machine.
$ curl http://localhost:9100/metrics
Example output
go_gc_duration_seconds{quantile="0"} 1.5244e-05 go_gc_duration_seconds{quantile="0.25"} 3.0449e-05 go_gc_duration_seconds{quantile="0.5"} 3.7913e-05
12.4.3. Creating a custom monitoring label for virtual machines
To enable queries to multiple virtual machines from a single service, add a custom label in the virtual machine’s YAML file.
Prerequisites
-
Install the OpenShift Container Platform CLI
oc
. -
Log in as a user with
cluster-admin
privileges. - Access to the web console for stop and restart a virtual machine.
Procedure
Edit the
template
spec of your virtual machine configuration file. In this example, the labelmonitor
has the valuemetrics
.spec: template: metadata: labels: monitor: metrics
-
Stop and restart the virtual machine to create a new pod with the label name given to the
monitor
label.
12.4.3.1. Querying the node-exporter service for metrics
Metrics are exposed for virtual machines through an HTTP service endpoint under the /metrics
canonical name. When you query for metrics, Prometheus directly scrapes the metrics from the metrics endpoint exposed by the virtual machines and presents these metrics for viewing.
Prerequisites
-
You have access to the cluster as a user with
cluster-admin
privileges or themonitoring-edit
role. - You have enabled monitoring for the user-defined project by configuring the node-exporter service.
Procedure
Obtain the HTTP service endpoint by specifying the namespace for the service:
$ oc get service -n <namespace> <node-exporter-service>
To list all available metrics for the node-exporter service, query the
metrics
resource.$ curl http://<172.30.226.162:9100>/metrics | grep -vE "^#|^$"
Example output
node_arp_entries{device="eth0"} 1 node_boot_time_seconds 1.643153218e+09 node_context_switches_total 4.4938158e+07 node_cooling_device_cur_state{name="0",type="Processor"} 0 node_cooling_device_max_state{name="0",type="Processor"} 0 node_cpu_guest_seconds_total{cpu="0",mode="nice"} 0 node_cpu_guest_seconds_total{cpu="0",mode="user"} 0 node_cpu_seconds_total{cpu="0",mode="idle"} 1.10586485e+06 node_cpu_seconds_total{cpu="0",mode="iowait"} 37.61 node_cpu_seconds_total{cpu="0",mode="irq"} 233.91 node_cpu_seconds_total{cpu="0",mode="nice"} 551.47 node_cpu_seconds_total{cpu="0",mode="softirq"} 87.3 node_cpu_seconds_total{cpu="0",mode="steal"} 86.12 node_cpu_seconds_total{cpu="0",mode="system"} 464.15 node_cpu_seconds_total{cpu="0",mode="user"} 1075.2 node_disk_discard_time_seconds_total{device="vda"} 0 node_disk_discard_time_seconds_total{device="vdb"} 0 node_disk_discarded_sectors_total{device="vda"} 0 node_disk_discarded_sectors_total{device="vdb"} 0 node_disk_discards_completed_total{device="vda"} 0 node_disk_discards_completed_total{device="vdb"} 0 node_disk_discards_merged_total{device="vda"} 0 node_disk_discards_merged_total{device="vdb"} 0 node_disk_info{device="vda",major="252",minor="0"} 1 node_disk_info{device="vdb",major="252",minor="16"} 1 node_disk_io_now{device="vda"} 0 node_disk_io_now{device="vdb"} 0 node_disk_io_time_seconds_total{device="vda"} 174 node_disk_io_time_seconds_total{device="vdb"} 0.054 node_disk_io_time_weighted_seconds_total{device="vda"} 259.79200000000003 node_disk_io_time_weighted_seconds_total{device="vdb"} 0.039 node_disk_read_bytes_total{device="vda"} 3.71867136e+08 node_disk_read_bytes_total{device="vdb"} 366592 node_disk_read_time_seconds_total{device="vda"} 19.128 node_disk_read_time_seconds_total{device="vdb"} 0.039 node_disk_reads_completed_total{device="vda"} 5619 node_disk_reads_completed_total{device="vdb"} 96 node_disk_reads_merged_total{device="vda"} 5 node_disk_reads_merged_total{device="vdb"} 0 node_disk_write_time_seconds_total{device="vda"} 240.66400000000002 node_disk_write_time_seconds_total{device="vdb"} 0 node_disk_writes_completed_total{device="vda"} 71584 node_disk_writes_completed_total{device="vdb"} 0 node_disk_writes_merged_total{device="vda"} 19761 node_disk_writes_merged_total{device="vdb"} 0 node_disk_written_bytes_total{device="vda"} 2.007924224e+09 node_disk_written_bytes_total{device="vdb"} 0
12.4.4. Creating a ServiceMonitor resource for the node exporter service
You can use a Prometheus client library and scrape metrics from the /metrics
endpoint to access and view the metrics exposed by the node-exporter service. Use a ServiceMonitor
custom resource definition (CRD) to monitor the node exporter service.
Prerequisites
-
You have access to the cluster as a user with
cluster-admin
privileges or themonitoring-edit
role. - You have enabled monitoring for the user-defined project by configuring the node-exporter service.
Procedure
Create a YAML file for the
ServiceMonitor
resource configuration. In this example, the service monitor matches any service with the labelmetrics
and queries theexmet
port every 30 seconds.apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: labels: k8s-app: node-exporter-metrics-monitor name: node-exporter-metrics-monitor 1 namespace: dynamation 2 spec: endpoints: - interval: 30s 3 port: exmet 4 scheme: http selector: matchLabels: servicetype: metrics
Create the
ServiceMonitor
configuration for the node-exporter service.$ oc create -f node-exporter-metrics-monitor.yaml
12.4.4.1. Accessing the node exporter service outside the cluster
You can access the node-exporter service outside the cluster and view the exposed metrics.
Prerequisites
-
You have access to the cluster as a user with
cluster-admin
privileges or themonitoring-edit
role. - You have enabled monitoring for the user-defined project by configuring the node-exporter service.
Procedure
Expose the node-exporter service.
$ oc expose service -n <namespace> <node_exporter_service_name>
Obtain the FQDN (Fully Qualified Domain Name) for the route.
$ oc get route -o=custom-columns=NAME:.metadata.name,DNS:.spec.host
Example output
NAME DNS node-exporter-service node-exporter-service-dynamation.apps.cluster.example.org
Use the
curl
command to display metrics for the node-exporter service.$ curl -s http://node-exporter-service-dynamation.apps.cluster.example.org/metrics
Example output
go_gc_duration_seconds{quantile="0"} 1.5382e-05 go_gc_duration_seconds{quantile="0.25"} 3.1163e-05 go_gc_duration_seconds{quantile="0.5"} 3.8546e-05 go_gc_duration_seconds{quantile="0.75"} 4.9139e-05 go_gc_duration_seconds{quantile="1"} 0.000189423
12.4.5. Additional resources
12.5. Exposing downward metrics for virtual machines
As an administrator, you can expose a limited set of host and virtual machine (VM) metrics to a guest VM by first enabling a downwardMetrics
feature gate and then configuring a downwardMetrics
device.
Users can view the metrics results by using the command line or the vm-dump-metrics tool
.
On Red Hat Enterprise Linux (RHEL) 9, use the command line to view downward metrics. See Viewing downward metrics by using the command line.
The vm-dump-metrics tool is not supported on the Red Hat Enterprise Linux (RHEL) 9 platform.
12.5.1. Enabling or disabling the downwardMetrics feature gate
You can enable or disable the downwardMetrics
feature gate by performing either of the following actions:
- Editing the HyperConverged custom resource (CR) in your default editor
- Using the command line
12.5.1.1. Enabling or disabling the downward metrics feature gate in a YAML file
To expose downward metrics for a host virtual machine, you can enable the downwardMetrics
feature gate by editing a YAML file.
Prerequisites
- You must have administrator privileges to enable the feature gate.
Procedure
Open the HyperConverged custom resource (CR) in your default editor by running the following command:
$ oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnv
Choose to enable or disable the downwardMetrics feature gate as follows:
To enable the
downwardMetrics
feature gate, add and then setspec.featureGates.downwardMetrics
totrue
. For example:apiVersion: hco.kubevirt.io/v1beta1 kind: HyperConverged metadata: name: kubevirt-hyperconverged namespace: openshift-cnv spec: featureGates: downwardMetrics: true # ...
To disable the
downwardMetrics
feature gate, setspec.featureGates.downwardMetrics
tofalse
. For example:apiVersion: hco.kubevirt.io/v1beta1 kind: HyperConverged metadata: name: kubevirt-hyperconverged namespace: openshift-cnv spec: featureGates: downwardMetrics: false # ...
12.5.1.2. Enabling or disabling the downward metrics feature gate from the command line
To expose downward metrics for a host virtual machine, you can enable the downwardMetrics
feature gate by using the command line.
Prerequisites
- You must have administrator privileges to enable the feature gate.
Procedure
Choose to enable or disable the
downwardMetrics
feature gate as follows:Enable the
downwardMetrics
feature gate by running the command shown in the following example:$ oc patch hco kubevirt-hyperconverged -n openshift-cnv \ --type json -p '[{"op": "replace", "path": \ "/spec/featureGates/downwardMetrics" \ "value": true}]'
Disable the
downwardMetrics
feature gate by running the command shown in the following example:$ oc patch hco kubevirt-hyperconverged -n openshift-cnv \ --type json -p '[{"op": "replace", "path": \ "/spec/featureGates/downwardMetrics" \ "value": false}]'
12.5.2. Configuring a downward metrics device
You enable the capturing of downward metrics for a host VM by creating a configuration file that includes a downwardMetrics
device. Adding this device establishes that the metrics are exposed through a virtio-serial
port.
Prerequisites
-
You must first enable the
downwardMetrics
feature gate.
Procedure
Edit or create a YAML file that includes a
downwardMetrics
device, as shown in the following example:Example downwardMetrics configuration file
apiVersion: kubevirt.io/v1 kind: VirtualMachine metadata: name: fedora namespace: default spec: dataVolumeTemplates: - metadata: name: fedora-volume spec: sourceRef: kind: DataSource name: fedora namespace: openshift-virtualization-os-images storage: resources: {} storageClassName: hostpath-csi-basic instancetype: name: u1.medium preference: name: fedora running: true template: metadata: labels: app.kubernetes.io/name: headless spec: domain: devices: downwardMetrics: {} 1 subdomain: headless volumes: - dataVolume: name: fedora-volume name: rootdisk - cloudInitNoCloud: userData: | #cloud-config chpasswd: expire: false password: '<password>' 2 user: fedora name: cloudinitdisk
12.5.3. Viewing downward metrics
You can view downward metrics by using either of the following options:
- The command line interface (CLI)
-
The
vm-dump-metrics
tool
On Red Hat Enterprise Linux (RHEL) 9, use the command line to view downward metrics. The vm-dump-metrics tool is not supported on the Red Hat Enterprise Linux (RHEL) 9 platform.
12.5.3.1. Viewing downward metrics by using the command line
You can view downward metrics by entering a command from inside a guest virtual machine (VM).
Procedure
Run the following commands:
$ sudo sh -c 'printf "GET /metrics/XML\n\n" > /dev/virtio-ports/org.github.vhostmd.1'
$ sudo cat /dev/virtio-ports/org.github.vhostmd.1
12.5.3.2. Viewing downward metrics by using the vm-dump-metrics tool
To view downward metrics, install the vm-dump-metrics
tool and then use the tool to expose the metrics results.
On Red Hat Enterprise Linux (RHEL) 9, use the command line to view downward metrics. The vm-dump-metrics tool is not supported on the Red Hat Enterprise Linux (RHEL) 9 platform.
Procedure
Install the
vm-dump-metrics
tool by running the following command:$ sudo dnf install -y vm-dump-metrics
Retrieve the metrics results by running the following command:
$ sudo vm-dump-metrics
Example output
<metrics> <metric type="string" context="host"> <name>HostName</name> <value>node01</value> [...] <metric type="int64" context="host" unit="s"> <name>Time</name> <value>1619008605</value> </metric> <metric type="string" context="host"> <name>VirtualizationVendor</name> <value>kubevirt.io</value> </metric> </metrics>
12.6. Virtual machine health checks
You can configure virtual machine (VM) health checks by defining readiness and liveness probes in the VirtualMachine
resource.
12.6.1. About readiness and liveness probes
Use readiness and liveness probes to detect and handle unhealthy virtual machines (VMs). You can include one or more probes in the specification of the VM to ensure that traffic does not reach a VM that is not ready for it and that a new VM is created when a VM becomes unresponsive.
A readiness probe determines whether a VM is ready to accept service requests. If the probe fails, the VM is removed from the list of available endpoints until the VM is ready.
A liveness probe determines whether a VM is responsive. If the probe fails, the VM is deleted and a new VM is created to restore responsiveness.
You can configure readiness and liveness probes by setting the spec.readinessProbe
and the spec.livenessProbe
fields of the VirtualMachine
object. These fields support the following tests:
- HTTP GET
- The probe determines the health of the VM by using a web hook. The test is successful if the HTTP response code is between 200 and 399. You can use an HTTP GET test with applications that return HTTP status codes when they are completely initialized.
- TCP socket
- The probe attempts to open a socket to the VM. The VM is only considered healthy if the probe can establish a connection. You can use a TCP socket test with applications that do not start listening until initialization is complete.
- Guest agent ping
-
The probe uses the
guest-ping
command to determine if the QEMU guest agent is running on the virtual machine.
12.6.1.1. Defining an HTTP readiness probe
Define an HTTP readiness probe by setting the spec.readinessProbe.httpGet
field of the virtual machine (VM) configuration.
Procedure
Include details of the readiness probe in the VM configuration file.
Sample readiness probe with an HTTP GET test
apiVersion: kubevirt.io/v1 kind: VirtualMachine metadata: annotations: name: fedora-vm namespace: example-namespace # ... spec: template: spec: readinessProbe: httpGet: 1 port: 1500 2 path: /healthz 3 httpHeaders: - name: Custom-Header value: Awesome initialDelaySeconds: 120 4 periodSeconds: 20 5 timeoutSeconds: 10 6 failureThreshold: 3 7 successThreshold: 3 8 # ...
- 1
- The HTTP GET request to perform to connect to the VM.
- 2
- The port of the VM that the probe queries. In the above example, the probe queries port 1500.
- 3
- The path to access on the HTTP server. In the above example, if the handler for the server’s /healthz path returns a success code, the VM is considered to be healthy. If the handler returns a failure code, the VM is removed from the list of available endpoints.
- 4
- The time, in seconds, after the VM starts before the readiness probe is initiated.
- 5
- The delay, in seconds, between performing probes. The default delay is 10 seconds. This value must be greater than
timeoutSeconds
. - 6
- The number of seconds of inactivity after which the probe times out and the VM is assumed to have failed. The default value is 1. This value must be lower than
periodSeconds
. - 7
- The number of times that the probe is allowed to fail. The default is 3. After the specified number of attempts, the pod is marked
Unready
. - 8
- The number of times that the probe must report success, after a failure, to be considered successful. The default is 1.
Create the VM by running the following command:
$ oc create -f <file_name>.yaml
12.6.1.2. Defining a TCP readiness probe
Define a TCP readiness probe by setting the spec.readinessProbe.tcpSocket
field of the virtual machine (VM) configuration.
Procedure
Include details of the TCP readiness probe in the VM configuration file.
Sample readiness probe with a TCP socket test
apiVersion: kubevirt.io/v1 kind: VirtualMachine metadata: annotations: name: fedora-vm namespace: example-namespace # ... spec: template: spec: readinessProbe: initialDelaySeconds: 120 1 periodSeconds: 20 2 tcpSocket: 3 port: 1500 4 timeoutSeconds: 10 5 # ...
- 1
- The time, in seconds, after the VM starts before the readiness probe is initiated.
- 2
- The delay, in seconds, between performing probes. The default delay is 10 seconds. This value must be greater than
timeoutSeconds
. - 3
- The TCP action to perform.
- 4
- The port of the VM that the probe queries.
- 5
- The number of seconds of inactivity after which the probe times out and the VM is assumed to have failed. The default value is 1. This value must be lower than
periodSeconds
.
Create the VM by running the following command:
$ oc create -f <file_name>.yaml
12.6.1.3. Defining an HTTP liveness probe
Define an HTTP liveness probe by setting the spec.livenessProbe.httpGet
field of the virtual machine (VM) configuration. You can define both HTTP and TCP tests for liveness probes in the same way as readiness probes. This procedure configures a sample liveness probe with an HTTP GET test.
Procedure
Include details of the HTTP liveness probe in the VM configuration file.
Sample liveness probe with an HTTP GET test
apiVersion: kubevirt.io/v1 kind: VirtualMachine metadata: annotations: name: fedora-vm namespace: example-namespace # ... spec: template: spec: livenessProbe: initialDelaySeconds: 120 1 periodSeconds: 20 2 httpGet: 3 port: 1500 4 path: /healthz 5 httpHeaders: - name: Custom-Header value: Awesome timeoutSeconds: 10 6 # ...
- 1
- The time, in seconds, after the VM starts before the liveness probe is initiated.
- 2
- The delay, in seconds, between performing probes. The default delay is 10 seconds. This value must be greater than
timeoutSeconds
. - 3
- The HTTP GET request to perform to connect to the VM.
- 4
- The port of the VM that the probe queries. In the above example, the probe queries port 1500. The VM installs and runs a minimal HTTP server on port 1500 via cloud-init.
- 5
- The path to access on the HTTP server. In the above example, if the handler for the server’s
/healthz
path returns a success code, the VM is considered to be healthy. If the handler returns a failure code, the VM is deleted and a new VM is created. - 6
- The number of seconds of inactivity after which the probe times out and the VM is assumed to have failed. The default value is 1. This value must be lower than
periodSeconds
.
Create the VM by running the following command:
$ oc create -f <file_name>.yaml
12.6.2. Defining a watchdog
You can define a watchdog to monitor the health of the guest operating system by performing the following steps:
- Configure a watchdog device for the virtual machine (VM).
- Install the watchdog agent on the guest.
The watchdog device monitors the agent and performs one of the following actions if the guest operating system is unresponsive:
-
poweroff
: The VM powers down immediately. Ifspec.running
is set totrue
orspec.runStrategy
is not set tomanual
, then the VM reboots. reset
: The VM reboots in place and the guest operating system cannot react.NoteThe reboot time might cause liveness probes to time out. If cluster-level protections detect a failed liveness probe, the VM might be forcibly rescheduled, increasing the reboot time.
-
shutdown
: The VM gracefully powers down by stopping all services.
Watchdog is not available for Windows VMs.
12.6.2.1. Configuring a watchdog device for the virtual machine
You configure a watchdog device for the virtual machine (VM).
Prerequisites
-
The VM must have kernel support for an
i6300esb
watchdog device. Red Hat Enterprise Linux (RHEL) images supporti6300esb
.
Procedure
Create a
YAML
file with the following contents:apiVersion: kubevirt.io/v1 kind: VirtualMachine metadata: labels: kubevirt.io/vm: vm2-rhel84-watchdog name: <vm-name> spec: running: false template: metadata: labels: kubevirt.io/vm: vm2-rhel84-watchdog spec: domain: devices: watchdog: name: <watchdog> i6300esb: action: "poweroff" 1 # ...
- 1
- Specify
poweroff
,reset
, orshutdown
.
The example above configures the
i6300esb
watchdog device on a RHEL8 VM with the poweroff action and exposes the device as/dev/watchdog
.This device can now be used by the watchdog binary.
Apply the YAML file to your cluster by running the following command:
$ oc apply -f <file_name>.yaml
This procedure is provided for testing watchdog functionality only and must not be run on production machines.
Run the following command to verify that the VM is connected to the watchdog device:
$ lspci | grep watchdog -i
Run one of the following commands to confirm the watchdog is active:
Trigger a kernel panic:
# echo c > /proc/sysrq-trigger
Stop the watchdog service:
# pkill -9 watchdog
12.6.2.2. Installing the watchdog agent on the guest
You install the watchdog agent on the guest and start the watchdog
service.
Procedure
- Log in to the virtual machine as root user.
Install the
watchdog
package and its dependencies:# yum install watchdog
Uncomment the following line in the
/etc/watchdog.conf
file and save the changes:#watchdog-device = /dev/watchdog
Enable the
watchdog
service to start on boot:# systemctl enable --now watchdog.service
12.6.3. Defining a guest agent ping probe
Define a guest agent ping probe by setting the spec.readinessProbe.guestAgentPing
field of the virtual machine (VM) configuration.
The guest agent ping probe is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
Prerequisites
- The QEMU guest agent must be installed and enabled on the virtual machine.
Procedure
Include details of the guest agent ping probe in the VM configuration file. For example:
Sample guest agent ping probe
apiVersion: kubevirt.io/v1 kind: VirtualMachine metadata: annotations: name: fedora-vm namespace: example-namespace # ... spec: template: spec: readinessProbe: guestAgentPing: {} 1 initialDelaySeconds: 120 2 periodSeconds: 20 3 timeoutSeconds: 10 4 failureThreshold: 3 5 successThreshold: 3 6 # ...
- 1
- The guest agent ping probe to connect to the VM.
- 2
- Optional: The time, in seconds, after the VM starts before the guest agent probe is initiated.
- 3
- Optional: The delay, in seconds, between performing probes. The default delay is 10 seconds. This value must be greater than
timeoutSeconds
. - 4
- Optional: The number of seconds of inactivity after which the probe times out and the VM is assumed to have failed. The default value is 1. This value must be lower than
periodSeconds
. - 5
- Optional: The number of times that the probe is allowed to fail. The default is 3. After the specified number of attempts, the pod is marked
Unready
. - 6
- Optional: The number of times that the probe must report success, after a failure, to be considered successful. The default is 1.
Create the VM by running the following command:
$ oc create -f <file_name>.yaml
12.6.4. Additional resources
12.7. OpenShift Virtualization runbooks
Runbooks for the OpenShift Virtualization Operator are maintained in the openshift/runbooks Git repository, and you can view them on GitHub. To diagnose and resolve issues that trigger OpenShift Virtualization alerts, follow the procedures in the runbooks.
OpenShift Virtualization alerts are displayed in the Virtualization
12.7.1. CDIDataImportCronOutdated
-
View the runbook for the
CDIDataImportCronOutdated
alert.
12.7.2. CDIDataVolumeUnusualRestartCount
-
View the runbook for the
CDIDataVolumeUnusualRestartCount
alert.
12.7.3. CDIDefaultStorageClassDegraded
-
View the runbook for the
CDIDefaultStorageClassDegraded
alert.
12.7.4. CDIMultipleDefaultVirtStorageClasses
-
View the runbook for the
CDIMultipleDefaultVirtStorageClasses
alert.
12.7.5. CDINoDefaultStorageClass
-
View the runbook for the
CDINoDefaultStorageClass
alert.
12.7.6. CDINotReady
-
View the runbook for the
CDINotReady
alert.
12.7.7. CDIOperatorDown
-
View the runbook for the
CDIOperatorDown
alert.
12.7.8. CDIStorageProfilesIncomplete
-
View the runbook for the
CDIStorageProfilesIncomplete
alert.
12.7.9. CnaoDown
-
View the runbook for the
CnaoDown
alert.
12.7.10. CnaoNMstateMigration
-
View the runbook for the
CnaoNMstateMigration
alert.
12.7.11. HCOInstallationIncomplete
-
View the runbook for the
HCOInstallationIncomplete
alert.
12.7.12. HPPNotReady
-
View the runbook for the
HPPNotReady
alert.
12.7.13. HPPOperatorDown
-
View the runbook for the
HPPOperatorDown
alert.
12.7.14. HPPSharingPoolPathWithOS
-
View the runbook for the
HPPSharingPoolPathWithOS
alert.
12.7.15. KubemacpoolDown
-
View the runbook for the
KubemacpoolDown
alert.
12.7.16. KubeMacPoolDuplicateMacsFound
-
View the runbook for the
KubeMacPoolDuplicateMacsFound
alert.
12.7.17. KubeVirtComponentExceedsRequestedCPU
-
The
KubeVirtComponentExceedsRequestedCPU
alert is deprecated.
12.7.18. KubeVirtComponentExceedsRequestedMemory
-
The
KubeVirtComponentExceedsRequestedMemory
alert is deprecated.
12.7.19. KubeVirtCRModified
-
View the runbook for the
KubeVirtCRModified
alert.
12.7.20. KubeVirtDeprecatedAPIRequested
-
View the runbook for the
KubeVirtDeprecatedAPIRequested
alert.
12.7.21. KubeVirtNoAvailableNodesToRunVMs
-
View the runbook for the
KubeVirtNoAvailableNodesToRunVMs
alert.
12.7.22. KubevirtVmHighMemoryUsage
-
View the runbook for the
KubevirtVmHighMemoryUsage
alert.
12.7.23. KubeVirtVMIExcessiveMigrations
-
View the runbook for the
KubeVirtVMIExcessiveMigrations
alert.
12.7.24. LowKVMNodesCount
-
View the runbook for the
LowKVMNodesCount
alert.
12.7.25. LowReadyVirtControllersCount
-
View the runbook for the
LowReadyVirtControllersCount
alert.
12.7.26. LowReadyVirtOperatorsCount
-
View the runbook for the
LowReadyVirtOperatorsCount
alert.
12.7.27. LowVirtAPICount
-
View the runbook for the
LowVirtAPICount
alert.
12.7.28. LowVirtControllersCount
-
View the runbook for the
LowVirtControllersCount
alert.
12.7.29. LowVirtOperatorCount
-
View the runbook for the
LowVirtOperatorCount
alert.
12.7.30. NetworkAddonsConfigNotReady
-
View the runbook for the
NetworkAddonsConfigNotReady
alert.
12.7.31. NoLeadingVirtOperator
-
View the runbook for the
NoLeadingVirtOperator
alert.
12.7.32. NoReadyVirtController
-
View the runbook for the
NoReadyVirtController
alert.
12.7.33. NoReadyVirtOperator
-
View the runbook for the
NoReadyVirtOperator
alert.
12.7.34. OrphanedVirtualMachineInstances
-
View the runbook for the
OrphanedVirtualMachineInstances
alert.
12.7.35. OutdatedVirtualMachineInstanceWorkloads
-
View the runbook for the
OutdatedVirtualMachineInstanceWorkloads
alert.
12.7.36. SingleStackIPv6Unsupported
-
View the runbook for the
SingleStackIPv6Unsupported
alert.
12.7.37. SSPCommonTemplatesModificationReverted
-
View the runbook for the
SSPCommonTemplatesModificationReverted
alert.
12.7.38. SSPDown
-
View the runbook for the
SSPDown
alert.
12.7.39. SSPFailingToReconcile
-
View the runbook for the
SSPFailingToReconcile
alert.
12.7.40. SSPHighRateRejectedVms
-
View the runbook for the
SSPHighRateRejectedVms
alert.
12.7.41. SSPTemplateValidatorDown
-
View the runbook for the
SSPTemplateValidatorDown
alert.
12.7.42. UnsupportedHCOModification
-
View the runbook for the
UnsupportedHCOModification
alert.
12.7.43. VirtAPIDown
-
View the runbook for the
VirtAPIDown
alert.
12.7.44. VirtApiRESTErrorsBurst
-
View the runbook for the
VirtApiRESTErrorsBurst
alert.
12.7.45. VirtApiRESTErrorsHigh
-
View the runbook for the
VirtApiRESTErrorsHigh
alert.
12.7.46. VirtControllerDown
-
View the runbook for the
VirtControllerDown
alert.
12.7.47. VirtControllerRESTErrorsBurst
-
View the runbook for the
VirtControllerRESTErrorsBurst
alert.
12.7.48. VirtControllerRESTErrorsHigh
-
View the runbook for the
VirtControllerRESTErrorsHigh
alert.
12.7.49. VirtHandlerDaemonSetRolloutFailing
-
View the runbook for the
VirtHandlerDaemonSetRolloutFailing
alert.
12.7.50. VirtHandlerRESTErrorsBurst
-
View the runbook for the
VirtHandlerRESTErrorsBurst
alert.
12.7.51. VirtHandlerRESTErrorsHigh
-
View the runbook for the
VirtHandlerRESTErrorsHigh
alert.
12.7.52. VirtOperatorDown
-
View the runbook for the
VirtOperatorDown
alert.
12.7.53. VirtOperatorRESTErrorsBurst
-
View the runbook for the
VirtOperatorRESTErrorsBurst
alert.
12.7.54. VirtOperatorRESTErrorsHigh
-
View the runbook for the
VirtOperatorRESTErrorsHigh
alert.
12.7.55. VirtualMachineCRCErrors
The runbook for the
VirtualMachineCRCErrors
alert is deprecated because the alert was renamed toVMStorageClassWarning
.-
View the runbook for the
VMStorageClassWarning
alert.
-
View the runbook for the
12.7.56. VMCannotBeEvicted
-
View the runbook for the
VMCannotBeEvicted
alert.
12.7.57. VMStorageClassWarning
-
View the runbook for the
VMStorageClassWarning
alert.