Red Hat Summit Red Hat SummitSupportConsoleDevelopersStart a trialContact

Chapter 15. Monitoring

15.1. Monitoring overview
Copy link

You can monitor the health of your cluster and virtual machines (VMs) with the following tools:

Monitoring OpenShift Virtualization VM health status
View the overall health of your OpenShift Virtualization environment in the web console by navigating to the Home Overview page in the OpenShift Container Platform web console. The Status card displays the overall health of OpenShift Virtualization based on the alerts and conditions.
OpenShift Container Platform cluster checkup framework

Run automated tests on your cluster with the OpenShift Container Platform cluster checkup framework to check the following conditions:

  • Network connectivity and latency between two VMs attached to a secondary network interface
  • VM running a Data Plane Development Kit (DPDK) workload with zero packet loss
  • Cluster storage is optimally configured for OpenShift Virtualization
Prometheus queries for virtual resources
Query vCPU, network, storage, and guest memory swapping usage and live migration progress.
VM custom metrics
Configure the node-exporter service to expose internal VM metrics and processes.
VM health checks
Configure readiness, liveness, and guest agent ping probes and a watchdog for VMs.
Runbooks
Diagnose and resolve issues that trigger OpenShift Virtualization alerts in the OpenShift Container Platform web console.

A checkup is an automated test workload that allows you to verify if a specific cluster functionality works as expected. The cluster checkup framework uses native Kubernetes resources to configure and execute the checkup.

Important

The OpenShift Virtualization cluster checkup framework is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

As a developer or cluster administrator, you can use predefined checkups to improve cluster maintainability, troubleshoot unexpected behavior, minimize errors, and save time. You can review the results of the checkup and share them with experts for further analysis. Vendors can write and publish checkups for features or services that they provide and verify that their customer environments are configured correctly.

15.2.1. Running predefined latency checkups
Copy link

You can use a latency checkup to verify network connectivity and measure latency between two virtual machines (VMs) that are attached to a secondary network interface. The predefined latency checkup uses the ping utility.

Important

Before you run a latency checkup, you must first create a bridge interface on the cluster nodes to connect the VM’s secondary interface to any interface on the node. If you do not create a bridge interface, the VMs do not start and the job fails.

Running a predefined checkup in an existing namespace involves setting up a service account for the checkup, creating the Role and RoleBinding objects for the service account, enabling permissions for the checkup, and creating the input config map and the checkup job. You can run a checkup multiple times.

Important

You must always:

  • Verify that the checkup image is from a trustworthy source before applying it.
  • Review the checkup permissions before creating the Role and RoleBinding objects.

Run a latency checkup to verify network connectivity and measure the latency between two virtual machines attached to a secondary network interface.

Prerequisites

  • You must add a NetworkAttachmentDefinition to the namespace.

Procedure

  1. Navigate to Virtualization Checkups in the web console.
  2. Click the Network latency tab.
  3. Click Install permissions.
  4. Click Run checkup.
  5. Enter a name for the checkup in the Name field.
  6. Select a NetworkAttachmentDefinition from the drop-down menu.
  7. Optional: Set a duration for the latency sample in the Sample duration (seconds) field.
  8. Optional: Define a maximum latency time interval by enabling Set maximum desired latency (milliseconds) and defining the time interval.
  9. Optional: Target specific nodes by enabling Select nodes and specifying the Source node and Target node.
  10. Click Run.

Verification

  • To view the status of the latency checkup, go to the Checkups list on the Latency checkup tab. Click on the name of the checkup for more details.

You run a latency checkup using the CLI by performing the following steps:

  1. Create a service account, roles, and rolebindings to provide cluster access permissions to the latency checkup.
  2. Create a config map to provide the input to run the checkup and to store the results.
  3. Create a job to run the checkup.
  4. Review the results in the config map.
  5. Optional: To rerun the checkup, delete the existing config map and job and then create a new config map and job.
  6. When you are finished, delete the latency checkup resources.

Prerequisites

  • You installed the OpenShift CLI (oc).
  • The cluster has at least two worker nodes.
  • You configured a network attachment definition for a namespace.

Procedure

  1. Create a ServiceAccount, Role, and RoleBinding manifest for the latency checkup:

    Example 15.1. Example role manifest file

    ---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: vm-latency-checkup-sa
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: kubevirt-vm-latency-checker
rules:
- apiGroups: ["kubevirt.io"]
  resources: ["virtualmachineinstances"]
  verbs: ["get", "create", "delete"]
- apiGroups: ["subresources.kubevirt.io"]
  resources: ["virtualmachineinstances/console"]
  verbs: ["get"]
- apiGroups: ["k8s.cni.cncf.io"]
  resources: ["network-attachment-definitions"]
  verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: kubevirt-vm-latency-checker
subjects:
- kind: ServiceAccount
  name: vm-latency-checkup-sa
roleRef:
  kind: Role
  name: kubevirt-vm-latency-checker
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: kiagnose-configmap-access
rules:
- apiGroups: [ "" ]
  resources: [ "configmaps" ]
  verbs: ["get", "update"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: kiagnose-configmap-access
subjects:
- kind: ServiceAccount
  name: vm-latency-checkup-sa
roleRef:
  kind: Role
  name: kiagnose-configmap-access
  apiGroup: rbac.authorization.k8s.io
    Copy to Clipboard Toggle word wrap

  2. Apply the ServiceAccount, Role, and RoleBinding manifest: 

    $ oc apply -n <target_namespace> -f <latency_sa_roles_rolebinding>.yaml
    1
    Copy to Clipboard Toggle word wrap
    1
    <target_namespace> is the namespace where the checkup is to be run. This must be an existing namespace where the NetworkAttachmentDefinition object resides.

  3. Create a ConfigMap manifest that contains the input parameters for the checkup:

    Example input config map

    apiVersion: v1
kind: ConfigMap
metadata:
  name: kubevirt-vm-latency-checkup-config
  labels:
    kiagnose/checkup-type: kubevirt-vm-latency
data:
  spec.timeout: 5m
  spec.param.networkAttachmentDefinitionNamespace: <target_namespace>
  spec.param.networkAttachmentDefinitionName: "blue-network"
    1 
    
  spec.param.maxDesiredLatencyMilliseconds: "10"
    2 
    
  spec.param.sampleDurationSeconds: "5"
    3 
    
  spec.param.sourceNode: "worker1"
    4 
    
  spec.param.targetNode: "worker2"
    5
    Copy to Clipboard Toggle word wrap

    1
    The name of the NetworkAttachmentDefinition object.
    2
    Optional: The maximum desired latency, in milliseconds, between the virtual machines. If the measured latency exceeds this value, the checkup fails.
    3
    Optional: The duration of the latency check, in seconds.
    4
    Optional: When specified, latency is measured from this node to the target node. If the source node is specified, the spec.param.targetNode field cannot be empty.
    5
    Optional: When specified, latency is measured from the source node to this node.

  4. Apply the config map manifest in the target namespace: 

    $ oc apply -n <target_namespace> -f <latency_config_map>.yaml
    Copy to Clipboard Toggle word wrap

  5. Create a Job manifest to run the checkup:

    Example job manifest

    apiVersion: batch/v1
kind: Job
metadata:
  name: kubevirt-vm-latency-checkup
  labels:
    kiagnose/checkup-type: kubevirt-vm-latency
spec:
  backoffLimit: 0
  template:
    spec:
      serviceAccountName: vm-latency-checkup-sa
      restartPolicy: Never
      containers:
        - name: vm-latency-checkup
          image: registry.redhat.io/container-native-virtualization/vm-network-latency-checkup-rhel9:v4.19.0
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]
            runAsNonRoot: true
            seccompProfile:
              type: "RuntimeDefault"
          env:
            - name: CONFIGMAP_NAMESPACE
              value: <target_namespace>
            - name: CONFIGMAP_NAME
              value: kubevirt-vm-latency-checkup-config
            - name: POD_UID
              valueFrom:
                fieldRef:
                  fieldPath: metadata.uid
    Copy to Clipboard Toggle word wrap

  6. Apply the Job manifest: 

    $ oc apply -n <target_namespace> -f <latency_job>.yaml
    Copy to Clipboard Toggle word wrap

  7. Wait for the job to complete: 

    $ oc wait job kubevirt-vm-latency-checkup -n <target_namespace> --for condition=complete --timeout 6m
    Copy to Clipboard Toggle word wrap

  8. Review the results of the latency checkup by running the following command. If the maximum measured latency is greater than the value of the spec.param.maxDesiredLatencyMilliseconds attribute, the checkup fails and returns an error. 

    $ oc get configmap kubevirt-vm-latency-checkup-config -n <target_namespace> -o yaml
    Copy to Clipboard Toggle word wrap

    Example output config map (success)

    apiVersion: v1
kind: ConfigMap
metadata:
  name: kubevirt-vm-latency-checkup-config
  namespace: <target_namespace>
  labels:
    kiagnose/checkup-type: kubevirt-vm-latency
data:
  spec.timeout: 5m
  spec.param.networkAttachmentDefinitionNamespace: <target_namespace>
  spec.param.networkAttachmentDefinitionName: "blue-network"
  spec.param.maxDesiredLatencyMilliseconds: "10"
  spec.param.sampleDurationSeconds: "5"
  spec.param.sourceNode: "worker1"
  spec.param.targetNode: "worker2"
  status.succeeded: "true"
  status.failureReason: ""
  status.completionTimestamp: "2022-01-01T09:00:00Z"
  status.startTimestamp: "2022-01-01T09:00:07Z"
  status.result.avgLatencyNanoSec: "177000"
  status.result.maxLatencyNanoSec: "244000"
    1 
    
  status.result.measurementDurationSec: "5"
  status.result.minLatencyNanoSec: "135000"
  status.result.sourceNode: "worker1"
  status.result.targetNode: "worker2"
    Copy to Clipboard Toggle word wrap

    1
    The maximum measured latency in nanoseconds.

  9. Optional: To view the detailed job log in case of checkup failure, use the following command: 

    $ oc logs job.batch/kubevirt-vm-latency-checkup -n <target_namespace>
    Copy to Clipboard Toggle word wrap

  10. Delete the job and config map that you previously created by running the following commands: 

    $ oc delete job -n <target_namespace> kubevirt-vm-latency-checkup
    Copy to Clipboard Toggle word wrap 
    $ oc delete config-map -n <target_namespace> kubevirt-vm-latency-checkup-config
    Copy to Clipboard Toggle word wrap

  11. Optional: If you do not plan to run another checkup, delete the roles manifest: 

    $ oc delete -f <latency_sa_roles_rolebinding>.yaml
    Copy to Clipboard Toggle word wrap

15.2.2. Running predefined storage checkups
Copy link

You can use a storage checkup to verify that the cluster storage is optimally configured for OpenShift Virtualization.

Running a predefined checkup in an existing namespace involves setting up a service account for the checkup, creating the Role and RoleBinding objects for the service account, enabling permissions for the checkup, and creating the input config map and the checkup job. You can run a checkup multiple times.

Important

You must always:

  • Verify that the checkup image is from a trustworthy source before applying it.
  • Review the checkup permissions before creating the Role and RoleBinding objects.

The predefined storage checkup includes skipTeardown configuration options, which control resource clean up after a storage checkup runs. By default, the skipTeardown field value is Never, which means that the checkup always performs teardown steps and deletes all resources after the checkup runs.

You can retain resources for further inspection in case a failure occurs by setting the skipTeardown field to onfailure.

Prerequisites

  • You have installed the OpenShift CLI (oc).

Procedure

  1. Run the following command to edit the storage-checkup-config config map: 

    $ oc edit configmap storage-checkup-config -n <checkup_namespace>
    Copy to Clipboard Toggle word wrap

  2. Configure the skipTeardown field to use the onfailure value. You can do this by modifying the storage-checkup-config config map, stored in the storage_checkup.yaml file: 

    apiVersion: v1
kind: ConfigMap
metadata:
  name: storage-checkup-config
  namespace: <checkup_namespace>
data:
  spec.param.skipTeardown: onfailure
# ...
    Copy to Clipboard Toggle word wrap

  3. Reapply the storage-checkup-config config map by running the following command: 

    $ oc apply -f storage_checkup.yaml -n <checkup_namespace>
    Copy to Clipboard Toggle word wrap

Run a storage checkup to validate that storage is working correctly for virtual machines.

Procedure

  1. Navigate to Virtualization Checkups in the web console.
  2. Click the Storage tab.
  3. Click Install permissions.
  4. Click Run checkup.
  5. Enter a name for the checkup in the Name field.
  6. Enter a timeout value for the checkup in the Timeout (minutes) fields.
  7. Click Run.

You can view the status of the storage checkup in the Checkups list on the Storage tab. Click on the name of the checkup for more details.

Use a predefined checkup to verify that the OpenShift Container Platform cluster storage is configured optimally to run OpenShift Virtualization workloads.

Prerequisites

  • You have installed the OpenShift CLI (oc).

  • The cluster administrator has created the required cluster-reader permissions for the storage checkup service account and namespace, such as in the following example: 

    apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubevirt-storage-checkup-clustereader
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-reader
subjects:
- kind: ServiceAccount
  name: storage-checkup-sa
  namespace: <target_namespace>
    1
    Copy to Clipboard Toggle word wrap
    1
    The namespace where the checkup is to be run.

Procedure

  1. Create a ServiceAccount, Role, and RoleBinding manifest file for the storage checkup:

    Example 15.2. Example service account, role, and rolebinding manifest

    ---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: storage-checkup-sa
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: storage-checkup-role
rules:
  - apiGroups: [ "" ]
    resources: [ "configmaps" ]
    verbs: ["get", "update"]
  - apiGroups: [ "kubevirt.io" ]
    resources: [ "virtualmachines" ]
    verbs: [ "create", "delete" ]
  - apiGroups: [ "kubevirt.io" ]
    resources: [ "virtualmachineinstances" ]
    verbs: [ "get" ]
  - apiGroups: [ "subresources.kubevirt.io" ]
    resources: [ "virtualmachineinstances/addvolume", "virtualmachineinstances/removevolume" ]
    verbs: [ "update" ]
  - apiGroups: [ "kubevirt.io" ]
    resources: [ "virtualmachineinstancemigrations" ]
    verbs: [ "create" ]
  - apiGroups: [ "cdi.kubevirt.io" ]
    resources: [ "datavolumes" ]
    verbs: [ "create", "delete" ]
  - apiGroups: [ "" ]
    resources: [ "persistentvolumeclaims" ]
    verbs: [ "delete" ]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: storage-checkup-role
subjects:
  - kind: ServiceAccount
    name: storage-checkup-sa
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: storage-checkup-role
    Copy to Clipboard Toggle word wrap

  2. Apply the ServiceAccount, Role, and RoleBinding manifest in the target namespace: 

    $ oc apply -n <target_namespace> -f <storage_sa_roles_rolebinding>.yaml
    Copy to Clipboard Toggle word wrap

  3. Create a ConfigMap and Job manifest file. The config map contains the input parameters for the checkup job.

    Example input config map and job manifest

    ---
apiVersion: v1
kind: ConfigMap
metadata:
  name: storage-checkup-config
  namespace: $CHECKUP_NAMESPACE
data:
  spec.timeout: 10m
  spec.param.storageClass: ocs-storagecluster-ceph-rbd-virtualization
  spec.param.vmiTimeout: 3m
---
apiVersion: batch/v1
kind: Job
metadata:
  name: storage-checkup
  namespace: $CHECKUP_NAMESPACE
spec:
  backoffLimit: 0
  template:
    spec:
      serviceAccount: storage-checkup-sa
      restartPolicy: Never
      containers:
        - name: storage-checkup
          image: quay.io/kiagnose/kubevirt-storage-checkup:main
          imagePullPolicy: Always
          env:
            - name: CONFIGMAP_NAMESPACE
              value: $CHECKUP_NAMESPACE
            - name: CONFIGMAP_NAME
              value: storage-checkup-config
    Copy to Clipboard Toggle word wrap

  4. Apply the ConfigMap and Job manifest file in the target namespace to run the checkup: 

    $ oc apply -n <target_namespace> -f <storage_configmap_job>.yaml
    Copy to Clipboard Toggle word wrap

  5. Wait for the job to complete: 

    $ oc wait job storage-checkup -n <target_namespace> --for condition=complete --timeout 10m
    Copy to Clipboard Toggle word wrap

  6. Review the results of the checkup by running the following command: 

    $ oc get configmap storage-checkup-config -n <target_namespace> -o yaml
    Copy to Clipboard Toggle word wrap

    Example output config map (success)

    apiVersion: v1
kind: ConfigMap
metadata:
  name: storage-checkup-config
  labels:
    kiagnose/checkup-type: kubevirt-storage
data:
  spec.timeout: 10m
  status.succeeded: "true"
    1 
    
  status.failureReason: ""
    2 
    
  status.startTimestamp: "2023-07-31T13:14:38Z"
    3 
    
  status.completionTimestamp: "2023-07-31T13:19:41Z"
    4 
    
  status.result.cnvVersion: 4.19.2
    5 
    
  status.result.defaultStorageClass: trident-nfs
    6 
    
  status.result.goldenImagesNoDataSource: <data_import_cron_list>
    7 
    
  status.result.goldenImagesNotUpToDate: <data_import_cron_list>
    8 
    
  status.result.ocpVersion: 4.19.0
    9 
    
  status.result.pvcBound: "true"
    10 
    
  status.result.storageProfileMissingVolumeSnapshotClass: <storage_class_list>
    11 
    
  status.result.storageProfilesWithEmptyClaimPropertySets: <storage_profile_list>
    12 
    
  status.result.storageProfilesWithSmartClone: <storage_profile_list>
    13 
    
  status.result.storageProfilesWithSpecClaimPropertySets: <storage_profile_list>
    14 
    
  status.result.storageProfilesWithRWX: |-
    ocs-storagecluster-ceph-rbd
    ocs-storagecluster-ceph-rbd-virtualization
    ocs-storagecluster-cephfs
    trident-iscsi
    trident-minio
    trident-nfs
    windows-vms
  status.result.vmBootFromGoldenImage: VMI "vmi-under-test-dhkb8" successfully booted
  status.result.vmHotplugVolume: |-
    VMI "vmi-under-test-dhkb8" hotplug volume ready
    VMI "vmi-under-test-dhkb8" hotplug volume removed
  status.result.vmLiveMigration: VMI "vmi-under-test-dhkb8" migration completed
  status.result.vmVolumeClone: 'DV cloneType: "csi-clone"'
  status.result.vmsWithNonVirtRbdStorageClass: <vm_list>
    15 
    
  status.result.vmsWithUnsetEfsStorageClass: <vm_list>
    16
    Copy to Clipboard Toggle word wrap

    1
    Specifies if the checkup is successful (true) or not (false).
    2
    The reason for failure if the checkup fails.
    3
    The time when the checkup started, in RFC 3339 time format.
    4
    The time when the checkup has completed, in RFC 3339 time format.
    5
    The OpenShift Virtualization version.
    6
    Specifies if there is a default storage class.
    7
    The list of golden images whose data source is not ready.
    8
    The list of golden images whose data import cron is not up-to-date.
    9
    The OpenShift Container Platform version.
    10
    Specifies if a PVC of 10Mi has been created and bound by the provisioner.
    11
    The list of storage profiles using snapshot-based clone but missing VolumeSnapshotClass.
    12
    The list of storage profiles with unknown provisioners.
    13
    The list of storage profiles with smart clone support (CSI/snapshot).
    14
    The list of storage profiles spec-overriden claimPropertySets.
    15
    The list of virtual machines that use the Ceph RBD storage class when the virtualization storage class exists.
    16
    The list of virtual machines that use an Elastic File Store (EFS) storage class where the GID and UID are not set in the storage class.

  7. Delete the job and config map that you previously created by running the following commands: 

    $ oc delete job -n <target_namespace> storage-checkup
    Copy to Clipboard Toggle word wrap 
    $ oc delete config-map -n <target_namespace> storage-checkup-config
    Copy to Clipboard Toggle word wrap

  8. Optional: If you do not plan to run another checkup, delete the ServiceAccount, Role, and RoleBinding manifest: 

    $ oc delete -f <storage_sa_roles_rolebinding>.yaml
    Copy to Clipboard Toggle word wrap

15.2.2.4. Troubleshooting a failed storage checkup
Copy link

If a storage checkup fails, there are steps that you can take to identify the reason for failure.

Prerequisites

  • You have installed the OpenShift CLI (oc).
  • You have downloaded the directory provided by the must-gather tool.

Procedure

  1. Review the status.failureReason field in the storage-checkup-config config map by running the following command and observing the output: 

    $ oc get configmap storage-checkup-config -n <namespace> -o yaml
    Copy to Clipboard Toggle word wrap

    Example output config map

    apiVersion: v1
kind: ConfigMap
metadata:
  name: storage-checkup-config
  labels:
    kiagnose/checkup-type: kubevirt-storage
data:
  spec.timeout: 10m
  status.succeeded: "false"
    1 
    
  status.failureReason: "ErrNoDefaultStorageClass"
    2 
    
# ...
    Copy to Clipboard Toggle word wrap

    1
    If the checkup has failed, the status.succeeded value is false.
    2
    If the checkup has failed, the status.failureReason field contains an error message. In this example output, the ErrNoDefaultStorageClass error message means that no default storage class is configured.
  2. Search the directory provided by the must-gather tool for logs, events, or terms related to the error in the data.status.failureReason field value.

15.2.2.5. Storage checkup error codes
Copy link

The following error codes might appear in the storage-checkup-config config map after a storage checkup fails.

Expand
Error codeMeaning

ErrNoDefaultStorageClass

No default storage class is configured.

ErrPvcNotBound

One or more persistent volume claims (PVCs) failed to bind.

ErrMultipleDefaultStorageClasses

Multiple default storage classes are configured.

ErrEmptyClaimPropertySets

There are StorageProfile objects containing empty ClaimPropertySets specs.

ErrVMsWithUnsetEfsStorageClass

There are VMs using elastic file system (EFS) storage classes, where the GID and UID are not set in the StorageClass object.

ErrGoldenImagesNotUpToDate

One or more golden images has a DataImportCron object that is either not up to date or has a DataSource object which is not ready.

ErrGoldenImageNoDataSource

The DataSource object of the golden image has either no PVC or no snapshot source configured.

ErrBootFailedOnSomeVMs

Some VMs failed to boot within the expected time.

15.2.3. Running predefined DPDK checkups
Copy link

You can use a DPDK checkup to verify that a node can run a VM with a Data Plane Development Kit (DPDK) workload with zero packet loss.

15.2.3.1. Running a DPDK checkup by using the CLI
Copy link

Use a predefined checkup to verify that your OpenShift Container Platform cluster node can run a virtual machine (VM) with a Data Plane Development Kit (DPDK) workload with zero packet loss. The DPDK checkup runs traffic between a traffic generator and a VM running a test DPDK application.

You run a DPDK checkup by performing the following steps:

  1. Create a service account, role, and role bindings for the DPDK checkup.
  2. Create a config map to provide the input to run the checkup and to store the results.
  3. Create a job to run the checkup.
  4. Review the results in the config map.
  5. Optional: To rerun the checkup, delete the existing config map and job and then create a new config map and job.
  6. When you are finished, delete the DPDK checkup resources.

Prerequisites

  • You have installed the OpenShift CLI (oc).
  • The cluster is configured to run DPDK applications.
  • The project is configured to run DPDK applications.

Procedure

  1. Create a ServiceAccount, Role, and RoleBinding manifest for the DPDK checkup:

    Example 15.3. Example service account, role, and rolebinding manifest file

    ---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: dpdk-checkup-sa
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: kiagnose-configmap-access
rules:
  - apiGroups: [ "" ]
    resources: [ "configmaps" ]
    verbs: [ "get", "update" ]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: kiagnose-configmap-access
subjects:
  - kind: ServiceAccount
    name: dpdk-checkup-sa
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: kiagnose-configmap-access
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: kubevirt-dpdk-checker
rules:
  - apiGroups: [ "kubevirt.io" ]
    resources: [ "virtualmachineinstances" ]
    verbs: [ "create", "get", "delete" ]
  - apiGroups: [ "subresources.kubevirt.io" ]
    resources: [ "virtualmachineinstances/console" ]
    verbs: [ "get" ]
  - apiGroups: [ "" ]
    resources: [ "configmaps" ]
    verbs: [ "create", "delete" ]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: kubevirt-dpdk-checker
subjects:
  - kind: ServiceAccount
    name: dpdk-checkup-sa
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: kubevirt-dpdk-checker
    Copy to Clipboard Toggle word wrap

  2. Apply the ServiceAccount, Role, and RoleBinding manifest: 

    $ oc apply -n <target_namespace> -f <dpdk_sa_roles_rolebinding>.yaml
    Copy to Clipboard Toggle word wrap

  3. Create a ConfigMap manifest that contains the input parameters for the checkup:

    Example input config map

    apiVersion: v1
kind: ConfigMap
metadata:
  name: dpdk-checkup-config
  labels:
    kiagnose/checkup-type: kubevirt-dpdk
data:
  spec.timeout: 10m
  spec.param.networkAttachmentDefinitionName: <network_name>
    1 
    
  spec.param.trafficGenContainerDiskImage: "quay.io/kiagnose/kubevirt-dpdk-checkup-traffic-gen:v0.4.0
    2 
    
  spec.param.vmUnderTestContainerDiskImage: "quay.io/kiagnose/kubevirt-dpdk-checkup-vm:v0.4.0"
    3
    Copy to Clipboard Toggle word wrap

    1
    The name of the NetworkAttachmentDefinition object.
    2
    The container disk image for the traffic generator. In this example, the image is pulled from the upstream Project Quay Container Registry.
    3
    The container disk image for the VM under test. In this example, the image is pulled from the upstream Project Quay Container Registry.

  4. Apply the ConfigMap manifest in the target namespace: 

    $ oc apply -n <target_namespace> -f <dpdk_config_map>.yaml
    Copy to Clipboard Toggle word wrap

  5. Create a Job manifest to run the checkup:

    Example job manifest

    apiVersion: batch/v1
kind: Job
metadata:
  name: dpdk-checkup
  labels:
    kiagnose/checkup-type: kubevirt-dpdk
spec:
  backoffLimit: 0
  template:
    spec:
      serviceAccountName: dpdk-checkup-sa
      restartPolicy: Never
      containers:
        - name: dpdk-checkup
          image: registry.redhat.io/container-native-virtualization/kubevirt-dpdk-checkup-rhel9:v4.19.0
          imagePullPolicy: Always
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]
            runAsNonRoot: true
            seccompProfile:
              type: "RuntimeDefault"
          env:
            - name: CONFIGMAP_NAMESPACE
              value: <target-namespace>
            - name: CONFIGMAP_NAME
              value: dpdk-checkup-config
            - name: POD_UID
              valueFrom:
                fieldRef:
                  fieldPath: metadata.uid
    Copy to Clipboard Toggle word wrap

  6. Apply the Job manifest: 

    $ oc apply -n <target_namespace> -f <dpdk_job>.yaml
    Copy to Clipboard Toggle word wrap

  7. Wait for the job to complete: 

    $ oc wait job dpdk-checkup -n <target_namespace> --for condition=complete --timeout 10m
    Copy to Clipboard Toggle word wrap

  8. Review the results of the checkup by running the following command: 

    $ oc get configmap dpdk-checkup-config -n <target_namespace> -o yaml
    Copy to Clipboard Toggle word wrap

    Example output config map (success)

    apiVersion: v1
kind: ConfigMap
metadata:
  name: dpdk-checkup-config
  labels:
    kiagnose/checkup-type: kubevirt-dpdk
data:
  spec.timeout: 10m
  spec.param.NetworkAttachmentDefinitionName: "dpdk-network-1"
  spec.param.trafficGenContainerDiskImage: "quay.io/kiagnose/kubevirt-dpdk-checkup-traffic-gen:v0.4.0"
  spec.param.vmUnderTestContainerDiskImage: "quay.io/kiagnose/kubevirt-dpdk-checkup-vm:v0.4.0"
  status.succeeded: "true"
    1 
    
  status.failureReason: ""
    2 
    
  status.startTimestamp: "2023-07-31T13:14:38Z"
    3 
    
  status.completionTimestamp: "2023-07-31T13:19:41Z"
    4 
    
  status.result.trafficGenSentPackets: "480000000"
    5 
    
  status.result.trafficGenOutputErrorPackets: "0"
    6 
    
  status.result.trafficGenInputErrorPackets: "0"
    7 
    
  status.result.trafficGenActualNodeName: worker-dpdk1
    8 
    
  status.result.vmUnderTestActualNodeName: worker-dpdk2
    9 
    
  status.result.vmUnderTestReceivedPackets: "480000000"
    10 
    
  status.result.vmUnderTestRxDroppedPackets: "0"
    11 
    
  status.result.vmUnderTestTxDroppedPackets: "0"
    12
    Copy to Clipboard Toggle word wrap

    1
    Specifies if the checkup is successful (true) or not (false).
    2
    The reason for failure if the checkup fails.
    3
    The time when the checkup started, in RFC 3339 time format.
    4
    The time when the checkup has completed, in RFC 3339 time format.
    5
    The number of packets sent from the traffic generator.
    6
    The number of error packets sent from the traffic generator.
    7
    The number of error packets received by the traffic generator.
    8
    The node on which the traffic generator VM was scheduled.
    9
    The node on which the VM under test was scheduled.
    10
    The number of packets received on the VM under test.
    11
    The ingress traffic packets that were dropped by the DPDK application.
    12
    The egress traffic packets that were dropped from the DPDK application.

  9. Delete the job and config map that you previously created by running the following commands: 

    $ oc delete job -n <target_namespace> dpdk-checkup
    Copy to Clipboard Toggle word wrap 
    $ oc delete config-map -n <target_namespace> dpdk-checkup-config
    Copy to Clipboard Toggle word wrap

  10. Optional: If you do not plan to run another checkup, delete the ServiceAccount, Role, and RoleBinding manifest: 

    $ oc delete -f <dpdk_sa_roles_rolebinding>.yaml
    Copy to Clipboard Toggle word wrap
15.2.3.1.1. DPDK checkup config map parameters
Copy link

The following table shows the mandatory and optional parameters that you can set in the data stanza of the input ConfigMap manifest when you run a cluster DPDK readiness checkup:

Expand
Table 15.1. DPDK checkup config map input parameters
ParameterDescriptionIs Mandatory

spec.timeout

The time, in minutes, before the checkup fails.

True

spec.param.networkAttachmentDefinitionName

The name of the NetworkAttachmentDefinition object of the SR-IOV NICs connected.

True

spec.param.trafficGenContainerDiskImage

The container disk image for the traffic generator.

True

spec.param.trafficGenTargetNodeName

The node on which the traffic generator VM is to be scheduled. The node should be configured to allow DPDK traffic.

False

spec.param.trafficGenPacketsPerSecond

The number of packets per second, in kilo (k) or million(m). The default value is 8m.

False

spec.param.vmUnderTestContainerDiskImage

The container disk image for the VM under test.

True

spec.param.vmUnderTestTargetNodeName

The node on which the VM under test is to be scheduled. The node should be configured to allow DPDK traffic.

False

spec.param.testDuration

The duration, in minutes, for which the traffic generator runs. The default value is 5 minutes.

False

spec.param.portBandwidthGbps

The maximum bandwidth of the SR-IOV NIC. The default value is 10Gbps.

False

spec.param.verbose

When set to true, it increases the verbosity of the checkup log. The default value is false.

False

You can build a custom Red Hat Enterprise Linux (RHEL) 9 OS image in qcow2 format and use it to create a container disk image. You can store the container disk image in a registry that is accessible from your cluster and specify the image location in the spec.param.vmContainerDiskImage attribute of the DPDK checkup config map.

To build a container disk image, you must create an image builder virtual machine (VM). The image builder VM is a RHEL 9 VM that can be used to build custom RHEL images.

Prerequisites

  • The image builder VM must run RHEL 9.4 and must have a minimum of 2 CPU cores, 4 GiB RAM, and 20 GB of free space in the /var directory.
  • You have installed the image builder tool and its CLI (composer-cli) on the VM. For more information, see "Additional resources".

  • You have installed the virt-customize tool: 

    # dnf install guestfs-tools
    Copy to Clipboard Toggle word wrap
  • You have installed the Podman CLI tool (podman).

Procedure

  1. Verify that you can build a RHEL 9.4 image: 

    # composer-cli distros list
    Copy to Clipboard Toggle word wrap
    Note

    To run the composer-cli commands as non-root, add your user to the weldr or root groups: 

    # usermod -a -G weldr <user>
    Copy to Clipboard Toggle word wrap 
    $ newgrp weldr
    Copy to Clipboard Toggle word wrap

  2. Enter the following command to create an image blueprint file in TOML format that contains the packages to be installed, kernel customizations, and the services to be disabled during boot time: 

    $ cat << EOF > dpdk-vm.toml
name = "dpdk_image"
description = "Image to use with the DPDK checkup"
version = "0.0.1"
distro = "rhel-9.4"

[[customizations.user]]
name = "root"
password = "redhat"

[[packages]]
name = "dpdk"

[[packages]]
name = "dpdk-tools"

[[packages]]
name = "driverctl"

[[packages]]
name = "tuned-profiles-cpu-partitioning"

[customizations.kernel]
append = "default_hugepagesz=1GB hugepagesz=1G hugepages=1"

[customizations.services]
disabled = ["NetworkManager-wait-online", "sshd"]
EOF
    Copy to Clipboard Toggle word wrap

  3. Push the blueprint file to the image builder tool by running the following command: 

    # composer-cli blueprints push dpdk-vm.toml
    Copy to Clipboard Toggle word wrap

  4. Generate the system image by specifying the blueprint name and output file format. The Universally Unique Identifier (UUID) of the image is displayed when you start the compose process. 

    # composer-cli compose start dpdk_image qcow2
    Copy to Clipboard Toggle word wrap

  5. Wait for the compose process to complete. The compose status must show FINISHED before you can continue to the next step. 

    # composer-cli compose status
    Copy to Clipboard Toggle word wrap

  6. Enter the following command to download the qcow2 image file by specifying its UUID: 

    # composer-cli compose image <UUID>
    Copy to Clipboard Toggle word wrap

  7. Create the customization scripts by running the following commands: 

    $ cat <<EOF >customize-vm
#!/bin/bash

# Setup hugepages mount
mkdir -p /mnt/huge
echo "hugetlbfs /mnt/huge hugetlbfs defaults,pagesize=1GB 0 0" >> /etc/fstab

# Create vfio-noiommu.conf
echo "options vfio enable_unsafe_noiommu_mode=1" > /etc/modprobe.d/vfio-noiommu.conf

# Enable guest-exec,guest-exec-status on the qemu-guest-agent configuration
sed -i 's/\(--allow-rpcs=[^"]*\)/\1,guest-exec-status,guest-exec/' /etc/sysconfig/qemu-ga

# Disable Bracketed-paste mode
echo "set enable-bracketed-paste off" >> /root/.inputrc
EOF
    Copy to Clipboard Toggle word wrap

  8. Use the virt-customize tool to customize the image generated by the image builder tool: 

    $ virt-customize -a <UUID>-disk.qcow2 --run=customize-vm --selinux-relabel
    Copy to Clipboard Toggle word wrap

  9. To create a Dockerfile that contains all the commands to build the container disk image, enter the following command: 

    $ cat << EOF > Dockerfile
FROM scratch
COPY --chown=107:107 <UUID>-disk.qcow2 /disk/
EOF
    Copy to Clipboard Toggle word wrap

    where:

    <UUID>-disk.qcow2
    Specifies the name of the custom image in qcow2 format.

  10. Build and tag the container by running the following command: 

    $ podman build . -t dpdk-rhel:latest
    Copy to Clipboard Toggle word wrap

  11. Push the container disk image to a registry that is accessible from your cluster by running the following command: 

    $ podman push dpdk-rhel:latest
    Copy to Clipboard Toggle word wrap
  12. Provide a link to the container disk image in the spec.param.vmUnderTestContainerDiskImage attribute in the DPDK checkup config map.

15.3. Prometheus queries for virtual resources
Copy link

OpenShift Virtualization provides metrics that you can use to monitor the consumption of cluster infrastructure resources, including vCPU, network, storage, and guest memory swapping. You can also use metrics to query live migration status.

15.3.1. Prerequisites
Copy link

  • To use the vCPU metric, the schedstats=enable kernel argument must be applied to the MachineConfig object. This kernel argument enables scheduler statistics used for debugging and performance tuning and adds a minor additional load to the scheduler. For more information, see Adding kernel arguments to nodes.
  • For guest memory swapping queries to return data, memory swapping must be enabled on the virtual guests.

You can use the OpenShift Container Platform metrics query browser to run Prometheus Query Language (PromQL) queries to examine metrics visualized on a plot. This functionality provides information about the state of a cluster and any user-defined workloads that you are monitoring.

As a cluster administrator or as a user with view permissions for all projects, you can access metrics for all default OpenShift Container Platform and user-defined projects in the Metrics UI.

The Metrics UI includes predefined queries, for example, CPU, memory, bandwidth, or network packet for all projects. You can also run custom Prometheus Query Language (PromQL) queries.

Prerequisites

  • You have access to the cluster as a user with the cluster-admin cluster role or with view permissions for all projects.
  • You have installed the OpenShift CLI (oc).

Procedure

  1. In the OpenShift Container Platform web console, click Observe Metrics.

  2. To add one or more queries, perform any of the following actions:

    Expand
    OptionDescription

    Select an existing query.

    From the Select query drop-down list, select an existing query.

    Create a custom query.

    Add your Prometheus Query Language (PromQL) query to the Expression field.

    As you type a PromQL expression, autocomplete suggestions appear in a drop-down list. These suggestions include functions, metrics, labels, and time tokens. Use the keyboard arrows to select one of these suggested items and then press Enter to add the item to your expression. Move your mouse pointer over a suggested item to view a brief description of that item.

    Add multiple queries.

    Click Add query.

    Duplicate an existing query.

    Click the options menu kebab next to the query, then choose Duplicate query.

    Disable a query from being run.

    Click the options menu kebab next to the query and choose Disable query.

  3. To run queries that you created, click Run queries. The metrics from the queries are visualized on the plot. If a query is invalid, the UI shows an error message.

    Note
    • When drawing time series graphs, queries that operate on large amounts of data might time out or overload the browser. To avoid this, click Hide graph and calibrate your query by using only the metrics table. Then, after finding a feasible query, enable the plot to draw the graphs.
    • By default, the query table shows an expanded view that lists every metric and its current value. Click the ˅ down arrowhead to minimize the expanded view for a query.
  4. Optional: Save the page URL to use this set of queries again in the future.

  5. Explore the visualized metrics. Initially, all metrics from all enabled queries are shown on the plot. Select which metrics are shown by performing any of the following actions:

    Expand
    OptionDescription

    Hide all metrics from a query.

    Click the options menu kebab for the query and click Hide all series.

    Hide a specific metric.

    Go to the query table and click the colored square near the metric name.

    Zoom into the plot and change the time range.

    Perform one of the following actions:

    • Visually select the time range by clicking and dragging on the plot horizontally.
    • Use the menu to select the time range.

    Reset the time range.

    Click Reset zoom.

    Display outputs for all queries at a specific point in time.

    Hover over the plot at the point you are interested in. The query outputs appear in a pop-up box.

    Hide the plot.

    Click Hide graph.

You can use the OpenShift Container Platform metrics query browser to run Prometheus Query Language (PromQL) queries to examine metrics visualized on a plot. This functionality provides information about any user-defined workloads that you are monitoring.

As a developer, you must specify a project name when querying metrics. You must have the required privileges to view metrics for the selected project.

The Metrics UI includes predefined queries, for example, CPU, memory, bandwidth, or network packet. These queries are restricted to the selected project. You can also run custom Prometheus Query Language (PromQL) queries for the project.

Prerequisites

  • You have access to the cluster as a developer or as a user with view permissions for the project that you are viewing metrics for.
  • You have enabled monitoring for user-defined projects.
  • You have deployed a service in a user-defined project.
  • You have created a ServiceMonitor custom resource definition (CRD) for the service to define how the service is monitored.

Procedure

  1. In the OpenShift Container Platform web console, click Observe Metrics.

  2. To add one or more queries, perform any of the following actions:

    Expand
    OptionDescription

    Select an existing query.

    From the Select query drop-down list, select an existing query.

    Create a custom query.

    Add your Prometheus Query Language (PromQL) query to the Expression field.

    As you type a PromQL expression, autocomplete suggestions appear in a drop-down list. These suggestions include functions, metrics, labels, and time tokens. Use the keyboard arrows to select one of these suggested items and then press Enter to add the item to your expression. Move your mouse pointer over a suggested item to view a brief description of that item.

    Add multiple queries.

    Click Add query.

    Duplicate an existing query.

    Click the options menu kebab next to the query, then choose Duplicate query.

    Disable a query from being run.

    Click the options menu kebab next to the query and choose Disable query.

  3. To run queries that you created, click Run queries. The metrics from the queries are visualized on the plot. If a query is invalid, the UI shows an error message.

    Note
    • When drawing time series graphs, queries that operate on large amounts of data might time out or overload the browser. To avoid this, click Hide graph and calibrate your query by using only the metrics table. Then, after finding a feasible query, enable the plot to draw the graphs.
    • By default, the query table shows an expanded view that lists every metric and its current value. Click the ˅ down arrowhead to minimize the expanded view for a query.
  4. Optional: Save the page URL to use this set of queries again in the future.

  5. Explore the visualized metrics. Initially, all metrics from all enabled queries are shown on the plot. Select which metrics are shown by performing any of the following actions:

    Expand
    OptionDescription

    Hide all metrics from a query.

    Click the options menu kebab for the query and click Hide all series.

    Hide a specific metric.

    Go to the query table and click the colored square near the metric name.

    Zoom into the plot and change the time range.

    Perform one of the following actions:

    • Visually select the time range by clicking and dragging on the plot horizontally.
    • Use the menu to select the time range.

    Reset the time range.

    Click Reset zoom.

    Display outputs for all queries at a specific point in time.

    Hover over the plot at the point you are interested in. The query outputs appear in a pop-up box.

    Hide the plot.

    Click Hide graph.

15.3.4. Virtualization metrics
Copy link

The following metric descriptions include example Prometheus Query Language (PromQL) queries. These metrics are not an API and might change between versions. For a complete list of virtualization metrics, see KubeVirt components metrics.

Note

The following examples use topk queries that specify a time period. If virtual machines are deleted during that time period, they can still appear in the query output.

15.3.4.1. vCPU metrics
Copy link

The following query can identify virtual machines that are waiting for Input/Output (I/O):

kubevirt_vmi_vcpu_wait_seconds_total
Returns the wait time (in seconds) on I/O for vCPUs of a virtual machine. Type: Counter.

A value above '0' means that the vCPU wants to run, but the host scheduler cannot run it yet. This inability to run indicates that there is an issue with I/O.

Note

To query the vCPU metric, the schedstats=enable kernel argument must first be applied to the MachineConfig object. This kernel argument enables scheduler statistics used for debugging and performance tuning and adds a minor additional load to the scheduler.

Example vCPU wait time query 

topk(3, sum by (name, namespace) (rate(kubevirt_vmi_vcpu_wait_seconds_total[6m]))) > 0
1
Copy to Clipboard Toggle word wrap
1
This query returns the top 3 VMs waiting for I/O at every given moment over a six-minute time period.

15.3.4.2. Network metrics
Copy link

The following queries can identify virtual machines that are saturating the network:

kubevirt_vmi_network_receive_bytes_total
Returns the total amount of traffic received (in bytes) on the virtual machine’s network. Type: Counter.
kubevirt_vmi_network_transmit_bytes_total
Returns the total amount of traffic transmitted (in bytes) on the virtual machine’s network. Type: Counter.

Example network traffic query 

topk(3, sum by (name, namespace) (rate(kubevirt_vmi_network_receive_bytes_total[6m])) + sum by (name, namespace) (rate(kubevirt_vmi_network_transmit_bytes_total[6m]))) > 0
1
Copy to Clipboard Toggle word wrap
1
This query returns the top 3 VMs transmitting the most network traffic at every given moment over a six-minute time period.

15.3.4.3. Storage metrics
Copy link

15.3.4.3.1. Storage-related traffic
Copy link

The following queries can identify VMs that are writing large amounts of data:

kubevirt_vmi_storage_read_traffic_bytes_total
Returns the total amount (in bytes) of the virtual machine’s storage-related traffic. Type: Counter.
kubevirt_vmi_storage_write_traffic_bytes_total
Returns the total amount of storage writes (in bytes) of the virtual machine’s storage-related traffic. Type: Counter.

Example storage-related traffic query 

topk(3, sum by (name, namespace) (rate(kubevirt_vmi_storage_read_traffic_bytes_total[6m])) + sum by (name, namespace) (rate(kubevirt_vmi_storage_write_traffic_bytes_total[6m]))) > 0
1
Copy to Clipboard Toggle word wrap
1
This query returns the top 3 VMs performing the most storage traffic at every given moment over a six-minute time period.
15.3.4.3.2. Storage snapshot data
Copy link
kubevirt_vmsnapshot_disks_restored_from_source
Returns the total number of virtual machine disks restored from the source virtual machine. Type: Gauge.
kubevirt_vmsnapshot_disks_restored_from_source_bytes
Returns the amount of space in bytes restored from the source virtual machine. Type: Gauge.

Examples of storage snapshot data queries 

kubevirt_vmsnapshot_disks_restored_from_source{vm_name="simple-vm", vm_namespace="default"}
1
Copy to Clipboard Toggle word wrap
1
This query returns the total number of virtual machine disks restored from the source virtual machine. 
kubevirt_vmsnapshot_disks_restored_from_source_bytes{vm_name="simple-vm", vm_namespace="default"}
1
Copy to Clipboard Toggle word wrap
1
This query returns the amount of space in bytes restored from the source virtual machine.
15.3.4.3.3. I/O performance
Copy link

The following queries can determine the I/O performance of storage devices:

kubevirt_vmi_storage_iops_read_total
Returns the amount of write I/O operations the virtual machine is performing per second. Type: Counter.
kubevirt_vmi_storage_iops_write_total
Returns the amount of read I/O operations the virtual machine is performing per second. Type: Counter.

Example I/O performance query 

topk(3, sum by (name, namespace) (rate(kubevirt_vmi_storage_iops_read_total[6m])) + sum by (name, namespace) (rate(kubevirt_vmi_storage_iops_write_total[6m]))) > 0
1
Copy to Clipboard Toggle word wrap
1
This query returns the top 3 VMs performing the most I/O operations per second at every given moment over a six-minute time period.

15.3.4.4. Guest memory swapping metrics
Copy link

The following queries can identify which swap-enabled guests are performing the most memory swapping:

kubevirt_vmi_memory_swap_in_traffic_bytes
Returns the total amount (in bytes) of memory the virtual guest is swapping in. Type: Gauge.
kubevirt_vmi_memory_swap_out_traffic_bytes
Returns the total amount (in bytes) of memory the virtual guest is swapping out. Type: Gauge.

Example memory swapping query 

topk(3, sum by (name, namespace) (rate(kubevirt_vmi_memory_swap_in_traffic_bytes[6m])) + sum by (name, namespace) (rate(kubevirt_vmi_memory_swap_out_traffic_bytes[6m]))) > 0
1
Copy to Clipboard Toggle word wrap
1
This query returns the top 3 VMs where the guest is performing the most memory swapping at every given moment over a six-minute time period.
Note

Memory swapping indicates that the virtual machine is under memory pressure. Increasing the memory allocation of the virtual machine can mitigate this issue.

15.3.4.5. Live migration metrics
Copy link

The following metrics can be queried to show live migration status:

kubevirt_vmi_migration_data_processed_bytes
The amount of guest operating system data that has migrated to the new virtual machine (VM). Type: Gauge.
kubevirt_vmi_migration_data_remaining_bytes
The amount of guest operating system data that remains to be migrated. Type: Gauge.
kubevirt_vmi_migration_memory_transfer_rate_bytes
The rate at which memory is becoming dirty in the guest operating system. Dirty memory is data that has been changed but not yet written to disk. Type: Gauge.
kubevirt_vmi_migrations_in_pending_phase
The number of pending migrations. Type: Gauge.
kubevirt_vmi_migrations_in_scheduling_phase
The number of scheduling migrations. Type: Gauge.
kubevirt_vmi_migrations_in_running_phase
The number of running migrations. Type: Gauge.
kubevirt_vmi_migration_succeeded
The number of successfully completed migrations. Type: Gauge.
kubevirt_vmi_migration_failed
The number of failed migrations. Type: Gauge.

15.4. Exposing custom metrics for virtual machines
Copy link

OpenShift Container Platform includes a preconfigured, preinstalled, and self-updating monitoring stack that provides monitoring for core platform components. This monitoring stack is based on the Prometheus monitoring system. Prometheus is a time-series database and a rule evaluation engine for metrics.

In addition to using the OpenShift Container Platform monitoring stack, you can enable monitoring for user-defined projects by using the CLI and query custom metrics that are exposed for virtual machines through the node-exporter service.

15.4.1. Configuring the node exporter service
Copy link

The node-exporter agent is deployed on every virtual machine in the cluster from which you want to collect metrics. Configure the node-exporter agent as a service to expose internal metrics and processes that are associated with virtual machines.

Prerequisites

  • Install the OpenShift CLI (oc).
  • Log in to the cluster as a user with cluster-admin privileges.
  • Create the cluster-monitoring-config ConfigMap object in the openshift-monitoring project.
  • Configure the user-workload-monitoring-config ConfigMap object in the openshift-user-workload-monitoring project by setting enableUserWorkload to true.

Procedure

  1. Create the Service YAML file. In the following example, the file is called node-exporter-service.yaml. 

    kind: Service
apiVersion: v1
metadata:
  name: node-exporter-service
    1 
    
  namespace: dynamation
    2 
    
  labels:
    servicetype: metrics
    3 
    
spec:
  ports:
    - name: exmet
    4 
    
      protocol: TCP
      port: 9100
    5 
    
      targetPort: 9100
    6 
    
  type: ClusterIP
  selector:
    monitor: metrics
    7
    Copy to Clipboard Toggle word wrap
    1
    The node-exporter service that exposes the metrics from the virtual machines.
    2
    The namespace where the service is created.
    3
    The label for the service. The ServiceMonitor uses this label to match this service.
    4
    The name given to the port that exposes metrics on port 9100 for the ClusterIP service.
    5
    The target port used by node-exporter-service to listen for requests.
    6
    The TCP port number of the virtual machine that is configured with the monitor label.
    7
    The label used to match the virtual machine’s pods. In this example, any virtual machine’s pod with the label monitor and a value of metrics will be matched.

  2. Create the node-exporter service: 

    $ oc create -f node-exporter-service.yaml
    Copy to Clipboard Toggle word wrap

Download the node-exporter file on to the virtual machine. Then, create a systemd service that runs the node-exporter service when the virtual machine boots.

Prerequisites

  • The pods for the component are running in the openshift-user-workload-monitoring project.
  • Grant the monitoring-edit role to users who need to monitor this user-defined project.

Procedure

  1. Log on to the virtual machine.

  2. Download the node-exporter file on to the virtual machine by using the directory path that applies to the version of node-exporter file. 

    $ wget https://github.com/prometheus/node_exporter/releases/download/<version>/node_exporter-<version>.linux-<architecture>.tar.gz
    Copy to Clipboard Toggle word wrap

  3. Extract the executable and place it in the /usr/bin directory. 

    $ sudo tar xvf node_exporter-<version>.linux-<architecture>.tar.gz \
    --directory /usr/bin --strip 1 "*/node_exporter"
    Copy to Clipboard Toggle word wrap

  4. Create a node_exporter.service file in this directory path: /etc/systemd/system. This systemd service file runs the node-exporter service when the virtual machine reboots. 

    [Unit]
Description=Prometheus Metrics Exporter
After=network.target
StartLimitIntervalSec=0

[Service]
Type=simple
Restart=always
RestartSec=1
User=root
ExecStart=/usr/bin/node_exporter

[Install]
WantedBy=multi-user.target
    Copy to Clipboard Toggle word wrap

  5. Enable and start the systemd service. 

    $ sudo systemctl enable node_exporter.service
$ sudo systemctl start node_exporter.service
    Copy to Clipboard Toggle word wrap

Verification

  • Verify that the node-exporter agent is reporting metrics from the virtual machine. 

    $ curl http://localhost:9100/metrics
    Copy to Clipboard Toggle word wrap

    Example output

    go_gc_duration_seconds{quantile="0"} 1.5244e-05
go_gc_duration_seconds{quantile="0.25"} 3.0449e-05
go_gc_duration_seconds{quantile="0.5"} 3.7913e-05
    Copy to Clipboard Toggle word wrap

To enable queries to multiple virtual machines from a single service, add a custom label in the virtual machine’s YAML file.

Prerequisites

  • Install the OpenShift CLI (oc).
  • Log in as a user with cluster-admin privileges.
  • Access to the web console for stop and restart a virtual machine.

Procedure

  1. Edit the template spec of your virtual machine configuration file. In this example, the label monitor has the value metrics. 

    spec:
  template:
    metadata:
      labels:
        monitor: metrics
    Copy to Clipboard Toggle word wrap
  2. Stop and restart the virtual machine to create a new pod with the label name given to the monitor label.

Metrics are exposed for virtual machines through an HTTP service endpoint under the /metrics canonical name. When you query for metrics, Prometheus directly scrapes the metrics from the metrics endpoint exposed by the virtual machines and presents these metrics for viewing.

Prerequisites

  • You have access to the cluster as a user with cluster-admin privileges or the monitoring-edit role.
  • You have enabled monitoring for the user-defined project by configuring the node-exporter service.
  • You have installed the OpenShift CLI (oc).

Procedure

  1. Obtain the HTTP service endpoint by specifying the namespace for the service: 

    $ oc get service -n <namespace> <node-exporter-service>
    Copy to Clipboard Toggle word wrap

  2. To list all available metrics for the node-exporter service, query the metrics resource. 

    $ curl http://<172.30.226.162:9100>/metrics | grep -vE "^#|^$"
    Copy to Clipboard Toggle word wrap

    Example output

    node_arp_entries{device="eth0"} 1
node_boot_time_seconds 1.643153218e+09
node_context_switches_total 4.4938158e+07
node_cooling_device_cur_state{name="0",type="Processor"} 0
node_cooling_device_max_state{name="0",type="Processor"} 0
node_cpu_guest_seconds_total{cpu="0",mode="nice"} 0
node_cpu_guest_seconds_total{cpu="0",mode="user"} 0
node_cpu_seconds_total{cpu="0",mode="idle"} 1.10586485e+06
node_cpu_seconds_total{cpu="0",mode="iowait"} 37.61
node_cpu_seconds_total{cpu="0",mode="irq"} 233.91
node_cpu_seconds_total{cpu="0",mode="nice"} 551.47
node_cpu_seconds_total{cpu="0",mode="softirq"} 87.3
node_cpu_seconds_total{cpu="0",mode="steal"} 86.12
node_cpu_seconds_total{cpu="0",mode="system"} 464.15
node_cpu_seconds_total{cpu="0",mode="user"} 1075.2
node_disk_discard_time_seconds_total{device="vda"} 0
node_disk_discard_time_seconds_total{device="vdb"} 0
node_disk_discarded_sectors_total{device="vda"} 0
node_disk_discarded_sectors_total{device="vdb"} 0
node_disk_discards_completed_total{device="vda"} 0
node_disk_discards_completed_total{device="vdb"} 0
node_disk_discards_merged_total{device="vda"} 0
node_disk_discards_merged_total{device="vdb"} 0
node_disk_info{device="vda",major="252",minor="0"} 1
node_disk_info{device="vdb",major="252",minor="16"} 1
node_disk_io_now{device="vda"} 0
node_disk_io_now{device="vdb"} 0
node_disk_io_time_seconds_total{device="vda"} 174
node_disk_io_time_seconds_total{device="vdb"} 0.054
node_disk_io_time_weighted_seconds_total{device="vda"} 259.79200000000003
node_disk_io_time_weighted_seconds_total{device="vdb"} 0.039
node_disk_read_bytes_total{device="vda"} 3.71867136e+08
node_disk_read_bytes_total{device="vdb"} 366592
node_disk_read_time_seconds_total{device="vda"} 19.128
node_disk_read_time_seconds_total{device="vdb"} 0.039
node_disk_reads_completed_total{device="vda"} 5619
node_disk_reads_completed_total{device="vdb"} 96
node_disk_reads_merged_total{device="vda"} 5
node_disk_reads_merged_total{device="vdb"} 0
node_disk_write_time_seconds_total{device="vda"} 240.66400000000002
node_disk_write_time_seconds_total{device="vdb"} 0
node_disk_writes_completed_total{device="vda"} 71584
node_disk_writes_completed_total{device="vdb"} 0
node_disk_writes_merged_total{device="vda"} 19761
node_disk_writes_merged_total{device="vdb"} 0
node_disk_written_bytes_total{device="vda"} 2.007924224e+09
node_disk_written_bytes_total{device="vdb"} 0
    Copy to Clipboard Toggle word wrap

You can use a Prometheus client library and scrape metrics from the /metrics endpoint to access and view the metrics exposed by the node-exporter service. Use a ServiceMonitor custom resource definition (CRD) to monitor the node exporter service.

Prerequisites

  • You have access to the cluster as a user with cluster-admin privileges or the monitoring-edit role.
  • You have enabled monitoring for the user-defined project by configuring the node-exporter service.
  • You have installed the OpenShift CLI (oc).

Procedure

  1. Create a YAML file for the ServiceMonitor resource configuration. In this example, the service monitor matches any service with the label metrics and queries the exmet port every 30 seconds. 

    apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    k8s-app: node-exporter-metrics-monitor
  name: node-exporter-metrics-monitor
    1 
    
  namespace: dynamation
    2 
    
spec:
  endpoints:
  - interval: 30s
    3 
    
    port: exmet
    4 
    
    scheme: http
  selector:
    matchLabels:
      servicetype: metrics
    Copy to Clipboard Toggle word wrap
    1
    The name of the ServiceMonitor.
    2
    The namespace where the ServiceMonitor is created.
    3
    The interval at which the port will be queried.
    4
    The name of the port that is queried every 30 seconds

  2. Create the ServiceMonitor configuration for the node-exporter service. 

    $ oc create -f node-exporter-metrics-monitor.yaml
    Copy to Clipboard Toggle word wrap

You can access the node-exporter service outside the cluster and view the exposed metrics.

Prerequisites

  • You have access to the cluster as a user with cluster-admin privileges or the monitoring-edit role.
  • You have enabled monitoring for the user-defined project by configuring the node-exporter service.
  • You have installed the OpenShift CLI (oc).

Procedure

  1. Expose the node-exporter service. 

    $ oc expose service -n <namespace> <node_exporter_service_name>
    Copy to Clipboard Toggle word wrap

  2. Obtain the FQDN (Fully Qualified Domain Name) for the route. 

    $ oc get route -o=custom-columns=NAME:.metadata.name,DNS:.spec.host
    Copy to Clipboard Toggle word wrap

    Example output

    NAME                    DNS
node-exporter-service   node-exporter-service-dynamation.apps.cluster.example.org
    Copy to Clipboard Toggle word wrap

  3. Use the curl command to display metrics for the node-exporter service. 

    $ curl -s http://node-exporter-service-dynamation.apps.cluster.example.org/metrics
    Copy to Clipboard Toggle word wrap

    Example output

    go_gc_duration_seconds{quantile="0"} 1.5382e-05
go_gc_duration_seconds{quantile="0.25"} 3.1163e-05
go_gc_duration_seconds{quantile="0.5"} 3.8546e-05
go_gc_duration_seconds{quantile="0.75"} 4.9139e-05
go_gc_duration_seconds{quantile="1"} 0.000189423
    Copy to Clipboard Toggle word wrap

As an administrator, you can expose a limited set of host and virtual machine (VM) metrics to a guest VM by first enabling a downwardMetrics feature gate and then configuring a downwardMetrics device.

Users can view the metrics results by using the command line or the vm-dump-metrics tool.

Note

On Red Hat Enterprise Linux (RHEL) 9, use the command line to view downward metrics. See Viewing downward metrics by using the command line.

The vm-dump-metrics tool is not supported on the Red Hat Enterprise Linux (RHEL) 9 platform.

You can enable or disable the downwardMetrics feature gate by performing either of the following actions:

  • Editing the HyperConverged custom resource (CR) in your default editor
  • Using the command line

To expose downward metrics for a host virtual machine, you can enable the downwardMetrics feature gate by editing a YAML file.

Prerequisites

  • You must have administrator privileges to enable the feature gate.
  • You have installed the OpenShift CLI (oc).

Procedure

  1. Open the HyperConverged custom resource (CR) in your default editor by running the following command: 

    $ oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnv
    Copy to Clipboard Toggle word wrap

  2. Choose to enable or disable the downwardMetrics feature gate as follows:

    • To enable the downwardMetrics feature gate, add and then set spec.featureGates.downwardMetrics to true. For example: 

      apiVersion: hco.kubevirt.io/v1beta1
kind: HyperConverged
metadata:
  name: kubevirt-hyperconverged
  namespace: openshift-cnv
spec:
    featureGates:
      downwardMetrics: true
# ...
      Copy to Clipboard Toggle word wrap

    • To disable the downwardMetrics feature gate, set spec.featureGates.downwardMetrics to false. For example: 

      apiVersion: hco.kubevirt.io/v1beta1
kind: HyperConverged
metadata:
  name: kubevirt-hyperconverged
  namespace: openshift-cnv
spec:
    featureGates:
      downwardMetrics: false
# ...
      Copy to Clipboard Toggle word wrap

To expose downward metrics for a host virtual machine, you can enable the downwardMetrics feature gate by using the command line.

Prerequisites

  • You must have administrator privileges to enable the feature gate.
  • You have installed the OpenShift CLI (oc).

Procedure

  • Choose to enable or disable the downwardMetrics feature gate as follows:

    • Enable the downwardMetrics feature gate by running the command shown in the following example: 

      $ oc patch hco kubevirt-hyperconverged -n openshift-cnv \
  --type json -p '[{"op": "replace", "path": \
  "/spec/featureGates/downwardMetrics", \
  "value": true}]'
      Copy to Clipboard Toggle word wrap

    • Disable the downwardMetrics feature gate by running the command shown in the following example: 

      $ oc patch hco kubevirt-hyperconverged -n openshift-cnv \
  --type json -p '[{"op": "replace", "path": \
  "/spec/featureGates/downwardMetrics", \
  "value": false}]'
      Copy to Clipboard Toggle word wrap

15.5.2. Configuring a downward metrics device
Copy link

You enable the capturing of downward metrics for a host VM by creating a configuration file that includes a downwardMetrics device. Adding this device establishes that the metrics are exposed through a virtio-serial port.

Prerequisites

  • You must first enable the downwardMetrics feature gate.

Procedure

  • Edit or create a YAML file that includes a downwardMetrics device, as shown in the following example:

    Example downwardMetrics configuration file

    apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: fedora
  namespace: default
spec:
  dataVolumeTemplates:
    - metadata:
        name: fedora-volume
      spec:
        sourceRef:
          kind: DataSource
          name: fedora
          namespace: openshift-virtualization-os-images
        storage:
          resources: {}
  instancetype:
    name: u1.medium
  runStrategy: Always
  template:
    metadata:
      labels:
        app.kubernetes.io/name: headless
    spec:
      domain:
        devices:
          downwardMetrics: {}
    1 
    
      subdomain: headless
      volumes:
        - dataVolume:
            name: fedora-volume
          name: rootdisk
        - cloudInitNoCloud:
            userData: |
              #cloud-config
              chpasswd:
                expire: false
              password: '<password>'
    2 
    
              user: fedora
          name: cloudinitdisk
    Copy to Clipboard Toggle word wrap

    1
    The downwardMetrics device.
    2
    The password for the fedora user.

15.5.3. Viewing downward metrics
Copy link

You can view downward metrics by using either of the following options:

  • The command-line interface (CLI)
  • The vm-dump-metrics tool
Note

On Red Hat Enterprise Linux (RHEL) 9, use the command line to view downward metrics. The vm-dump-metrics tool is not supported on the Red Hat Enterprise Linux (RHEL) 9 platform.

You can view downward metrics by entering a command from inside a guest virtual machine (VM).

Procedure

  • Run the following commands: 

    $ sudo sh -c 'printf "GET /metrics/XML\n\n" > /dev/virtio-ports/org.github.vhostmd.1'
    Copy to Clipboard Toggle word wrap 
    $ sudo cat /dev/virtio-ports/org.github.vhostmd.1
    Copy to Clipboard Toggle word wrap

To view downward metrics, install the vm-dump-metrics tool and then use the tool to expose the metrics results.

Note

On Red Hat Enterprise Linux (RHEL) 9, use the command line to view downward metrics. The vm-dump-metrics tool is not supported on the Red Hat Enterprise Linux (RHEL) 9 platform.

Procedure

  1. Install the vm-dump-metrics tool by running the following command: 

    $ sudo dnf install -y vm-dump-metrics
    Copy to Clipboard Toggle word wrap

  2. Retrieve the metrics results by running the following command: 

    $ sudo vm-dump-metrics
    Copy to Clipboard Toggle word wrap

    Example output

    <metrics>
  <metric type="string" context="host">
    <name>HostName</name>
    <value>node01</value>
[...]
  <metric type="int64" context="host" unit="s">
    <name>Time</name>
    <value>1619008605</value>
  </metric>
  <metric type="string" context="host">
    <name>VirtualizationVendor</name>
    <value>kubevirt.io</value>
  </metric>
</metrics>
    Copy to Clipboard Toggle word wrap

15.6. Virtual machine health checks
Copy link

You can configure virtual machine (VM) health checks by defining readiness and liveness probes in the VirtualMachine resource.

15.6.1. About readiness and liveness probes
Copy link

Use readiness and liveness probes to detect and handle unhealthy virtual machines (VMs). You can include one or more probes in the specification of the VM to ensure that traffic does not reach a VM that is not ready for it and that a new VM is created when a VM becomes unresponsive.

A readiness probe determines whether a VM is ready to accept service requests. If the probe fails, the VM is removed from the list of available endpoints until the VM is ready.

A liveness probe determines whether a VM is responsive. If the probe fails, the VM is deleted and a new VM is created to restore responsiveness.

You can configure readiness and liveness probes by setting the spec.readinessProbe and the spec.livenessProbe fields of the VirtualMachine object. These fields support the following tests:

HTTP GET
The probe determines the health of the VM by using a web hook. The test is successful if the HTTP response code is between 200 and 399. You can use an HTTP GET test with applications that return HTTP status codes when they are completely initialized.
TCP socket
The probe attempts to open a socket to the VM. The VM is only considered healthy if the probe can establish a connection. You can use a TCP socket test with applications that do not start listening until initialization is complete.
Guest agent ping
The probe uses the guest-ping command to determine if the QEMU guest agent is running on the virtual machine.

15.6.1.1. Defining an HTTP readiness probe
Copy link

Define an HTTP readiness probe by setting the spec.readinessProbe.httpGet field of the virtual machine (VM) configuration.

Prerequisites

  • You have installed the OpenShift CLI (oc).

Procedure

  1. Include details of the readiness probe in the VM configuration file.

    Sample readiness probe with an HTTP GET test

    apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  annotations:
  name: fedora-vm
  namespace: example-namespace
# ...
spec:
  template:
    spec:
      readinessProbe:
        httpGet:
    1 
    
          port: 1500
    2 
    
          path: /healthz
    3 
    
          httpHeaders:
          - name: Custom-Header
            value: Awesome
        initialDelaySeconds: 120
    4 
    
        periodSeconds: 20
    5 
    
        timeoutSeconds: 10
    6 
    
        failureThreshold: 3
    7 
    
        successThreshold: 3
    8 
    
# ...
    Copy to Clipboard Toggle word wrap

    1
    The HTTP GET request to perform to connect to the VM.
    2
    The port of the VM that the probe queries. In the above example, the probe queries port 1500.
    3
    The path to access on the HTTP server. In the above example, if the handler for the server’s /healthz path returns a success code, the VM is considered to be healthy. If the handler returns a failure code, the VM is removed from the list of available endpoints.
    4
    The time, in seconds, after the VM starts before the readiness probe is initiated.
    5
    The delay, in seconds, between performing probes. The default delay is 10 seconds. This value must be greater than timeoutSeconds.
    6
    The number of seconds of inactivity after which the probe times out and the VM is assumed to have failed. The default value is 1. This value must be lower than periodSeconds.
    7
    The number of times that the probe is allowed to fail. The default is 3. After the specified number of attempts, the pod is marked Unready.
    8
    The number of times that the probe must report success, after a failure, to be considered successful. The default is 1.

  2. Create the VM by running the following command: 

    $ oc create -f <file_name>.yaml
    Copy to Clipboard Toggle word wrap

15.6.1.2. Defining a TCP readiness probe
Copy link

Define a TCP readiness probe by setting the spec.readinessProbe.tcpSocket field of the virtual machine (VM) configuration.

Prerequisites

  • You have installed the OpenShift CLI (oc).

Procedure

  1. Include details of the TCP readiness probe in the VM configuration file.

    Sample readiness probe with a TCP socket test

    apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  annotations:
  name: fedora-vm
  namespace: example-namespace
# ...
spec:
  template:
    spec:
      readinessProbe:
        initialDelaySeconds: 120
    1 
    
        periodSeconds: 20
    2 
    
        tcpSocket:
    3 
    
          port: 1500
    4 
    
        timeoutSeconds: 10
    5 
    
# ...
    Copy to Clipboard Toggle word wrap

    1
    The time, in seconds, after the VM starts before the readiness probe is initiated.
    2
    The delay, in seconds, between performing probes. The default delay is 10 seconds. This value must be greater than timeoutSeconds.
    3
    The TCP action to perform.
    4
    The port of the VM that the probe queries.
    5
    The number of seconds of inactivity after which the probe times out and the VM is assumed to have failed. The default value is 1. This value must be lower than periodSeconds.

  2. Create the VM by running the following command: 

    $ oc create -f <file_name>.yaml
    Copy to Clipboard Toggle word wrap

15.6.1.3. Defining an HTTP liveness probe
Copy link

Define an HTTP liveness probe by setting the spec.livenessProbe.httpGet field of the virtual machine (VM) configuration. You can define both HTTP and TCP tests for liveness probes in the same way as readiness probes. This procedure configures a sample liveness probe with an HTTP GET test.

Prerequisites

  • You have installed the OpenShift CLI (oc).

Procedure

  1. Include details of the HTTP liveness probe in the VM configuration file.

    Sample liveness probe with an HTTP GET test

    apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  annotations:
  name: fedora-vm
  namespace: example-namespace
# ...
spec:
  template:
    spec:
      livenessProbe:
        initialDelaySeconds: 120
    1 
    
        periodSeconds: 20
    2 
    
        httpGet:
    3 
    
          port: 1500
    4 
    
          path: /healthz
    5 
    
          httpHeaders:
          - name: Custom-Header
            value: Awesome
        timeoutSeconds: 10
    6 
    
# ...
    Copy to Clipboard Toggle word wrap

    1
    The time, in seconds, after the VM starts before the liveness probe is initiated.
    2
    The delay, in seconds, between performing probes. The default delay is 10 seconds. This value must be greater than timeoutSeconds.
    3
    The HTTP GET request to perform to connect to the VM.
    4
    The port of the VM that the probe queries. In the above example, the probe queries port 1500. The VM installs and runs a minimal HTTP server on port 1500 via cloud-init.
    5
    The path to access on the HTTP server. In the above example, if the handler for the server’s /healthz path returns a success code, the VM is considered to be healthy. If the handler returns a failure code, the VM is deleted and a new VM is created.
    6
    The number of seconds of inactivity after which the probe times out and the VM is assumed to have failed. The default value is 1. This value must be lower than periodSeconds.

  2. Create the VM by running the following command: 

    $ oc create -f <file_name>.yaml
    Copy to Clipboard Toggle word wrap

15.6.2. Defining a watchdog
Copy link

You can define a watchdog to monitor the health of the guest operating system by performing the following steps:

  1. Configure a watchdog device for the virtual machine (VM).
  2. Install the watchdog agent on the guest.

The watchdog device monitors the agent and performs one of the following actions if the guest operating system is unresponsive:

  • poweroff: The VM powers down immediately. If spec.runStrategy is not set to manual, the VM reboots.

  • reset: The VM reboots in place and the guest operating system cannot react.

    Note

    The reboot time might cause liveness probes to time out. If cluster-level protections detect a failed liveness probe, the VM might be forcibly rescheduled, increasing the reboot time.

  • shutdown: The VM gracefully powers down by stopping all services.
Note

Watchdog is not available for Windows VMs.

You configure a watchdog device for the virtual machine (VM).

Prerequisites

  • For x86 systems, the VM must use a kernel that works with the i6300esb watchdog device. If you use s390x architecture, the kernel must be enabled for diag288. Red Hat Enterprise Linux (RHEL) images support i6300esb and diag288.
  • You have installed the OpenShift CLI (oc).

Procedure

  1. Create a YAML file with the following contents: 

    apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  labels:
    kubevirt.io/vm: <vm-label>
  name: <vm-name>
spec:
  runStrategy: Halted
  template:
    metadata:
      labels:
        kubevirt.io/vm: <vm-label>
    spec:
      domain:
        devices:
          watchdog:
            name: <watchdog>
            <watchdog-device-model>:
    1 
    
              action: "poweroff"
    2 
    
# ...
    Copy to Clipboard Toggle word wrap
    1
    The watchdog device model to use. For x86 specify i6300esb. For s390x specify diag288.
    2
    Specify poweroff, reset, or shutdown. The shutdown action requires that the guest virtual machine is responsive to ACPI signals. Therefore, using shutdown is not recommended.

    The example above configures the watchdog device on a VM with the poweroff action and exposes the device as /dev/watchdog.

    This device can now be used by the watchdog binary.

  2. Apply the YAML file to your cluster by running the following command: 

    $ oc apply -f <file_name>.yaml
    Copy to Clipboard Toggle word wrap

Verification

Important

This procedure is provided for testing watchdog functionality only and must not be run on production machines.

  1. Run the following command to verify that the VM is connected to the watchdog device: 

    $ lspci | grep watchdog -i
    Copy to Clipboard Toggle word wrap

  2. Run one of the following commands to confirm the watchdog is active:

    • Trigger a kernel panic: 

      # echo c > /proc/sysrq-trigger
      Copy to Clipboard Toggle word wrap

    • Stop the watchdog service: 

      # pkill -9 watchdog
      Copy to Clipboard Toggle word wrap

You install the watchdog agent on the guest and start the watchdog service.

Procedure

  1. Log in to the virtual machine as root user.

  2. This step is only required when installing on IBM Z® (s390x). Enable watchdog by running the following command: 

    # modprobe diag288_wdt
    Copy to Clipboard Toggle word wrap

  3. Verify that the /dev/watchdog file path is present in the VM by running the following command: 

    # ls /dev/watchdog
    Copy to Clipboard Toggle word wrap

  4. Install the watchdog package and its dependencies: 

    # yum install watchdog
    Copy to Clipboard Toggle word wrap

  5. Uncomment the following line in the /etc/watchdog.conf file and save the changes: 

    #watchdog-device = /dev/watchdog
    Copy to Clipboard Toggle word wrap

  6. Enable the watchdog service to start on boot: 

    # systemctl enable --now watchdog.service
    Copy to Clipboard Toggle word wrap

15.6.3. Defining a guest agent ping probe
Copy link

Define a guest agent ping probe by setting the spec.readinessProbe.guestAgentPing field of the virtual machine (VM) configuration.

Important

The guest agent ping probe is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

Prerequisites

  • The QEMU guest agent must be installed and enabled on the virtual machine.
  • You have installed the OpenShift CLI (oc).

Procedure

  1. Include details of the guest agent ping probe in the VM configuration file. For example:

    Sample guest agent ping probe

    apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  annotations:
  name: fedora-vm
  namespace: example-namespace
# ...
spec:
  template:
    spec:
      readinessProbe:
        guestAgentPing: {}
    1 
    
        initialDelaySeconds: 120
    2 
    
        periodSeconds: 20
    3 
    
        timeoutSeconds: 10
    4 
    
        failureThreshold: 3
    5 
    
        successThreshold: 3
    6 
    
# ...
    Copy to Clipboard Toggle word wrap

    1
    The guest agent ping probe to connect to the VM.
    2
    Optional: The time, in seconds, after the VM starts before the guest agent probe is initiated.
    3
    Optional: The delay, in seconds, between performing probes. The default delay is 10 seconds. This value must be greater than timeoutSeconds.
    4
    Optional: The number of seconds of inactivity after which the probe times out and the VM is assumed to have failed. The default value is 1. This value must be lower than periodSeconds.
    5
    Optional: The number of times that the probe is allowed to fail. The default is 3. After the specified number of attempts, the pod is marked Unready.
    6
    Optional: The number of times that the probe must report success, after a failure, to be considered successful. The default is 1.

  2. Create the VM by running the following command: 

    $ oc create -f <file_name>.yaml
    Copy to Clipboard Toggle word wrap

15.7. OpenShift Virtualization runbooks
Copy link

To diagnose and resolve issues that trigger OpenShift Virtualization alerts, follow the procedures in the runbooks for the OpenShift Virtualization Operator. Triggered OpenShift Virtualization alerts can be viewed in the main Observe Alerts tab in the web console, and also in the Virtualization Overview tab.

Runbooks for the OpenShift Virtualization Operator are maintained in the openshift/runbooks Git repository, and you can view them on GitHub.

15.7.1. CDIDataImportCronOutdated
Copy link

15.7.2. CDIDataVolumeUnusualRestartCount
Copy link

15.7.3. CDIDefaultStorageClassDegraded
Copy link

15.7.4. CDIMultipleDefaultVirtStorageClasses
Copy link

15.7.5. CDINoDefaultStorageClass
Copy link

15.7.6. CDINotReady
Copy link

15.7.7. CDIOperatorDown
Copy link

15.7.8. CDIStorageProfilesIncomplete
Copy link

15.7.9. CnaoDown
Copy link

15.7.10. CnaoNMstateMigration
Copy link

15.7.11. HAControlPlaneDown
Copy link

15.7.12. HCOInstallationIncomplete
Copy link

15.7.13. HCOMisconfiguredDescheduler
Copy link

15.7.14. HPPNotReady
Copy link

15.7.15. HPPOperatorDown
Copy link

15.7.16. HPPSharingPoolPathWithOS
Copy link

15.7.17. HighCPUWorkload
Copy link

15.7.18. KubemacpoolDown
Copy link

15.7.19. KubeMacPoolDuplicateMacsFound
Copy link

15.7.20. KubeVirtComponentExceedsRequestedCPU
Copy link

  • The KubeVirtComponentExceedsRequestedCPU alert is deprecated.

15.7.21. KubeVirtComponentExceedsRequestedMemory
Copy link

  • The KubeVirtComponentExceedsRequestedMemory alert is deprecated.

15.7.22. KubeVirtCRModified
Copy link

15.7.23. KubeVirtDeprecatedAPIRequested
Copy link

15.7.24. KubeVirtNoAvailableNodesToRunVMs
Copy link

15.7.25. KubevirtVmHighMemoryUsage
Copy link

15.7.26. KubeVirtVMIExcessiveMigrations
Copy link

15.7.27. LowKVMNodesCount
Copy link

15.7.28. LowReadyVirtControllersCount
Copy link

15.7.29. LowReadyVirtOperatorsCount
Copy link

15.7.30. LowVirtAPICount
Copy link

15.7.31. LowVirtControllersCount
Copy link

15.7.32. LowVirtOperatorCount
Copy link

15.7.33. NetworkAddonsConfigNotReady
Copy link

15.7.34. NoLeadingVirtOperator
Copy link

15.7.35. NoReadyVirtController
Copy link

15.7.36. NoReadyVirtOperator
Copy link

15.7.37. NodeNetworkInterfaceDown
Copy link

15.7.38. OperatorConditionsUnhealthy
Copy link

  • The OperatorConditionsUnhealthy alert is deprecated.

15.7.39. OrphanedVirtualMachineInstances
Copy link

15.7.40. OutdatedVirtualMachineInstanceWorkloads
Copy link

15.7.41. SingleStackIPv6Unsupported
Copy link

15.7.42. SSPCommonTemplatesModificationReverted
Copy link

15.7.43. SSPDown
Copy link

15.7.44. SSPFailingToReconcile
Copy link

15.7.45. SSPHighRateRejectedVms
Copy link

15.7.46. SSPOperatorDown
Copy link

15.7.47. SSPTemplateValidatorDown
Copy link

15.7.48. UnsupportedHCOModification
Copy link

15.7.49. VirtAPIDown
Copy link

15.7.50. VirtApiRESTErrorsBurst
Copy link

15.7.51. VirtApiRESTErrorsHigh
Copy link

15.7.52. VirtControllerDown
Copy link

15.7.53. VirtControllerRESTErrorsBurst
Copy link

15.7.54. VirtControllerRESTErrorsHigh
Copy link

15.7.55. VirtHandlerDaemonSetRolloutFailing
Copy link

15.7.56. VirtHandlerRESTErrorsBurst
Copy link

15.7.57. VirtHandlerRESTErrorsHigh
Copy link

15.7.58. VirtOperatorDown
Copy link

15.7.59. VirtOperatorRESTErrorsBurst
Copy link

15.7.60. VirtOperatorRESTErrorsHigh
Copy link

15.7.61. VirtualMachineCRCErrors
Copy link

  • The VirtualMachineCRCErrors alert is deprecated.

    The alert is now called VMStorageClassWarning.

15.7.62. VMCannotBeEvicted
Copy link

15.7.63. VMStorageClassWarning
Copy link

Back to top
Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Theme

© 2025 Red Hat