Chapter 12. Monitoring

Procedure

1. Navigate to Virtualization Checkups in the web console.
2. Click the Network latency tab.
3. Click Install permissions.
4. Click Run checkup.
5. Enter a name for the checkup in the Name field.
6. Select a NetworkAttachmentDefinition from the drop-down menu.
7. Optional: Set a duration for the latency sample in the Sample duration (seconds) field.
8. Optional: Define a maximum latency time interval by enabling Set maximum desired latency (milliseconds) and defining the time interval.
9. Optional: Target specific nodes by enabling Select nodes and specifying the Source node and Target node.
10. Click Run.

Procedure

1. Navigate to Virtualization Checkups in the web console.
2. Click the Storage tab.
3. Click Install permissions.
4. Click Run checkup.
5. Enter a name for the checkup in the Name field.
6. Enter a timeout value for the checkup in the Timeout (minutes) fields.
7. Click Run.

1. Create a service account, roles, and rolebindings to provide cluster access permissions to the latency checkup.
2. Create a config map to provide the input to run the checkup and to store the results.
3. Create a job to run the checkup.
4. Review the results in the config map.
5. Optional: To rerun the checkup, delete the existing config map and job and then create a new config map and job.
6. When you are finished, delete the latency checkup resources.

Procedure

1. Create a ServiceAccount, Role, and RoleBinding manifest for the latency checkup:

   Example 12.1. Example role manifest file

   ---
   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: vm-latency-checkup-sa
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: Role
   metadata:
     name: kubevirt-vm-latency-checker
   rules:
   - apiGroups: ["kubevirt.io"]
     resources: ["virtualmachineinstances"]
     verbs: ["get", "create", "delete"]
   - apiGroups: ["subresources.kubevirt.io"]
     resources: ["virtualmachineinstances/console"]
     verbs: ["get"]
   - apiGroups: ["k8s.cni.cncf.io"]
     resources: ["network-attachment-definitions"]
     verbs: ["get"]
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: RoleBinding
   metadata:
     name: kubevirt-vm-latency-checker
   subjects:
   - kind: ServiceAccount
     name: vm-latency-checkup-sa
   roleRef:
     kind: Role
     name: kubevirt-vm-latency-checker
     apiGroup: rbac.authorization.k8s.io
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: Role
   metadata:
     name: kiagnose-configmap-access
   rules:
   - apiGroups: [ "" ]
     resources: [ "configmaps" ]
     verbs: ["get", "update"]
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: RoleBinding
   metadata:
     name: kiagnose-configmap-access
   subjects:
   - kind: ServiceAccount
     name: vm-latency-checkup-sa
   roleRef:
     kind: Role
     name: kiagnose-configmap-access
     apiGroup: rbac.authorization.k8s.io

2. Apply the ServiceAccount, Role, and RoleBinding manifest:

   $ oc apply -n <target_namespace> -f <latency_sa_roles_rolebinding>.yaml 1

   1

   <target_namespace> is the namespace where the checkup is to be run. This must be an existing namespace where the NetworkAttachmentDefinition object resides.

3. Create a ConfigMap manifest that contains the input parameters for the checkup:

   Example input config map

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: kubevirt-vm-latency-checkup-config
     labels:
       kiagnose/checkup-type: kubevirt-vm-latency
   data:
     spec.timeout: 5m
     spec.param.networkAttachmentDefinitionNamespace: <target_namespace>
     spec.param.networkAttachmentDefinitionName: "blue-network" 1
     spec.param.maxDesiredLatencyMilliseconds: "10" 2
     spec.param.sampleDurationSeconds: "5" 3
     spec.param.sourceNode: "worker1" 4
     spec.param.targetNode: "worker2" 5

   1

   The name of the NetworkAttachmentDefinition object.

   2

   Optional: The maximum desired latency, in milliseconds, between the virtual machines. If the measured latency exceeds this value, the checkup fails.

   3

   Optional: The duration of the latency check, in seconds.

   4

   Optional: When specified, latency is measured from this node to the target node. If the source node is specified, the spec.param.targetNode field cannot be empty.

   5

   Optional: When specified, latency is measured from the source node to this node.

4. Apply the config map manifest in the target namespace:

   $ oc apply -n <target_namespace> -f <latency_config_map>.yaml

5. Create a Job manifest to run the checkup:

   Example job manifest

   apiVersion: batch/v1
   kind: Job
   metadata:
     name: kubevirt-vm-latency-checkup
     labels:
       kiagnose/checkup-type: kubevirt-vm-latency
   spec:
     backoffLimit: 0
     template:
       spec:
         serviceAccountName: vm-latency-checkup-sa
         restartPolicy: Never
         containers:
           - name: vm-latency-checkup
             image: registry.redhat.io/container-native-virtualization/vm-network-latency-checkup-rhel9:v4.16.0
             securityContext:
               allowPrivilegeEscalation: false
               capabilities:
                 drop: ["ALL"]
               runAsNonRoot: true
               seccompProfile:
                 type: "RuntimeDefault"
             env:
               - name: CONFIGMAP_NAMESPACE
                 value: <target_namespace>
               - name: CONFIGMAP_NAME
                 value: kubevirt-vm-latency-checkup-config
               - name: POD_UID
                 valueFrom:
                   fieldRef:
                     fieldPath: metadata.uid

6. Apply the Job manifest:

   $ oc apply -n <target_namespace> -f <latency_job>.yaml

7. Wait for the job to complete:

   $ oc wait job kubevirt-vm-latency-checkup -n <target_namespace> --for condition=complete --timeout 6m

8. Review the results of the latency checkup by running the following command. If the maximum measured latency is greater than the value of the spec.param.maxDesiredLatencyMilliseconds attribute, the checkup fails and returns an error.

   $ oc get configmap kubevirt-vm-latency-checkup-config -n <target_namespace> -o yaml

   Example output config map (success)

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: kubevirt-vm-latency-checkup-config
     namespace: <target_namespace>
     labels:
       kiagnose/checkup-type: kubevirt-vm-latency
   data:
     spec.timeout: 5m
     spec.param.networkAttachmentDefinitionNamespace: <target_namespace>
     spec.param.networkAttachmentDefinitionName: "blue-network"
     spec.param.maxDesiredLatencyMilliseconds: "10"
     spec.param.sampleDurationSeconds: "5"
     spec.param.sourceNode: "worker1"
     spec.param.targetNode: "worker2"
     status.succeeded: "true"
     status.failureReason: ""
     status.completionTimestamp: "2022-01-01T09:00:00Z"
     status.startTimestamp: "2022-01-01T09:00:07Z"
     status.result.avgLatencyNanoSec: "177000"
     status.result.maxLatencyNanoSec: "244000" 1
     status.result.measurementDurationSec: "5"
     status.result.minLatencyNanoSec: "135000"
     status.result.sourceNode: "worker1"
     status.result.targetNode: "worker2"

   1

   The maximum measured latency in nanoseconds.

9. Optional: To view the detailed job log in case of checkup failure, use the following command:

   $ oc logs job.batch/kubevirt-vm-latency-checkup -n <target_namespace>

10. Delete the job and config map that you previously created by running the following commands:

    $ oc delete job -n <target_namespace> kubevirt-vm-latency-checkup

    $ oc delete config-map -n <target_namespace> kubevirt-vm-latency-checkup-config

11. Optional: If you do not plan to run another checkup, delete the roles manifest:

    $ oc delete -f <latency_sa_roles_rolebinding>.yaml

Procedure

1. Create a ServiceAccount, Role, and RoleBinding manifest file for the storage checkup:

   Example 12.2. Example service account, role, and rolebinding manifest

   ---
   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: storage-checkup-sa
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: Role
   metadata:
     name: storage-checkup-role
   rules:
     - apiGroups: [ "" ]
       resources: [ "configmaps" ]
       verbs: ["get", "update"]
     - apiGroups: [ "kubevirt.io" ]
       resources: [ "virtualmachines" ]
       verbs: [ "create", "delete" ]
     - apiGroups: [ "kubevirt.io" ]
       resources: [ "virtualmachineinstances" ]
       verbs: [ "get" ]
     - apiGroups: [ "subresources.kubevirt.io" ]
       resources: [ "virtualmachineinstances/addvolume", "virtualmachineinstances/removevolume" ]
       verbs: [ "update" ]
     - apiGroups: [ "kubevirt.io" ]
       resources: [ "virtualmachineinstancemigrations" ]
       verbs: [ "create" ]
     - apiGroups: [ "cdi.kubevirt.io" ]
       resources: [ "datavolumes" ]
       verbs: [ "create", "delete" ]
     - apiGroups: [ "" ]
       resources: [ "persistentvolumeclaims" ]
       verbs: [ "delete" ]
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: RoleBinding
   metadata:
     name: storage-checkup-role
   subjects:
     - kind: ServiceAccount
       name: storage-checkup-sa
   roleRef:
     apiGroup: rbac.authorization.k8s.io
     kind: Role
     name: storage-checkup-role

2. Apply the ServiceAccount, Role, and RoleBinding manifest in the target namespace:

   $ oc apply -n <target_namespace> -f <storage_sa_roles_rolebinding>.yaml

3. Create a ConfigMap and Job manifest file. The config map contains the input parameters for the checkup job.

   Example input config map and job manifest

   ---
   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: storage-checkup-config
     namespace: $CHECKUP_NAMESPACE
   data:
     spec.timeout: 10m
     spec.param.storageClass: ocs-storagecluster-ceph-rbd-virtualization
     spec.param.vmiTimeout: 3m
   ---
   apiVersion: batch/v1
   kind: Job
   metadata:
     name: storage-checkup
     namespace: $CHECKUP_NAMESPACE
   spec:
     backoffLimit: 0
     template:
       spec:
         serviceAccount: storage-checkup-sa
         restartPolicy: Never
         containers:
           - name: storage-checkup
             image: quay.io/kiagnose/kubevirt-storage-checkup:main
             imagePullPolicy: Always
             env:
               - name: CONFIGMAP_NAMESPACE
                 value: $CHECKUP_NAMESPACE
               - name: CONFIGMAP_NAME
                 value: storage-checkup-config

4. Apply the ConfigMap and Job manifest file in the target namespace to run the checkup:

   $ oc apply -n <target_namespace> -f <storage_configmap_job>.yaml

5. Wait for the job to complete:

   $ oc wait job storage-checkup -n <target_namespace> --for condition=complete --timeout 10m

6. Review the results of the checkup by running the following command:

   $ oc get configmap storage-checkup-config -n <target_namespace> -o yaml

   Example output config map (success)

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: storage-checkup-config
     labels:
       kiagnose/checkup-type: kubevirt-storage
   data:
     spec.timeout: 10m
     status.succeeded: "true" 1
     status.failureReason: "" 2
     status.startTimestamp: "2023-07-31T13:14:38Z" 3
     status.completionTimestamp: "2023-07-31T13:19:41Z" 4
     status.result.cnvVersion: 4.16.2 5
     status.result.defaultStorageClass: trident-nfs 6
     status.result.goldenImagesNoDataSource: <data_import_cron_list> 7
     status.result.goldenImagesNotUpToDate: <data_import_cron_list> 8
     status.result.ocpVersion: 4.16.0 9
     status.result.pvcBound: "true" 10
     status.result.storageProfileMissingVolumeSnapshotClass: <storage_class_list> 11
     status.result.storageProfilesWithEmptyClaimPropertySets: <storage_profile_list> 12
     status.result.storageProfilesWithSmartClone: <storage_profile_list> 13
     status.result.storageProfilesWithSpecClaimPropertySets: <storage_profile_list> 14
     status.result.storageProfilesWithRWX: |-
       ocs-storagecluster-ceph-rbd
       ocs-storagecluster-ceph-rbd-virtualization
       ocs-storagecluster-cephfs
       trident-iscsi
       trident-minio
       trident-nfs
       windows-vms
     status.result.vmBootFromGoldenImage: VMI "vmi-under-test-dhkb8" successfully booted
     status.result.vmHotplugVolume: |-
       VMI "vmi-under-test-dhkb8" hotplug volume ready
       VMI "vmi-under-test-dhkb8" hotplug volume removed
     status.result.vmLiveMigration: VMI "vmi-under-test-dhkb8" migration completed
     status.result.vmVolumeClone: 'DV cloneType: "csi-clone"'
     status.result.vmsWithNonVirtRbdStorageClass: <vm_list> 15
     status.result.vmsWithUnsetEfsStorageClass: <vm_list> 16

   1

   Specifies if the checkup is successful (true) or not (false).

   2

   The reason for failure if the checkup fails.

   3

   The time when the checkup started, in RFC 3339 time format.

   4

   The time when the checkup has completed, in RFC 3339 time format.

   5

   The OpenShift Virtualization version.

   6

   Specifies if there is a default storage class.

   7

   The list of golden images whose data source is not ready.

   8

   The list of golden images whose data import cron is not up-to-date.

   9

   The OpenShift Container Platform version.

   10

   Specifies if a PVC of 10Mi has been created and bound by the provisioner.

   11

   The list of storage profiles using snapshot-based clone but missing VolumeSnapshotClass.

   12

   The list of storage profiles with unknown provisioners.

   13

   The list of storage profiles with smart clone support (CSI/snapshot).

   14

   The list of storage profiles spec-overriden claimPropertySets.

   15

   The list of virtual machines that use the Ceph RBD storage class when the virtualization storage class exists.

   16

   The list of virtual machines that use an Elastic File Store (EFS) storage class where the GID and UID are not set in the storage class.

7. Delete the job and config map that you previously created by running the following commands:

   $ oc delete job -n <target_namespace> storage-checkup

   $ oc delete config-map -n <target_namespace> storage-checkup-config

8. Optional: If you do not plan to run another checkup, delete the ServiceAccount, Role, and RoleBinding manifest:

   $ oc delete -f <storage_sa_roles_rolebinding>.yaml

1. Create a service account, role, and role bindings for the DPDK checkup.
2. Create a config map to provide the input to run the checkup and to store the results.
3. Create a job to run the checkup.
4. Review the results in the config map.
5. Optional: To rerun the checkup, delete the existing config map and job and then create a new config map and job.
6. When you are finished, delete the DPDK checkup resources.

Procedure

1. Create a ServiceAccount, Role, and RoleBinding manifest for the DPDK checkup:

   Example 12.3. Example service account, role, and rolebinding manifest file

   ---
   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: dpdk-checkup-sa
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: Role
   metadata:
     name: kiagnose-configmap-access
   rules:
     - apiGroups: [ "" ]
       resources: [ "configmaps" ]
       verbs: [ "get", "update" ]
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: RoleBinding
   metadata:
     name: kiagnose-configmap-access
   subjects:
     - kind: ServiceAccount
       name: dpdk-checkup-sa
   roleRef:
     apiGroup: rbac.authorization.k8s.io
     kind: Role
     name: kiagnose-configmap-access
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: Role
   metadata:
     name: kubevirt-dpdk-checker
   rules:
     - apiGroups: [ "kubevirt.io" ]
       resources: [ "virtualmachineinstances" ]
       verbs: [ "create", "get", "delete" ]
     - apiGroups: [ "subresources.kubevirt.io" ]
       resources: [ "virtualmachineinstances/console" ]
       verbs: [ "get" ]
     - apiGroups: [ "" ]
       resources: [ "configmaps" ]
       verbs: [ "create", "delete" ]
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: RoleBinding
   metadata:
     name: kubevirt-dpdk-checker
   subjects:
     - kind: ServiceAccount
       name: dpdk-checkup-sa
   roleRef:
     apiGroup: rbac.authorization.k8s.io
     kind: Role
     name: kubevirt-dpdk-checker

2. Apply the ServiceAccount, Role, and RoleBinding manifest:

   $ oc apply -n <target_namespace> -f <dpdk_sa_roles_rolebinding>.yaml

3. Create a ConfigMap manifest that contains the input parameters for the checkup:

   Example input config map

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: dpdk-checkup-config
     labels:
       kiagnose/checkup-type: kubevirt-dpdk
   data:
     spec.timeout: 10m
     spec.param.networkAttachmentDefinitionName: <network_name> 1
     spec.param.trafficGenContainerDiskImage: "quay.io/kiagnose/kubevirt-dpdk-checkup-traffic-gen:v0.4.0 2
     spec.param.vmUnderTestContainerDiskImage: "quay.io/kiagnose/kubevirt-dpdk-checkup-vm:v0.4.0" 3

   1

   The name of the NetworkAttachmentDefinition object.

   2

   The container disk image for the traffic generator. In this example, the image is pulled from the upstream Project Quay Container Registry.

   3

   The container disk image for the VM under test. In this example, the image is pulled from the upstream Project Quay Container Registry.

4. Apply the ConfigMap manifest in the target namespace:

   $ oc apply -n <target_namespace> -f <dpdk_config_map>.yaml

5. Create a Job manifest to run the checkup:

   Example job manifest

   apiVersion: batch/v1
   kind: Job
   metadata:
     name: dpdk-checkup
     labels:
       kiagnose/checkup-type: kubevirt-dpdk
   spec:
     backoffLimit: 0
     template:
       spec:
         serviceAccountName: dpdk-checkup-sa
         restartPolicy: Never
         containers:
           - name: dpdk-checkup
             image: registry.redhat.io/container-native-virtualization/kubevirt-dpdk-checkup-rhel9:v4.16.0
             imagePullPolicy: Always
             securityContext:
               allowPrivilegeEscalation: false
               capabilities:
                 drop: ["ALL"]
               runAsNonRoot: true
               seccompProfile:
                 type: "RuntimeDefault"
             env:
               - name: CONFIGMAP_NAMESPACE
                 value: <target-namespace>
               - name: CONFIGMAP_NAME
                 value: dpdk-checkup-config
               - name: POD_UID
                 valueFrom:
                   fieldRef:
                     fieldPath: metadata.uid

6. Apply the Job manifest:

   $ oc apply -n <target_namespace> -f <dpdk_job>.yaml

7. Wait for the job to complete:

   $ oc wait job dpdk-checkup -n <target_namespace> --for condition=complete --timeout 10m

8. Review the results of the checkup by running the following command:

   $ oc get configmap dpdk-checkup-config -n <target_namespace> -o yaml

   Example output config map (success)

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: dpdk-checkup-config
     labels:
       kiagnose/checkup-type: kubevirt-dpdk
   data:
     spec.timeout: 10m
     spec.param.NetworkAttachmentDefinitionName: "dpdk-network-1"
     spec.param.trafficGenContainerDiskImage: "quay.io/kiagnose/kubevirt-dpdk-checkup-traffic-gen:v0.4.0"
     spec.param.vmUnderTestContainerDiskImage: "quay.io/kiagnose/kubevirt-dpdk-checkup-vm:v0.4.0"
     status.succeeded: "true" 1
     status.failureReason: "" 2
     status.startTimestamp: "2023-07-31T13:14:38Z" 3
     status.completionTimestamp: "2023-07-31T13:19:41Z" 4
     status.result.trafficGenSentPackets: "480000000" 5
     status.result.trafficGenOutputErrorPackets: "0" 6
     status.result.trafficGenInputErrorPackets: "0" 7
     status.result.trafficGenActualNodeName: worker-dpdk1 8
     status.result.vmUnderTestActualNodeName: worker-dpdk2 9
     status.result.vmUnderTestReceivedPackets: "480000000" 10
     status.result.vmUnderTestRxDroppedPackets: "0" 11
     status.result.vmUnderTestTxDroppedPackets: "0" 12

   1

   Specifies if the checkup is successful (true) or not (false).

   2

   The reason for failure if the checkup fails.

   3

   The time when the checkup started, in RFC 3339 time format.

   4

   The time when the checkup has completed, in RFC 3339 time format.

   5

   The number of packets sent from the traffic generator.

   6

   The number of error packets sent from the traffic generator.

   7

   The number of error packets received by the traffic generator.

   8

   The node on which the traffic generator VM was scheduled.

   9

   The node on which the VM under test was scheduled.

   10

   The number of packets received on the VM under test.

   11

   The ingress traffic packets that were dropped by the DPDK application.

   12

   The egress traffic packets that were dropped from the DPDK application.

9. Delete the job and config map that you previously created by running the following commands:

   $ oc delete job -n <target_namespace> dpdk-checkup

   $ oc delete config-map -n <target_namespace> dpdk-checkup-config

10. Optional: If you do not plan to run another checkup, delete the ServiceAccount, Role, and RoleBinding manifest:

    $ oc delete -f <dpdk_sa_roles_rolebinding>.yaml

Procedure

1. From the Administrator perspective in the OpenShift Container Platform web console, select Observe Metrics.

2. To add one or more queries, do any of the following:

   Option	Description
   Create a custom query.	Add your Prometheus Query Language (PromQL) query to the Expression field. As you type a PromQL expression, autocomplete suggestions appear in a drop-down list. These suggestions include functions, metrics, labels, and time tokens. You can use the keyboard arrows to select one of these suggested items and then press Enter to add the item to your expression. You can also move your mouse pointer over a suggested item to view a brief description of that item.
   Add multiple queries.	Select Add query.
   Duplicate an existing query.	Select the Options menu next to the query, then choose Duplicate query.
   Disable a query from being run.	Select the Options menu next to the query and choose Disable query.

3. To run queries that you created, select Run queries. The metrics from the queries are visualized on the plot. If a query is invalid, the UI shows an error message.

   Note

   Queries that operate on large amounts of data might time out or overload the browser when drawing time series graphs. To avoid this, select Hide graph and calibrate your query using only the metrics table. Then, after finding a feasible query, enable the plot to draw the graphs.

   Note

   By default, the query table shows an expanded view that lists every metric and its current value. You can select ˅ to minimize the expanded view for a query.

4. Optional: The page URL now contains the queries you ran. To use this set of queries again in the future, save this URL.

5. Explore the visualized metrics. Initially, all metrics from all enabled queries are shown on the plot. You can select which metrics are shown by doing any of the following:

   Option	Description
   Hide all metrics from a query.	Click the Options menu for the query and click Hide all series.
   Hide a specific metric.	Go to the query table and click the colored square near the metric name.
   Zoom into the plot and change the time range.	Either: Visually select the time range by clicking and dragging on the plot horizontally. Use the menu in the left upper corner to select the time range.
   Reset the time range.	Select Reset zoom.
   Display outputs for all queries at a specific point in time.	Hold the mouse cursor on the plot at that point. The query outputs will appear in a pop-up box.
   Hide the plot.	Select Hide graph.

Procedure

1. From the Developer perspective in the OpenShift Container Platform web console, select Observe Metrics.
2. Select the project that you want to view metrics for in the Project: list.

3. Select a query from the Select query list, or create a custom PromQL query based on the selected query by selecting Show PromQL. The metrics from the queries are visualized on the plot.

   Note

   In the Developer perspective, you can only run one query at a time.

4. Explore the visualized metrics by doing any of the following:

   Option	Description
   Zoom into the plot and change the time range.	Either: Visually select the time range by clicking and dragging on the plot horizontally. Use the menu in the left upper corner to select the time range.
   Reset the time range.	Select Reset zoom.
   Display outputs for all queries at a specific point in time.	Hold the mouse cursor on the plot at that point. The query outputs appear in a pop-up box.

Procedure

1. Obtain the HTTP service endpoint by specifying the namespace for the service:

   $ oc get service -n <namespace> <node-exporter-service>

2. To list all available metrics for the node-exporter service, query the metrics resource.

   $ curl http://<172.30.226.162:9100>/metrics | grep -vE "^#|^$"

   Example output

   node_arp_entries{device="eth0"} 1
   node_boot_time_seconds 1.643153218e+09
   node_context_switches_total 4.4938158e+07
   node_cooling_device_cur_state{name="0",type="Processor"} 0
   node_cooling_device_max_state{name="0",type="Processor"} 0
   node_cpu_guest_seconds_total{cpu="0",mode="nice"} 0
   node_cpu_guest_seconds_total{cpu="0",mode="user"} 0
   node_cpu_seconds_total{cpu="0",mode="idle"} 1.10586485e+06
   node_cpu_seconds_total{cpu="0",mode="iowait"} 37.61
   node_cpu_seconds_total{cpu="0",mode="irq"} 233.91
   node_cpu_seconds_total{cpu="0",mode="nice"} 551.47
   node_cpu_seconds_total{cpu="0",mode="softirq"} 87.3
   node_cpu_seconds_total{cpu="0",mode="steal"} 86.12
   node_cpu_seconds_total{cpu="0",mode="system"} 464.15
   node_cpu_seconds_total{cpu="0",mode="user"} 1075.2
   node_disk_discard_time_seconds_total{device="vda"} 0
   node_disk_discard_time_seconds_total{device="vdb"} 0
   node_disk_discarded_sectors_total{device="vda"} 0
   node_disk_discarded_sectors_total{device="vdb"} 0
   node_disk_discards_completed_total{device="vda"} 0
   node_disk_discards_completed_total{device="vdb"} 0
   node_disk_discards_merged_total{device="vda"} 0
   node_disk_discards_merged_total{device="vdb"} 0
   node_disk_info{device="vda",major="252",minor="0"} 1
   node_disk_info{device="vdb",major="252",minor="16"} 1
   node_disk_io_now{device="vda"} 0
   node_disk_io_now{device="vdb"} 0
   node_disk_io_time_seconds_total{device="vda"} 174
   node_disk_io_time_seconds_total{device="vdb"} 0.054
   node_disk_io_time_weighted_seconds_total{device="vda"} 259.79200000000003
   node_disk_io_time_weighted_seconds_total{device="vdb"} 0.039
   node_disk_read_bytes_total{device="vda"} 3.71867136e+08
   node_disk_read_bytes_total{device="vdb"} 366592
   node_disk_read_time_seconds_total{device="vda"} 19.128
   node_disk_read_time_seconds_total{device="vdb"} 0.039
   node_disk_reads_completed_total{device="vda"} 5619
   node_disk_reads_completed_total{device="vdb"} 96
   node_disk_reads_merged_total{device="vda"} 5
   node_disk_reads_merged_total{device="vdb"} 0
   node_disk_write_time_seconds_total{device="vda"} 240.66400000000002
   node_disk_write_time_seconds_total{device="vdb"} 0
   node_disk_writes_completed_total{device="vda"} 71584
   node_disk_writes_completed_total{device="vdb"} 0
   node_disk_writes_merged_total{device="vda"} 19761
   node_disk_writes_merged_total{device="vdb"} 0
   node_disk_written_bytes_total{device="vda"} 2.007924224e+09
   node_disk_written_bytes_total{device="vdb"} 0

Procedure

1. Expose the node-exporter service.

   $ oc expose service -n <namespace> <node_exporter_service_name>

2. Obtain the FQDN (Fully Qualified Domain Name) for the route.

   $ oc get route -o=custom-columns=NAME:.metadata.name,DNS:.spec.host

   Example output

   NAME                    DNS
   node-exporter-service   node-exporter-service-dynamation.apps.cluster.example.org

3. Use the curl command to display metrics for the node-exporter service.

   $ curl -s http://node-exporter-service-dynamation.apps.cluster.example.org/metrics

   Example output

   go_gc_duration_seconds{quantile="0"} 1.5382e-05
   go_gc_duration_seconds{quantile="0.25"} 3.1163e-05
   go_gc_duration_seconds{quantile="0.5"} 3.8546e-05
   go_gc_duration_seconds{quantile="0.75"} 4.9139e-05
   go_gc_duration_seconds{quantile="1"} 0.000189423

Procedure

1. Open the HyperConverged custom resource (CR) in your default editor by running the following command:

   $ oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnv

2. Choose to enable or disable the downwardMetrics feature gate as follows:

   • To enable the downwardMetrics feature gate, add and then set spec.featureGates.downwardMetrics to true. For example:

     apiVersion: hco.kubevirt.io/v1beta1
     kind: HyperConverged
     metadata:
       name: kubevirt-hyperconverged
       namespace: openshift-cnv
     spec:
         featureGates:
           downwardMetrics: true
     # ...

   • To disable the downwardMetrics feature gate, set spec.featureGates.downwardMetrics to false. For example:

     apiVersion: hco.kubevirt.io/v1beta1
     kind: HyperConverged
     metadata:
       name: kubevirt-hyperconverged
       namespace: openshift-cnv
     spec:
         featureGates:
           downwardMetrics: false
     # ...

Procedure

1. Install the vm-dump-metrics tool by running the following command:

   $ sudo dnf install -y vm-dump-metrics

2. Retrieve the metrics results by running the following command:

   $ sudo vm-dump-metrics

   Example output

   <metrics>
     <metric type="string" context="host">
       <name>HostName</name>
       <value>node01</value>
   [...]
     <metric type="int64" context="host" unit="s">
       <name>Time</name>
       <value>1619008605</value>
     </metric>
     <metric type="string" context="host">
       <name>VirtualizationVendor</name>
       <value>kubevirt.io</value>
     </metric>
   </metrics>

Procedure

1. Include details of the readiness probe in the VM configuration file.

   Sample readiness probe with an HTTP GET test

   apiVersion: kubevirt.io/v1
   kind: VirtualMachine
   metadata:
     annotations:
     name: fedora-vm
     namespace: example-namespace
   # ...
   spec:
     template:
       spec:
         readinessProbe:
           httpGet: 1
             port: 1500 2
             path: /healthz 3
             httpHeaders:
             - name: Custom-Header
               value: Awesome
           initialDelaySeconds: 120 4
           periodSeconds: 20 5
           timeoutSeconds: 10 6
           failureThreshold: 3 7
           successThreshold: 3 8
   # ...

   1

   The HTTP GET request to perform to connect to the VM.

   2

   The port of the VM that the probe queries. In the above example, the probe queries port 1500.

   3

   The path to access on the HTTP server. In the above example, if the handler for the server’s /healthz path returns a success code, the VM is considered to be healthy. If the handler returns a failure code, the VM is removed from the list of available endpoints.

   4

   The time, in seconds, after the VM starts before the readiness probe is initiated.

   5

   The delay, in seconds, between performing probes. The default delay is 10 seconds. This value must be greater than timeoutSeconds.

   6

   The number of seconds of inactivity after which the probe times out and the VM is assumed to have failed. The default value is 1. This value must be lower than periodSeconds.

   7

   The number of times that the probe is allowed to fail. The default is 3. After the specified number of attempts, the pod is marked Unready.

   8

   The number of times that the probe must report success, after a failure, to be considered successful. The default is 1.

2. Create the VM by running the following command:

   $ oc create -f <file_name>.yaml

Procedure

1. Include details of the TCP readiness probe in the VM configuration file.

   Sample readiness probe with a TCP socket test

   apiVersion: kubevirt.io/v1
   kind: VirtualMachine
   metadata:
     annotations:
     name: fedora-vm
     namespace: example-namespace
   # ...
   spec:
     template:
       spec:
         readinessProbe:
           initialDelaySeconds: 120 1
           periodSeconds: 20 2
           tcpSocket: 3
             port: 1500 4
           timeoutSeconds: 10 5
   # ...

   1

   The time, in seconds, after the VM starts before the readiness probe is initiated.

   2

   The delay, in seconds, between performing probes. The default delay is 10 seconds. This value must be greater than timeoutSeconds.

   3

   The TCP action to perform.

   4

   The port of the VM that the probe queries.

   5

   The number of seconds of inactivity after which the probe times out and the VM is assumed to have failed. The default value is 1. This value must be lower than periodSeconds.

2. Create the VM by running the following command:

   $ oc create -f <file_name>.yaml

Procedure

1. Include details of the HTTP liveness probe in the VM configuration file.

   Sample liveness probe with an HTTP GET test

   apiVersion: kubevirt.io/v1
   kind: VirtualMachine
   metadata:
     annotations:
     name: fedora-vm
     namespace: example-namespace
   # ...
   spec:
     template:
       spec:
         livenessProbe:
           initialDelaySeconds: 120 1
           periodSeconds: 20 2
           httpGet: 3
             port: 1500 4
             path: /healthz 5
             httpHeaders:
             - name: Custom-Header
               value: Awesome
           timeoutSeconds: 10 6
   # ...

   1

   The time, in seconds, after the VM starts before the liveness probe is initiated.

   2

   The delay, in seconds, between performing probes. The default delay is 10 seconds. This value must be greater than timeoutSeconds.

   3

   The HTTP GET request to perform to connect to the VM.

   4

   The port of the VM that the probe queries. In the above example, the probe queries port 1500. The VM installs and runs a minimal HTTP server on port 1500 via cloud-init.

   5

   The path to access on the HTTP server. In the above example, if the handler for the server’s /healthz path returns a success code, the VM is considered to be healthy. If the handler returns a failure code, the VM is deleted and a new VM is created.

   6

   The number of seconds of inactivity after which the probe times out and the VM is assumed to have failed. The default value is 1. This value must be lower than periodSeconds.

2. Create the VM by running the following command:

   $ oc create -f <file_name>.yaml

Procedure

1. Create a YAML file with the following contents:

   apiVersion: kubevirt.io/v1
   kind: VirtualMachine
   metadata:
     labels:
       kubevirt.io/vm: vm2-rhel84-watchdog
     name: <vm-name>
   spec:
     running: false
     template:
       metadata:
         labels:
           kubevirt.io/vm: vm2-rhel84-watchdog
       spec:
         domain:
           devices:
             watchdog:
               name: <watchdog>
               i6300esb:
                 action: "poweroff" 1
   # ...

   1

   Specify poweroff, reset, or shutdown.

   The example above configures the i6300esb watchdog device on a RHEL8 VM with the poweroff action and exposes the device as /dev/watchdog.

   This device can now be used by the watchdog binary.

2. Apply the YAML file to your cluster by running the following command:

   $ oc apply -f <file_name>.yaml

1. Run the following command to verify that the VM is connected to the watchdog device:

   $ lspci | grep watchdog -i

2. Run one of the following commands to confirm the watchdog is active:

   • Trigger a kernel panic:

     # echo c > /proc/sysrq-trigger

   • Stop the watchdog service:

     # pkill -9 watchdog

Procedure

1. Log in to the virtual machine as root user.

2. Install the watchdog package and its dependencies:

   # yum install watchdog

3. Uncomment the following line in the /etc/watchdog.conf file and save the changes:

   #watchdog-device = /dev/watchdog

4. Enable the watchdog service to start on boot:

   # systemctl enable --now watchdog.service