Chapter 13. Monitoring

Procedure

1. Run the following command to edit the storage-checkup-config config map:

   $ oc edit configmap storage-checkup-config -n <checkup_namespace>

2. Configure the skipTeardown field to use the onfailure value. You can do this by modifying the storage-checkup-config config map, stored in the storage_checkup.yaml file:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: storage-checkup-config
     namespace: <checkup_namespace>
   data:
     spec.param.skipTeardown: onfailure
   # ...

3. Reapply the storage-checkup-config config map by running the following command:

   $ oc apply -f storage_checkup.yaml -n <checkup_namespace>

Procedure

1. Navigate to Virtualization Checkups in the web console.
2. Click the Storage tab.
3. Click Install permissions.
4. Click Run checkup.
5. Enter a name for the checkup in the Name field.
6. Enter a timeout value for the checkup in the Timeout (minutes) fields.
7. Click Run.

Procedure

1. Create a ServiceAccount, Role, and RoleBinding manifest file for the storage checkup.

   Example service account, role, and rolebinding manifest:

   ---
   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: storage-checkup-sa
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: Role
   metadata:
     name: storage-checkup-role
   rules:
     - apiGroups: [ "" ]
       resources: [ "configmaps" ]
       verbs: ["get", "update"]
     - apiGroups: [ "kubevirt.io" ]
       resources: [ "virtualmachines" ]
       verbs: [ "create", "delete" ]
     - apiGroups: [ "kubevirt.io" ]
       resources: [ "virtualmachineinstances" ]
       verbs: [ "get" ]
     - apiGroups: [ "subresources.kubevirt.io" ]
       resources: [ "virtualmachineinstances/addvolume", "virtualmachineinstances/removevolume" ]
       verbs: [ "update" ]
     - apiGroups: [ "kubevirt.io" ]
       resources: [ "virtualmachineinstancemigrations" ]
       verbs: [ "create" ]
     - apiGroups: [ "cdi.kubevirt.io" ]
       resources: [ "datavolumes" ]
       verbs: [ "create", "delete" ]
     - apiGroups: [ "" ]
       resources: [ "persistentvolumeclaims" ]
       verbs: [ "delete" ]
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: RoleBinding
   metadata:
     name: storage-checkup-role
   subjects:
     - kind: ServiceAccount
       name: storage-checkup-sa
   roleRef:
     apiGroup: rbac.authorization.k8s.io
     kind: Role
     name: storage-checkup-role

2. Apply the ServiceAccount, Role, and RoleBinding manifest in the target namespace:

   $ oc apply -n <target_namespace> -f <storage_sa_roles_rolebinding>.yaml

3. Create a ConfigMap and Job manifest file. The config map contains the input parameters for the checkup job.

   Example input config map and job manifest:

   ---
   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: storage-checkup-config
     namespace: $CHECKUP_NAMESPACE
   data:
     spec.timeout: 10m
     spec.param.storageClass: ocs-storagecluster-ceph-rbd-virtualization
     spec.param.vmiTimeout: 3m
   ---
   apiVersion: batch/v1
   kind: Job
   metadata:
     name: storage-checkup
     namespace: $CHECKUP_NAMESPACE
   spec:
     backoffLimit: 0
     template:
       spec:
         serviceAccount: storage-checkup-sa
         restartPolicy: Never
         containers:
           - name: storage-checkup
             image: quay.io/kiagnose/kubevirt-storage-checkup:main
             imagePullPolicy: Always
             env:
               - name: CONFIGMAP_NAMESPACE
                 value: $CHECKUP_NAMESPACE
               - name: CONFIGMAP_NAME
                 value: storage-checkup-config

4. Apply the ConfigMap and Job manifest file in the target namespace to run the checkup:

   $ oc apply -n <target_namespace> -f <storage_configmap_job>.yaml

5. Wait for the job to complete:

   $ oc wait job storage-checkup -n <target_namespace> --for condition=complete --timeout 10m

6. Review the results of the checkup by running the following command:

   $ oc get configmap storage-checkup-config -n <target_namespace> -o yaml

   Example output config map (success):

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: storage-checkup-config
     labels:
       kiagnose/checkup-type: kubevirt-storage
   data:
     spec.timeout: 10m
     status.succeeded: "true"
     status.failureReason: ""
     status.startTimestamp: "2023-07-31T13:14:38Z"
     status.completionTimestamp: "2023-07-31T13:19:41Z"
     status.result.cnvVersion: 4.21.2
     status.result.defaultStorageClass: trident-nfs
     status.result.goldenImagesNoDataSource: <data_import_cron_list>
     status.result.goldenImagesNotUpToDate: <data_import_cron_list>
     status.result.ocpVersion: 4.21.0
     status.result.pvcBound: "true"
     status.result.storageProfileMissingVolumeSnapshotClass: <storage_class_list>
     status.result.storageProfilesWithEmptyClaimPropertySets: <storage_profile_list>
     status.result.storageProfilesWithSmartClone: <storage_profile_list>
     status.result.storageProfilesWithSpecClaimPropertySets: <storage_profile_list>
     status.result.storageProfilesWithRWX: |-
       ocs-storagecluster-ceph-rbd
       ocs-storagecluster-ceph-rbd-virtualization
       ocs-storagecluster-cephfs
       trident-iscsi
       trident-minio
       trident-nfs
       windows-vms
     status.result.vmBootFromGoldenImage: VMI "vmi-under-test-dhkb8" successfully booted
     status.result.vmHotplugVolume: |-
       VMI "vmi-under-test-dhkb8" hotplug volume ready
       VMI "vmi-under-test-dhkb8" hotplug volume removed
     status.result.vmLiveMigration: VMI "vmi-under-test-dhkb8" migration completed
     status.result.vmVolumeClone: 'DV cloneType: "csi-clone"'
     status.result.vmsWithNonVirtRbdStorageClass: <vm_list>
     status.result.vmsWithUnsetEfsStorageClass: <vm_list>

   • data.status.succeeded defines if the checkup is successful (true) or not (false).
   • data.status.failureReason defines the reason for failure if the checkup fails.
   • data.status.startTimestamp defines the time when the checkup started, in RFC 3339 time format.
   • data.status.completionTimestamp defines the time when the checkup has completed, in RFC 3339 time format.
   • data.status.result.cnvVersion defines the OpenShift Virtualization version.
   • data.status.result.defaultStorageClass defines if there is a default storage class.
   • data.status.result.goldenImagesNoDataSource defines the list of golden images whose data source is not ready.
   • data.status.result.goldenImagesNotUpToDate defines the list of golden images whose data import cron is not up-to-date.
   • data.status.result.ocpVersion defines the OpenShift Container Platform version.
   • data.status.result.pvcBound defines if a PVC of 10Mi has been created and bound by the provisioner.
   • data.status.result.storageProfileMissingVolumeSnapshotClass defines the list of storage profiles using snapshot-based clone but missing VolumeSnapshotClass.
   • data.status.result.storageProfilesWithEmptyClaimPropertySets defines the list of storage profiles with unknown provisioners.
   • data.status.result.storageProfilesWithSmartClone defines the list of storage profiles with smart clone support (CSI/snapshot).
   • data.status.result.storageProfilesWithSpecClaimPropertySets defines the list of storage profiles spec-overriden claimPropertySets.
   • data.status.result.vmsWithNonVirtRbdStorageClass defines the list of virtual machines that use the Ceph RBD storage class when the virtualization storage class exists.
   • data.status.result.vmsWithUnsetEfsStorageClass defines the list of virtual machines that use an Elastic File Store (EFS) storage class where the GID and UID are not set in the storage class.

7. Delete the job and config map that you previously created by running the following commands:

   $ oc delete job -n <target_namespace> storage-checkup

   $ oc delete config-map -n <target_namespace> storage-checkup-config

8. Optional: If you do not plan to run another checkup, delete the ServiceAccount, Role, and RoleBinding manifest:

   $ oc delete -f <storage_sa_roles_rolebinding>.yaml

Procedure

1. Review the status.failureReason field in the storage-checkup-config config map by running the following command and observing the output:

   $ oc get configmap storage-checkup-config -n <namespace> -o yaml

   Example output config map:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: storage-checkup-config
     labels:
       kiagnose/checkup-type: kubevirt-storage
   data:
     spec.timeout: 10m
     status.succeeded: "false"
     status.failureReason: "ErrNoDefaultStorageClass"
   # ...

   • If the checkup has failed, the status.succeeded value is false.
   • If the checkup has failed, the status.failureReason field contains an error message. In this example output, the ErrNoDefaultStorageClass error message means that no default storage class is configured.
2. Search the directory provided by the must-gather tool for logs, events, or terms related to the error in the data.status.failureReason field value.

Procedure

1. In the OpenShift Container Platform web console, click Observe Metrics.

2. To add one or more queries, perform any of the following actions:

   Option	Description
   Select an existing query.	From the Select query drop-down list, select an existing query.
   Create a custom query.	Add your Prometheus Query Language (PromQL) query to the Expression field. As you type a PromQL expression, autocomplete suggestions appear in a drop-down list. These suggestions include functions, metrics, labels, and time tokens. Use the keyboard arrows to select one of these suggested items and then press Enter to add the item to your expression. Move your mouse pointer over a suggested item to view a brief description of that item.
   Add multiple queries.	Click Add query.
   Duplicate an existing query.	Click the options menu next to the query, then choose Duplicate query.
   Disable a query from being run.	Click the options menu next to the query and choose Disable query.

3. To run queries that you created, click Run queries. The metrics from the queries are visualized on the plot. If a query is invalid, the UI shows an error message.

   Note
   • When drawing time series graphs, queries that operate on large amounts of data might time out or overload the browser. To avoid this, click Hide graph and calibrate your query by using only the metrics table. Then, after finding a feasible query, enable the plot to draw the graphs.
   • By default, the query table shows an expanded view that lists every metric and its current value. Click the ˅ down arrowhead to minimize the expanded view for a query.
4. Optional: Save the page URL to use this set of queries again in the future.

5. Explore the visualized metrics. Initially, all metrics from all enabled queries are shown on the plot. Select which metrics are shown by performing any of the following actions:

   Option	Description
   Hide all metrics from a query.	Click the options menu for the query and click Hide all series.
   Hide a specific metric.	Go to the query table and click the colored square near the metric name.
   Zoom into the plot and change the time range.	Perform one of the following actions: Visually select the time range by clicking and dragging on the plot horizontally. Use the menu to select the time range.
   Reset the time range.	Click Reset zoom.
   Display outputs for all queries at a specific point in time.	Hover over the plot at the point you are interested in. The query outputs appear in a pop-up box.
   Hide the plot.	Click Hide graph.

Procedure

1. In the OpenShift Container Platform web console, click Observe Metrics.

2. To add one or more queries, perform any of the following actions:

   Option	Description
   Select an existing query.	From the Select query drop-down list, select an existing query.
   Create a custom query.	Add your Prometheus Query Language (PromQL) query to the Expression field. As you type a PromQL expression, autocomplete suggestions appear in a drop-down list. These suggestions include functions, metrics, labels, and time tokens. Use the keyboard arrows to select one of these suggested items and then press Enter to add the item to your expression. Move your mouse pointer over a suggested item to view a brief description of that item.
   Add multiple queries.	Click Add query.
   Duplicate an existing query.	Click the options menu next to the query, then choose Duplicate query.
   Disable a query from being run.	Click the options menu next to the query and choose Disable query.

3. To run queries that you created, click Run queries. The metrics from the queries are visualized on the plot. If a query is invalid, the UI shows an error message.

   Note
   • When drawing time series graphs, queries that operate on large amounts of data might time out or overload the browser. To avoid this, click Hide graph and calibrate your query by using only the metrics table. Then, after finding a feasible query, enable the plot to draw the graphs.
   • By default, the query table shows an expanded view that lists every metric and its current value. Click the ˅ down arrowhead to minimize the expanded view for a query.
4. Optional: Save the page URL to use this set of queries again in the future.

5. Explore the visualized metrics. Initially, all metrics from all enabled queries are shown on the plot. Select which metrics are shown by performing any of the following actions:

   Option	Description
   Hide all metrics from a query.	Click the options menu for the query and click Hide all series.
   Hide a specific metric.	Go to the query table and click the colored square near the metric name.
   Zoom into the plot and change the time range.	Perform one of the following actions: Visually select the time range by clicking and dragging on the plot horizontally. Use the menu to select the time range.
   Reset the time range.	Click Reset zoom.
   Display outputs for all queries at a specific point in time.	Hover over the plot at the point you are interested in. The query outputs appear in a pop-up box.
   Hide the plot.	Click Hide graph.

Procedure

1. Create the Service YAML file. In the following example, the file is called node-exporter-service.yaml.

   kind: Service
   apiVersion: v1
   metadata:
     name: node-exporter-service
     namespace: dynamation
     labels:
       servicetype: metrics
   spec:
     ports:
       - name: exmet
         protocol: TCP
         port: 9100
         targetPort: 9100
     type: ClusterIP
     selector:
       monitor: metrics

   • metadata.name defines the node-exporter service that exposes the metrics from the virtual machines.
   • metadata.namespace defines the namespace where the service is created.
   • metadata.labels.servicetype defines the label for the service. The ServiceMonitor uses this label to match this service.
   • spec.ports.name defines the name given to the port that exposes metrics on port 9100 for the ClusterIP service.
   • spec.ports.port defines the target port used by node-exporter-service to listen for requests.
   • spec.ports.targetPort defines the TCP port number of the virtual machine that is configured with the monitor label.
   • spec.selector.monitor defines the label used to match the virtual machine’s pods. In this example, any virtual machine’s pod with the label monitor and a value of metrics will be matched.

2. Create the node-exporter service:

   $ oc create -f node-exporter-service.yaml

Procedure

1. Log on to the virtual machine.

2. Download the node-exporter file on to the virtual machine by using the directory path that applies to the version of node-exporter file.

   $ wget https://github.com/prometheus/node_exporter/releases/download/<version>/node_exporter-<version>.linux-<architecture>.tar.gz

3. Extract the executable and place it in the /usr/bin directory.

   $ sudo tar xvf node_exporter-<version>.linux-<architecture>.tar.gz \
       --directory /usr/bin --strip 1 "*/node_exporter"

4. Create a node_exporter.service file in this directory path: /etc/systemd/system. This systemd service file runs the node-exporter service when the virtual machine reboots.

   [Unit]
   Description=Prometheus Metrics Exporter
   After=network.target
   StartLimitIntervalSec=0
   
   [Service]
   Type=simple
   Restart=always
   RestartSec=1
   User=root
   ExecStart=/usr/bin/node_exporter
   
   [Install]
   WantedBy=multi-user.target

5. Enable and start the systemd service.

   $ sudo systemctl enable node_exporter.service

   $ sudo systemctl start node_exporter.service

Procedure

1. Edit the template spec of your virtual machine configuration file. In this example, the label monitor has the value metrics.

   spec:
     template:
       metadata:
         labels:
           monitor: metrics

2. Stop and restart the virtual machine to create a new pod with the label name given to the monitor label.

Procedure

1. Create a YAML file for the ServiceMonitor resource configuration. In this example, the service monitor matches any service with the label metrics and queries the exmet port every 30 seconds.

   apiVersion: monitoring.coreos.com/v1
   kind: ServiceMonitor
   metadata:
     labels:
       k8s-app: node-exporter-metrics-monitor
     name: node-exporter-metrics-monitor
     namespace: dynamation
   spec:
     endpoints:
     - interval: 30s
       port: exmet
       scheme: http
     selector:
       matchLabels:
         servicetype: metrics

   • metadata.name defines the name of the ServiceMonitor.
   • metadata.namespace defines the namespace where the ServiceMonitor is created.
   • spec.endpoints.interval defines the interval at which the port will be queried.
   • spec.endpoints.port defines the name of the port that is queried every 30 seconds

2. Create the ServiceMonitor configuration for the node-exporter service.

   $ oc create -f node-exporter-metrics-monitor.yaml

Procedure

1. Open the HyperConverged custom resource (CR) in your default editor by running the following command:

   $ oc edit hco kubevirt-hyperconverged -n openshift-cnv

2. Choose to enable or disable the downwardMetrics feature gate as follows:

   • To enable the downwardMetrics feature gate, add a feature gate entry with name: downwardMetrics. For example:

     apiVersion: hco.kubevirt.io/v1
     kind: HyperConverged
     metadata:
       name: kubevirt-hyperconverged
       namespace: openshift-cnv
     spec:
       featureGates:
         - name: downwardMetrics
     # ...

   • To disable the downwardMetrics feature gate, set the state to Disabled. For example:

     apiVersion: hco.kubevirt.io/v1
     kind: HyperConverged
     metadata:
       name: kubevirt-hyperconverged
       namespace: openshift-cnv
     spec:
       featureGates:
         - name: downwardMetrics
           state: Disabled
     # ...

Procedure

1. Install the vm-dump-metrics tool by running the following command:

   $ sudo dnf install -y vm-dump-metrics

2. Retrieve the metrics results by running the following command:

   $ sudo vm-dump-metrics

   Example output:

   <metrics>
     <metric type="string" context="host">
       <name>HostName</name>
       <value>node01</value>
   [...]
     <metric type="int64" context="host" unit="s">
       <name>Time</name>
       <value>1619008605</value>
     </metric>
     <metric type="string" context="host">
       <name>VirtualizationVendor</name>
       <value>kubevirt.io</value>
     </metric>
   </metrics>

Procedure

1. Include details of the guest agent ping probe in the VM configuration file. For example:

   apiVersion: kubevirt.io/v1
   kind: VirtualMachine
   metadata:
     annotations:
     name: fedora-vm
     namespace: example-namespace
   # ...
   spec:
     template:
       spec:
         readinessProbe:
           guestAgentPing: {}
           initialDelaySeconds: 120
           periodSeconds: 20
           timeoutSeconds: 10
           failureThreshold: 3
           successThreshold: 3
   # ...

   • spec.template.spec.readinessProbe.guestAgentPing defines the guest agent ping probe to connect to the VM.
   • spec.template.spec.readinessProbe.initialDelaySeconds defines the time, in seconds, after the VM starts before the guest agent probe is initiated. This value is optional.
   • spec.template.spec.readinessProbe.periodSeconds defines the delay, in seconds, between performing probes. The default delay is 10 seconds. This value must be greater than timeoutSeconds. This value is optional
   • spec.template.spec.readinessProbe.timeoutSeconds defines the number of seconds of inactivity after which the probe times out and the VM is assumed to have failed. The default value is 1. This value must be lower than periodSeconds. This value is optional.
   • spec.template.spec.readinessProbe.failureThreshold defines the number of times that the probe is allowed to fail. The default is 3. After the specified number of attempts, the pod is marked Unready. This value is optional.
   • spec.template.spec.readinessProbe.successThreshold defines the number of times that the probe must report success, after a failure, to be considered successful. The default is 1. This value is optional.

2. Create the VM by running the following command:

   $ oc create -f <file_name>.yaml