Chapter 14. Support

Procedure

1. Verify that you can build a RHEL 8.7 image:

   # composer-cli distros list

   Note

   To run the composer-cli commands as non-root, add your user to the weldr or root groups:

   # usermod -a -G weldr user

   $ newgrp weldr

2. Enter the following command to create an image blueprint file in TOML format that contains the packages to be installed, kernel customizations, and the services to be disabled during boot time:

   $ cat << EOF > dpdk-vm.toml
   name = "dpdk_image"
   description = "Image to use with the DPDK checkup"
   version = "0.0.1"
   distro = "rhel-87"
   
   [[packages]]
   name = "dpdk"
   
   [[packages]]
   name = "dpdk-tools"
   
   [[packages]]
   name = "driverctl"
   
   [[packages]]
   name = "tuned-profiles-cpu-partitioning"
   
   [customizations.kernel]
   append = "default_hugepagesz=1GB hugepagesz=1G hugepages=8 isolcpus=2-7"
   
   [customizations.services]
   disabled = ["NetworkManager-wait-online", "sshd"]
   EOF

3. Push the blueprint file to the image builder tool by running the following command:

   # composer-cli blueprints push dpdk-vm.toml

4. Generate the system image by specifying the blueprint name and output file format. The Universally Unique Identifier (UUID) of the image is displayed when you start the compose process.

   # composer-cli compose start dpdk_image qcow2

5. Wait for the compose process to complete. The compose status must show FINISHED before you can continue to the next step.

   # composer-cli compose status

6. Enter the following command to download the qcow2 image file by specifying its UUID:

   # composer-cli compose image <UUID>

7. Create the customization scripts by running the following commands:

   $ cat <<EOF >customize-vm
   echo  isolated_cores=2-7 > /etc/tuned/cpu-partitioning-variables.conf
   tuned-adm profile cpu-partitioning
   echo "options vfio enable_unsafe_noiommu_mode=1" > /etc/modprobe.d/vfio-noiommu.conf
   EOF

   $ cat <<EOF >first-boot
   driverctl set-override 0000:06:00.0 vfio-pci
   driverctl set-override 0000:07:00.0 vfio-pci
   
   mkdir /mnt/huge
   mount /mnt/huge --source nodev -t hugetlbfs -o pagesize=1GB
   EOF

8. Use the virt-customize tool to customize the image generated by the image builder tool:

   $ virt-customize -a <UUID>.qcow2 --run=customize-vm --firstboot=first-boot --selinux-relabel

9. To create a Dockerfile that contains all the commands to build the container disk image, enter the following command:

   $ cat << EOF > Dockerfile
   FROM scratch
   COPY <uuid>-disk.qcow2 /disk/
   EOF

   where:

   <uuid>-disk.qcow2

   Specifies the name of the custom image in qcow2 format.

10. Build and tag the container by running the following command:

    $ podman build . -t dpdk-rhel:latest

11. Push the container disk image to a registry that is accessible from your cluster by running the following command:

    $ podman push dpdk-rhel:latest

12. Provide a link to the container disk image in the spec.param.vmContainerDiskImage attribute in the DPDK checkup config map.

Procedure

1. From the Administrator perspective in the OpenShift Container Platform web console, select Observe Metrics.

2. To add one or more queries, do any of the following:

   Option	Description
   Create a custom query.	Add your Prometheus Query Language (PromQL) query to the Expression field. As you type a PromQL expression, autocomplete suggestions appear in a drop-down list. These suggestions include functions, metrics, labels, and time tokens. You can use the keyboard arrows to select one of these suggested items and then press Enter to add the item to your expression. You can also move your mouse pointer over a suggested item to view a brief description of that item.
   Add multiple queries.	Select Add query.
   Duplicate an existing query.	Select the Options menu next to the query, then choose Duplicate query.
   Disable a query from being run.	Select the Options menu next to the query and choose Disable query.

3. To run queries that you created, select Run queries. The metrics from the queries are visualized on the plot. If a query is invalid, the UI shows an error message.

   Note

   Queries that operate on large amounts of data might time out or overload the browser when drawing time series graphs. To avoid this, select Hide graph and calibrate your query using only the metrics table. Then, after finding a feasible query, enable the plot to draw the graphs.

   Note

   By default, the query table shows an expanded view that lists every metric and its current value. You can select ˅ to minimize the expanded view for a query.

4. Optional: The page URL now contains the queries you ran. To use this set of queries again in the future, save this URL.

5. Explore the visualized metrics. Initially, all metrics from all enabled queries are shown on the plot. You can select which metrics are shown by doing any of the following:

   Option	Description
   Hide all metrics from a query.	Click the Options menu for the query and click Hide all series.
   Hide a specific metric.	Go to the query table and click the colored square near the metric name.
   Zoom into the plot and change the time range.	Either: Visually select the time range by clicking and dragging on the plot horizontally. Use the menu in the left upper corner to select the time range.
   Reset the time range.	Select Reset zoom.
   Display outputs for all queries at a specific point in time.	Hold the mouse cursor on the plot at that point. The query outputs will appear in a pop-up box.
   Hide the plot.	Select Hide graph.

Procedure

1. From the Developer perspective in the OpenShift Container Platform web console, select Observe Metrics.
2. Select the project that you want to view metrics for in the Project: list.

3. Select a query from the Select query list, or create a custom PromQL query based on the selected query by selecting Show PromQL. The metrics from the queries are visualized on the plot.

   Note

   In the Developer perspective, you can only run one query at a time.

4. Explore the visualized metrics by doing any of the following:

   Option	Description
   Zoom into the plot and change the time range.	Either: Visually select the time range by clicking and dragging on the plot horizontally. Use the menu in the left upper corner to select the time range.
   Reset the time range.	Select Reset zoom.
   Display outputs for all queries at a specific point in time.	Hold the mouse cursor on the plot at that point. The query outputs appear in a pop-up box.

Procedure

1. Obtain the HTTP service endpoint by specifying the namespace for the service:

   $ oc get service -n <namespace> <node-exporter-service>

2. To list all available metrics for the node-exporter service, query the metrics resource.

   $ curl http://<172.30.226.162:9100>/metrics | grep -vE "^#|^$"

   Example output

   node_arp_entries{device="eth0"} 1
   node_boot_time_seconds 1.643153218e+09
   node_context_switches_total 4.4938158e+07
   node_cooling_device_cur_state{name="0",type="Processor"} 0
   node_cooling_device_max_state{name="0",type="Processor"} 0
   node_cpu_guest_seconds_total{cpu="0",mode="nice"} 0
   node_cpu_guest_seconds_total{cpu="0",mode="user"} 0
   node_cpu_seconds_total{cpu="0",mode="idle"} 1.10586485e+06
   node_cpu_seconds_total{cpu="0",mode="iowait"} 37.61
   node_cpu_seconds_total{cpu="0",mode="irq"} 233.91
   node_cpu_seconds_total{cpu="0",mode="nice"} 551.47
   node_cpu_seconds_total{cpu="0",mode="softirq"} 87.3
   node_cpu_seconds_total{cpu="0",mode="steal"} 86.12
   node_cpu_seconds_total{cpu="0",mode="system"} 464.15
   node_cpu_seconds_total{cpu="0",mode="user"} 1075.2
   node_disk_discard_time_seconds_total{device="vda"} 0
   node_disk_discard_time_seconds_total{device="vdb"} 0
   node_disk_discarded_sectors_total{device="vda"} 0
   node_disk_discarded_sectors_total{device="vdb"} 0
   node_disk_discards_completed_total{device="vda"} 0
   node_disk_discards_completed_total{device="vdb"} 0
   node_disk_discards_merged_total{device="vda"} 0
   node_disk_discards_merged_total{device="vdb"} 0
   node_disk_info{device="vda",major="252",minor="0"} 1
   node_disk_info{device="vdb",major="252",minor="16"} 1
   node_disk_io_now{device="vda"} 0
   node_disk_io_now{device="vdb"} 0
   node_disk_io_time_seconds_total{device="vda"} 174
   node_disk_io_time_seconds_total{device="vdb"} 0.054
   node_disk_io_time_weighted_seconds_total{device="vda"} 259.79200000000003
   node_disk_io_time_weighted_seconds_total{device="vdb"} 0.039
   node_disk_read_bytes_total{device="vda"} 3.71867136e+08
   node_disk_read_bytes_total{device="vdb"} 366592
   node_disk_read_time_seconds_total{device="vda"} 19.128
   node_disk_read_time_seconds_total{device="vdb"} 0.039
   node_disk_reads_completed_total{device="vda"} 5619
   node_disk_reads_completed_total{device="vdb"} 96
   node_disk_reads_merged_total{device="vda"} 5
   node_disk_reads_merged_total{device="vdb"} 0
   node_disk_write_time_seconds_total{device="vda"} 240.66400000000002
   node_disk_write_time_seconds_total{device="vdb"} 0
   node_disk_writes_completed_total{device="vda"} 71584
   node_disk_writes_completed_total{device="vdb"} 0
   node_disk_writes_merged_total{device="vda"} 19761
   node_disk_writes_merged_total{device="vdb"} 0
   node_disk_written_bytes_total{device="vda"} 2.007924224e+09
   node_disk_written_bytes_total{device="vdb"} 0

Procedure

1. Expose the node-exporter service.

   $ oc expose service -n <namespace> <node_exporter_service_name>

2. Obtain the FQDN (Fully Qualified Domain Name) for the route.

   $ oc get route -o=custom-columns=NAME:.metadata.name,DNS:.spec.host

   Example output

   NAME                    DNS
   node-exporter-service   node-exporter-service-dynamation.apps.cluster.example.org

3. Use the curl command to display metrics for the node-exporter service.

   $ curl -s http://node-exporter-service-dynamation.apps.cluster.example.org/metrics

   Example output

   go_gc_duration_seconds{quantile="0"} 1.5382e-05
   go_gc_duration_seconds{quantile="0.25"} 3.1163e-05
   go_gc_duration_seconds{quantile="0.5"} 3.8546e-05
   go_gc_duration_seconds{quantile="0.75"} 4.9139e-05
   go_gc_duration_seconds{quantile="1"} 0.000189423

Procedure

1. Include details of the readiness probe in the VM configuration file.

   Sample readiness probe with an HTTP GET test

   # ...
   spec:
     readinessProbe:
       httpGet: 1
         port: 1500 2
         path: /healthz 3
         httpHeaders:
         - name: Custom-Header
           value: Awesome
       initialDelaySeconds: 120 4
       periodSeconds: 20 5
       timeoutSeconds: 10 6
       failureThreshold: 3 7
       successThreshold: 3 8
   # ...

   1

   The HTTP GET request to perform to connect to the VM.

   2

   The port of the VM that the probe queries. In the above example, the probe queries port 1500.

   3

   The path to access on the HTTP server. In the above example, if the handler for the server’s /healthz path returns a success code, the VM is considered to be healthy. If the handler returns a failure code, the VM is removed from the list of available endpoints.

   4

   The time, in seconds, after the VM starts before the readiness probe is initiated.

   5

   The delay, in seconds, between performing probes. The default delay is 10 seconds. This value must be greater than timeoutSeconds.

   6

   The number of seconds of inactivity after which the probe times out and the VM is assumed to have failed. The default value is 1. This value must be lower than periodSeconds.

   7

   The number of times that the probe is allowed to fail. The default is 3. After the specified number of attempts, the pod is marked Unready.

   8

   The number of times that the probe must report success, after a failure, to be considered successful. The default is 1.

2. Create the VM by running the following command:

   $ oc create -f <file_name>.yaml

Procedure

1. Include details of the TCP readiness probe in the VM configuration file.

   Sample readiness probe with a TCP socket test

   # ...
   spec:
     readinessProbe:
       initialDelaySeconds: 120 1
       periodSeconds: 20 2
       tcpSocket: 3
         port: 1500 4
       timeoutSeconds: 10 5
   # ...

   1

   The time, in seconds, after the VM starts before the readiness probe is initiated.

   2

   The delay, in seconds, between performing probes. The default delay is 10 seconds. This value must be greater than timeoutSeconds.

   3

   The TCP action to perform.

   4

   The port of the VM that the probe queries.

   5

   The number of seconds of inactivity after which the probe times out and the VM is assumed to have failed. The default value is 1. This value must be lower than periodSeconds.

2. Create the VM by running the following command:

   $ oc create -f <file_name>.yaml

Procedure

1. Include details of the HTTP liveness probe in the VM configuration file.

   Sample liveness probe with an HTTP GET test

   # ...
   spec:
     livenessProbe:
       initialDelaySeconds: 120 1
       periodSeconds: 20 2
       httpGet: 3
         port: 1500 4
         path: /healthz 5
         httpHeaders:
         - name: Custom-Header
           value: Awesome
       timeoutSeconds: 10 6
   # ...

   1

   The time, in seconds, after the VM starts before the liveness probe is initiated.

   2

   The delay, in seconds, between performing probes. The default delay is 10 seconds. This value must be greater than timeoutSeconds.

   3

   The HTTP GET request to perform to connect to the VM.

   4

   The port of the VM that the probe queries. In the above example, the probe queries port 1500. The VM installs and runs a minimal HTTP server on port 1500 via cloud-init.

   5

   The path to access on the HTTP server. In the above example, if the handler for the server’s /healthz path returns a success code, the VM is considered to be healthy. If the handler returns a failure code, the VM is deleted and a new VM is created.

   6

   The number of seconds of inactivity after which the probe times out and the VM is assumed to have failed. The default value is 1. This value must be lower than periodSeconds.

2. Create the VM by running the following command:

   $ oc create -f <file_name>.yaml

Procedure

1. Create a YAML file with the following contents:

   apiVersion: kubevirt.io/v1
   kind: VirtualMachine
   metadata:
     labels:
       kubevirt.io/vm: vm2-rhel84-watchdog
     name: <vm-name>
   spec:
     running: false
     template:
       metadata:
         labels:
           kubevirt.io/vm: vm2-rhel84-watchdog
       spec:
         domain:
           devices:
             watchdog:
               name: <watchdog>
               i6300esb:
                 action: "poweroff" 1
   # ...

   1

   Specify poweroff, reset, or shutdown.

   The example above configures the i6300esb watchdog device on a RHEL8 VM with the poweroff action and exposes the device as /dev/watchdog.

   This device can now be used by the watchdog binary.

2. Apply the YAML file to your cluster by running the following command:

   $ oc apply -f <file_name>.yaml

1. Run the following command to verify that the VM is connected to the watchdog device:

   $ lspci | grep watchdog -i

2. Run one of the following commands to confirm the watchdog is active:

   • Trigger a kernel panic:

     # echo c > /proc/sysrq-trigger

   • Stop the watchdog service:

     # pkill -9 watchdog

Procedure

1. Log in to the virtual machine as root user.

2. Install the watchdog package and its dependencies:

   # yum install watchdog

3. Uncomment the following line in the /etc/watchdog.conf file and save the changes:

   #watchdog-device = /dev/watchdog

4. Enable the watchdog service to start on boot:

   # systemctl enable --now watchdog.service

Procedure

1. View a list of pods in the OpenShift Virtualization namespace by running the following command:

   $ oc get pods -n openshift-cnv

   Example 14.3. Example output

   NAME                               READY   STATUS    RESTARTS   AGE
   disks-images-provider-7gqbc        1/1     Running   0          32m
   disks-images-provider-vg4kx        1/1     Running   0          32m
   virt-api-57fcc4497b-7qfmc          1/1     Running   0          31m
   virt-api-57fcc4497b-tx9nc          1/1     Running   0          31m
   virt-controller-76c784655f-7fp6m   1/1     Running   0          30m
   virt-controller-76c784655f-f4pbd   1/1     Running   0          30m
   virt-handler-2m86x                 1/1     Running   0          30m
   virt-handler-9qs6z                 1/1     Running   0          30m
   virt-operator-7ccfdbf65f-q5snk     1/1     Running   0          32m
   virt-operator-7ccfdbf65f-vllz8     1/1     Running   0          32m

2. View the pod log by running the following command:

   $ oc logs -n openshift-cnv <pod_name>

   Note

   If a pod fails to start, you can use the --previous option to view logs from the last attempt.

   To monitor log output in real time, use the -f option.

   Example 14.4. Example output

   {"component":"virt-handler","level":"info","msg":"set verbosity to 2","pos":"virt-handler.go:453","timestamp":"2022-04-17T08:58:37.373695Z"}
   {"component":"virt-handler","level":"info","msg":"set verbosity to 2","pos":"virt-handler.go:453","timestamp":"2022-04-17T08:58:37.373726Z"}
   {"component":"virt-handler","level":"info","msg":"setting rate limiter to 5 QPS and 10 Burst","pos":"virt-handler.go:462","timestamp":"2022-04-17T08:58:37.373782Z"}
   {"component":"virt-handler","level":"info","msg":"CPU features of a minimum baseline CPU model: map[apic:true clflush:true cmov:true cx16:true cx8:true de:true fpu:true fxsr:true lahf_lm:true lm:true mca:true mce:true mmx:true msr:true mtrr:true nx:true pae:true pat:true pge:true pni:true pse:true pse36:true sep:true sse:true sse2:true sse4.1:true ssse3:true syscall:true tsc:true]","pos":"cpu_plugin.go:96","timestamp":"2022-04-17T08:58:37.390221Z"}
   {"component":"virt-handler","level":"warning","msg":"host model mode is expected to contain only one model","pos":"cpu_plugin.go:103","timestamp":"2022-04-17T08:58:37.390263Z"}
   {"component":"virt-handler","level":"info","msg":"node-labeller is running","pos":"node_labeller.go:94","timestamp":"2022-04-17T08:58:37.391011Z"}

Procedure

1. To set log verbosity for specific components, open the HyperConverged CR in your default text editor by running the following command:

   $ oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnv

2. Set the log level for one or more components by editing the spec.logVerbosityConfig stanza. For example:

   apiVersion: hco.kubevirt.io/v1beta1
   kind: HyperConverged
   metadata:
     name: kubevirt-hyperconverged
   spec:
     logVerbosityConfig:
       kubevirt:
         virtAPI: 5 1
         virtController: 4
         virtHandler: 3
         virtLauncher: 2
         virtOperator: 6

   1

   The log verbosity value must be an integer in the range 1–9, where a higher number indicates a more detailed log. In this example, the virtAPI component logs are exposed if their priority level is 5 or higher.

3. Apply your changes by saving and exiting the editor.