Chapter 14. Logging, events, and monitoring

Procedure

1. In the Administrator perspective in the OpenShift Container Platform web console, navigate to Virtualization Overview.
2. Click the Top consumers tab.
3. Optional: You can filter the results by selecting a time period or by selecting the 5 or 10 top consumers.

Procedure

1. To set log verbosity for specific components, open the HyperConverged CR in your default text editor by running the following command:

   $ oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnv

2. Set the log level for one or more components by editing the spec.logVerbosityConfig stanza. For example:

   apiVersion: hco.kubevirt.io/v1beta1
   kind: HyperConverged
   metadata:
     name: kubevirt-hyperconverged
   spec:
     logVerbosityConfig:
       kubevirt:
         virtAPI: 5 1
         virtController: 4
         virtHandler: 3
         virtLauncher: 2
         virtOperator: 6

   1

   The log verbosity value must be an integer in the range 1–9, where a higher number indicates a more detailed log. In this example, the virtAPI component logs are exposed if their priority level is 5 or higher.

3. Apply your changes by saving and exiting the editor.

4. View a list of pods in the OpenShift Virtualization namespace by running the following command:

   $ oc get pods -n openshift-cnv

   Example 14.1. Example output

   NAME                               READY   STATUS    RESTARTS   AGE
   disks-images-provider-7gqbc        1/1     Running   0          32m
   disks-images-provider-vg4kx        1/1     Running   0          32m
   virt-api-57fcc4497b-7qfmc          1/1     Running   0          31m
   virt-api-57fcc4497b-tx9nc          1/1     Running   0          31m
   virt-controller-76c784655f-7fp6m   1/1     Running   0          30m
   virt-controller-76c784655f-f4pbd   1/1     Running   0          30m
   virt-handler-2m86x                 1/1     Running   0          30m
   virt-handler-9qs6z                 1/1     Running   0          30m
   virt-operator-7ccfdbf65f-q5snk     1/1     Running   0          32m
   virt-operator-7ccfdbf65f-vllz8     1/1     Running   0          32m

5. To view logs for a component pod, run the following command:

   $ oc logs -n openshift-cnv <pod_name>

   For example:

   $ oc logs -n openshift-cnv virt-handler-2m86x

   Note

   If a pod fails to start, you can use the --previous option to view logs from the last attempt.

   To monitor log output in real time, use the -f option.

   Example 14.2. Example output

   {"component":"virt-handler","level":"info","msg":"set verbosity to 2","pos":"virt-handler.go:453","timestamp":"2022-04-17T08:58:37.373695Z"}
   {"component":"virt-handler","level":"info","msg":"set verbosity to 2","pos":"virt-handler.go:453","timestamp":"2022-04-17T08:58:37.373726Z"}
   {"component":"virt-handler","level":"info","msg":"setting rate limiter to 5 QPS and 10 Burst","pos":"virt-handler.go:462","timestamp":"2022-04-17T08:58:37.373782Z"}
   {"component":"virt-handler","level":"info","msg":"CPU features of a minimum baseline CPU model: map[apic:true clflush:true cmov:true cx16:true cx8:true de:true fpu:true fxsr:true lahf_lm:true lm:true mca:true mce:true mmx:true msr:true mtrr:true nx:true pae:true pat:true pge:true pni:true pse:true pse36:true sep:true sse:true sse2:true sse4.1:true ssse3:true syscall:true tsc:true]","pos":"cpu_plugin.go:96","timestamp":"2022-04-17T08:58:37.390221Z"}
   {"component":"virt-handler","level":"warning","msg":"host model mode is expected to contain only one model","pos":"cpu_plugin.go:103","timestamp":"2022-04-17T08:58:37.390263Z"}
   {"component":"virt-handler","level":"info","msg":"node-labeller is running","pos":"node_labeller.go:94","timestamp":"2022-04-17T08:58:37.391011Z"}

Procedure

1. In the OpenShift Container Platform console, click Virtualization VirtualMachines from the side menu.
2. Select a virtual machine to open the VirtualMachine details page.
3. Click the Details tab.
4. Click the virt-launcher-<name> pod in the Pod section to open the Pod details page.
5. Click the Logs tab to view the pod logs.

Procedure

1. Click Virtualization VirtualMachines from the side menu.
2. Select a virtual machine to open the VirtualMachine details page.

3. Click the Events tab to view streaming events for the virtual machine.

   • The ▮▮ button pauses the events stream.
   • The ▶ button resumes a paused events stream.

Procedure

1. Include details of the readiness probe in the VMI configuration file.

   Sample readiness probe with an HTTP GET test

   # ...
   spec:
     readinessProbe:
       httpGet: 1
         port: 1500 2
         path: /healthz 3
         httpHeaders:
         - name: Custom-Header
           value: Awesome
       initialDelaySeconds: 120 4
       periodSeconds: 20 5
       timeoutSeconds: 10 6
       failureThreshold: 3 7
       successThreshold: 3 8
   # ...

   1

   The HTTP GET request to perform to connect to the VMI.

   2

   The port of the VMI that the probe queries. In the above example, the probe queries port 1500.

   3

   The path to access on the HTTP server. In the above example, if the handler for the server’s /healthz path returns a success code, the VMI is considered to be healthy. If the handler returns a failure code, the VMI is removed from the list of available endpoints.

   4

   The time, in seconds, after the VMI starts before the readiness probe is initiated.

   5

   The delay, in seconds, between performing probes. The default delay is 10 seconds. This value must be greater than timeoutSeconds.

   6

   The number of seconds of inactivity after which the probe times out and the VMI is assumed to have failed. The default value is 1. This value must be lower than periodSeconds.

   7

   The number of times that the probe is allowed to fail. The default is 3. After the specified number of attempts, the pod is marked Unready.

   8

   The number of times that the probe must report success, after a failure, to be considered successful. The default is 1.

2. Create the VMI by running the following command:

   $ oc create -f <file_name>.yaml

Procedure

1. Include details of the TCP readiness probe in the VMI configuration file.

   Sample readiness probe with a TCP socket test

   ...
   spec:
     readinessProbe:
       initialDelaySeconds: 120 1
       periodSeconds: 20 2
       tcpSocket: 3
         port: 1500 4
       timeoutSeconds: 10 5
   ...

   1

   The time, in seconds, after the VMI starts before the readiness probe is initiated.

   2

   The delay, in seconds, between performing probes. The default delay is 10 seconds. This value must be greater than timeoutSeconds.

   3

   The TCP action to perform.

   4

   The port of the VMI that the probe queries.

   5

   The number of seconds of inactivity after which the probe times out and the VMI is assumed to have failed. The default value is 1. This value must be lower than periodSeconds.

2. Create the VMI by running the following command:

   $ oc create -f <file_name>.yaml

Procedure

1. Include details of the HTTP liveness probe in the VMI configuration file.

   Sample liveness probe with an HTTP GET test

   # ...
   spec:
     livenessProbe:
       initialDelaySeconds: 120 1
       periodSeconds: 20 2
       httpGet: 3
         port: 1500 4
         path: /healthz 5
         httpHeaders:
         - name: Custom-Header
           value: Awesome
       timeoutSeconds: 10 6
   # ...

   1

   The time, in seconds, after the VMI starts before the liveness probe is initiated.

   2

   The delay, in seconds, between performing probes. The default delay is 10 seconds. This value must be greater than timeoutSeconds.

   3

   The HTTP GET request to perform to connect to the VMI.

   4

   The port of the VMI that the probe queries. In the above example, the probe queries port 1500. The VMI installs and runs a minimal HTTP server on port 1500 via cloud-init.

   5

   The path to access on the HTTP server. In the above example, if the handler for the server’s /healthz path returns a success code, the VMI is considered to be healthy. If the handler returns a failure code, the VMI is deleted and a new instance is created.

   6

   The number of seconds of inactivity after which the probe times out and the VMI is assumed to have failed. The default value is 1. This value must be lower than periodSeconds.

2. Create the VMI by running the following command:

   $ oc create -f <file_name>.yaml

Procedure

1. Include details of the guest agent ping probe in the VMI configuration file. For example:

   Sample guest agent ping probe

   # ...
   spec:
     readinessProbe:
       guestAgentPing: {} 1
       initialDelaySeconds: 120 2
       periodSeconds: 20 3
       timeoutSeconds: 10 4
       failureThreshold: 3 5
       successThreshold: 3 6
   # ...

   1

   The guest agent ping probe to connect to the VMI.

   2

   Optional: The time, in seconds, after the VMI starts before the guest agent probe is initiated.

   3

   Optional: The delay, in seconds, between performing probes. The default delay is 10 seconds. This value must be greater than timeoutSeconds.

   4

   Optional: The number of seconds of inactivity after which the probe times out and the VMI is assumed to have failed. The default value is 1. This value must be lower than periodSeconds.

   5

   Optional: The number of times that the probe is allowed to fail. The default is 3. After the specified number of attempts, the pod is marked Unready.

   6

   Optional: The number of times that the probe must report success, after a failure, to be considered successful. The default is 1.

2. Create the VMI by running the following command:

   $ oc create -f <file_name>.yaml

Procedure

1. In the Administrator perspective in the OpenShift Virtualization web console, navigate to Observe Dashboards.
2. Select the KubeVirt/Infrastructure Resources/Top Consumers dashboard from the Dashboard list.
3. Select a predefined time period from the drop-down menu for Period. You can review the data for top consumers in the tables.
4. Optional: Click Inspect to view or edit the Prometheus Query Language (PromQL) query associated with the top consumers for a table.

Procedure

1. Create a manifest file that contains the ServiceAccount, Role, and RoleBinding objects with permissions that the checkup requires for cluster access:

   Example 14.3. Example role manifest file

   ---
   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: vm-latency-checkup-sa
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: Role
   metadata:
     name: kubevirt-vm-latency-checker
   rules:
   - apiGroups: ["kubevirt.io"]
     resources: ["virtualmachineinstances"]
     verbs: ["get", "create", "delete"]
   - apiGroups: ["subresources.kubevirt.io"]
     resources: ["virtualmachineinstances/console"]
     verbs: ["get"]
   - apiGroups: ["k8s.cni.cncf.io"]
     resources: ["network-attachment-definitions"]
     verbs: ["get"]
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: RoleBinding
   metadata:
     name: kubevirt-vm-latency-checker
   subjects:
   - kind: ServiceAccount
     name: vm-latency-checkup-sa
   roleRef:
     kind: Role
     name: kubevirt-vm-latency-checker
     apiGroup: rbac.authorization.k8s.io
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: Role
   metadata:
     name: kiagnose-configmap-access
   rules:
   - apiGroups: [ "" ]
     resources: [ "configmaps" ]
     verbs: ["get", "update"]
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: RoleBinding
   metadata:
     name: kiagnose-configmap-access
   subjects:
   - kind: ServiceAccount
     name: vm-latency-checkup-sa
   roleRef:
     kind: Role
     name: kiagnose-configmap-access
     apiGroup: rbac.authorization.k8s.io

2. Apply the checkup roles manifest:

   $ oc apply -n <target_namespace> -f <latency_roles>.yaml 1

   1

   <target_namespace> is the namespace where the checkup is to be run. This must be an existing namespace where the NetworkAttachmentDefinition object resides.

3. Create a ConfigMap manifest that contains the input parameters for the checkup. The config map provides the input for the framework to run the checkup and also stores the results of the checkup.

   Example input config map

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: kubevirt-vm-latency-checkup-config
   data:
     spec.timeout: 5m
     spec.param.network_attachment_definition_namespace: <target_namespace>
     spec.param.network_attachment_definition_name: "blue-network" 1
     spec.param.max_desired_latency_milliseconds: "10" 2
     spec.param.sample_duration_seconds: "5" 3
     spec.param.source_node: "worker1" 4
     spec.param.target_node: "worker2" 5

   1

   The name of the NetworkAttachmentDefinition object.

   2

   Optional: The maximum desired latency, in milliseconds, between the virtual machines. If the measured latency exceeds this value, the checkup fails.

   3

   Optional: The duration of the latency check, in seconds.

   4

   Optional: When specified, latency is measured from this node to the target node. If the source node is specified, the spec.param.target_node field cannot be empty.

   5

   Optional: When specified, latency is measured from the source node to this node.

4. Apply the config map manifest in the target namespace:

   $ oc apply -n <target_namespace> -f <latency_config_map>.yaml

5. Create a Job object to run the checkup:

   Example job manifest

   apiVersion: batch/v1
   kind: Job
   metadata:
     name: kubevirt-vm-latency-checkup
   spec:
     backoffLimit: 0
     template:
       spec:
         serviceAccountName: vm-latency-checkup-sa
         restartPolicy: Never
         containers:
           - name: vm-latency-checkup
             image: registry.redhat.io/container-native-virtualization/vm-network-latency-checkup:v4.12.0
             securityContext:
               allowPrivilegeEscalation: false
               capabilities:
                 drop: ["ALL"]
               runAsNonRoot: true
               seccompProfile:
                 type: "RuntimeDefault"
             env:
               - name: CONFIGMAP_NAMESPACE
                 value: <target_namespace>
               - name: CONFIGMAP_NAME
                 value: kubevirt-vm-latency-checkup-config

6. Apply the Job manifest. The checkup uses the ping utility to verify connectivity and measure latency.

   $ oc apply -n <target_namespace> -f <latency_job>.yaml

7. Wait for the job to complete:

   $ oc wait job kubevirt-vm-latency-checkup -n <target_namespace> --for condition=complete --timeout 6m

8. Review the results of the latency checkup by running the following command. If the maximum measured latency is greater than the value of the spec.param.max_desired_latency_milliseconds attribute, the checkup fails and returns an error.

   $ oc get configmap kubevirt-vm-latency-checkup-config -n <target_namespace> -o yaml

   Example output config map (success)

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: kubevirt-vm-latency-checkup-config
     namespace: <target_namespace>
   data:
     spec.timeout: 5m
     spec.param.network_attachment_definition_namespace: <target_namespace>
     spec.param.network_attachment_definition_name: "blue-network"
     spec.param.max_desired_latency_milliseconds: "10"
     spec.param.sample_duration_seconds: "5"
     spec.param.source_node: "worker1"
     spec.param.target_node: "worker2"
     status.succeeded: "true"
     status.failureReason: ""
     status.completionTimestamp: "2022-01-01T09:00:00Z"
     status.startTimestamp: "2022-01-01T09:00:07Z"
     status.result.avgLatencyNanoSec: "177000"
     status.result.maxLatencyNanoSec: "244000" 1
     status.result.measurementDurationSec: "5"
     status.result.minLatencyNanoSec: "135000"
     status.result.sourceNode: "worker1"
     status.result.targetNode: "worker2"

   1

   The maximum measured latency in nanoseconds.

9. Optional: To view the detailed job log in case of checkup failure, use the following command:

   $ oc logs job.batch/kubevirt-vm-latency-checkup -n <target_namespace>

10. Delete the job and config map resources that you previously created by running the following commands:

    $ oc delete job -n <target_namespace> kubevirt-vm-latency-checkup

    $ oc delete config-map -n <target_namespace> kubevirt-vm-latency-checkup-config

11. Optional: If you do not plan to run another checkup, delete the checkup role and framework manifest files.

    $ oc delete -f <file_name>.yaml

Procedure

1. Create the Service YAML file. In the following example, the file is called node-exporter-service.yaml.

   kind: Service
   apiVersion: v1
   metadata:
     name: node-exporter-service 1
     namespace: dynamation 2
     labels:
       servicetype: metrics 3
   spec:
     ports:
       - name: exmet 4
         protocol: TCP
         port: 9100 5
         targetPort: 9100 6
     type: ClusterIP
     selector:
       monitor: metrics 7

   1

   The node-exporter service that exposes the metrics from the virtual machines.

   2

   The namespace where the service is created.

   3

   The label for the service. The ServiceMonitor uses this label to match this service.

   4

   The name given to the port that exposes metrics on port 9100 for the ClusterIP service.

   5

   The target port used by node-exporter-service to listen for requests.

   6

   The TCP port number of the virtual machine that is configured with the monitor label.

   7

   The label used to match the virtual machine’s pods. In this example, any virtual machine’s pod with the label monitor and a value of metrics will be matched.

2. Create the node-exporter service:

   $ oc create -f node-exporter-service.yaml

Procedure

1. Log on to the virtual machine.

2. Download the node-exporter file on to the virtual machine by using the directory path that applies to the version of node-exporter file.

   $ wget https://github.com/prometheus/node_exporter/releases/download/v1.3.1/node_exporter-1.3.1.linux-amd64.tar.gz

3. Extract the executable and place it in the /usr/bin directory.

   $ sudo tar xvf node_exporter-1.3.1.linux-amd64.tar.gz \
       --directory /usr/bin --strip 1 "*/node_exporter"

4. Create a node_exporter.service file in this directory path: /etc/systemd/system. This systemd service file runs the node-exporter service when the virtual machine reboots.

   [Unit]
   Description=Prometheus Metrics Exporter
   After=network.target
   StartLimitIntervalSec=0
   
   [Service]
   Type=simple
   Restart=always
   RestartSec=1
   User=root
   ExecStart=/usr/bin/node_exporter
   
   [Install]
   WantedBy=multi-user.target

5. Enable and start the systemd service.

   $ sudo systemctl enable node_exporter.service
   $ sudo systemctl start node_exporter.service

Procedure

1. Edit the template spec of your virtual machine configuration file. In this example, the label monitor has the value metrics.

   spec:
     template:
       metadata:
         labels:
           monitor: metrics

2. Stop and restart the virtual machine to create a new pod with the label name given to the monitor label.

Procedure

1. Create a YAML file for the ServiceMonitor resource configuration. In this example, the service monitor matches any service with the label metrics and queries the exmet port every 30 seconds.

   apiVersion: monitoring.coreos.com/v1
   kind: ServiceMonitor
   metadata:
     labels:
       k8s-app: node-exporter-metrics-monitor
     name: node-exporter-metrics-monitor 1
     namespace: dynamation 2
   spec:
     endpoints:
     - interval: 30s 3
       port: exmet 4
       scheme: http
     selector:
       matchLabels:
         servicetype: metrics

   1

   The name of the ServiceMonitor.

   2

   The namespace where the ServiceMonitor is created.

   3

   The interval at which the port will be queried.

   4

   The name of the port that is queried every 30 seconds

2. Create the ServiceMonitor configuration for the node-exporter service.

   $ oc create -f node-exporter-metrics-monitor.yaml

Procedure

1. Collect must-gather data for the cluster by using the default must-gather image.
2. Collect must-gather data for Red Hat OpenShift Data Foundation, if necessary.
3. Collect must-gather data for OpenShift Virtualization by using the OpenShift Virtualization must-gather image.
4. Collect Prometheus metrics for the cluster.

Procedure

1. Collect detailed must-gather data about the malfunctioning VMs.
2. Collect screenshots of VMs that have crashed before you restart them.
3. Record factors that the malfunctioning VMs have in common. For example, the VMs have the same host or network.