Chapter 13. Logging, events, and monitoring

Procedure

1. In the Administrator perspective in the OpenShift Virtualization web console, navigate to Virtualization Overview.
2. Navigate to the Top Consumers cards.
3. From the drop-down menu, select Show top 5 or Show top 10.
4. For a Top Consumer card, select the type of resource from the drop-down menu: CPU, Memory, Used Filesystem, Memory Swap, vCPU Wait, or Storage Throughput.
5. Select By Project, By VM, or By Node. A list of the top five or top ten consumers of the selected resource is displayed.

Procedure

1. To set log verbosity for specific components, open the

   HyperConverged

   CR in your default text editor by running the following command:

   $ oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnv

2. Set the log level for one or more components by editing the

   spec.logVerbosityConfig

   stanza. For example:

   apiVersion: hco.kubevirt.io/v1beta1
   kind: HyperConverged
   metadata:
     name: kubevirt-hyperconverged
   spec:
     logVerbosityConfig:
       kubevirt:
         virtAPI: 5 

   1

   
         virtController: 4
         virtHandler: 3
         virtLauncher: 2
         virtOperator: 6

   1

   The log verbosity value must be an integer in the range 1–9, where a higher number indicates a more detailed log. In this example, the virtAPI component logs are exposed if their priority level is 5 or higher.

3. Apply your changes by saving and exiting the editor.

4. View a list of pods in the OpenShift Virtualization namespace by running the following command:

   $ oc get pods -n openshift-cnv

   Example 13.1. Example output

   NAME                               READY   STATUS    RESTARTS   AGE
   disks-images-provider-7gqbc        1/1     Running   0          32m
   disks-images-provider-vg4kx        1/1     Running   0          32m
   virt-api-57fcc4497b-7qfmc          1/1     Running   0          31m
   virt-api-57fcc4497b-tx9nc          1/1     Running   0          31m
   virt-controller-76c784655f-7fp6m   1/1     Running   0          30m
   virt-controller-76c784655f-f4pbd   1/1     Running   0          30m
   virt-handler-2m86x                 1/1     Running   0          30m
   virt-handler-9qs6z                 1/1     Running   0          30m
   virt-operator-7ccfdbf65f-q5snk     1/1     Running   0          32m
   virt-operator-7ccfdbf65f-vllz8     1/1     Running   0          32m

5. To view logs for a component pod, run the following command:

   $ oc logs -n openshift-cnv <pod_name>

   For example:

   $ oc logs -n openshift-cnv virt-handler-2m86x

   Note

   If a pod fails to start, you can use the

   --previous

   option to view logs from the last attempt.

   To monitor log output in real time, use the

   -f

   option.

   Example 13.2. Example output

   {"component":"virt-handler","level":"info","msg":"set verbosity to 2","pos":"virt-handler.go:453","timestamp":"2022-04-17T08:58:37.373695Z"}
   {"component":"virt-handler","level":"info","msg":"set verbosity to 2","pos":"virt-handler.go:453","timestamp":"2022-04-17T08:58:37.373726Z"}
   {"component":"virt-handler","level":"info","msg":"setting rate limiter to 5 QPS and 10 Burst","pos":"virt-handler.go:462","timestamp":"2022-04-17T08:58:37.373782Z"}
   {"component":"virt-handler","level":"info","msg":"CPU features of a minimum baseline CPU model: map[apic:true clflush:true cmov:true cx16:true cx8:true de:true fpu:true fxsr:true lahf_lm:true lm:true mca:true mce:true mmx:true msr:true mtrr:true nx:true pae:true pat:true pge:true pni:true pse:true pse36:true sep:true sse:true sse2:true sse4.1:true ssse3:true syscall:true tsc:true]","pos":"cpu_plugin.go:96","timestamp":"2022-04-17T08:58:37.390221Z"}
   {"component":"virt-handler","level":"warning","msg":"host model mode is expected to contain only one model","pos":"cpu_plugin.go:103","timestamp":"2022-04-17T08:58:37.390263Z"}
   {"component":"virt-handler","level":"info","msg":"node-labeller is running","pos":"node_labeller.go:94","timestamp":"2022-04-17T08:58:37.391011Z"}

Procedure

1. In the OpenShift Container Platform console, click Virtualization VirtualMachines from the side menu.
2. Select a virtual machine to open the VirtualMachine details page.
3. Click the Details tab.
4. Click the

   virt-launcher-<name>

   pod in the Pod section to open the Pod details page.
5. Click the Logs tab to view the pod logs.

Procedure

1. Click Virtualization VirtualMachines from the side menu.
2. Select a virtual machine to open the VirtualMachine details page.

3. Click the Events tab to view streaming events for the virtual machine.

   • The ▮▮ button pauses the events stream.
   • The ▶ button resumes a paused events stream.

Procedure

1. Include details of the readiness probe in the VMI configuration file.

   Sample readiness probe with an HTTP GET test

   # ...
   spec:
     readinessProbe:
       httpGet: 

   1

   
         port: 1500 

   2

   
         path: /healthz 

   3

   
         httpHeaders:
         - name: Custom-Header
           value: Awesome
       initialDelaySeconds: 120 

   4

   
       periodSeconds: 20 

   5

   
       timeoutSeconds: 10 

   6

   
       failureThreshold: 3 

   7

   
       successThreshold: 3 

   8

   
   # ...

   1

   The HTTP GET request to perform to connect to the VMI.

   2

   The port of the VMI that the probe queries. In the above example, the probe queries port 1500.

   3

   The path to access on the HTTP server. In the above example, if the handler for the server’s /healthz path returns a success code, the VMI is considered to be healthy. If the handler returns a failure code, the VMI is removed from the list of available endpoints.

   4

   The time, in seconds, after the VMI starts before the readiness probe is initiated.

   5

   The delay, in seconds, between performing probes. The default delay is 10 seconds. This value must be greater than timeoutSeconds.

   6

   The number of seconds of inactivity after which the probe times out and the VMI is assumed to have failed. The default value is 1. This value must be lower than periodSeconds.

   7

   The number of times that the probe is allowed to fail. The default is 3. After the specified number of attempts, the pod is marked Unready.

   8

   The number of times that the probe must report success, after a failure, to be considered successful. The default is 1.

2. Create the VMI by running the following command:

   $ oc create -f <file_name>.yaml

Procedure

1. Include details of the TCP readiness probe in the VMI configuration file.

   Sample readiness probe with a TCP socket test

   ...
   spec:
     readinessProbe:
       initialDelaySeconds: 120 

   1

   
       periodSeconds: 20 

   2

   
       tcpSocket: 

   3

   
         port: 1500 

   4

   
       timeoutSeconds: 10 

   5

   
   ...

   1

   The time, in seconds, after the VMI starts before the readiness probe is initiated.

   2

   The delay, in seconds, between performing probes. The default delay is 10 seconds. This value must be greater than timeoutSeconds.

   3

   The TCP action to perform.

   4

   The port of the VMI that the probe queries.

   5

   The number of seconds of inactivity after which the probe times out and the VMI is assumed to have failed. The default value is 1. This value must be lower than periodSeconds.

2. Create the VMI by running the following command:

   $ oc create -f <file_name>.yaml

Procedure

1. Include details of the HTTP liveness probe in the VMI configuration file.

   Sample liveness probe with an HTTP GET test

   # ...
   spec:
     livenessProbe:
       initialDelaySeconds: 120 

   1

   
       periodSeconds: 20 

   2

   
       httpGet: 

   3

   
         port: 1500 

   4

   
         path: /healthz 

   5

   
         httpHeaders:
         - name: Custom-Header
           value: Awesome
       timeoutSeconds: 10 

   6

   
   # ...

   1

   The time, in seconds, after the VMI starts before the liveness probe is initiated.

   2

   The delay, in seconds, between performing probes. The default delay is 10 seconds. This value must be greater than timeoutSeconds.

   3

   The HTTP GET request to perform to connect to the VMI.

   4

   The port of the VMI that the probe queries. In the above example, the probe queries port 1500. The VMI installs and runs a minimal HTTP server on port 1500 via cloud-init.

   5

   The path to access on the HTTP server. In the above example, if the handler for the server’s /healthz path returns a success code, the VMI is considered to be healthy. If the handler returns a failure code, the VMI is deleted and a new instance is created.

   6

   The number of seconds of inactivity after which the probe times out and the VMI is assumed to have failed. The default value is 1. This value must be lower than periodSeconds.

2. Create the VMI by running the following command:

   $ oc create -f <file_name>.yaml

Procedure

1. In the Administrator perspective in the OpenShift Virtualization web console, navigate to Observe Dashboards.
2. Select the KubeVirt/Infrastructure Resources/Top Consumers dashboard from the Dashboard list.
3. Select a predefined time period from the drop-down menu for Period. You can review the data for top consumers in the tables.
4. Optional: Click Inspect to view or edit the Prometheus Query Language (PromQL) query associated with the top consumers for a table.

Procedure

1. Create a configuration file that contains the resources to set up the framework. This includes a namespace and service account for the framework, and the

   ClusterRole

   and

   ClusterRoleBinding

   objects to define permissions for the service account.

   Example 13.3. Example framework manifest file

   ---
   apiVersion: v1
   kind: Namespace
   metadata:
     name: kiagnose
   ---
   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: kiagnose
     namespace: kiagnose
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRole
   metadata:
     name: kiagnose
   rules:
     - apiGroups: [ "" ]
       resources: [ "configmaps" ]
       verbs:
         - get
         - list
         - create
         - update
         - patch
     - apiGroups: [ "" ]
       resources: [ "namespaces" ]
       verbs:
         - get
         - list
         - create
         - delete
         - watch
     - apiGroups: [ "" ]
       resources: [ "serviceaccounts" ]
       verbs:
         - get
         - list
         - create
     - apiGroups: [ "rbac.authorization.k8s.io" ]
       resources:
         - roles
         - rolebindings
         - clusterrolebindings
       verbs:
         - get
         - list
         - create
         - delete
     - apiGroups: [ "rbac.authorization.k8s.io" ]
       resources:
         - clusterroles
       verbs:
         - get
         - list
         - create
         - bind
     - apiGroups: [ "batch" ]
       resources: [ "jobs" ]
       verbs:
         - get
         - list
         - create
         - delete
         - watch
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRoleBinding
   metadata:
     name: kiagnose
   roleRef:
     apiGroup: rbac.authorization.k8s.io
     kind: ClusterRole
     name: kiagnose
   subjects:
     - kind: ServiceAccount
       name: kiagnose
       namespace: kiagnose
   ...

2. Apply the framework manifest:

   $ oc apply -f <framework_manifest>.yaml

3. Create a configuration file that contains the

   ClusterRole

   and

   Role

   objects with permissions that the checkup requires for cluster access:

   Example cluster role manifest file

   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRole
   metadata:
     name: kubevirt-vm-latency-checker
   rules:
   - apiGroups: ["kubevirt.io"]
     resources: ["virtualmachineinstances"]
     verbs: ["get", "create", "delete"]
   - apiGroups: ["subresources.kubevirt.io"]
     resources: ["virtualmachineinstances/console"]
     verbs: ["get"]
   - apiGroups: ["k8s.cni.cncf.io"]
     resources: ["network-attachment-definitions"]
     verbs: ["get"]

4. Apply the checkup roles manifest:

   $ oc apply -f <latency_roles>.yaml

5. Create a

   ConfigMap

   manifest that contains the input parameters for the checkup. The config map provides the input for the framework to run the checkup and also stores the results of the checkup.

   Example input config map

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: kubevirt-vm-latency-checkup
     namespace: kiagnose
   data:
     spec.image: registry.redhat.io/container-native-virtualization/vm-network-latency-checkup:v4.11.0
     spec.timeout: 10m
     spec.clusterRoles: |
       kubevirt-vmis-manager
     spec.param.network_attachment_definition_namespace: "default" 

   1

   
     spec.param.network_attachment_definition_name: "bridge-network" 

   2

   
     spec.param.max_desired_latency_milliseconds: "10" 

   3

   
     spec.param.sample_duration_seconds: "5" 

   4

   1

   The namespace where the NetworkAttachmentDefinition object resides.

   2

   The name of the NetworkAttachmentDefinition object.

   3

   Optional: The maximum desired latency, in milliseconds, between the virtual machines. If the measured latency exceeds this value, the check fails.

   4

   Optional: The duration of the latency check, in seconds.

6. Create the config map in the framework’s namespace:

   $ oc apply -f <latency_config_map>.yaml

7. Create a

   Job

   object to run the checkup:

   Example job manifest

   apiVersion: batch/v1
   kind: Job
   metadata:
     name: kubevirt-vm-latency-checkup
     namespace: kiagnose
   spec:
     backoffLimit: 0
     template:
       spec:
         serviceAccount: kiagnose
         restartPolicy: Never
         containers:
           - name: framework
             image: registry.redhat.io/container-native-virtualization/checkup-framework:v4.11.0
             env:
               - name: CONFIGMAP_NAMESPACE
                 value: kiagnose
               - name: CONFIGMAP_NAME
                 value: kubevirt-vm-latency-checkup

8. Apply the

   Job

   manifest. The checkup uses the ping utility to verify connectivity and measure latency.

   $ oc apply -f <latency_job>.yaml

9. Wait for the job to complete:

   $ oc wait --for=condition=complete --timeout=10m job.batch/kubevirt-vm-latency-checkup -n kiagnose

10. Review the results of the latency checkup by retrieving the status of the

    ConfigMap

    object. If the measured latency is greater than the value of the

    spec.param.max_desired_latency_milliseconds

    attribute, the checkup fails and returns an error.

    $ oc get configmap kubevirt-vm-latency-checkup -n kiagnose -o yaml

    Example output config map (success)

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: kubevirt-vm-latency-checkup
      namespace: kiagnose
    ...
      status.succeeded: "true"
      status.failureReason: ""
      status.result.minLatencyNanoSec: 2000
      status.result.maxLatencyNanoSec: 3000
      status.result.avgLatencyNanoSec: 2500
      status.results.measurementDurationSec: 300
    ...

11. Delete the framework and checkup resources that you previously created. This includes the job, config map, cluster role, and framework manifest files.

    Note

    Do not delete the framework and cluster role manifest files if you plan to run another checkup.

    $ oc delete -f <file_name>.yaml

Procedure

1. Create the

   Service

   YAML file. In the following example, the file is called

   node-exporter-service.yaml

   .

   kind: Service
   apiVersion: v1
   metadata:
     name: node-exporter-service 

   1

   
     namespace: dynamation 

   2

   
     labels:
       servicetype: metrics 

   3

   
   spec:
     ports:
       - name: exmet 

   4

   
         protocol: TCP
         port: 9100 

   5

   
         targetPort: 9100 

   6

   
     type: ClusterIP
     selector:
       monitor: metrics 

   7

   1

   The node-exporter service that exposes the metrics from the virtual machines.

   2

   The namespace where the service is created.

   3

   The label for the service. The ServiceMonitor uses this label to match this service.

   4

   The name given to the port that exposes metrics on port 9100 for the ClusterIP service.

   5

   The target port used by node-exporter-service to listen for requests.

   6

   The TCP port number of the virtual machine that is configured with the monitor label.

   7

   The label used to match the virtual machine’s pods. In this example, any virtual machine’s pod with the label monitor and a value of metrics will be matched.

2. Create the node-exporter service:

   $ oc create -f node-exporter-service.yaml

Procedure

1. Log on to the virtual machine.

2. Download the

   node-exporter

   file on to the virtual machine by using the directory path that applies to the version of

   node-exporter

   file.

   $ wget https://github.com/prometheus/node_exporter/releases/download/v1.3.1/node_exporter-1.3.1.linux-amd64.tar.gz

3. Extract the executable and place it in the

   /usr/bin

   directory.

   $ sudo tar xvf node_exporter-1.3.1.linux-amd64.tar.gz \
       --directory /usr/bin --strip 1 "*/node_exporter"

4. Create a

   node_exporter.service

   file in this directory path:

   /etc/systemd/system

   . This

   systemd

   service file runs the node-exporter service when the virtual machine reboots.

   [Unit]
   Description=Prometheus Metrics Exporter
   After=network.target
   StartLimitIntervalSec=0
   
   [Service]
   Type=simple
   Restart=always
   RestartSec=1
   User=root
   ExecStart=/usr/bin/node_exporter
   
   [Install]
   WantedBy=multi-user.target

5. Enable and start the

   systemd

   service.

   $ sudo systemctl enable node_exporter.service
   $ sudo systemctl start node_exporter.service

Procedure

1. Edit the

   template

   spec of your virtual machine configuration file. In this example, the label

   monitor

   has the value

   metrics

   .

   spec:
     template:
       metadata:
         labels:
           monitor: metrics

2. Stop and restart the virtual machine to create a new pod with the label name given to the

   monitor

   label.

Procedure

1. Create a YAML file for the

   ServiceMonitor

   resource configuration. In this example, the service monitor matches any service with the label

   metrics

   and queries the

   exmet

   port every 30 seconds.

   apiVersion: monitoring.coreos.com/v1
   kind: ServiceMonitor
   metadata:
     labels:
       k8s-app: node-exporter-metrics-monitor
     name: node-exporter-metrics-monitor 

   1

   
     namespace: dynamation 

   2

   
   spec:
     endpoints:
     - interval: 30s 

   3

   
       port: exmet 

   4

   
       scheme: http
     selector:
       matchLabels:
         servicetype: metrics

   1

   The name of the ServiceMonitor.

   2

   The namespace where the ServiceMonitor is created.

   3

   The interval at which the port will be queried.

   4

   The name of the port that is queried every 30 seconds

2. Create the

   ServiceMonitor

   configuration for the node-exporter service.

   $ oc create -f node-exporter-metrics-monitor.yaml

Procedure

1. Collect

   must-gather

   data for the cluster by using the default

   must-gather

   image.
2. Collect

   must-gather

   data for Red Hat OpenShift Data Foundation, if necessary.
3. Collect

   must-gather

   data for OpenShift Virtualization by using the OpenShift Virtualization

   must-gather

   image.
4. Collect Prometheus metrics for the cluster.

Procedure

1. Collect detailed

   must-gather

   data about the malfunctioning VMs.
2. Collect screenshots of VMs that have crashed before you restart them.
3. Record factors that the malfunctioning VMs have in common. For example, the VMs have the same host or network.