Red Hat SummitChapter 6. Observe
You can monitor the health of your OpenShift sandboxed containers environment.
The following tools are available:
- OpenShift Container Platform web console. Administrators can access and query raw metrics through Prometheus.
- Logging
6.1. Metrics
You can monitor system health by querying metrics displayed in the OpenShift Container Platform web console.
You can access the following metrics:
- Kata agent metrics
-
Kata agent metrics display information about the kata agent process running in the VM embedded in your sandboxed containers. These metrics include data from
/proc/<pid>/[io, stat, status]. - Kata guest operating system metrics
-
Kata guest operating system metrics display data from the guest operating system running in your sandboxed containers. These metrics include data from
/proc/[stats, diskstats, meminfo, vmstats]and/proc/net/dev. - Hypervisor metrics
-
Hypervisor metrics display data regarding the hypervisor running the VM embedded in your sandboxed containers. These metrics mainly include data from
/proc/<pid>/[io, stat, status]. - Kata monitor metrics
- Kata monitor is the process that gathers metric data and makes it available to Prometheus. The kata monitor metrics display detailed information about the resource usage of the kata-monitor process itself. These metrics also include counters from Prometheus data collection.
- Kata containerd shim v2 metrics
-
Kata containerd shim v2 metrics display detailed information about the kata shim process. These metrics include data from
/proc/<pid>/[io, stat, status]and detailed resource usage metrics.
6.2. Viewing metrics
You can access the metrics for OpenShift sandboxed containers in the Metrics page In the OpenShift Container Platform web console.
Prerequisites
-
You have access to the cluster as a user with the
cluster-adminrole or with view permissions for all projects.
Procedure
-
In the OpenShift Container Platform web console, navigate to Observe
Metrics. In the input field, enter the query for the metric you want to observe.
All kata-related metrics begin with kata. Typing kata displays a list of all available kata metrics.
The metrics from your query are visualized on the page.
6.3. Enabling debug logs for CRI-O runtime
You can enable debug logs by updating the logLevel field in the KataConfig CR. This changes the log level in the CRI-O runtime for the worker nodes running OpenShift sandboxed containers.
Prerequisites
-
You have installed the OpenShift CLI (
oc). -
You have access to the cluster as a user with the
cluster-adminrole.
Procedure
Change the
logLevelfield in your existingKataConfigCR todebug:$ oc patch kataconfig <kataconfig> --type merge --patch '{"spec":{"logLevel":"debug"}}'Monitor the
kata-ocmachine config pool until the value ofUPDATEDisTrue, indicating that all worker nodes are updated:$ oc get mcp kata-ocExample output
NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE kata-oc rendered-kata-oc-169 False True False 3 1 1 0 9h
Verification
Start a debug session with a node in the machine config pool:
$ oc debug node/<node_name>Change the root directory to
/host:# chroot /hostVerify the changes in the
crio.conffile:# crio config | egrep 'log_levelExample output
log_level = "debug"
6.4. Viewing debug logs for components
Cluster administrators can use the debug logs to troubleshoot issues. The logs for each node are printed to the node journal.
You can review the logs for the following OpenShift sandboxed containers components:
- Kata agent
-
Kata runtime (
containerd-shim-kata-v2) -
virtiofsd
QEMU only generates warning and error logs. These warnings and errors print to the node journal in both the Kata runtime logs and the CRI-O logs with an extra qemuPid field.
Example of QEMU logs
Mar 11 11:57:28 openshift-worker-0 kata[2241647]: time="2023-03-11T11:57:28.587116986Z" level=info msg="Start logging QEMU (qemuPid=2241693)" name=containerd-shim-v2 pid=2241647 sandbox=d1d4d68efc35e5ccb4331af73da459c13f46269b512774aa6bde7da34db48987 source=virtcontainers/hypervisor subsystem=qemu
Mar 11 11:57:28 openshift-worker-0 kata[2241647]: time="2023-03-11T11:57:28.607339014Z" level=error msg="qemu-kvm: -machine q35,accel=kvm,kernel_irqchip=split,foo: Expected '=' after parameter 'foo'" name=containerd-shim-v2 pid=2241647 qemuPid=2241693 sandbox=d1d4d68efc35e5ccb4331af73da459c13f46269b512774aa6bde7da34db48987 source=virtcontainers/hypervisor subsystem=qemu
Mar 11 11:57:28 openshift-worker-0 kata[2241647]: time="2023-03-11T11:57:28.60890737Z" level=info msg="Stop logging QEMU (qemuPid=2241693)" name=containerd-shim-v2 pid=2241647 sandbox=d1d4d68efc35e5ccb4331af73da459c13f46269b512774aa6bde7da34db48987 source=virtcontainers/hypervisor subsystem=qemu
The Kata runtime prints Start logging QEMU when QEMU starts, and Stop Logging QEMU when QEMU stops. The error appears in between these two log messages with the qemuPid field. The actual error message from QEMU appears in red.
The console of the QEMU guest is printed to the node journal as well. You can view the guest console logs together with the Kata agent logs.
Prerequisites
-
You have installed the OpenShift CLI (
oc). -
You have access to the cluster as a user with the
cluster-adminrole.
Procedure
To review the Kata agent logs and guest console logs, run the following command:
$ oc debug node/<nodename> -- journalctl -D /host/var/log/journal -t kata -g “reading guest console”To review the Kata runtime logs, run the following command:
$ oc debug node/<nodename> -- journalctl -D /host/var/log/journal -t kataTo review the
virtiofsdlogs, run the following command:$ oc debug node/<nodename> -- journalctl -D /host/var/log/journal -t virtiofsdTo review the QEMU logs, run the following command:
$ oc debug node/<nodename> -- journalctl -D /host/var/log/journal -t kata -g "qemuPid=\d+"
'%20d='M201.5%20305.5c-13.8%200-24.9-11.1-24.9-24.6%200-13.8%2011.1-24.9%2024.9-24.9%2013.6%200%2024.6%2011.1%2024.6%2024.9%200%2013.6-11.1%2024.6-24.6%2024.6zM504%20256c0%20137-111%20248-248%20248S8%20393%208%20256%20119%208%20256%208s248%20111%20248%20248zm-132.3-41.2c-9.4%200-17.7%203.9-23.8%2010-22.4-15.5-52.6-25.5-86.1-26.6l17.4-78.3%2055.4%2012.5c0%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.3%2024.9-24.9s-11.1-24.9-24.9-24.9c-9.7%200-18%205.8-22.1%2013.8l-61.2-13.6c-3-.8-6.1%201.4-6.9%204.4l-19.1%2086.4c-33.2%201.4-63.1%2011.3-85.5%2026.8-6.1-6.4-14.7-10.2-24.1-10.2-34.9%200-46.3%2046.9-14.4%2062.8-1.1%205-1.7%2010.2-1.7%2015.5%200%2052.6%2059.2%2095.2%20132%2095.2%2073.1%200%20132.3-42.6%20132.3-95.2%200-5.3-.6-10.8-1.9-15.8%2031.3-16%2019.8-62.5-14.9-62.5zM302.8%20331c-18.2%2018.2-76.1%2017.9-93.6%200-2.2-2.2-6.1-2.2-8.3%200-2.5%202.5-2.5%206.4%200%208.6%2022.8%2022.8%2087.3%2022.8%20110.2%200%202.5-2.2%202.5-6.1%200-8.6-2.2-2.2-6.1-2.2-8.3%200zm7.7-75c-13.6%200-24.6%2011.1-24.6%2024.9%200%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.1%2024.9-24.6%200-13.8-11-24.9-24.9-24.9z'/%3e%3c/svg%3e){kind=small}