Chapter 12. Logging, events, and monitoring

Procedure

• Use the following command:

  $ oc logs <virt-launcher-name>

Procedure

• In the namespace, use the oc get command:

  $ oc get events

Procedure

• In the namespace, use the oc describe command. The following example shows how to get the events for a virtual machine, a virtual machine instance, and the virt-launcher pod for a virtual machine:

  $ oc describe vm <vm>

  $ oc describe vmi <vmi>

  $ oc describe pod virt-launcher-<name>

• Bound
• Running
• Ready

• Type of event
• Reason for logging
• Source of the event
• Message containing additional diagnostic information.

• Bound – A successfully bound PVC displays in this example.

  Note that the Type is Bound, so the Status is True. If the PVC is not bound, the Status is False.

  When the PVC is bound, an event is generated stating that the PVC is bound. In this case, the Reason is Bound and Status is True. The Message indicates which PVC owns the data volume.

  Message, in the Events section, provides further details including how long the PVC has been bound (Age) and by what resource (From), in this case datavolume-controller:

  Example output

  Status:
  	Conditions:
  		Last Heart Beat Time:  2020-07-15T03:58:24Z
  		Last Transition Time:  2020-07-15T03:58:24Z
  		Message:               PVC win10-rootdisk Bound
  		Reason:                Bound
  		Status:                True
  		Type:                  Bound
  
  	Events:
  		Type     Reason     Age    From                   Message
  		----     ------     ----   ----                   -------
  		Normal   Bound      24s    datavolume-controller  PVC example-dv Bound

• Running – In this case, note that Type is Running and Status is False, indicating that an event has occurred that caused an attempted operation to fail, changing the Status from True to False.

  However, note that Reason is Completed and the Message field indicates Import Complete.

  In the Events section, the Reason and Message contain additional troubleshooting information about the failed operation. In this example, the Message displays an inability to connect due to a 404, listed in the Events section’s first Warning.

  From this information, you conclude that an import operation was running, creating contention for other operations that are attempting to access the data volume:

  Example output

  Status:
  	 Conditions:
  		 Last Heart Beat Time:  2020-07-15T04:31:39Z
  		 Last Transition Time:  2020-07-15T04:31:39Z
  		 Message:               Import Complete
  		 Reason:                Completed
  		 Status:                False
  		 Type:                  Running
  
  	Events:
  		Type     Reason           Age                From                   Message
  		----     ------           ----               ----                   -------
  		Warning  Error            12s (x2 over 14s)  datavolume-controller  Unable to connect
  		to http data source: expected status code 200, got 404. Status: 404 Not Found

• Ready – If Type is Ready and Status is True, then the data volume is ready to be used, as in the following example. If the data volume is not ready to be used, the Status is False:

  Example output

  Status:
  	 Conditions:
  		 Last Heart Beat Time: 2020-07-15T04:31:39Z
  		 Last Transition Time:  2020-07-15T04:31:39Z
  		 Status:                True
  		 Type:                  Ready

• Virtual Machines
• Virtual Machine Templates

• Details provides identifying information about the virtual machine, including:

  • Name
  • Namespace
  • Date of creation
  • Node name
  • IP address

• Inventory lists the virtual machine’s resources, including:

  • Network interface controllers (NICs)
  • Disks

• Status includes:

  • The current status of the virtual machine
  • A note indicating whether or not the QEMU guest agent is installed on the virtual machine

• Utilization includes charts that display usage data for:

  • CPU
  • Memory
  • Filesystem
  • Network transfer

• Events lists messages about virtual machine activity over the past hour. To view additional events, click View all.

• Details provides a brief overview of informational cluster details.

  Status include ok, error, warning, in progress, and unknown. Resources can add custom status names.

  • Cluster ID
  • Provider
  • Version

• Cluster Inventory details number of resources and associated statuses. It is helpful when intervention is required to resolve problems, including information about:

  • Number of nodes
  • Number of pods
  • Persistent storage volume claims
  • Virtual machines (available if OpenShift Virtualization is installed)
  • Bare metal hosts in the cluster, listed according to their state (only available in metal3 environment).
• Cluster Health summarizes the current health of the cluster as a whole, including relevant alerts and descriptions. If OpenShift Virtualization is installed, the overall health of OpenShift Virtualization is diagnosed as well. If more than one subsystem is present, click See All to view the status of each subsystem.

• Cluster Capacity charts help administrators understand when additional resources are required in the cluster. The charts contain an inner ring that displays current consumption, while an outer ring displays thresholds configured for the resource, including information about:

  • CPU time
  • Memory allocation
  • Storage consumed
  • Network resources consumed
• Cluster Utilization shows the capacity of various resources over a specified period of time, to help administrators understand the scale and frequency of high resource consumption.
• Events lists messages related to recent activity in the cluster, such as pod creation or virtual machine migration to another host.
• Top Consumers helps administrators understand how cluster resources are consumed. Click on a resource to jump to a detailed page listing pods and nodes that consume the largest amount of the specified cluster resource (CPU, memory, or storage).

• collection - This is the component that collects logs from the cluster, formats them, and forwards them to the log store. The current implementation is Fluentd.
• log store - This is where the logs are stored. The default implementation is Elasticsearch. You can use the default Elasticsearch log store or forward logs to external log stores. The default log store is optimized and tested for short-term storage.
• visualization - This is the UI component you can use to view logs, graphs, charts, and so forth. The current implementation is Kibana.

Prerequisites

• Set the retention time for Prometheus metrics data to a minimum of seven days.
• Configure the Alertmanager to capture relevant alerts and to send them to a dedicated mailbox so that they can be viewed and persisted outside the cluster.
• Record the exact number of affected nodes and virtual machines.

Prerequisites

• Windows VMs:

  • Record the Windows patch update details for Red Hat Support.
  • Install the latest version of the VirtIO drivers. The VirtIO drivers include the QEMU guest agent.
  • If Remote Desktop Protocol (RDP) is enabled, try to connect to the VMs with RDP to determine whether there is a problem with the connection software.

• OpenShift Virtualization Operator namespaces, including child objects
• OpenShift Virtualization custom resource definitions
• Namespaces that contain virtual machines
• Basic virtual machine definitions

Procedure

• Run the following command to collect data about OpenShift Virtualization:

  $ oc adm must-gather --image-stream=openshift/must-gather \
    --image=registry.redhat.io/container-native-virtualization/cnv-must-gather-rhel8:v{HCOVersion}