Este conteúdo não está disponível no idioma selecionado.
Chapter 14. Support
14.1. Support overview Copiar o linkLink copiado para a área de transferência!
Accelerate the resolution of cluster and virtual machine (VM) issues by using the integrated diagnostic tools and support provided by OpenShift Virtualization.
14.1.1. Opening a support case Copiar o linkLink copiado para a área de transferência!
Open a support case with Red Hat Support when you encounter an issue that requires immediate assistance.
14.1.1.1. Collecting data for Red Hat Support Copiar o linkLink copiado para a área de transferência!
Gather information about the issue affecting your environment to submit with your support case. This aids Red Hat Support in effectively diagnosing your issue.
Gather troubleshooting information by using the following tools:
- Configure Prometheus and Alertmanager.
14.1.1.2. Submitting a support case Copiar o linkLink copiado para a área de transferência!
Submit a support case to resolve a cluster issue that is affecting the ability of OpenShift Virtualization to function properly in your environment.
You can submit a support case to Red Hat Support by using the Customer Support page. Include data that you collected about your issue with your support request.
14.1.1.3. Creating a Jira issue Copiar o linkLink copiado para a área de transferência!
To report an issue with your environment to Red Hat Support, create a Jira issue in the OpenShift Virtualization (CNV) Jira project.
Procedure
- Log in to Red Hat Atlassian Jira.
- Click the following link to open a Create Issue page: Create issue.
- Select OpenShift Virtualization (CNV) as the Project.
- Select Bug as the Issue Type.
- Click Next.
- Complete the Summary and Description fields. In the Description field, include a detailed description of the issue.
Submit any collected troubleshooting information.
- Add any textual troubleshooting information, such as command outputs, in the Description field.
- Add troubleshooting files using the Attachment field.
- Select the appropriate component from Components.
- Click Create.
- Review the details of the bug you created.
14.1.2. Web console monitoring Copiar o linkLink copiado para a área de transferência!
Monitor cluster and virtual machine (VM) health with the Red Hat OpenShift Service on AWS web console.
The Red Hat OpenShift Service on AWS web console displays resource usage, alerts, events, and trends for your cluster and for OpenShift Virtualization components and resources.
| Page | Description |
|---|---|
| Overview page | Cluster details, status, alerts, inventory, and resource usage |
|
Virtualization | OpenShift Virtualization resources, usage, alerts, and status |
|
Virtualization | Top consumers of CPU, memory, and storage |
|
Virtualization | Progress of live migrations |
|
Virtualization | CPU, memory, and storage usage summary |
|
Virtualization | VM resource usage, storage, network, and migration |
|
Virtualization | List of VM events |
|
Virtualization | VM status conditions and volume snapshot status |
14.2. Collecting data for Red Hat Support Copiar o linkLink copiado para a área de transferência!
When you submit a support case to Red Hat Support, it is helpful to provide debugging information for Red Hat OpenShift Service on AWS and OpenShift Virtualization by using the following tools:
- Prometheus
- Prometheus is a time-series database and a rule evaluation engine for metrics. Prometheus sends alerts to Alertmanager for processing.
- Alertmanager
- The Alertmanager service handles alerts received from Prometheus. The Alertmanager is also responsible for sending the alerts to external notification systems.
14.2.1. Collecting data about your environment Copiar o linkLink copiado para a área de transferência!
Collecting data about your environment minimizes the time required to analyze and determine the root cause.
Prerequisites
- You have recorded the exact number of affected nodes and virtual machines.
14.2.2. Collecting data about virtual machines Copiar o linkLink copiado para a área de transferência!
Procedure
Collecting data about malfunctioning virtual machines (VMs) minimizes the time required to analyze and determine the root cause.
Prerequisites
- For Linux VMs, you have installed the latest QEMU guest agent.
For Windows VMs, you have:
- Recorded the Windows patch update details.
- Installed the latest VirtIO drivers.
- Installed the latest QEMU guest agent.
- If Remote Desktop Protocol (RDP) is enabled, you have connected by using the desktop viewer to determine whether there is a problem with the connection software.
Procedure
- Collect screenshots of VMs that have crashed before you restart them.
- Collect memory dumps from VMs before remediation attempts.
- Record factors that the malfunctioning VMs have in common. For example, the VMs have the same host or network.
14.2.3. Generating a VM memory dump Copiar o linkLink copiado para a área de transferência!
When a virtual machine (VM) terminates unexpectedly, you can use the virtctl memory-dump to generate a memory dump command to output a VM memory dump and save it on a persistent volume claim (PVC). Afterwards, you can analyze the memory dump to diagnose and troubleshoot issues on the VM.
Procedure
Optional: You have an existing PVC on which you want to save the memory dump.
-
The PVC volume mode must be
FileSystem. The PVC must be large enough to contain the memory dump.
The formula for calculating the PVC size is
(VMMemorySize + 100Mi) * FileSystemOverhead, where100Miis the memory dump overhead, andFileSystemOverheadis defined in theHCOobject.
-
The PVC volume mode must be
Create a memory dump of the required VM:
If you have an existing PVC selected on which you want to save the memory dump:
$ virtctl memory-dump get <vm_name> --claim-name=<pvc_name>If you want to create a new PVC for the memory dump:
$ virtctl memory-dump get <vm_name> --claim-name=<new_pvc_name> --create-claim
Download the memory dump:
$ virtctl memory-dump download <vm_name> --output=<output_file>Attach the memory dump to a Red Hat Support case.
Alternatively, you can inspect the memory dump, for example by using the volatility3 tool.
Optional: Remove the memory dump:
$ virtctl memory-dump remove <vm_name>
14.3. Troubleshooting Copiar o linkLink copiado para a área de transferência!
To diagnose and resolve issues with virtual machines (VMs) and cluster components, you can troubleshoot OpenShift Virtualization by using the web console or the oc CLI tool. These practices help ensure your virtualized infrastructure remains healthy.
14.3.1. Events Copiar o linkLink copiado para a área de transferência!
To monitor and troubleshoot virtual machine (VM), namespace, and resource issues, you can review Red Hat OpenShift Service on AWS events. Tracking this life-cycle information helps ensure you maintain a healthy cluster environment.
Procedure
-
To view VM events, go to VirtualMachine details
Events in the web console. To view namespace events, run the following command:
$ oc get events -n <namespace>To view resource events, run the following command:
$ oc describe <resource> <resource_name>
14.3.2. Pod logs Copiar o linkLink copiado para a área de transferência!
To diagnose issues and monitor OpenShift Virtualization pods, you can view logs using the web console or the CLI. You can also view aggregated logs using the LokiStack in the web console.
14.3.2.1. Configuring OpenShift Virtualization pod log verbosity Copiar o linkLink copiado para a área de transferência!
To gather more detailed diagnostic information for troubleshooting, you can configure the verbosity level of OpenShift Virtualization pod logs. Edit the HyperConverged custom resource (CR) to configure this setting.
Prerequisites
-
You have installed the OpenShift CLI (
oc).
Procedure
To set log verbosity for specific components, open the
HyperConvergedCR in your default text editor by running the following command:$ oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnvSet the log level for one or more components by editing the
spec.logVerbosityConfigstanza. For example:apiVersion: hco.kubevirt.io/v1beta1 kind: HyperConverged metadata: name: kubevirt-hyperconverged spec: logVerbosityConfig: kubevirt: virtAPI: 5 virtController: 4 virtHandler: 3 virtLauncher: 2 virtOperator: 6The log verbosity value must be an integer in the range
1–9, where a higher number indicates a more detailed log. In this example, thevirtAPIcomponent logs are exposed if their priority level is5or higher.- Apply your changes by saving and exiting the editor.
14.3.2.2. Viewing virt-launcher pod logs with the web console Copiar o linkLink copiado para a área de transferência!
To diagnose and troubleshoot virtual machine issues, you can view the virt-launcher pod logs by using the Red Hat OpenShift Service on AWS web console.
Procedure
-
Navigate to Virtualization
VirtualMachines. - Select a virtual machine to open the VirtualMachine details page.
- On the General tile, click the pod name to open the Pod details page.
- Click the Logs tab to view the logs.
14.3.2.3. Viewing OpenShift Virtualization pod logs with the CLI Copiar o linkLink copiado para a área de transferência!
To diagnose issues and monitor OpenShift Virtualization pods, you can view logs by using the OpenShift CLI (oc).
Prerequisites
-
You have installed the OpenShift CLI (
oc).
Procedure
View a list of pods in the OpenShift Virtualization namespace by running the following command:
$ oc get pods -n openshift-cnvExample output:
NAME READY STATUS RESTARTS AGE disks-images-provider-7gqbc 1/1 Running 0 32m disks-images-provider-vg4kx 1/1 Running 0 32m virt-api-57fcc4497b-7qfmc 1/1 Running 0 31m virt-api-57fcc4497b-tx9nc 1/1 Running 0 31m virt-controller-76c784655f-7fp6m 1/1 Running 0 30m virt-controller-76c784655f-f4pbd 1/1 Running 0 30m virt-handler-2m86x 1/1 Running 0 30m virt-handler-9qs6z 1/1 Running 0 30m virt-operator-7ccfdbf65f-q5snk 1/1 Running 0 32m virt-operator-7ccfdbf65f-vllz8 1/1 Running 0 32mView the pod log by running the following command:
$ oc logs -n openshift-cnv <pod_name>NoteIf a pod fails to start, you can use the
--previousoption to view logs from the last attempt.To monitor log output in real time, use the
-foption.Example output:
{"component":"virt-handler","level":"info","msg":"set verbosity to 2","pos":"virt-handler.go:453","timestamp":"2022-04-17T08:58:37.373695Z"} {"component":"virt-handler","level":"info","msg":"set verbosity to 2","pos":"virt-handler.go:453","timestamp":"2022-04-17T08:58:37.373726Z"} {"component":"virt-handler","level":"info","msg":"setting rate limiter to 5 QPS and 10 Burst","pos":"virt-handler.go:462","timestamp":"2022-04-17T08:58:37.373782Z"} {"component":"virt-handler","level":"info","msg":"CPU features of a minimum baseline CPU model: map[apic:true clflush:true cmov:true cx16:true cx8:true de:true fpu:true fxsr:true lahf_lm:true lm:true mca:true mce:true mmx:true msr:true mtrr:true nx:true pae:true pat:true pge:true pni:true pse:true pse36:true sep:true sse:true sse2:true sse4.1:true ssse3:true syscall:true tsc:true]","pos":"cpu_plugin.go:96","timestamp":"2022-04-17T08:58:37.390221Z"} {"component":"virt-handler","level":"warning","msg":"host model mode is expected to contain only one model","pos":"cpu_plugin.go:103","timestamp":"2022-04-17T08:58:37.390263Z"} {"component":"virt-handler","level":"info","msg":"node-labeller is running","pos":"node_labeller.go:94","timestamp":"2022-04-17T08:58:37.391011Z"}
14.3.3. Guest system logs Copiar o linkLink copiado para a área de transferência!
To diagnose issues with virtual machine (VM) guests, you can configure access to and view their boot logs using the Red Hat OpenShift Service on AWS web console or the OpenShift CLI (oc).
If the guest VM has no network, you can access it using its VNC or serial console.
This feature is disabled by default. If a VM does not explicitly have this setting enabled or disabled, it inherits the cluster-wide default setting.
If sensitive information such as credentials or other personally identifiable information (PII) is written to the serial console, it is logged with all other visible text. Use SSH to send sensitive data.
14.3.3.1. Enabling default access to guest system logs with the web console Copiar o linkLink copiado para a área de transferência!
To troubleshoot issues more easily, you can enable default access to virtual machine (VM) guest system logs by using the web console.
Procedure
-
From the side menu, click Virtualization
Overview. - Click the Settings tab.
-
Click Cluster
Guest management. - Set Enable guest system log access to on.
- Optional: If you want to hide the VM user credentials that were set by using cloud-init, set Hide guest credentials for non-privileged users to on.
14.3.3.2. Enabling default access to guest system logs with the CLI Copiar o linkLink copiado para a área de transferência!
To troubleshoot issues more easily, you can enable default access to virtual machine (VM) guest system logs by editing the HyperConverged custom resource (CR).
Prerequisites
-
You have installed the OpenShift CLI (
oc).
Procedure
Open the
HyperConvergedCR in your default editor by running the following command:$ oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnvUpdate the
disableSerialConsoleLogvalue. For example:kind: HyperConverged metadata: name: kubevirt-hyperconverged spec: virtualMachineOptions: disableSerialConsoleLog: true #...Set the value of
disableSerialConsoleLogtofalseif you want serial console access to be enabled on VMs by default.
14.3.3.3. Setting guest system log access for a single VM with the web console Copiar o linkLink copiado para a área de transferência!
To troubleshoot a specific virtual machine (VM) without changing global settings, you can configure the guest system log access by using the web console.
Procedure
-
Click Virtualization
VirtualMachines from the side menu. - Select a virtual machine to open the VirtualMachine details page.
- Click the Configuration tab.
- Set Guest system log access to on or off.
14.3.3.4. Setting guest system log access for a single VM with the CLI Copiar o linkLink copiado para a área de transferência!
To troubleshoot a specific virtual machine (VM) without changing global settings, you can configure the guest system log access by editing the VirtualMachine CR.
Prerequisites
-
You have installed the OpenShift CLI (
oc).
Procedure
Edit the virtual machine manifest by running the following command:
$ oc edit vm <vm_name>Update the value of the
logSerialConsolefield. For example:apiVersion: kubevirt.io/v1 kind: VirtualMachine metadata: name: example-vm spec: template: spec: domain: devices: logSerialConsole: true #...To enable access to the guest serial console log, set the
logSerialConsolevalue totrue.Apply the new configuration to the VM by running the following command:
$ oc apply vm <vm_name>Optional: If you edited a running VM, restart the VM to apply the new configuration. For example:
$ virtctl restart <vm_name> -n <namespace>
14.3.3.5. Viewing guest system logs with the web console Copiar o linkLink copiado para a área de transferência!
To diagnose and troubleshoot issues with a virtual machine (VM) guest, you can view the serial console logs by using the web console.
Prerequisites
- Guest system log access is enabled.
Procedure
-
Click Virtualization
VirtualMachines from the side menu. - Select a virtual machine to open the VirtualMachine details page.
- Click the Diagnostics tab.
- Click Guest system logs to load the serial console.
14.3.3.6. Viewing guest system logs with the CLI Copiar o linkLink copiado para a área de transferência!
To diagnose and troubleshoot issues with a virtual machine (VM) guest, you can view the serial console logs by running the oc logs command.
Prerequisites
- Guest system log access is enabled.
-
You have installed the OpenShift CLI (
oc).
Procedure
View the logs by running the following command, substituting your own values for
<namespace>and<vm_name>:$ oc logs -n <namespace> -l kubevirt.io/domain=<vm_name> --tail=-1 -c guest-console-log
14.3.4. Log aggregation Copiar o linkLink copiado para a área de transferência!
To more easily diagnose and troubleshoot issues, you can aggregate and filter your logs.
14.3.4.1. Viewing aggregated OpenShift Virtualization logs with Loki Copiar o linkLink copiado para a área de transferência!
To troubleshoot issues and monitor the health of your virtualization environment, you can view aggregated logs for OpenShift Virtualization pods and containers in the web console. This process requires Loki, a logging component that provides a short-term, horizontally scalable log store and log aggregation. For more information about Loki, see "Additional resources".
Prerequisites
-
You have installed the Loki Operator and deployed the
LokiStackcustom resource (CR).
Procedure
-
Navigate to Observe
Logs in the web console. -
Select application, for
virt-launcherpod logs, or infrastructure, for OpenShift Virtualization control plane pods and containers, from the log type list. - Click Show Query to display the query field.
- Enter the LogQL query in the query field and click Run Query to display the filtered logs.
14.3.4.2. OpenShift Virtualization LogQL queries Copiar o linkLink copiado para a área de transferência!
To diagnose issues and monitor OpenShift Virtualization components, you can view and filter aggregated logs by running Loki Query Language (LogQL) queries on the Observe
The default log type is infrastructure. The virt-launcher log type is application.
Optional: You can include or exclude strings or regular expressions by using line filter expressions.
If the query matches a large number of logs, the query might time out.
| Component | LogQL query |
|---|---|
| All |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| Container |
Specify one or more containers separated by a pipe ( |
|
| You must select application from the log type list before running this query.
|
You can filter log lines to include or exclude strings or regular expressions by using line filter expressions.
| Line filter expression | Description |
|---|---|
|
| Log line contains string |
|
| Log line does not contain string |
|
| Log line contains regular expression |
|
| Log line does not contain regular expression |
Example 14.1. Example line filter expression
{log_type=~".+"}|json
|kubernetes_labels_app_kubernetes_io_part_of="hyperconverged-cluster"
|= "error" != "timeout"
14.3.4.3. Common error messages Copiar o linkLink copiado para a área de transferência!
Troubleshoot OpenShift Virtualization by reviewing common error messages found in the logs.
ErrImagePullorImagePullBackOff- Indicates an incorrect deployment configuration or problems with the images that are referenced.
14.3.5. Troubleshooting data volumes Copiar o linkLink copiado para a área de transferência!
To analyze and resolve issues, you can check the Conditions and Events sections of the DataVolume object.
14.3.5.1. About data volume conditions and events Copiar o linkLink copiado para a área de transferência!
To diagnose data volume issues, you can examine the Conditions and Events sections of the oc describe command output.
Run the following command to inspect the data volume:
$ oc describe dv <DataVolume>
The Conditions section displays the following Types:
-
Bound -
Running -
Ready
The Events section provides the following additional information:
-
Typeof event -
Reasonfor logging -
Sourceof the event -
Messagecontaining additional diagnostic information.
The output from oc describe does not always contains Events.
An event is generated when the Status, Reason, or Message changes. Both conditions and events react to changes in the state of the data volume.
For example, if you misspell the URL during an import operation, the import generates a 404 message. That message change generates an event with a reason. The output in the Conditions section is updated as well.
14.3.5.2. Analyzing data volume conditions and events Copiar o linkLink copiado para a área de transferência!
To determine the state of a data volume in relation to a Persistent Volume Claim (PVC) and whether or not an operation is actively running on the data volume, inspect the Conditions and Events section of the oc describe command output.
You might also receive messages that offer specific details about the status of the data volume, and how it came to be in its current state.
There are many different combinations of conditions. Each must be evaluated in its unique context.
Examples of various combinations follow.
Bound- A successfully bound PVC displays in this example.Note that the
TypeisBound, so theStatusisTrue. If the PVC is not bound, theStatusisFalse.When the PVC is bound, an event is generated stating that the PVC is bound. In this case, the
ReasonisBoundandStatusisTrue. TheMessageindicates which PVC owns the data volume.Message, in theEventssection, provides further details including how long the PVC has been bound (Age) and by what resource (From), in this casedatavolume-controller.Example output:
Status: Conditions: Last Heart Beat Time: 2020-07-15T03:58:24Z Last Transition Time: 2020-07-15T03:58:24Z Message: PVC win10-rootdisk Bound Reason: Bound Status: True Type: Bound ... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Bound 24s datavolume-controller PVC example-dv BoundRunning- In this case, note thatTypeisRunningandStatusisFalse, indicating that an event has occurred that caused an attempted operation to fail, changing the Status fromTruetoFalse.However, note that
ReasonisCompletedand theMessagefield indicatesImport Complete.In the
Eventssection, theReasonandMessagecontain additional troubleshooting information about the failed operation. In this example, theMessagedisplays an inability to connect due to a404, listed in theEventssection’s firstWarning.From this information, you conclude that an import operation was running, creating contention for other operations that are attempting to access the data volume.
Example output:
Status: Conditions: Last Heart Beat Time: 2020-07-15T04:31:39Z Last Transition Time: 2020-07-15T04:31:39Z Message: Import Complete Reason: Completed Status: False Type: Running ... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning Error 12s (x2 over 14s) datavolume-controller Unable to connect to http data source: expected status code 200, got 404. Status: 404 Not FoundReady– IfTypeisReadyandStatusisTrue, then the data volume is ready to be used, as in the following example. If the data volume is not ready to be used, theStatusisFalse.Example output:
Status: Conditions: Last Heart Beat Time: 2020-07-15T04:31:39Z Last Transition Time: 2020-07-15T04:31:39Z Status: True Type: Ready