SupportConsoleDevelopersStart a trialContact

Chapter 23. Diagnosing virtual machine problems

When working with virtual machines (VMs), you may encounter problems with varying levels of severity. Some problems may have a quick and easy fix, while for others, you may have to capture VM-related data and logs to report or diagnose the problems.

The following sections provide detailed information about generating logs and diagnosing some common VM problems, as well as about reporting these problems.

23.1. Generating libvirt debug logs

To diagnose virtual machine (VM) problems, it is helpful to generate and review libvirt debug logs. Attaching debug logs is also useful when asking for support to resolve VM-related problems.

The following sections explain what debug logs are, how you can set them to be persistent, enable them during runtime, and attach them when reporting problems.

23.1.1. Understanding libvirt debug logs

Debug logs are text files that contain data about events that occur during virtual machine (VM) runtime. The logs provide information about fundamental server-side functionalities, such as host libraries and the libvirt daemon. The log files also contain the standard error output (stderr) of all running VMs.

Debug logging is not enabled by default and has to be enabled when libvirt starts. You can enable logging for a single session or persistently. You can also enable logging when a libvirt daemon session is already running by modifying the daemon run-time settings.

Attaching the libvirt debug logs is also useful when requesting support with a VM problem.

23.1.2. Enabling persistent settings for libvirt debug logs

You can configure libvirt debug logging to be automatically enabled whenever libvirt starts. By default, virtqemud is the main libvirt daemon in RHEL 9. To make persistent changes in the libvirt configuration, you must edit the virtqemud.conf file, located in the /etc/libvirt directory.

Note

In some cases, for example when you upgrade from RHEL 8, libvirtd might still be the enabled libvirt daemon. In that case, you must edit the libvirtd.conf file instead.

Procedure

  1. Open the virtqemud.conf file in an editor.

  2. Replace or set the filters according to your requirements.

    Table 23.1. Debugging filter values

    1

    logs all messages generated by libvirt.

    2

    logs all non-debugging information.

    3

    logs all warning and error messages. This is the default value.

    4

    logs only error messages.

    Example 23.1. Sample daemon settings for logging filters

    The following settings:

    • Log all error and warning messages from the remote, util.json, and rpc layers
    • Log only error messages from the event layer.
    • Save the filtered logs to /var/log/libvirt/libvirt.log 
    log_filters="3:remote 4:event 3:util.json 3:rpc"
log_outputs="1:file:/var/log/libvirt/libvirt.log"
  3. Save and exit.

  4. Restart the libvirt daemon. 

    $ systemctl restart virtqemud.service

23.1.3. Enabling libvirt debug logs during runtime

You can modify the libvirt daemon’s runtime settings to enable debug logs and save them to an output file.

This is useful when restarting the libvirt daemon is not possible because restarting fixes the problem, or because there is another process, such as migration or backup, running at the same time. Modifying runtime settings is also useful if you want to try a command without editing the configuration files or restarting the daemon.

Prerequisites

  • Make sure the libvirt-admin package is installed.

Procedure

  1. Optional: Back up the active set of log filters. 

    # virt-admin -c virtqemud:///system daemon-log-filters >> virt-filters-backup
    Note

    It is recommended that you back up the active set of filters so that you can restore them after generating the logs. If you do not restore the filters, the messages will continue to be logged which may affect system performance.

  2. Use the virt-admin utility to enable debugging and set the filters according to your requirements.

    Table 23.2. Debugging filter values

    1

    logs all messages generated by libvirt.

    2

    logs all non-debugging information.

    3

    logs all warning and error messages. This is the default value.

    4

    logs only error messages.

    Example 23.2. Sample virt-admin setting for logging filters

    The following command:

    • Logs all error and warning messages from the remote, util.json, and rpc layers
    • Logs only error messages from the event layer. 
    # virt-admin -c virtqemud:///system daemon-log-filters "3:remote 4:event 3:util.json 3:rpc"

  3. Use the virt-admin utility to save the logs to a specific file or directory.

    For example, the following command saves the log output to the libvirt.log file in the /var/log/libvirt/ directory. 

    # virt-admin -c virtqemud:///system daemon-log-outputs "1:file:/var/log/libvirt/libvirt.log"

  4. Optional: You can also remove the filters to generate a log file that contains all VM-related information. However, it is not recommended since this file may contain a large amount of redundant information produced by libvirt’s modules.

    • Use the virt-admin utility to specify an empty set of filters. 

      # virt-admin -c virtqemud:///system daemon-log-filters
  Logging filters:
  5. Optional: Restore the filters to their original state using the backup file.
    Perform the second step with the saved values to restore the filters.

23.1.4. Attaching libvirt debug logs to support requests

You may have to request additional support to diagnose and resolve virtual machine (VM) problems. Attaching the debug logs to the support request is highly recommended to ensure that the support team has access to all the information they need to provide a quick resolution of the VM-related problem.

Procedure

  • To report a problem and request support, open a support case.

  • Based on the encountered problems, attach the following logs along with your report:

    • For problems with the libvirt service, attach the /var/log/libvirt/libvirt.log file from the host.

    • For problems with a specific VM, attach its respective log file.

      For example, for the testguest1 VM, attach the testguest1.log file, which can be found at /var/log/libvirt/qemu/testguest1.log.

Additional resources

23.2. Dumping a virtual machine core

To analyze why a virtual machine (VM) crashed or malfunctioned, you can dump the VM core to a file on disk for later analysis and diagnostics.

This section provides a brief introduction to core dumping and explains how you can dump a VM core to a specific file.

23.2.1. How virtual machine core dumping works

A virtual machine (VM) requires numerous running processes to work accurately and efficiently. In some cases, a running VM may terminate unexpectedly or malfunction while you are using it. Restarting the VM may cause the data to be reset or lost, which makes it difficult to diagnose the exact problem that caused the VM to crash.

In such cases, you can use the virsh dump utility to save (or dump) the core of a VM to a file before you reboot the VM. The core dump file contains a raw physical memory image of the VM which contains detailed information about the VM. This information can be used to diagnose VM problems, either manually, or by using a tool such as the crash utility.

Additional resources

23.2.2. Creating a virtual machine core dump file

A virtual machine (VM) core dump contains detailed information about the state of a VM at any given time. This information, which is similar to a snapshot of the VM, can help you detect problems if a VM malfunctions or shuts down suddenly.

Prerequisites

  • Make sure you have sufficient disk space to save the file. Note that the space occupied by the VM depends on the amount of RAM allocated to the VM.

Procedure

  • Use the virsh dump utility.

    For example, the following command dumps the lander1 VM’s cores, its memory and the CPU common register file to gargantua.file in the /core/file directory. 

    # virsh dump lander1 /core/file/gargantua.file --memory-only
Domain 'lander1' dumped to /core/file/gargantua.file
Important

The crash utility no longer supports the default file format of the virsh dump command. To analyze a core dump file by using crash, you must create the file with the --memory-only option.

Additionally, you must use the --memory-only option when creating a core dump file to attach to a Red Hat Support Case.

Troubleshooting

If the virsh dump command fails with a System is deadlocked on memory error, ensure you are assigning sufficient memory for the core dump file. To do so, use the following crashkernel option value. Alternatively, do not use crashkernel at all, which assigns core dump memory automatically. 

crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M

Additional resources

23.3. Backtracing virtual machine processes

When a process related to a virtual machine (VM) malfunctions, you can use the gstack command along with the process identifier (PID) to generate an execution stack trace of the malfunctioning process. If the process is a part of a thread group then all the threads are traced as well.

Prerequisites

  • Ensure that the GDB package is installed.

    For details about installing GDB and the available components, see Installing the GNU Debugger.

  • Make sure you know the PID of the processes that you want to backtrace.

    You can find the PID by using the pgrep command followed by the name of the process. For example: 

    # pgrep libvirt
22014
22025

Procedure

  • Use the gstack utility followed by the PID of the process you wish to backtrace.

    For example, the following command backtraces the libvirt process with the PID 22014. 

    # gstack 22014
Thread 3 (Thread 0x7f33edaf7700 (LWP 22017)):
#0  0x00007f33f81aef21 in poll () from /lib64/libc.so.6
#1  0x00007f33f89059b6 in g_main_context_iterate.isra () from /lib64/libglib-2.0.so.0
#2  0x00007f33f8905d72 in g_main_loop_run () from /lib64/libglib-2.0.so.0
...

Additional resources

Additional resources for reporting virtual machine problems and providing logs

To request additional help and support, you can:

Red Hat logoGithubRedditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

© 2024 Red Hat, Inc.