Red Hat Summit Red Hat SummitSupportConsoleDevelopersStart a trialContact

Chapter 20. Diagnosing virtual machine problems

When working with virtual machines (VMs), you may encounter problems with varying levels of severity. Some problems may have a quick and easy fix, while for others, you may have to capture VM-related data and logs to report or diagnose the problems.

The following sections provide detailed information about generating logs and diagnosing some common VM problems, as well as about reporting these problems.

20.1. Generating libvirt debug logs
Copy link

To diagnose virtual machine (VM) problems, it is helpful to generate and review libvirt debug logs. Attaching debug logs is also useful when asking for support to resolve VM-related problems.

20.1.1. Understanding libvirt debug logs
Copy link

Debug logs are text files that contain data about events that occur during virtual machine (VM) runtime. The logs provide information about fundamental server-side functionalities, such as host libraries and the libvirt daemon. The log files also contain the standard error output (stderr) of all running VMs.

Debug logging is not enabled by default and has to be enabled when libvirt starts.

Afterwards, you can attach the logs when requesting support with a VM problem. For details, see Attaching libvirt debug logs to support requests.

20.1.2. Enabling libvirt debug logs persistently
Copy link

You can configure libvirt debug logging to be automatically enabled whenever libvirt starts. By default, virtqemud is the main libvirt daemon in RHEL 10. To make persistent changes in the libvirt configuration, you must edit the virtqemud.conf file, located in the /etc/libvirt directory.

Procedure

  1. Open the virtqemud.conf file in an editor.

  2. Replace or set the filters according to your requirements.

    Expand
    Table 20.1. Debugging filter values

    1

    logs all messages generated by libvirt.

    2

    logs all non-debugging information.

    3

    logs all warning and error messages. This is the default value.

    4

    logs only error messages.

    Example 20.1. Sample daemon settings for logging filters

    The following settings:

    • Log all error and warning messages from the remote, util.json, and rpc layers
    • Log only error messages from the event layer.
    • Save the filtered logs to /var/log/libvirt/libvirt.log 
    log_filters="3:remote 4:event 3:util.json 3:rpc"
log_outputs="1:file:/var/log/libvirt/libvirt.log"
    Copy to Clipboard Toggle word wrap
  3. Save and exit.

  4. Restart the libvirt daemon. 

    $ systemctl restart virtqemud.service
    Copy to Clipboard Toggle word wrap

20.1.3. Enabling libvirt debug logs during runtime
Copy link

You can modify the libvirt daemon’s runtime settings to enable debug logs and save them to an output file.

This is useful when restarting the libvirt daemon is not possible because restarting fixes the problem, or because there is another process, such as migration or backup, running at the same time. Modifying runtime settings is also useful if you want to try a command without editing the configuration files or restarting the daemon.

Prerequisites

  • Make sure the libvirt-admin package is installed.

Procedure

  1. Optional: Back up the active set of log filters. 

    # virt-admin -c virtqemud:///system daemon-log-filters >> virt-filters-backup
    Copy to Clipboard Toggle word wrap

    This makes it possible to restore the active set of filter after generating the logs. If you do not restore the filters, the messages continue to be logged, which may affect system performance.

  2. Use the virt-admin utility to enable debugging and set the filters according to your requirements.

    Expand
    Table 20.2. Debugging filter values

    1

    logs all messages generated by libvirt.

    2

    logs all non-debugging information.

    3

    logs all warning and error messages. This is the default value.

    4

    logs only error messages.

    Example 20.2. Sample virt-admin setting for logging filters

    The following command:

    • Logs all error and warning messages from the remote, util.json, and rpc layers
    • Logs only error messages from the event layer. 
    # virt-admin -c virtqemud:///system daemon-log-filters "3:remote 4:event 3:util.json 3:rpc"
    Copy to Clipboard Toggle word wrap

  3. Use the virt-admin utility to save the logs to a specific file or directory.

    For example, the following command saves the log output to the libvirt.log file in the /var/log/libvirt/ directory. 

    # virt-admin -c virtqemud:///system daemon-log-outputs "1:file:/var/log/libvirt/libvirt.log"
    Copy to Clipboard Toggle word wrap

  4. Optional: You can also remove the filters to generate a log file that contains all VM-related information. However, it is not recommended since this file may contain a large amount of redundant information produced by libvirt’s modules.

    • Use the virt-admin utility to specify an empty set of filters. 

      # virt-admin -c virtqemud:///system daemon-log-filters
  Logging filters:
      Copy to Clipboard Toggle word wrap

  5. Optional: Restore the filters to their original state by using the backup file that you created previously. 

    # virt-admin -c virtqemud:///system daemon-log-filters "<original-filters>"
    Copy to Clipboard Toggle word wrap

    In this command, replace <original-filters> with the content of virt-filters-backup.

    Note that if you do not restore the filters, the messages continue to be logged, which may affect system performance.

You may have to request additional support to diagnose and resolve virtual machine (VM) problems. Attaching the debug logs to the support request is highly recommended to ensure that the support team has access to all the information they need to provide a quick resolution of the VM-related problem.

Procedure

  • To report a problem and request support, open a support case.

  • Based on the encountered problems, attach the following logs along with your report:

    • For problems with the libvirt service, attach the /var/log/libvirt/libvirt.log file from the host.

    • For problems with a specific VM, attach its respective log file.

      For example, for the testguest1 VM, attach the testguest1.log file, which can be found at /var/log/libvirt/qemu/testguest1.log.

20.2. Dumping a virtual machine core
Copy link

To analyze why a virtual machine (VM) crashed or malfunctioned, you can dump the VM core to a file on disk for later analysis and diagnostics.

20.2.1. How virtual machine core dumping works
Copy link

A virtual machine (VM) requires numerous running processes to work accurately and efficiently. In some cases, a running VM may terminate unexpectedly or malfunction while you are using it. Restarting the VM may cause the data to be reset or lost, which makes it difficult to diagnose the exact problem that caused the VM to crash.

In such cases, you can use the virsh dump utility to save (or dump) the core of a VM to a file before you reboot the VM. The core dump file contains a raw physical memory image of the VM which contains detailed information about the VM. This information can be used to diagnose VM problems, either manually, or by using a tool such as the crash utility.

20.2.2. Creating a virtual machine core dump file
Copy link

A virtual machine (VM) core dump contains detailed information about the state of a VM at any given time. This information, which is similar to a snapshot of the VM, can help you detect problems if a VM malfunctions or shuts down suddenly.

Prerequisites

  • Make sure you have sufficient disk space to save the file. Note that the space occupied by the VM depends on the amount of RAM allocated to the VM.

Procedure

  • Use the virsh dump utility.

    For example, the following command dumps the test-guest1 VM’s cores, its memory and the CPU common register file to sample-core.file in the /core/file directory. 

    # virsh dump test-guest1 /core/file/sample-core.file --memory-only
Domain 'test-guest1' dumped to /core/file/sample-core.file
    Copy to Clipboard Toggle word wrap
Important

The crash utility no longer supports the default file format of the virsh dump command. To analyze a core dump file by using crash, you must create the file with the --memory-only option.

Additionally, you must use the --memory-only option when creating a core dump file to attach to a Red Hat Support Case.

Troubleshooting

If the virsh dump command fails with a System is deadlocked on memory error, ensure you are assigning sufficient memory for the core dump file. To do so, use the following crashkernel option value. Alternatively, do not use crashkernel at all, which assigns core dump memory automatically. 

crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M
Copy to Clipboard Toggle word wrap

20.3. Backtracing virtual machine processes
Copy link

When a process related to a virtual machine (VM) malfunctions, you can use the gstack command along with the process identifier (PID) to generate an execution stack trace of the malfunctioning process. If the process is a part of a thread group then all the threads are traced as well.

Prerequisites

  • Ensure that the GDB package is installed.

    For details about installing GDB and the available components, see Installing the GNU Debugger.

  • Make sure you know the PID of the processes that you want to backtrace.

    You can find the PID by using the pgrep command followed by the name of the process. For example: 

    # pgrep libvirt
22014
22025
    Copy to Clipboard Toggle word wrap

Procedure

  • Use the gstack utility followed by the PID of the process you wish to backtrace.

    For example, the following command backtraces the libvirt process with the PID 22014. 

    # gstack 22014
Thread 3 (Thread 0x7f33edaf7700 (LWP 22017)):
#0  0x00007f33f81aef21 in poll () from /lib64/libc.so.6
#1  0x00007f33f89059b6 in g_main_context_iterate.isra () from /lib64/libglib-2.0.so.0
#2  0x00007f33f8905d72 in g_main_loop_run () from /lib64/libglib-2.0.so.0
...
    Copy to Clipboard Toggle word wrap

    For more information on using gstack, see the gstack man page on your system.

To request additional help and support, you can:

Back to top
Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Theme

© 2025 Red Hat