Chapter 5. Troubleshooting

Verifying the status of all virt-who instances is the recommended first step in troubleshooting. All virt-who instances configured and deployed using either the Satellite web UI or hammer are reported in the Satellite web UI. Those virt-who instances which were configured manually are not available in the Satellite web UI.

hammer virt-who-config list

# hammer virt-who-config list

By default, virt-who logs all its activity to the file /var/log/rhsm/rhsm.log. When troubleshooting, check the log file as this might reveal useful information. To enable more detailed logging, change the debugging line in the global configuration file /etc/sysconfig/virt-who to VIRTWHO_DEBUG=1. If virt-who is running as a service, you must restart it for the configuration change to take effect. When you have resolved the underlying issue, disable diagnostic logging by changing the debugging line back to VIRTWHO_DEBUG=0 and restarting the virt-who service.

Since there can be multiple configuration files, both global and hypervisor-specific, duplicate configuration lines might result in virt-who behaving differently to what you intend.

To detect duplicate lines in the virt-who configuration files, use the following command. The output of this command is a list of all lines in the specified files, prefixed by the number of times it occurs. Check all instances where the same line is listed as occurring twice or more, remove the duplicate line and restart the virt-who service. For instructions see Section 3.6, “Restarting the virt-who Service”.

cat /etc/sysconfig/virt-who /etc/virt-who.d/* | sort | uniq -c

# cat /etc/sysconfig/virt-who /etc/virt-who.d/* | sort | uniq -c

The comment character # must be the first character on the line in a configuration file. Any white space at the start of the line results in the line being included in the configuration.

Incorrect credentials can be a source of virt-who failure. If possible, test the credentials configured for use by virt-who by logging in to the virtualization manager or hypervisor. For example, if you can log in to the VMware vSphere management console and the expected hosts are visible, then credentials are correct.

When troubleshooting, a common method of determining the root cause of a problem is to make a change and test the result, repeating as needed. The virt-who agent provides an option to help with this technique.

Run the command virt-who --one-shot which reads all configuration files, retrieves the list of virtual machines from all sources, then exits immediately. This tests the configuration files, credentials and connectivity to configured virtualization platforms.

virt-who --one-shot

# virt-who --one-shot

The output you can expect is a list of hypervisors and the hosted guest virtual machines, in JSON format. The following is an extract from virt-who output from a VMware vSphere instance. The output from all hypervisors follows the same structure.

{
    "guestId": "422f24ed-71f1-8ddf-de53-86da7900df12",
    "state": 5,
    "attributes": {
        "active": 0,
        "virtWhoType": "esx",
        "hypervisorType": "vmware"
    }
},

{
    "guestId": "422f24ed-71f1-8ddf-de53-86da7900df12",
    "state": 5,
    "attributes": {
        "active": 0,
        "virtWhoType": "esx",
        "hypervisorType": "vmware"
    }
},

If you have multiple virt-who configuration files, move one virt-who configuration file at a time to your /root directory while testing after each file move. If the issue no longer occurs, the cause is associated with the most recently moved file. After you have resolved the issue, return the virt-who configuration files to their original location.