Chapter 5. Troubleshooting

5.1. Verifying virt-who Status

Verifying the status of all virt-who instances is the recommended first step in troubleshooting. All virt-who instances configured and deployed using either the Satellite web UI or hammer are reported in the Satellite web UI. Those virt-who instances which were configured manually are not available in the Satellite web UI.

5.1.1. Verifying virt-who Status in Satellite web UI

Navigate to Infrastructure Virt-who configurations
View the Status value for each virt-who instance.

A status value of OK indicates the virt-who instance is successfully connecting to the Satellite Server and reporting those virtual machines managed by the hypervisor.

5.1.2. Verifying virt-who Status using Hammer

On the Satellite Server, list the status of all virt-who instances.

# hammer virt-who-config list

The hammer output includes the date and time at which each virt-who instance reported to the Satellite Server.

5.2. Debug Logging

By default, virt-who logs all its activity to the file /var/log/rhsm/rhsm.log. When troubleshooting, check the log file as this might reveal useful information. To enable more detailed logging, change the debugging line in the global configuration file /etc/sysconfig/virt-who to VIRTWHO_DEBUG=1. If virt-who is running as a service, you must restart it for the configuration change to take effect. When you have resolved the underlying issue, disable diagnostic logging by changing the debugging line back to VIRTWHO_DEBUG=0 and restarting the virt-who service.

5.3. Duplicate Configuration Lines

Since there can be multiple configuration files, both global and hypervisor-specific, duplicate configuration lines might result in virt-who behaving differently to what you intend.

To detect duplicate lines in the virt-who configuration files, use the following command. The output of this command is a list of all lines in the specified files, prefixed by the number of times it occurs. Check all instances where the same line is listed as occurring twice or more, remove the duplicate line and restart the virt-who service. For instructions see Section 3.4.1, “Restarting the virt-who Service”.

# cat /etc/sysconfig/virt-who /etc/virt-who.d/* | sort | uniq -c

5.4. Comment Character in a Configuration File

The comment character # must be the first character on the line in a configuration file. Any white space at the start of the line results in the line being included in the configuration.

5.5. Credentials

Incorrect credentials can be a source of virt-who failure. If possible, test the credentials configured for use by virt-who by logging in to the virtualization manager or hypervisor. For example, if you can log in to the VMware vSphere management console and the expected hosts are visible, then credentials are correct.

5.6. Testing Configuration Options

When troubleshooting, a common method of determining the root cause of a problem is to make a change and test the result, repeating as needed. The virt-who agent provides an option to help with this technique.

Run the command virt-who --one-shot which reads all configuration files, retrieves the list of virtual machines from all sources, then exits immediately. This tests the configuration files, credentials and connectivity to configured virtualization platforms.

# virt-who --one-shot

The output you can expect is a list of hypervisors and the hosted guest virtual machines, in JSON format. The following is an extract from virt-who output from a VMware vSphere instance. The output from all hypervisors follows the same structure.

{
    "guestId": "422f24ed-71f1-8ddf-de53-86da7900df12",
    "state": 5,
    "attributes": {
        "active": 0,
        "virtWhoType": "esx",
        "hypervisorType": "vmware"
    }
},

5.7. Identifying issues when using multiple virt-who configuration files

If you have multiple virt-who configuration files, move one virt-who configuration file at a time to your /root directory while testing after each file move. If the issue no longer occurs, the cause is associated with the most recently moved file. After you have resolved the issue, return the virt-who configuration files to their original location.

5.8. Example Scenarios

5.8.1. Virt-who fails to connect with the virtualization platform

If virt-who fails to connect to the virtualization platform check the Red Hat Subscription Manager log file /var/log/rhsm/rhsm.log. If you find the message No route to host, one possible reason is that the hypervisor is listening on a port other than what you expect. For example, Red Hat Virtualization Manager defaults to port 8443 for backward compatibility, but virt-who defaults to using port 443. In this case, edit the hypervisor’s configuration file in directory /etc/virt-who.d/ and append :443 to the value for the server line, resulting in the line: server=https://rhevmhost1.example.com:443.

5.8.2. Hypervisors are listed in the Satellite web UI by their UUID, not their host name

By default, hypervisors are identified in the Satellite web UI by their UUID. It is possible to change this so that they are identified by host name, but this configuration change must be made before Satellite is started, otherwise the hypervisors will be duplicated. If you need to change this, raise a Support Ticket with Red Hat Support.

5.8.3. Virt-who attempts to connect to virtualization manager or hypervisor via an HTTP proxy on the local network fails

There are three workarounds:

Configure the proxy to allow local traffic to pass through. (Recommended)
If allowing local traffic to pass through is not possible, install a Squid proxy server on the Satellite Server. For further details, see the Red Hat Knowledgebase solution How to bypass proxy for certain repository URLs on Satellite 6.
You can also configure virt-who to ignore proxy network settings by adding NO_PROXY=* to /etc/sysconfig/virt-who. Note that the values in /etc/sysconfig/virt-who are environment variables, and are sourced during daemon runs. If running virt-who in one-shot mode, export the values in /etc/sysconfig/virt-who first. Note that the required package versions are: python-rhsm >= 1.17.9-1 and virt-who >= 0.17-11.
```
# set -a
# source /etc/sysconfig/virt-who
# virt-who -o
```

5.8.4. Configure virt-who to use an internal proxy

To configure virt-who to use an internal proxy, add options rhsm_proxy_hostname and rhsm_proxy_port to the virt-who configuration file in /etc/virt-who.d/. Note that the virt-who version must be >= 0.14. For example:

# vi /etc/virt-who.d/fabric-1.conf

rhsm_proxy_hostname = internal-proxy.example.com
rhsm_proxy_port = 3128

Optionally specify rhsm_proxy_user and rhsm_proxy_password in the same configuration file if required.

5.9. Renewing Host Subscriptions

This section covers three methods to reattach subscriptions for multiple hosts. The following use cases apply:

When host subscriptions have expired and you need to attach new valid subscriptions.
When host subscriptions are still valid but you need to attach additional subscriptions.

If host subscriptions have expired, but you have configured auto-attach and virt-who previously, subscription manager will attempt to reattach a valid subscription that covers the host and its virtual machines based on a set of criteria. No action is required.

5.9.1. Using Web UI

The web UI method allows you to attach multiple subscriptions to multiple hosts at the same time.

Click Hosts Content Hosts. If prompted, select the desired organization.
Select the desired hosts. You can use the filter function to narrow down the list of hosts you want to attach subscriptions to. Use the check box at the top to select all hosts listed.
Click Select Action Manage Subscriptions.
Select the desired subscriptions, and click Add Selected.

When all selected subscriptions have been attached, the task result displays success. To confirm, go to Hosts Content Hosts and select the desired host. Click Subscriptions Subscriptions, and verify that the newly attached subscriptions are listed.

5.9.2. Using Hammer CLI

The Hammer CLI method allows you to update the subscriptions iteratively per host, or script and automate the action for multiple hosts.

List available subscriptions in the organization.

# hammer --output json subscription list --organization example

[
{
  "ID": 192,
  "UUID": "2c918093561eaa39015630f5cd841d56",
  "Name": "Red Hat Enterprise Linux Server, Premium (Physical or Virtual Nodes)",
   ...
}]

Search for hosts that do not have a valid subscription.

# hammer host list --search "subscription_status = invalid"

---|---------------------------|------------------|---------------
ID | NAME                      | OPERATING SYSTEM | HOST GROUP
---|---------------------------|------------------|---------------
45 | cloudforms.example.com    | RedHat 7.2       | Infrastructure
84 | devnode-146.example.com   | RedHat 7.2       | Wordpress
82 | virt-testing.example.com  | RedHat 7.1       | Development
---|---------------------------|------------------|---------------

Attach a subscription to the desired host.

# hammer host subscription attach --host devnode-146.example.com --quantity 2 --subscription-id 192

Subscription attached to the host successfully

Confirm the subscription has been successfully attached.

# hammer host list --search "subscription_status = invalid"

---|---------------------------|------------------|---------------
ID | NAME                      | OPERATING SYSTEM | HOST GROUP
---|---------------------------|------------------|---------------
45 | cloudforms.example.com    | RedHat 7.2       | Infrastructure
82 | virt-testing.example.com  | RedHat 7.1       | Development
---|---------------------------|------------------|---------------

5.9.3. Using CSV Export and Import

The CSV method also uses the Hammer CLI tool and allows you to back up the mapping information of subscriptions, hosts, and activation keys and import it back to Satellite to ensure that each hosts gets the right subscriptions attached. To use this method for subscription renewal, you need to export the CSV file before the subscriptions expire.

Export the CSV file. It is recommended to add this to a cron job so that the subscription status of all the hosts are always backed up.
```
# hammer csv content-hosts --export --file content-hosts-export.csv --itemized-subscriptions --organization example
```
Edit the CSV file to include new subscription details, for example the new contract numbers.
Import the CSV file back to the host when your hosts' subscriptions expire, and you need to re-attach subscriptions.
```
# hammer csv content-hosts --file content-hosts-export.csv --itemized-subscriptions --organization example
```