Chapter 6. Troubleshooting
6.1. Debug Logging
By default, virt-who logs all its activity to the file /var/log/rhsm/rhsm.log. When troubleshooting, check the log file as this might reveal useful information. To enable more detailed logging, change the debugging line in the global configuration file /etc/sysconfig/virt-who to VIRTWHO_DEBUG=1. If virt-who is running as a service, you must restart it for the configuration change to take effect. When you have resolved the underlying issue, disable diagnostic logging by changing the debugging line back to VIRTWHO_DEBUG=0 and restarting the virt-who service.
6.2. Duplicate Configuration Lines
Since there can be multiple configuration files, both global and hypervisor-specific, duplicate configuration lines might result in virt-who behaving differently to what you intend.
To detect duplicate lines in the virt-who configuration files, use the following command. The output of this command is a list of all lines in the specified files, prefixed by the number of times it occurs. Check all instances where the same line is listed as occurring twice or more, remove the duplicate line and restart the virt-who service. For instructions see Section 5.8.1, “Restarting the virt-who Service”.
# cat /etc/sysconfig/virt-who /etc/virt-who.d/* | sort | uniq -c
6.3. Credentials
Incorrect credentials can be a source of virt-who failure. If possible, test the credentials configured for use by virt-who by logging in to the virtualization manager or hypervisor. For example, if you can log in to the VMware vSphere management console and the expected hosts are visible, then credentials are correct.
6.4. Testing Configuration Options
When troubleshooting, a common method of determining the root cause of a problem is to make a change and test the result, repeating as needed. The virt-who agent provides an option to help with this technique.
Run the command virt-who --one-shot which reads all configuration files, retrieves the list of virtual machines from all sources, then exits immediately. This tests the configuration files, credentials and connectivity to configured virtualization platforms.
# virt-who --one-shot
The output you can expect is a list of hypervisors and the hosted guest virtual machines, in JSON format. The following is an extract from virt-who output from a VMware vSphere instance. The output from all hypervisors follows the same structure.
{
"guestId": "422f24ed-71f1-8ddf-de53-86da7900df12",
"state": 5,
"attributes": {
"active": 0,
"virtWhoType": "esx",
"hypervisorType": "vmware"
}
},6.5. Example Scenarios
6.5.1. Virt-who fails to connect with the virtualization platform
Check the Red Hat Subscription Manager log file - /var/log/rhsm/rhsm.log - if virt-who fails to connect with the virtualization platform. If you find the message No route to host, one possible reason is that the hypervisor is listening on a port other than what you expect. For example, Red Hat Virtualization Manager defaults to port 8443 for backward compatibility, but virt-who defaults to using port 443. In this case, you would edit the hypervisor’s configuration file in /etc/virt-who.d/ and append :443 to the value for the server line, resulting in the line: server=https://rhevmhost1.example.com:443.
6.5.2. Hypervisors are listed in the Satellite web UI by their UUID, not their host name
By default, hypervisors are identified in the Satellite web UI by their UUID. It is possible to change this so that they are identified by host name, but this configuration change must be made before Satellite is started, otherwise the hypervisors will be duplicated. If you need to change this, raise a Support Ticket with Red Hat Support.
6.5.3. Virt-who attempts to connect to virtualization manager or hypervisor via an HTTP proxy on the local network fails
There are three workarounds:
- Configure the proxy to allow local traffic to pass through. (Recommended)
- If allowing local traffic to pass through is not possible, install a Squid proxy server on the Satellite Server. For further details, see the Red Hat Knowledgebase solution How to bypass proxy for certain repository URLs on Satellite 6.
You can also consider to configure virt-who to use no proxy by adding
NO_PROXY=*to/etc/sysconfig/virt-who. Note that the values in/etc/sysconfig/virt-whoare environment variables, and are sourced during daemon runs. If running virt-who in one-shot mode, export the values in/etc/sysconfig/virt-whofirst. Note that the required package versions are: python-rhsm >= 1.17.9-1 and virt-who >= 0.17-11.# set -a # source /etc/sysconfig/virt-who # virt-who -o
6.5.4. Configure virt-who to use an internal proxy
To configure virt-who to use an internal proxy instead of the external proxy Satellite Server uses to connect to the external networks（for example, the CDN), add rhsm_proxy_hostname and rhsm_proxy_port to the virt-who configuration file in /etc/virt-who.d/. Note that the virt-who version must be >= 0.14. For example:
# vi /etc/virt-who.d/fabric-1.conf rhsm_proxy_hostname = internal-proxy.example.com rhsm_proxy_port = 3128
Optionally specify rhsm_proxy_user and rhsm_proxy_password in the same configuration file if required.
6.6. Renewing Host Subscriptions
This section covers three methods to reattach subscriptions for multiple hosts. The following use cases apply:
- When host subscriptions have expired and you need to attach new valid subscriptions.
- When host subscriptions are still valid but you need to attach additional subscriptions.
If host subscriptions have expired, but you have configured auto-attach and virt-who previously, subscription manager will attempt to reattach a valid subscription that covers the host and its virtual machines based on a set of criteria. No action is required.
6.6.1. Using Web UI
The web UI method allows you to attach multiple subscriptions to multiple hosts at the same time.
-
Click Hosts
Content Hosts. If prompted, select the desired organization. - Select the desired hosts. You can use the filter function to narrow down the list of hosts you want to attach subscriptions to. Use the check box at the top to select all hosts listed.
-
Click Select Action
Manage Subscriptions. - Select the desired subscriptions, and click Add Selected.
When all selected subscriptions have been attached, the task result displays success. To confirm, go to Hosts
6.6.2. Using Hammer CLI
The Hammer CLI method allows you to update the subscriptions iteratively per host, or script and automate the action for multiple hosts.
List available subscriptions in the organization.
# hammer --output json subscription list --organization example [ { "ID": 192, "UUID": "2c918093561eaa39015630f5cd841d56", "Name": "Red Hat Enterprise Linux Server, Premium (Physical or Virtual Nodes)", ... }]Search for hosts that do not have a valid subscription.
# hammer host list --search "subscription_status = invalid" ---|---------------------------|------------------|--------------- ID | NAME | OPERATING SYSTEM | HOST GROUP ---|---------------------------|------------------|--------------- 45 | cloudforms.example.com | RedHat 7.2 | Infrastructure 84 | devnode-146.example.com | RedHat 7.2 | Wordpress 82 | virt-testing.example.com | RedHat 7.1 | Development ---|---------------------------|------------------|---------------
Attach a subscription to the desired host.
# hammer host subscription attach --host devnode-146.example.com --quantity 2 --subscription-id 192 Subscription attached to the host successfully
Confirm the subscription has been successfully attached.
# hammer host list --search "subscription_status = invalid" ---|---------------------------|------------------|--------------- ID | NAME | OPERATING SYSTEM | HOST GROUP ---|---------------------------|------------------|--------------- 45 | cloudforms.example.com | RedHat 7.2 | Infrastructure 82 | virt-testing.example.com | RedHat 7.1 | Development ---|---------------------------|------------------|---------------
6.6.3. Using CSV Export and Import
The CSV method also uses the Hammer CLI tool and allows you to back up the mapping information of subscriptions, hosts, and activation keys and import it back to Satellite to ensure that each hosts gets the right subscriptions attached. To use this method for subscription renewal, you need to export the CSV file before the subscriptions expire.
Export the CSV file. It is recommended to add this to a cron job so that the subscription status of all the hosts are always backed up.
# hammer csv content-hosts --export --file content-hosts-export.csv --itemized-subscriptions --organization example
- Edit the CSV file to include new subscription details, for example the new contract numbers.
Import the CSV file back to the host when your hosts' subscriptions expire, and you need to re-attach subscriptions.
# hammer csv content-hosts --file content-hosts-export.csv --itemized-subscriptions --organization example
