Chapter 7. Troubleshooting CDK Problems

Vagrant is strictly command-line oriented. There are no menu entries. To run Vagrant, launch a command prompt or a terminal session on your machine. Vagrant should be automatically added to your path, so you only need to type vagrant.

There are a number of provisioning steps performed when Vagrant launches a virtual machine. These may come from plugins, such as the Vagrant Registration plugin for Red Hat, or from steps that are included in the Vagrantfile for provisioning the box on startup. Vagrant runs these commands by using SSH.

In many cases, errors during the provisioning step are not fatal, so the Vagrant box will still be running. Use the vagrant status command to see what the state of your box is. You can stop the Vagrant box with the vagrant halt command. If the machine is running, you should be able to log into it using the vagrant ssh command to examine the VM and see what went wrong.

While the box is running, you can rerun the provisioning steps by running the vagrant provision command.

Note that you need to be in the same directory where your Vagrantfile is. If you have lost track of where your Vagrantfile is, use the vagrant global-status command to list the boxes that you have started and the directory where the Vagrantfile and state is stored.

Any system running the CDK (Windows, Mac, or RHEL), requires not only that the computer it is running on have Virtualization Support, but also that the support be enabled in the BIOS. On a RHEL or Fedora system, if Virtualization Support is not enabled, when you run vagrant up, you will see a message similar to the following: "Error while creating domain: Error saving the server: Call to virDomainDefineXML failed: invalid argument: could not find capabilities for domaintype=kvm". Enabling Virtualization Support in the computer’s BIOS should fix the problem.

Run the virt-host-validate command to check whether or not Virtualization has been enabled on the computer.

You can run the subscription-manager tool to log in, register or unregister the box, check your subscription status, or enable available software repositories as you would on any other Red Hat Enterprise Linux installation. However, you need to use the sudo -i command to be root because the root password for the Red Hat Enterprise Linux Vagrant boxes is not distributed. The sudo utility has been setup for the vagrant user to run any commands.

For more information, see Red Hat Subscription Management.

The Vagrant Registration plugin automatically detaches the box from the Red Hat subscription when you shut down the box using the vagrant halt or vagrant destroy commands. If the box is not shut down by Vagrant, this does not happen. If you still have the box set up, you should be able to bring the box up again using the vagrant up command and then shut it down correctly with vagrant halt or vagrant destroy, which will unregister the box from Red Hat Subscription Management.

Alternatively, you can use the subscription management on the Red Hat Customer Portal to find and delete the virtual system that is no longer being used.

The sudo utility has been set up for the vagrant user to run any commands as root. You can use the sudo command to run a single command as root, or use sudo -i to start an interactive root shell.

Run the id command to check whether your regular user ID is a member of the vagrant group. If you skipped this step before, log out and log back in for the changes to take effect. Adding the user to the vagrant group works in conjuction with the PolicyKit rule that was added during the steps run under root to allow non-root users to work with libvirt.

One of the Software Installation and Configuration steps installs a PolicyKit rule in the /etc/polkit-1/rules.d/ directory that allows non-root users to run Vagrant and perform libvirt operations that are normally restricted to root users as long as the user is a member of the vagrant group. The error message is:

Error while connecting to libvirt: Error making a connection to libvirt URI qemu:///system:
Call to virConnectOpen failed: authentication failed: no agent is available to authenticate

Error while connecting to libvirt: Error making a connection to libvirt URI qemu:///system:
Call to virConnectOpen failed: authentication failed: no agent is available to authenticate

To resolve this, make sure your user ID is a member of the vagrant group. You can do this by running the id command. Review the installation steps listed above to verify that the PolicyKit rule is installed in /etc/polkit-1/rules.d/. Restart the PolicyKit and libvirtd services for the changes to take effect:

systemctl restart libvirtd
systemctl restart polkit

# systemctl restart libvirtd
# systemctl restart polkit

Normally, Vagrant takes care of all interaction with libvirt, creating and destroying resources as necessary. Running the vagrant destroy command should free up any resources that were allocated to an environment. However, if the Vagrant state directory, .vagrant is deleted, or if a Vagrant operation is interrupted, before it can clean up, it is possible that libvirt resources are left around that cannot be removed using vagrant destroy. If that happens, you need to use the virsh command-line utility or virt-manager, a graphical tool, to clean up.

Note that these tools must be run as root (or with the sudo command) to see all of the libvirt resources on the system. If you do not run them as root, it appears that there are no allocated resources.

For the CDK with libvirt, old Vagrant boxes are not automatically removed when the new one is installed. As a result, even after adding a new box, the old one is used in place of the new box. To add a new Vagrant box for an updated CDK, you need to destroy the running boxes, delete the .vagrant/ directory in each directory where you ran vagrant up and remove the image created by each box.

If you ran both the OSE and Kubernetes Vagrantfiles, here’s how you could delete them completely before adding the updated box. First go to each directory to stop the running box and remove the .vagrant/ directory.

cd ~/cdk/components/rhel/misc/rhel-k8s-singlenode-setup
vagrant destroy && rm -fr .vagrant/
cd ~/cdk/components/rhel/rhel-ose
vagrant destroy && rm -fr .vagrant/

$ cd ~/cdk/components/rhel/misc/rhel-k8s-singlenode-setup
$ vagrant destroy && rm -fr .vagrant/
$ cd ~/cdk/components/rhel/rhel-ose
$ vagrant destroy && rm -fr .vagrant/

Next remove the images representing the Vagrant boxes in the libvirt images directory and restart libvirt.

sudo rm -fr /var/lib/libvirt/images/*_0.img && sudo systemctl restart libvirtd

$ sudo rm -fr /var/lib/libvirt/images/*_0.img && sudo systemctl restart libvirtd

You can now proceed to adding the new Vagrant box.

In order to use vagrant ssh on a Windows system, you need to have ssh.exe in your path. It is recommended to install Cygwin and ensure that ssh.exe is installed and in your path.

If rsync is being used for Vagrant synchronized folders on a Windows system, either because it was explicitly selected, or it was the last available choice, Vagrant will not start a Vagrant box unless rsync.exe is available in the path. It is recommended to install Cygwin and ensure that rsync.exe is in your path.

If your Vagrant box is not configured with VirtualBox Guest Additions, then Vagrant attempts to use rsync for the shared folders. Using rsync is the supported, recommended way (VirtualBox Guest Additions may work, but is not officially supported). If rsync cannot be found, Vagrant generates an error and does not start.

Note that if a new kernel gets installed, you may need to rebuild VirtualBox Guest Additions on the Vagrant box. You can do this by running:

/etc/init.d/vboxdrv reload

/etc/init.d/vboxdrv reload

There are a number of provisioning steps performed when Vagrant launches a virtual machine. These may come from plugins, such as the Vagrant Registration plugin for Red Hat, or from steps that are included in the Vagrantfile for provisioning the box on startup. Vagrant runs these commands by using SSH.

In many cases, errors during the provisioning step are not fatal, so the Vagrant box will still be running. Use the vagrant status command to see what the state of your box is. You can stop the Vagrant box with the vagrant halt command. If the machine is running, you should be able to log into it using the vagrant ssh command to examine the VM and see what went wrong.

While the box is running, you can rerun the provision steps by running the vagrant provision command.

Note that you need to be in the same directory where your Vagrant file is. If you have lost track of where your Vagrantfile is, use the vagrant global-status command to list all the boxes that you have started and the directory where the Vagrantfile and state is stored.