Chapter 9. Guest virtual machine device configuration

Procedure 9.3. Assigning a PCI device to a guest virtual machine with virsh

1. Identify the device

   First, identify the PCI device designated for device assignment to the virtual machine. Use the lspci command to list the available PCI devices. You can refine the output of lspci with grep.
   This example uses the Ethernet controller highlighted in the following output:

   # lspci | grep Ethernet
   00:19.0 Ethernet controller: Intel Corporation 82567LM-2 Gigabit Network Connection
   01:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
   01:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)

   This Ethernet controller is shown with the short identifier 00:19.0. We need to find out the full identifier used by virsh in order to assign this PCI device to a virtual machine.
   To do so, use the virsh nodedev-list command to list all devices of a particular type (pci) that are attached to the host machine. Then look at the output for the string that maps to the short identifier of the device you wish to use.
   This example highlights the string that maps to the Ethernet controller with the short identifier 00:19.0. In this example, the : and . characters are replaced with underscores in the full identifier.

   # virsh nodedev-list --cap pci
   pci_0000_00_00_0
   pci_0000_00_01_0
   pci_0000_00_03_0
   pci_0000_00_07_0
   pci_0000_00_10_0
   pci_0000_00_10_1
   pci_0000_00_14_0
   pci_0000_00_14_1
   pci_0000_00_14_2
   pci_0000_00_14_3
   pci_0000_00_19_0
   pci_0000_00_1a_0
   pci_0000_00_1a_1
   pci_0000_00_1a_2
   pci_0000_00_1a_7
   pci_0000_00_1b_0
   pci_0000_00_1c_0
   pci_0000_00_1c_1
   pci_0000_00_1c_4
   pci_0000_00_1d_0
   pci_0000_00_1d_1
   pci_0000_00_1d_2
   pci_0000_00_1d_7
   pci_0000_00_1e_0
   pci_0000_00_1f_0
   pci_0000_00_1f_2
   pci_0000_00_1f_3
   pci_0000_01_00_0
   pci_0000_01_00_1
   pci_0000_02_00_0
   pci_0000_02_00_1
   pci_0000_06_00_0
   pci_0000_07_02_0
   pci_0000_07_03_0

   Record the PCI device number that maps to the device you want to use; this is required in other steps.

2. Review device information

   Information on the domain, bus, and function are available from output of the virsh nodedev-dumpxml command:

   virsh nodedev-dumpxml pci_0000_00_19_0
   <device>
     <name>pci_0000_00_19_0</name>
     <parent>computer</parent>
     <driver>
       <name>e1000e</name>
     </driver>
     <capability type='pci'>
       <domain>0</domain>
       <bus>0</bus>
       <slot>25</slot>
       <function>0</function>
       <product id='0x1502'>82579LM Gigabit Network Connection</product>
       <vendor id='0x8086'>Intel Corporation</vendor>
       <iommuGroup number='7'>
         <address domain='0x0000' bus='0x00' slot='0x19' function='0x0'/>
       </iommuGroup>
     </capability>
   </device>

   Note

   An IOMMU group is determined based on the visibility and isolation of devices from the perspective of the IOMMU. Each IOMMU group may contain one or more devices. When multiple devices are present, all endpoints within the IOMMU group must be claimed for any device within the group to be assigned to a guest. This can be accomplished either by also assigning the extra endpoints to the guest or by detaching them from the host driver using virsh nodedev-detach. Devices contained within a single group may not be split between multiple guests or split between host and guest. Non-endpoint devices such as PCIe root ports, switch ports, and bridges should not be detached from the host drivers and will not interfere with assignment of endpoints.

   Devices within an IOMMU group can be determined using the iommuGroup section of the virsh nodedev-dumpxml output. Each member of the group is provided via a separate "address" field. This information may also be found in sysfs using the following:

   $ ls /sys/bus/pci/devices/0000:01:00.0/iommu_group/devices/

   An example of the output from this would be:

   0000:01:00.0  0000:01:00.1

   To assign only 0000.01.00.0 to the guest, the unused endpoint should be detached from the host before starting the guest:

   $ virsh nodedev-detach pci_0000_01_00_1

3. Determine required configuration details

   Refer to the output from the virsh nodedev-dumpxml pci_0000_00_19_0 command for the values required for the configuration file.
   The example device has the following values: bus = 0, slot = 25 and function = 0. The decimal configuration uses those three values:

   bus='0'
   slot='25'
   function='0'

4. Add configuration details

   Run virsh edit, specifying the virtual machine name, and add a device entry in the <source> section to assign the PCI device to the guest virtual machine.

   # virsh edit guest1-rhel6-64
   <hostdev mode='subsystem' type='pci' managed='yes'>
     <source>
        <address domain='0' bus='0' slot='25' function='0'/>
     </source>
   </hostdev>

   Alternately, run virsh attach-device, specifying the virtual machine name and the guest's XML file:

   virsh attach-device guest1-rhel6-64 file.xml

5. Start the virtual machine

   # virsh start guest1-rhel6-64

Procedure 9.4. Assigning a PCI device to a guest virtual machine using virt-manager

1. Open the hardware settings

   Open the guest virtual machine and click the Add Hardware button to add a new device to the virtual machine.
   

   Figure 9.1. The virtual machine hardware information window

2. Select a PCI device

   Select PCI Host Device from the Hardware list on the left.
   Select an unused PCI device. If you select a PCI device that is in use by another guest an error may result. In this example, a spare 82576 network device is used. Click Finish to complete setup.
   

   Figure 9.2. The Add new virtual hardware wizard

3. Add the new device

   The setup is complete and the guest virtual machine now has direct access to the PCI device.
   

   Figure 9.3. The virtual machine hardware information window

Procedure 9.5. Assigning a PCI device to a virtual machine with virt-install

1. Identify the device

   Identify the PCI device designated for device assignment to the guest virtual machine.

   # lspci | grep Ethernet
   00:19.0 Ethernet controller: Intel Corporation 82567LM-2 Gigabit Network Connection
   01:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
   01:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)

   The virsh nodedev-list command lists all devices attached to the system, and identifies each PCI device with a string. To limit output to only PCI devices, run the following command:

   # virsh nodedev-list --cap pci
   pci_0000_00_00_0
   pci_0000_00_01_0
   pci_0000_00_03_0
   pci_0000_00_07_0
   pci_0000_00_10_0
   pci_0000_00_10_1
   pci_0000_00_14_0
   pci_0000_00_14_1
   pci_0000_00_14_2
   pci_0000_00_14_3
   pci_0000_00_19_0
   pci_0000_00_1a_0
   pci_0000_00_1a_1
   pci_0000_00_1a_2
   pci_0000_00_1a_7
   pci_0000_00_1b_0
   pci_0000_00_1c_0
   pci_0000_00_1c_1
   pci_0000_00_1c_4
   pci_0000_00_1d_0
   pci_0000_00_1d_1
   pci_0000_00_1d_2
   pci_0000_00_1d_7
   pci_0000_00_1e_0
   pci_0000_00_1f_0
   pci_0000_00_1f_2
   pci_0000_00_1f_3
   pci_0000_01_00_0
   pci_0000_01_00_1
   pci_0000_02_00_0
   pci_0000_02_00_1
   pci_0000_06_00_0
   pci_0000_07_02_0
   pci_0000_07_03_0

   Record the PCI device number; the number is needed in other steps.
   Information on the domain, bus and function are available from output of the virsh nodedev-dumpxml command:

   # virsh nodedev-dumpxml pci_0000_01_00_0
   <device>
     <name>pci_0000_01_00_0</name>
     <parent>pci_0000_00_01_0</parent>
     <driver>
       <name>igb</name>
     </driver>
     <capability type='pci'>
       <domain>0</domain>
       <bus>1</bus>
       <slot>0</slot>
       <function>0</function>
       <product id='0x10c9'>82576 Gigabit Network Connection</product>
       <vendor id='0x8086'>Intel Corporation</vendor>
       <iommuGroup number='7'>
         <address domain='0x0000' bus='0x00' slot='0x19' function='0x0'/>
       </iommuGroup>
     </capability>
   </device>

   Note

   If there are multiple endpoints in the IOMMU group and not all of them are assigned to the guest, you will need to manually detach the other endpoint(s) from the host by running the following command before you start the guest:

   $ virsh nodedev-detach pci_0000_00_19_1

   Refer to the Note in Section 9.1.1, “Assigning a PCI Device with virsh” for more information on IOMMU groups.

2. Add the device

   Use the PCI identifier output from the virsh nodedev command as the value for the --host-device parameter.

   virt-install \
   --name=guest1-rhel6-64 \
   --disk path=/var/lib/libvirt/images/guest1-rhel6-64.img,size=8 \
   --nonsparse --graphics spice \
   --vcpus=2 --ram=2048 \
   --location=http://example1.com/installation_tree/RHEL6.0-Server-x86_64/os \
   --nonetworks \
   --os-type=linux \
   --os-variant=rhel6
   --host-device=pci_0000_01_00_0

3. Complete the installation

   Complete the guest installation. The PCI device should be attached to the guest.

Procedure 9.6. Detaching a PCI device from a guest with virsh

1. Detach the device

   Use the following command to detach the PCI device from the guest by removing it in the guest's XML file:

   # virsh detach-device name_of_guest file.xml

2. Re-attach the device to the host (optional)

   If the device is in managed mode, skip this step. The device will be returned to the host automatically.
   If the device is not using managed mode, use the following command to re-attach the PCI device to the host machine:

   # virsh nodedev-reattach device

   For example, to re-attach the pci_0000_01_00_0 device to the host:

   virsh nodedev-reattach pci_0000_01_00_0

   The device is now available for host use.

Procedure 9.7. Detaching a PCI Device from a guest with virt-manager

1. Open the virtual hardware details screen

   In virt-manager, double-click on the virtual machine that contains the device. Select the Show virtual hardware details button to display a list of virtual hardware.
   

   Figure 9.4. The virtual hardware details button

2. Select and remove the device

   Select the PCI device to be detached from the list of virtual devices in the left panel.
   

   Figure 9.5. Selecting the PCI device to be detached

   Click the Remove button to confirm. The device is now available for host use.

Procedure 9.8. Configuring MAC addresses, vLAN, and virtual ports for assigning PCI devices on SR-IOV

1. Shutdown the guest virtual machine

   Using virsh shutdown command (refer to Section 14.9.1, “Shutting Down a Guest Virtual Machine”), shutdown the guest virtual machine named guestVM.

   # virsh shutdown guestVM

2. Gather information

   In order to use <interface type='hostdev'>, you must have an SR-IOV-capable network card, host physical machine hardware that supports either the Intel VT-d or AMD IOMMU extensions, and you must know the PCI address of the VF that you wish to assign.

3. Open the XML file for editing

   Run the # virsh save-image-edit command to open the XML file for editing (refer to Section 14.8.10, “Edit Domain XML Configuration Files” for more information). As you would want to restore the guest virtual machine to its former running state, the --running would be used in this case. The name of the configuration file in this example is guestVM.xml, as the name of the guest virtual machine is guestVM.

    # virsh save-image-edit guestVM.xml --running 

   The guestVM.xml opens in your default editor.

4. Edit the XML file

   Update the configuration file (guestVM.xml) to have a <devices> entry similar to the following:

   
    <devices>
      ...
      <interface type='hostdev' managed='yes'>
        <source>
          <address type='pci' domain='0x0' bus='0x00' slot='0x07' function='0x0'/> <!--these values can be decimal as well-->
        </source>
        <mac address='52:54:00:6d:90:02'/>                                         <!--sets the mac address-->
        <virtualport type='802.1Qbh'>                                              <!--sets the virtual port for the 802.1Qbh switch-->
          <parameters profileid='finance'/>
        </virtualport>
        <vlan>                                                                     <!--sets the vlan tag-->
         <tag id='42'/>
        </vlan>
      </interface>
      ...
    </devices>
   
   

   Figure 9.7. Sample domain XML for hostdev interface type

   Note that if you do not provide a MAC address, one will be automatically generated, just as with any other type of interface device. Also, the <virtualport> element is only used if you are connecting to an 802.11Qgh hardware switch (802.11Qbg (a.k.a. "VEPA") switches are currently not supported.

5. Re-start the guest virtual machine

   Run the virsh start command to restart the guest virtual machine you shutdown in the first step (example uses guestVM as the guest virtual machine's domain name). Refer to Section 14.8.1, “Starting a Defined Domain” for more information.

    # virsh start guestVM 

   When the guest virtual machine starts, it sees the network device provided to it by the physical host machine's adapter, with the configured MAC address. This MAC address will remain unchanged across guest virtual machine and host physical machine reboots.

Procedure 9.9. Creating a device pool

1. Shutdown the guest virtual machine

   Using virsh shutdown command (refer to Section 14.9, “Shutting Down, Rebooting, and Forcing Shutdown of a Guest Virtual Machine”), shutdown the guest virtual machine named guestVM.

   # virsh shutdown guestVM

2. Create a configuration file

   Using your editor of choice create an XML file (named passthrough.xml, for example) in the /tmp directory. Make sure to replace pf dev='eth3' with the netdev name of your own SR-IOV device's PF
   The following is an example network definition that will make available a pool of all VFs for the SR-IOV adapter with its physical function (PF) at "eth3' on the host physical machine:

         
   <network>
      <name>passthrough</name>                                                <!--This is the name of the file you created-->
      <forward mode='hostdev' managed='yes'>
        <pf dev='myNetDevName'/>                                              <!--Use the netdev name of your SR-IOV devices PF here-->
      </forward>
   </network>
         
   
   

   Figure 9.8. Sample network definition domain XML

3. Load the new XML file

   Run the following command, replacing /tmp/passthrough.xml, with the name and location of your XML file you created in the previous step:

   # virsh net-define /tmp/passthrough.xml

4. Restarting the guest

   Run the following replacing passthrough.xml, with the name of your XML file you created in the previous step:

    # virsh net-autostart passthrough # virsh net-start passthrough 

5. Re-start the guest virtual machine

   Run the virsh start command to restart the guest virtual machine you shutdown in the first step (example uses guestVM as the guest virtual machine's domain name). Refer to Section 14.8.1, “Starting a Defined Domain” for more information.

    # virsh start guestVM 

6. Initiating passthrough for devices

   Although only a single device is shown, libvirt will automatically derive the list of all VFs associated with that PF the first time a guest virtual machine is started with an interface definition in its domain XML like the following:

            
   <interface type='network'>
      <source network='passthrough'>
   </interface>
         
   
   

   Figure 9.9. Sample domain XML for interface network definition

7. Verification

   You can verify this by running virsh net-dumpxml passthrough command after starting the first guest that uses the network; you will get output similar to the following:

         
   <network connections='1'>
      <name>passthrough</name>
      <uuid>a6b49429-d353-d7ad-3185-4451cc786437</uuid>
      <forward mode='hostdev' managed='yes'>
        <pf dev='eth3'/>
        <address type='pci' domain='0x0000' bus='0x02' slot='0x10' function='0x1'/>
        <address type='pci' domain='0x0000' bus='0x02' slot='0x10' function='0x3'/>
        <address type='pci' domain='0x0000' bus='0x02' slot='0x10' function='0x5'/>
        <address type='pci' domain='0x0000' bus='0x02' slot='0x10' function='0x7'/>
        <address type='pci' domain='0x0000' bus='0x02' slot='0x11' function='0x1'/>
        <address type='pci' domain='0x0000' bus='0x02' slot='0x11' function='0x3'/>
        <address type='pci' domain='0x0000' bus='0x02' slot='0x11' function='0x5'/>
      </forward>
   </network>
         
   
   

   Figure 9.10. XML dump file passthrough contents