16.2. Using SR-IOV

Procedure 16.1. Attach an SR-IOV network device

1. Enable Intel VT-d in BIOS and in the kernel

   Enable Intel VT-D in BIOS. See Procedure 15.1, “Preparing an Intel system for PCI passthrough” for more information on enabling Intel VT-d in BIOS and the kernel, or see your system manufacturer's documentation for specific instructions.

2. Verify support

   Verify if the PCI device with SR-IOV capabilities are detected. This example lists an Intel 82576 network interface card which supports SR-IOV. Use the lspci command to verify if the device was detected.

   # lspci
   03:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
   03:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
   

   Copy to Clipboard Toggle word wrap

   Note

   Note that the output has been modified to remove all other devices.

3. Start the SR-IOV kernel modules

   If the device is supported the driver kernel module should be loaded automatically by the kernel. Optional parameters can be passed to the module using the modprobe command. The Intel 82576 network interface card uses the igb driver kernel module.

   # modprobe igb [<option>=<VAL1>,<VAL2>,]
   # lsmod |grep igb
   igb    87592  0
   dca    6708    1 igb
   

   Copy to Clipboard Toggle word wrap

4. Activate Virtual Functions

   The max_vfs parameter of the igb module allocates the maximum number of Virtual Functions. The max_vfs parameter causes the driver to spawn, up to the value of the parameter in, Virtual Functions. For this particular card the valid range is 0 to 7.
   Remove the module to change the variable.

   # modprobe -r igb

   Copy to Clipboard Toggle word wrap
   Restart the module with the max_vfs set to 1 or any number of Virtual Functions up to the maximum supported by your device.

   # modprobe igb max_vfs=1
   

   Copy to Clipboard Toggle word wrap

5. Inspect the new Virtual Functions

   Using the lspci command, list the newly added Virtual Functions attached to the Intel 82576 network device.

   # lspci | grep 82576
   03:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
   03:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
   03:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
   03:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
   

   Copy to Clipboard Toggle word wrap
   The identifier for the PCI device is found with the -n parameter of the lspci command.

   # lspci -n | grep 03:00.0
   03:00.0 0200: 8086:10c9 (rev 01)
   # lspci -n | grep 03:10.0
   03:10.0 0200: 8086:10ca (rev 01)
   

   Copy to Clipboard Toggle word wrap

   The Physical Function corresponds to 8086:10c9 and the Virtual Function to 8086:10ca.

6. Find the devices with virsh

   The libvirt service must find the device to add a device to a guest. Use the virsh nodedev-list command to list available host devices.

   # virsh nodedev-list | grep 8086
   pci_8086_10c9
   pci_8086_10c9_0
   pci_8086_10ca
   pci_8086_10ca_0
   [output truncated]

   Copy to Clipboard Toggle word wrap
   The serial numbers for the Virtual Functions and Physical Functions should be in the list.

7. Get advanced details

   The pci_8086_10c9 is one of the Physical Functions and pci_8086_10ca_0 is the first corresponding Virtual Function for that Physical Function. Use the virsh nodedev-dumpxml command to get advanced output for both devices.

   # virsh nodedev-dumpxml pci_8086_10ca
   # virsh nodedev-dumpxml pci_8086_10ca_0
   <device>
     <name>pci_8086_10ca_0</name>
     <parent>pci_8086_3408</parent>
     <driver>
       <name>igbvf</name>
     </driver>
     <capability type='pci'>
       <domain>0</domain>
       <bus>3</bus>
       <slot>16</slot>
       <function>1</function>
       <product id='0x10ca'>82576 Virtual Function</product>
       <vendor id='0x8086'>Intel Corporation</vendor>
     </capability>
   </device>
   

   Copy to Clipboard Toggle word wrap
   This example adds the Virtual Function pci_8086_10ca_0 to the guest in Step 8. Note the bus, slot and function parameters of the Virtual Function, these are required for adding the device.

8. Add the Virtual Function to the guest

   1. Shut down the guest.
   2. Use the output from the virsh nodedev-dumpxml pci_8086_10ca_0 command to calculate the values for the configuration file. Convert slot and function values to hexadecimal values (from decimal) to get the PCI bus addresses. Append "0x" to the beginning of the output to tell the computer that the value is a hexadecimal number.
      The example device has the following values: bus = 3, slot = 16 and function = 1. Use the printf utility to convert decimal values to hexadecimal values.

      $ printf %x 3
      3
      $ printf %x 16
      10
      $ printf %x 1
      1

      Copy to Clipboard Toggle word wrap
      This example would use the following values in the configuration file:

      bus='0x03'
      slot='0x10'
      function='0x01'

      Copy to Clipboard Toggle word wrap
   3. Open the XML configuration file with the virsh edit command. This example edits a guest named MyGuest.

      # virsh edit MyGuest

      Copy to Clipboard Toggle word wrap
   4. The default text editor will open the libvirt configuration file for the guest. Add the new device to the devices section of the XML configuration file.

      <hostdev mode='subsystem' type='pci' managed='yes'>
            <source>
              <address bus='0x03' slot='0x10' function='0x01'/>
            </source>
      </hostdev>
      

      Copy to Clipboard Toggle word wrap
   5. Save the configuration.

9. Restart

   Restart the guest to complete the installation.

   # virsh start MyGuest

   Copy to Clipboard Toggle word wrap