12.6. Live migrating a virtual machine with an attached Mellanox virtual function

Procedure

1. On the source host, set the Mellanox networking device to the switchdev mode.

   # devlink dev eswitch set pci/<device_pci_address> mode switchdev

2. On the source host, create a virtual function on the Mellanox device.

   # echo 1 > /sys/bus/pci/devices/0000\:e1\:00.0/sriov_numvfs

   The /0000\:e1\:00.0/ part of the file path is based on the PCI address of the device. In the example it is: 0000:e1:00.0

3. On the source host, unbind the VF from its driver.

   # virsh nodedev-detach <vf_pci_address> --driver pci-stub

   You can view the PCI address of the VF by using the following command:

   # lshw -c network -businfo
   
   Bus info            Device         Class          Description
   =============================================================================
   pci@0000:e1:00.0    enp225s0np0    network        MT2910 Family [ConnectX-7]
   pci@0000:e1:00.1    enp225s0v0     network        ConnectX Family mlx5Gen Virtual Function

4. On the source host, enable the migration function of the VF.

   # devlink port function set pci/0000:e1:00.0/1 migratable enable

   In this example, pci/0000:e1:00.0/1 refers to the first VF on the Mellanox device with the given PCI address.

5. On the source host, configure Open vSwitch (OVS) for the migration of the VF. If the Mellanox device is in switchdev mode, it cannot transfer data over the network.

   1. Ensure the openvswitch service is running.

      # systemctl start openvswitch

   2. Enable hardware offloading to improve networking performance.

      # ovs-vsctl set Open_vSwitch . other_config:hw-offload=true

   3. Increase the maximum idle time to ensure network connections remain open during the migration.

      # ovs-vsctl set Open_vSwitch . other_config:max-idle=300000

   4. Create a new bridge in the OVS instance.

      # ovs-vsctl add-br <bridge_name>

   5. Restart the openvswitch service.

      # systemctl restart openvswitch

   6. Add the physical Mellanox device to the OVS bridge.

      # ovs-vsctl add-port <bridge_name> enp225s0np0

      In this example, enp225s0np0 is the network interface name of the Mellanox device.

   7. Add the VF of the Mellanox device to the OVS bridge.

      # ovs-vsctl add-port <bridge_name> enp225s0npf0vf0

      In this example, enp225s0npf0vf0 is the network interface name of the VF.

6. Repeat steps 1-5 on the destination host.

7. On the source host, open a new file, such as mlx_vf.xml, and add the following XML configuration of the VF:

    <interface type='hostdev' managed='yes'>
         <mac address='52:54:00:56:8c:f7'/>
         <source>
           <address type='pci' domain='0x0000' bus='0xe1' slot='0x00' function='0x1'/>
         </source>
    </interface>

   This example configures a pass-through of the VF as a network interface for the VM. Ensure the MAC address is unique, and use the PCI address of the VF on the source host.

8. On the source host, attach the VF XML file to the VM.

   # virsh attach-device <vm_name> mlx_vf.xml --live --config

   In this example, mlx_vf.xml is the name of the XML file with the VF configuration. Use the --live option to attach the device to a running VM.

9. On the source host, start the live migration of the running VM with the attached VF.

   # virsh migrate --live --domain <vm_name> --desturi qemu+ssh://<destination_host_ip_address>/system

   For more details about performing a live migration, see Migrating a virtual machine by using the command line.

Verification

1. In the migrated VM, view the network interface name of the Mellanox VF.

   # ifconfig
   
   eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
           inet 192.168.1.10  netmask 255.255.255.0  broadcast 192.168.1.255
           inet6 fe80::a00:27ff:fe4e:66a1  prefixlen 64  scopeid 0x20<link>
           ether 08:00:27:4e:66:a1  txqueuelen 1000  (Ethernet)
           RX packets 100000  bytes 6543210 (6.5 MB)
           RX errors 0  dropped 0  overruns 0  frame 0
           TX packets 100000  bytes 6543210 (6.5 MB)
           TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
   
   enp4s0f0v0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
           inet 192.168.3.10  netmask 255.255.255.0  broadcast 192.168.3.255
           inet6 fe80::a00:27ff:fe4e:66c3  prefixlen 64  scopeid 0x20<link>
           ether 08:00:27:4e:66:c3  txqueuelen 1000  (Ethernet)
           RX packets 200000  bytes 12345678 (12.3 MB)
           RX errors 0  dropped 0  overruns 0  frame 0
           TX packets 200000  bytes 12345678 (12.3 MB)
           TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

2. In the migrated VM, check that the Mellanox VF works, for example:

   # ping -I <VF_interface_name> 8.8.8.8
   
   PING 8.8.8.8 (8.8.8.8) from 192.168.3.10 <VF_interface_name>: 56(84) bytes of data.
   64 bytes from 8.8.8.8: icmp_seq=1 ttl=57 time=27.4 ms
   64 bytes from 8.8.8.8: icmp_seq=2 ttl=57 time=26.9 ms
   
   --- 8.8.8.8 ping statistics ---
   2 packets transmitted, 2 received, 0% packet loss, time 1002ms
   rtt min/avg/max/mdev = 26.944/27.046/27.148/0.102 ms