Chapter 8. Networking

1. Configure an OVN-Kubernetes secondary network by creating a network attachment definition (NAD).

   Note

   For localnet topology, you must configure an OVS bridge by creating a NodeNetworkConfigurationPolicy object before creating the NAD.

2. Connect the VM to the OVN-Kubernetes secondary network by adding the network details to the VM specification.

Procedure

1. Edit the interfaces spec of your virtual machine configuration file:

   apiVersion: kubevirt.io/v1
   kind: VirtualMachine
   metadata:
     name: example-vm
   spec:
     template:
       spec:
         domain:
           devices:
             interfaces:
               - name: default
                 masquerade: {} 1
                 ports: 2
                   - port: 80
   # ...
         networks:
         - name: default
           pod: {}

   1

   Connect using masquerade mode.

   2

   Optional: List the ports that you want to expose from the virtual machine, each specified by the port field. The port value must be a number between 0 and 65536. When the ports array is not used, all ports in the valid range are open to incoming traffic. In this example, incoming traffic is allowed on port 80.

   Note

   Ports 49152 and 49153 are reserved for use by the libvirt platform and all other incoming traffic to these ports is dropped.

2. Create the virtual machine:

   $ oc create -f <vm-name>.yaml

Procedure

1. In a new virtual machine configuration, include an interface with masquerade and configure the IPv6 address and default gateway by using cloud-init.

   apiVersion: kubevirt.io/v1
   kind: VirtualMachine
   metadata:
     name: example-vm-ipv6
   spec:
     template:
       spec:
         domain:
           devices:
             interfaces:
               - name: default
                 masquerade: {} 1
                 ports:
                   - port: 80 2
   # ...
         networks:
         - name: default
           pod: {}
         volumes:
         - cloudInitNoCloud:
             networkData: |
               version: 2
               ethernets:
                 eth0:
                   dhcp4: true
                   addresses: [ fd10:0:2::2/120 ] 3
                   gateway6: fd10:0:2::1 4

   1

   Connect using masquerade mode.

   2

   Allows incoming traffic on port 80 to the virtual machine.

   3

   The static IPv6 address as determined by the Network.pod.vmIPv6NetworkCIDR field in the virtual machine instance configuration. The default value is fd10:0:2::2/120.

   4

   The gateway IP address as determined by the Network.pod.vmIPv6NetworkCIDR field in the virtual machine instance configuration. The default value is fd10:0:2::1.

2. Create the virtual machine in the namespace:

   $ oc create -f example-vm-ipv6.yaml

Procedure

1. Edit the VirtualMachine manifest to add the label for service creation:

   apiVersion: kubevirt.io/v1
   kind: VirtualMachine
   metadata:
     name: example-vm
     namespace: example-namespace
   spec:
     running: false
     template:
       metadata:
         labels:
           special: key 1
   # ...

   1

   Add special: key to the spec.template.metadata.labels stanza.

   Note

   Labels on a virtual machine are passed through to the pod. The special: key label must match the label in the spec.selector attribute of the Service manifest.

2. Save the VirtualMachine manifest file to apply your changes.

3. Create a Service manifest to expose the VM:

   apiVersion: v1
   kind: Service
   metadata:
     name: example-service
     namespace: example-namespace
   spec:
   # ...
     selector:
       special: key 1
     type: NodePort 2
     ports: 3
       protocol: TCP
       port: 80
       targetPort: 9376
       nodePort: 30000

   1

   Specify the label that you added to the spec.template.metadata.labels stanza of the VirtualMachine manifest.

   2

   Specify ClusterIP, NodePort, or LoadBalancer.

   3

   Specifies a collection of network ports and protocols that you want to expose from the virtual machine.

4. Save the Service manifest file.

5. Create the service by running the following command:

   $ oc create -f example-service.yaml

6. Restart the VM to apply the changes.

Procedure

1. Create a Service manifest to expose the VM, such as the following example:

   apiVersion: v1
   kind: Service
   metadata:
     name: mysubdomain 1
   spec:
     selector:
       expose: me 2
     clusterIP: None 3
     ports: 4
     - protocol: TCP
       port: 1234
       targetPort: 1234

   1

   The name of the service. This must match the spec.subdomain attribute in the VirtualMachine manifest file.

   2

   This service selector must match the expose:me label in the VirtualMachine manifest file.

   3

   Specifies a headless service.

   4

   The list of ports that are exposed by the service. You must define at least one port. This can be any arbitrary value as it does not affect the headless service.

2. Save the Service manifest file.

3. Create the service by running the following command:

   $ oc create -f headless_service.yaml

Procedure

1. Edit the VirtualMachine manifest to add the service selector label and subdomain by running the following command:

   $ oc edit vm <vm_name>

   Example VirtualMachine manifest file

   apiVersion: kubevirt.io/v1
   kind: VirtualMachine
   metadata:
     name: vm-fedora
   spec:
     template:
       metadata:
         labels:
           expose: me 1
       spec:
         hostname: "myvm" 2
         subdomain: "mysubdomain" 3
   # ...

   1

   The expose:me label must match the spec.selector attribute of the Service manifest that you previously created.

   2

   If this attribute is not specified, the resulting DNS A record takes the form of <vm.metadata.name>.<vm.spec.subdomain>.<vm.metadata.namespace>.svc.cluster.local.

   3

   The spec.subdomain attribute must match the metadata.name value of the Service object.

2. Save your changes and exit the editor.
3. Restart the VM to apply the changes.

Procedure

1. Connect to the VM console by entering the following command:

   $ virtctl console vm-fedora

2. To connect to the VM by using the requested FQDN, run the following command:

   $ ping myvm.mysubdomain.<namespace>.svc.cluster.local

   Example output

   PING myvm.mysubdomain.default.svc.cluster.local (10.244.0.57) 56(84) bytes of data.
   64 bytes from myvm.mysubdomain.default.svc.cluster.local (10.244.0.57): icmp_seq=1 ttl=64 time=0.029 ms

   In the preceding example, the DNS entry for myvm.mysubdomain.default.svc.cluster.local points to 10.244.0.57, which is the cluster IP address that is currently assigned to the VM.

Procedure

1. Create an SriovNetworkNodePolicy object, and then save the YAML in the <name>-sriov-node-network.yaml file. Replace <name> with the name for this configuration.

   apiVersion: sriovnetwork.openshift.io/v1
   kind: SriovNetworkNodePolicy
   metadata:
     name: <name> 1
     namespace: openshift-sriov-network-operator 2
   spec:
     resourceName: <sriov_resource_name> 3
     nodeSelector:
       feature.node.kubernetes.io/network-sriov.capable: "true" 4
     priority: <priority> 5
     mtu: <mtu> 6
     numVfs: <num> 7
     nicSelector: 8
       vendor: "<vendor_code>" 9
       deviceID: "<device_id>" 10
       pfNames: ["<pf_name>", ...] 11
       rootDevices: ["<pci_bus_id>", "..."] 12
     deviceType: vfio-pci 13
     isRdma: false 14

   1

   Specify a name for the CR object.

   2

   Specify the namespace where the SR-IOV Operator is installed.

   3

   Specify the resource name of the SR-IOV device plugin. You can create multiple SriovNetworkNodePolicy objects for a resource name.

   4

   Specify the node selector to select which nodes are configured. Only SR-IOV network devices on selected nodes are configured. The SR-IOV Container Network Interface (CNI) plugin and device plugin are deployed only on selected nodes.

   5

   Optional: Specify an integer value between 0 and 99. A smaller number gets higher priority, so a priority of 10 is higher than a priority of 99. The default value is 99.

   6

   Optional: Specify a value for the maximum transmission unit (MTU) of the virtual function. The maximum MTU value can vary for different NIC models.

   7

   Specify the number of the virtual functions (VF) to create for the SR-IOV physical network device. For an Intel network interface controller (NIC), the number of VFs cannot be larger than the total VFs supported by the device. For a Mellanox NIC, the number of VFs cannot be larger than 127.

   8

   The nicSelector mapping selects the Ethernet device for the Operator to configure. You do not need to specify values for all the parameters. It is recommended to identify the Ethernet adapter with enough precision to minimize the possibility of selecting an Ethernet device unintentionally. If you specify rootDevices, you must also specify a value for vendor, deviceID, or pfNames. If you specify both pfNames and rootDevices at the same time, ensure that they point to an identical device.

   9

   Optional: Specify the vendor hex code of the SR-IOV network device. The only allowed values are either 8086 or 15b3.

   10

   Optional: Specify the device hex code of SR-IOV network device. The only allowed values are 158b, 1015, 1017.

   11

   Optional: The parameter accepts an array of one or more physical function (PF) names for the Ethernet device.

   12

   The parameter accepts an array of one or more PCI bus addresses for the physical function of the Ethernet device. Provide the address in the following format: 0000:02:00.1.

   13

   The vfio-pci driver type is required for virtual functions in OpenShift Virtualization.

   14

   Optional: Specify whether to enable remote direct memory access (RDMA) mode. For a Mellanox card, set isRdma to false. The default value is false.

   Note

   If isRDMA flag is set to true, you can continue to use the RDMA enabled VF as a normal network device. A device can be used in either mode.

2. Optional: Label the SR-IOV capable cluster nodes with SriovNetworkNodePolicy.Spec.NodeSelector if they are not already labeled. For more information about labeling nodes, see "Understanding how to update labels on nodes".

3. Create the SriovNetworkNodePolicy object:

   $ oc create -f <name>-sriov-node-network.yaml

   where <name> specifies the name for this configuration.

   After applying the configuration update, all the pods in sriov-network-operator namespace transition to the Running status.

4. To verify that the SR-IOV network device is configured, enter the following command. Replace <node_name> with the name of a node with the SR-IOV network device that you just configured.

   $ oc get sriovnetworknodestates -n openshift-sriov-network-operator <node_name> -o jsonpath='{.status.syncStatus}'

Procedure

1. Create the following SriovNetwork object, and then save the YAML in the <name>-sriov-network.yaml file. Replace <name> with a name for this additional network.

2. To create the object, enter the following command. Replace <name> with a name for this additional network.

   $ oc create -f <name>-sriov-network.yaml

3. Optional: To confirm that the NetworkAttachmentDefinition object associated with the SriovNetwork object that you created in the previous step exists, enter the following command. Replace <namespace> with the namespace you specified in the SriovNetwork object.

   $ oc get net-attach-def -n <namespace>

Procedure

1. Add the SR-IOV network details to the spec.domain.devices.interfaces and spec.networks stanzas of the VM configuration as in the following example:

   apiVersion: kubevirt.io/v1
   kind: VirtualMachine
   metadata:
     name: example-vm
   spec:
     domain:
       devices:
         interfaces:
         - name: nic1 1
           sriov: {}
     networks:
     - name: nic1 2
       multus:
           networkName: sriov-network 3
   # ...

   1

   Specify a unique name for the SR-IOV interface.

   2

   Specify the name of the SR-IOV interface. This must be the same as the interfaces.name that you defined earlier.

   3

   Specify the name of the SR-IOV network attachment definition.

2. Apply the virtual machine configuration:

   $ oc apply -f <vm_sriov>.yaml 1

   1

   The name of the virtual machine YAML file.

Procedure

1. Navigate to Virtualization VirtualMachines.
2. Click a VM to view the VirtualMachine details page.
3. On the Configuration tab, click the Network interfaces tab.
4. Click Add network interface.
5. Enter the interface name.
6. Select an SR-IOV network attachment definition from the Network list.
7. Select SR-IOV from the Type list.
8. Optional: Add a network Model or Mac address.
9. Click Save.
10. Restart or live-migrate the VM to apply the changes.

Procedure

1. Map your compute nodes topology to determine which Non-Uniform Memory Access (NUMA) CPUs are isolated for DPDK applications and which ones are reserved for the operating system (OS).

2. Label a subset of the compute nodes with a custom role; for example, worker-dpdk:

   $ oc label node <node_name> node-role.kubernetes.io/worker-dpdk=""

3. Create a new MachineConfigPool manifest that contains the worker-dpdk label in the spec.machineConfigSelector object:

   Example MachineConfigPool manifest

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfigPool
   metadata:
     name: worker-dpdk
     labels:
       machineconfiguration.openshift.io/role: worker-dpdk
   spec:
     machineConfigSelector:
       matchExpressions:
         - key: machineconfiguration.openshift.io/role
           operator: In
           values:
             - worker
             - worker-dpdk
     nodeSelector:
       matchLabels:
         node-role.kubernetes.io/worker-dpdk: ""

4. Create a PerformanceProfile manifest that applies to the labeled nodes and the machine config pool that you created in the previous steps. The performance profile specifies the CPUs that are isolated for DPDK applications and the CPUs that are reserved for house keeping.

   Example PerformanceProfile manifest

   apiVersion: performance.openshift.io/v2
   kind: PerformanceProfile
   metadata:
     name: profile-1
   spec:
     cpu:
       isolated: 4-39,44-79
       reserved: 0-3,40-43
     globallyDisableIrqLoadBalancing: true
     hugepages:
       defaultHugepagesSize: 1G
       pages:
       - count: 8
         node: 0
         size: 1G
     net:
       userLevelNetworking: true
     nodeSelector:
       node-role.kubernetes.io/worker-dpdk: ""
     numa:
       topologyPolicy: single-numa-node

   Note

   The compute nodes automatically restart after you apply the MachineConfigPool and PerformanceProfile manifests.

5. Retrieve the name of the generated RuntimeClass resource from the status.runtimeClass field of the PerformanceProfile object:

   $ oc get performanceprofiles.performance.openshift.io profile-1 -o=jsonpath='{.status.runtimeClass}{"\n"}'

6. Set the previously obtained RuntimeClass name as the default container runtime class for the virt-launcher pods by editing the HyperConverged custom resource (CR):

   $ oc patch hyperconverged kubevirt-hyperconverged -n openshift-cnv \
       --type='json' -p='[{"op": "add", "path": "/spec/defaultRuntimeClass", "value":"<runtimeclass-name>"}]'

   Note

   Editing the HyperConverged CR changes a global setting that affects all VMs that are created after the change is applied.

7. If your DPDK-enabled compute nodes use Simultaneous multithreading (SMT), enable the AlignCPUs enabler by editing the HyperConverged CR:

   $ oc patch hyperconverged kubevirt-hyperconverged -n openshift-cnv \
       --type='json' -p='[{"op": "replace", "path": "/spec/featureGates/alignCPUs", "value": true}]'

   Note

   Enabling AlignCPUs allows OpenShift Virtualization to request up to two additional dedicated CPUs to bring the total CPU count to an even parity when using emulator thread isolation.

8. Create an SriovNetworkNodePolicy object with the spec.deviceType field set to vfio-pci:

   Example SriovNetworkNodePolicy manifest

   apiVersion: sriovnetwork.openshift.io/v1
   kind: SriovNetworkNodePolicy
   metadata:
     name: policy-1
     namespace: openshift-sriov-network-operator
   spec:
     resourceName: intel_nics_dpdk
     deviceType: vfio-pci
     mtu: 9000
     numVfs: 4
     priority: 99
     nicSelector:
       vendor: "8086"
       deviceID: "1572"
       pfNames:
         - eno3
       rootDevices:
         - "0000:19:00.2"
     nodeSelector:
       feature.node.kubernetes.io/network-sriov.capable: "true"

Procedure

1. Create a namespace for your DPDK applications:

   $ oc create ns dpdk-checkup-ns

2. Create an SriovNetwork object that references the SriovNetworkNodePolicy object. When you create an SriovNetwork object, the SR-IOV Network Operator automatically creates a NetworkAttachmentDefinition object.

   Example SriovNetwork manifest

   apiVersion: sriovnetwork.openshift.io/v1
   kind: SriovNetwork
   metadata:
     name: dpdk-sriovnetwork
     namespace: openshift-sriov-network-operator
   spec:
     ipam: |
       {
         "type": "host-local",
         "subnet": "10.56.217.0/24",
         "rangeStart": "10.56.217.171",
         "rangeEnd": "10.56.217.181",
         "routes": [{
           "dst": "0.0.0.0/0"
         }],
         "gateway": "10.56.217.1"
       }
     networkNamespace: dpdk-checkup-ns 1
     resourceName: intel_nics_dpdk 2
     spoofChk: "off"
     trust: "on"
     vlan: 1019

   1

   The namespace where the NetworkAttachmentDefinition object is deployed.

   2

   The value of the spec.resourceName attribute of the SriovNetworkNodePolicy object that was created when configuring the cluster for DPDK workloads.

3. Optional: Run the virtual machine latency checkup to verify that the network is properly configured.
4. Optional: Run the DPDK checkup to verify that the namespace is ready for DPDK workloads.

Procedure

1. Edit the VirtualMachine manifest to include information about the SR-IOV network interface, CPU topology, CRI-O annotations, and huge pages:

   Example VirtualMachine manifest

   apiVersion: kubevirt.io/v1
   kind: VirtualMachine
   metadata:
     name: rhel-dpdk-vm
   spec:
     running: true
     template:
       metadata:
         annotations:
           cpu-load-balancing.crio.io: disable 1
           cpu-quota.crio.io: disable 2
           irq-load-balancing.crio.io: disable 3
       spec:
         domain:
           cpu:
             sockets: 1 4
             cores: 5 5
             threads: 2
             dedicatedCpuPlacement: true
             isolateEmulatorThread: true
           interfaces:
             - masquerade: {}
               name: default
             - model: virtio
               name: nic-east
               pciAddress: '0000:07:00.0'
               sriov: {}
             networkInterfaceMultiqueue: true
             rng: {}
         memory:
           hugepages:
             pageSize: 1Gi 6
             guest: 8Gi
         networks:
           - name: default
             pod: {}
           - multus:
               networkName: dpdk-net 7
             name: nic-east
   # ...

   1

   This annotation specifies that load balancing is disabled for CPUs that are used by the container.

   2

   This annotation specifies that the CPU quota is disabled for CPUs that are used by the container.

   3

   This annotation specifies that Interrupt Request (IRQ) load balancing is disabled for CPUs that are used by the container.

   4

   The number of sockets inside the VM. This field must be set to 1 for the CPUs to be scheduled from the same Non-Uniform Memory Access (NUMA) node.

   5

   The number of cores inside the VM. This must be a value greater than or equal to 1. In this example, the VM is scheduled with 5 hyper-threads or 10 CPUs.

   6

   The size of the huge pages. The possible values for x86-64 architecture are 1Gi and 2Mi. In this example, the request is for 8 huge pages of size 1Gi.

   7

   The name of the SR-IOV NetworkAttachmentDefinition object.

2. Save and exit the editor.

3. Apply the VirtualMachine manifest:

   $ oc apply -f <file_name>.yaml

4. Configure the guest operating system. The following example shows the configuration steps for RHEL 9 operating system:

   1. Configure huge pages by using the GRUB bootloader command-line interface. In the following example, 8 1G huge pages are specified.

      $ grubby --update-kernel=ALL --args="default_hugepagesz=1GB hugepagesz=1G hugepages=8"

   2. To achieve low-latency tuning by using the cpu-partitioning profile in the TuneD application, run the following commands:

      $ dnf install -y tuned-profiles-cpu-partitioning

      $ echo isolated_cores=2-9 > /etc/tuned/cpu-partitioning-variables.conf

      The first two CPUs (0 and 1) are set aside for house keeping tasks and the rest are isolated for the DPDK application.

      $ tuned-adm profile cpu-partitioning

   3. Override the SR-IOV NIC driver by using the driverctl device driver control utility:

      $ dnf install -y driverctl

      $ driverctl set-override 0000:07:00.0 vfio-pci

5. Restart the VM to apply the changes.

Procedure

1. If the VM to which you want to hot plug the network interface is not running, start it by using the following command:

   $ virtctl start <vm_name> -n <namespace>

2. Use the following command to add the new network interface to the running VM. Editing the VM specification adds the new network interface to the VM and virtual machine instance (VMI) configuration but does not attach it to the running VM.

   $ oc edit vm <vm_name>

   Example VM configuration

   apiVersion: kubevirt.io/v1
   kind: VirtualMachine
   metadata:
     name: vm-fedora
   template:
     spec:
       domain:
         devices:
           interfaces:
           - name: defaultnetwork
             masquerade: {}
           # new interface
           - name: <secondary_nic> 1
             bridge: {}
       networks:
       - name: defaultnetwork
         pod: {}
       # new network
       - name: <secondary_nic> 2
         multus:
           networkName: <nad_name> 3
   # ...

   1

   Specifies the name of the new network interface.

   2

   Specifies the name of the network. This must be the same as the name of the new network interface that you defined in the template.spec.domain.devices.interfaces list.

   3

   Specifies the name of the NetworkAttachmentDefinition object.

3. To attach the network interface to the running VM, live migrate the VM by running the following command:

   $ virtctl migrate <vm_name>

Verification

1. Verify that the VM live migration is successful by using the following command:

   $ oc get VirtualMachineInstanceMigration -w

   Example output

   NAME                        PHASE             VMI
   kubevirt-migrate-vm-lj62q   Scheduling        vm-fedora
   kubevirt-migrate-vm-lj62q   Scheduled         vm-fedora
   kubevirt-migrate-vm-lj62q   PreparingTarget   vm-fedora
   kubevirt-migrate-vm-lj62q   TargetReady       vm-fedora
   kubevirt-migrate-vm-lj62q   Running           vm-fedora
   kubevirt-migrate-vm-lj62q   Succeeded         vm-fedora

2. Verify that the new interface is added to the VM by checking the VMI status:

   $ oc get vmi vm-fedora -ojsonpath="{ @.status.interfaces }"

   Example output

   [
     {
       "infoSource": "domain, guest-agent",
       "interfaceName": "eth0",
       "ipAddress": "10.130.0.195",
       "ipAddresses": [
         "10.130.0.195",
         "fd02:0:0:3::43c"
       ],
       "mac": "52:54:00:0e:ab:25",
       "name": "default",
       "queueCount": 1
     },
     {
       "infoSource": "domain, guest-agent, multus-status",
       "interfaceName": "eth1",
       "mac": "02:d8:b8:00:00:2a",
       "name": "bridge-interface", 1
       "queueCount": 1
     }
   ]

   1

   The hot plugged interface appears in the VMI status.

Procedure

1. Edit the VM specification to hot unplug a secondary network interface. Setting the interface state to absent detaches the network interface from the guest, but the interface still exists in the pod.

   $ oc edit vm <vm_name>

   Example VM configuration

   apiVersion: kubevirt.io/v1
   kind: VirtualMachine
   metadata:
     name: vm-fedora
   template:
     spec:
       domain:
         devices:
           interfaces:
             - name: defaultnetwork
               masquerade: {}
             # set the interface state to absent
             - name: <secondary_nic>
               state: absent 1
               bridge: {}
       networks:
         - name: defaultnetwork
           pod: {}
         - name: <secondary_nic>
           multus:
             networkName: <nad_name>
   # ...

   1

   Set the interface state to absent to detach it from the running VM. Removing the interface details from the VM specification does not hot unplug the secondary network interface.

2. Remove the interface from the pod by migrating the VM:

   $ virtctl migrate <vm_name>

Procedure

1. Edit the VM configuration file to add the sidecar.istio.io/inject: "true" annotation:

   Example configuration file

   apiVersion: kubevirt.io/v1
   kind: VirtualMachine
   metadata:
     labels:
       kubevirt.io/vm: vm-istio
     name: vm-istio
   spec:
     runStrategy: Always
     template:
       metadata:
         labels:
           kubevirt.io/vm: vm-istio
           app: vm-istio 1
         annotations:
           sidecar.istio.io/inject: "true" 2
       spec:
         domain:
           devices:
             interfaces:
             - name: default
               masquerade: {} 3
             disks:
             - disk:
                 bus: virtio
               name: containerdisk
             - disk:
                 bus: virtio
               name: cloudinitdisk
           resources:
             requests:
               memory: 1024M
         networks:
         - name: default
           pod: {}
         terminationGracePeriodSeconds: 180
         volumes:
         - containerDisk:
             image: registry:5000/kubevirt/fedora-cloud-container-disk-demo:devel
           name: containerdisk

   1

   The key/value pair (label) that must be matched to the service selector attribute.

   2

   The annotation to enable automatic sidecar injection.

   3

   The binding method (masquerade mode) for use with the default pod network.

2. Apply the VM configuration:

   $ oc apply -f <vm_name>.yaml 1

   1

   The name of the virtual machine YAML file.

3. Create a Service object to expose your VM to the service mesh.

   apiVersion: v1
   kind: Service
   metadata:
     name: vm-istio
   spec:
     selector:
       app: vm-istio 1
     ports:
       - port: 8080
         name: http
         protocol: TCP

   1

   The service selector that determines the set of pods targeted by a service. This attribute corresponds to the spec.metadata.labels field in the VM configuration file. In the above example, the Service object named vm-istio targets TCP port 8080 on any pod with the label app=vm-istio.

4. Create the service:

   $ oc create -f <service_name>.yaml 1

   1

   The name of the service YAML file.

Procedure

1. Create a NetworkAttachmentDefinition manifest according to the following example:

   Example configuration file

   apiVersion: "k8s.cni.cncf.io/v1"
   kind: NetworkAttachmentDefinition
   metadata:
     name: my-secondary-network 1
     namespace: openshift-cnv 2
   spec:
     config: '{
       "cniVersion": "0.3.1",
       "name": "migration-bridge",
       "type": "macvlan",
       "master": "eth1", 3
       "mode": "bridge",
       "ipam": {
         "type": "whereabouts", 4
         "range": "10.200.5.0/24" 5
       }
     }'

   1

   Specify the name of the NetworkAttachmentDefinition object.

   2 3

   Specify the name of the NIC to be used for live migration.

   4

   Specify the name of the CNI plugin that provides the network for the NAD.

   5

   Specify an IP address range for the secondary network. This range must not overlap the IP addresses of the main network.

2. Open the HyperConverged CR in your default editor by running the following command:

   oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnv

3. Add the name of the NetworkAttachmentDefinition object to the spec.liveMigrationConfig stanza of the HyperConverged CR:

   Example HyperConverged manifest

   apiVersion: hco.kubevirt.io/v1beta1
   kind: HyperConverged
   metadata:
     name: kubevirt-hyperconverged
   spec:
     liveMigrationConfig:
       completionTimeoutPerGiB: 800
       network: <network> 1
       parallelMigrationsPerCluster: 5
       parallelOutboundMigrationsPerNode: 2
       progressTimeout: 150
   # ...

   1

   Specify the name of the Multus NetworkAttachmentDefinition object to be used for live migrations.

4. Save your changes and exit the editor. The virt-handler pods restart and connect to the secondary network.

Procedure

1. Navigate to Virtualization > Overview in the OpenShift Container Platform web console.
2. Click the Settings tab and then click Live migration.
3. Select the network from the Live migration network list.

Procedure

1. Create a load balancer service to expose the DNS server outside the cluster by running the oc expose command according to the following example:

   $ oc expose -n openshift-cnv deployment/secondary-dns --name=dns-lb \
     --type=LoadBalancer --port=53 --target-port=5353 --protocol='UDP'

2. Retrieve the external IP address by running the following command:

   $ oc get service -n openshift-cnv

   Example output

   NAME       TYPE             CLUSTER-IP     EXTERNAL-IP      PORT(S)          AGE
   dns-lb     LoadBalancer     172.30.27.5    10.46.41.94      53:31829/TCP     5s

3. Edit the HyperConverged CR in your default editor by running the following command:

   $ oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnv

4. Enable the DNS server and monitoring components according to the following example:

   apiVersion: hco.kubevirt.io/v1beta1
   kind: HyperConverged
   metadata:
     name: kubevirt-hyperconverged
     namespace: openshift-cnv
   spec:
     featureGates:
       deployKubeSecondaryDNS: true
     kubeSecondaryDNSNameServerIP: "10.46.41.94" 1
   # ...

   1

   Specify the external IP address exposed by the load balancer service.

5. Save the file and exit the editor.

6. Retrieve the cluster FQDN by running the following command:

    $ oc get dnses.config.openshift.io cluster -o jsonpath='{.spec.baseDomain}'

   Example output

   openshift.example.com

7. Point to the DNS server by using one of the following methods:

   • Add the kubeSecondaryDNSNameServerIP value to the resolv.conf file on your local machine.

     Note

     Editing the resolv.conf file overwrites existing DNS settings.

   • Add the kubeSecondaryDNSNameServerIP value and the cluster FQDN to the enterprise DNS server records. For example:

     vm.<FQDN>. IN NS ns.vm.<FQDN>.

     ns.vm.<FQDN>. IN A 10.46.41.94

Procedure

1. Retrieve the network interface name from the VM configuration by running the following command:

   $ oc get vm -n <namespace> <vm_name> -o yaml

   Example output

   apiVersion: kubevirt.io/v1
   kind: VirtualMachine
   metadata:
     name: example-vm
     namespace: example-namespace
   spec:
     running: true
     template:
       spec:
         domain:
           devices:
             interfaces:
               - bridge: {}
                 name: example-nic
   # ...
         networks:
         - multus:
             networkName: bridge-conf
           name: example-nic 1

   1

   Note the name of the network interface.

2. Connect to the VM by using the ssh command:

   $ ssh <user_name>@<interface_name>.<vm_name>.<namespace>.vm.<cluster_fqdn>