Chapter 11. High availability for pod-level bonds on SR-IOV networks

Procedure

1. Create the openshift-pf-status-relay-operator namespace by entering the following command:

   $ cat << EOF| oc create -f -
   apiVersion: v1
   kind: Namespace
   metadata:
     name: openshift-pf-status-relay-operator
     annotations:
       workload.openshift.io/allowed: management
   EOF

   Copy to Clipboard Toggle word wrap

2. Create an OperatorGroup custom resource (CR) by entering the following command:

   $ cat << EOF| oc create -f -
   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: pf-status-relay-operators
     namespace: openshift-pf-status-relay-operator
   spec:
     targetNamespaces:
     - openshift-pf-status-relay-operator
   EOF

   Copy to Clipboard Toggle word wrap

3. Create a Subscription CR for the PF Status Relay Operator by entering the following command:

   $ cat << EOF| oc create -f -
   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: pf-status-relay-operator-subscription
     namespace: openshift-pf-status-relay-operator
   spec:
     channel: stable
     name: pf-status-relay-operator
     source: redhat-operators
     sourceNamespace: openshift-marketplace
   EOF

   Copy to Clipboard Toggle word wrap

Procedure

1. Install the PF Status Relay Operator:

   1. In the OpenShift Container Platform web console, click Ecosystem Software Catalog.
   2. Select PF Status Relay Operator from the list of available Operators, and then click Install.
   3. On the Install Operator page, under Installed Namespace, select Operator recommended Namespace.
   4. Click Install.

Procedure

1. Create the project namespace by creating a namespace.yaml file such as the following example:

   Example namespace.yaml file

   apiVersion: v1
   kind: Namespace
   metadata:
     labels:
       kubernetes.io/metadata.name: sriov-operator-tests
       pod-security.kubernetes.io/audit: privileged
       pod-security.kubernetes.io/enforce: privileged
       pod-security.kubernetes.io/warn: privileged
       security.openshift.io/scc.podSecurityLabelSync: "false"
     name: sriov-operator-tests 

   1

   Copy to Clipboard Toggle word wrap

   1

   The namespace where you deploy the high-availability pod.

2. Apply the namespace by running the following command:

   $ oc apply -f namespace.yaml

   Copy to Clipboard Toggle word wrap

3. Configure host-level LACP bonds:

   1. Create a YAML file that defines the NodeNetworkConfigurationPolicy resource for the ens5f0 interface on the worker-0 node:

      Example nncpBondF0Worker0.yaml file

      apiVersion: nmstate.io/v1
      kind: NodeNetworkConfigurationPolicy
      metadata:
        name: example-bond-f0
      spec:
        nodeSelector:
          kubernetes.io/hostname: worker-0 

      1

      
        desiredState:
          interfaces:
            - name: example-bond-f0
              description: example-bond-f0
              type: bond
              state: up
              mtu: 9216
              link-aggregation:
                mode: 802.3ad 

      2

      
                options:
                  miimon: '100'
                  lacp_rate: 'fast' 

      3

      
                  min_links: '1'
                port:
                  - ens5f0 

      4

      
            - name: ens5f0
              type: ethernet
              state: up
              mtu: 9216

      Copy to Clipboard Toggle word wrap

      1

      The node where the bonded interface is created.

      2

      You must set the LACP mode to 802.3ad to enable LACP on the bond.

      3

      You must set the LACP rate fast on the interface and on the switch. The fast rate sends LACP packets every second.

      4

      The PF that you want to include in the bond.

   2. Create a YAML file that defines the NodeNetworkConfigurationPolicy resource for the ens5f1 interface on the worker-0 node:

      Example nncpBondF1Worker0.yaml file

      apiVersion: nmstate.io/v1
      kind: NodeNetworkConfigurationPolicy
      metadata:
        name: example-bond-f1
      spec:
        nodeSelector:
          kubernetes.io/hostname: worker-0 

      1

      
        desiredState:
          interfaces:
            - name: example-bond-f1
              description: example-bond-f1
              type: bond
              state: up
              mtu: 9216
              link-aggregation:
                mode: 802.3ad 

      2

      
                options:
                  miimon: '100'
                  lacp_rate: 'fast' 

      3

      
                  min_links: '1'
                port:
                  - ens5f1 

      4

      
            - name: ens5f1
              type: ethernet
              state: up
              mtu: 9216

      Copy to Clipboard Toggle word wrap

      1

      The node where the bonded interface is created.

      2

      You must set the LACP mode to 802.3ad to enable LACP on the bond.

      3

      You must set the LACP rate fast on the interface and on the switch. The fast rate sends LACP packets every second.

      4

      The PF that you want to include in the bond.

   3. Apply the resources by running the following commands:

      $ oc apply -f nncpBondF0Worker0.yaml
      $ oc apply -f nncpBondF1Worker0.yaml

      Copy to Clipboard Toggle word wrap

4. Create SR-IOV network VFs for the bonded interfaces:

   1. Create a YAML file that defines the SriovNetworkNodePolicy resource for the ens5f0 interface on the worker-0 node:

      Example sriovnetworkpolicy-port1.yaml file

      apiVersion: sriovnetwork.openshift.io/v1
      kind: SriovNetworkNodePolicy
      metadata:
        name: sriovnetpolicy-port-0
        namespace: openshift-sriov-network-operator
      spec:
        deviceType: netdevice
        nicSelector:
          pfNames:
            - ens5f0 

      1

      
        nodeSelector:
          kubernetes.io/hostname: worker-0 

      2

      
        numVfs: 10 

      3

      
        priority: 99
        resourceName: resourceport0 

      4

      Copy to Clipboard Toggle word wrap

      1

      The PF to create the VFs from.

      2

      The node where the VFs are created.

      3

      The number of VFs to create on the PF.

      4

      The resource name used by pods to request these VFs.

   2. Create a YAML file that defines the SriovNetworkNodePolicy resource for the ens5f1 interface on the worker-0 node:

      Example sriovnetworkpolicy-port2.yaml file

      apiVersion: sriovnetwork.openshift.io/v1
      kind: SriovNetworkNodePolicy
      metadata:
        name: sriovnetpolicy-port-1
        namespace: openshift-sriov-network-operator
      spec:
        deviceType: netdevice
        nicSelector:
          pfNames:
            - ens5f1 

      1

      
        nodeSelector:
          kubernetes.io/hostname: worker-0 

      2

      
        numVfs: 10 

      3

      
        priority: 99
        resourceName: resourceport1 

      4

      Copy to Clipboard Toggle word wrap

      1

      The PF to create the VFs from.

      2

      The node where the VFs are created.

      3

      The number of VFs to create on the PF.

      4

      The resource name used by pods to request these VFs.

   3. Apply the resources by running the following commands:

      $ oc apply -f sriovnetworkpolicy-port1.yaml
      $ oc apply -f sriovnetworkpolicy-port2.yaml

      Copy to Clipboard Toggle word wrap

5. Configure the PF Status Relay Operator:

   1. Create a YAML file that defines the PFLACPMonitor resource. This example file configures the Operator to monitor the LACP status of ens5f0 and ens5f1 bonded interfaces on the worker-0 node:

      Example pflacpmonitor.yaml file

      apiVersion: pfstatusrelay.openshift.io/v1alpha1
      kind: PFLACPMonitor
      metadata:
        namespace: openshift-pf-status-relay-operator
        labels:
          app.kubernetes.io/name: pf-status-relay-operator
        name: pflacpmonitor-worker-0
      spec:
        interfaces:
          - ens5f0 

      1

      
          - ens5f1
        pollingInterval: 1000 

      2

      
        nodeSelector:
          kubernetes.io/hostname: worker-0 

      3

      Copy to Clipboard Toggle word wrap

      1

      The list of PFs to monitor.

      2

      The polling interval in milliseconds to check the LACP status on the monitored interfaces. The minimum value is 1000.

      3

      The node for the target interfaces.

      Important

      Use only one PFLACPMonitor custom resource to monitor each network interface on a node. If you create multiple resources that target the same interface, the PF Status Relay Operator will not process the conflicting configurations.

   2. Apply the PFLACPMonitor resource by running the following command:

      $ oc apply -f pflacpmonitor.yaml

      Copy to Clipboard Toggle word wrap

Verification

1. Check the logs of the PF Status Relay Operator to verify that it is monitoring the LACP state:

   $ oc logs -n openshift-pf-status-relay-operator <pf_status_relay_operator_pod_name>

   Copy to Clipboard Toggle word wrap

   Example output

   {"time":"2025-07-24T13:35:54.653201692Z","level":"INFO","msg":"lacp is up","interface":"ens5f0"}
   {"time":"2025-07-24T13:35:54.65347273Z","level":"INFO","msg":"vf link state was set","id":0,"state":"auto","interface":"ens5f0"}
   ...

   Copy to Clipboard Toggle word wrap

2. Apply the SriovNetwork resources to make the VFs available for use within the sriov-operator-tests namespace:

   1. Create a YAML file that defines the SriovNetwork resource for the VFs created on ens5f0:

      Example sriovnetwork-port1.yaml file

      apiVersion: sriovnetwork.openshift.io/v1
      kind: SriovNetwork
      metadata:
        name: sriovnetwork-port0
        namespace: openshift-sriov-network-operator
      spec:
        capabilities: '{ "mac": true }'
        networkNamespace: sriov-operator-tests
        resourceName: resourceport0

      Copy to Clipboard Toggle word wrap

   2. Create a YAML file that defines the SriovNetwork resource for the VFs created on ens5f1:

      Example sriovnetwork-port2.yaml file

      apiVersion: sriovnetwork.openshift.io/v1
      kind: SriovNetwork
      metadata:
        name: sriovnetwork-port1
        namespace: openshift-sriov-network-operator
      spec:
        capabilities: '{ "mac": true }'
        networkNamespace: sriov-operator-tests
        resourceName: resourceport1

      Copy to Clipboard Toggle word wrap

   3. Apply the resources by running the following commands:

      $ oc apply -f sriovnetwork-port1.yaml
      $ oc apply -f sriovnetwork-port2.yaml

      Copy to Clipboard Toggle word wrap

3. Define a high-availability pod that uses the SR-IOV VFs:

   1. Apply the NetworkAttachmentDefinition resource to create an active-backup bond using the two SR-IOV networks:

      Example nad-bond.yaml file

      apiVersion: k8s.cni.cncf.io/v1
      kind: NetworkAttachmentDefinition
      metadata:
        name: nad-bond-1
        namespace: sriov-operator-tests
      spec:
        config: |-
          {"type": "bond", "cniVersion": "0.3.1", "name": "bond-net1",
          "mode": "active-backup", "failOverMac": 1, "linksInContainer": true, "miimon": "100", "mtu": 1450,
          "links": [{"name": "net1"},{"name": "net2"}], "capabilities": {"ips": true}, "ipam": {"type": "static"}}

      Copy to Clipboard Toggle word wrap

      • linksInContainer: true creates the bond inside the pod’s network namespace.
      • mode: active-backup configures the bond to use active-backup mode.

      • links specifies the pod-level interfaces to include in the bond.

        Important

        The PF Status Relay Operator provides LACP state monitoring for pod-level bonding with the mode: active-backup configuration only.

   2. Apply the NetworkAttachmentDefinition resource by running the following command:

      $ oc apply -f nad-bond.yaml

      Copy to Clipboard Toggle word wrap

   3. Create a YAML file that defines the Pod resource that uses the VFs from the bonded interfaces in active-backup mode:

      Example client-bond.yaml file

      apiVersion: v1
      kind: Pod
      metadata:
        name: client-bond
        namespace: sriov-operator-tests
        annotations:
          k8s.v1.cni.cncf.io/networks: |- 

      1

      
            [{
                "name": "sriovnetwork-port0",
                "interface": "net1",
                "mac": "<mac_address>"
              },{
                "name": "sriovnetwork-port1",
                "interface": "net2",
                "mac": "<mac_address>"
              },{
                "name": "nad-bond-1",
                "interface": "bond0",
                "ips": ["192.168.10.254/24","2001:100::254/64"],
                "mac": "<mac_address>"
            }]
      spec:
        nodeName: worker-0
        containers:
          - name: client-bond
            image: quay.io/nginx/nginx-unprivileged
            imagePullPolicy: IfNotPresent
            command: ["/bin/sh", "-c", "sleep 3650d"]
            securityContext:
              privileged: true
            command: ["/bin/sleep", "3650d"]

      Copy to Clipboard Toggle word wrap

      1

      The annotation requests three networks: two SR-IOV VFs, net1 and net2 and one bond, bond0, which uses them.

   4. Apply the Pod resource by running the following command:

      $ oc apply -f client-bond.yaml

      Copy to Clipboard Toggle word wrap

4. Check that the failover mechanism:

   1. Log in to the client-bond pod by running the following command:

      $ oc rsh -n sriov-operator-tests client-bond

      Copy to Clipboard Toggle word wrap

   2. Check the initial status of the pod-level bond by running the following command:

      sh-4.4# cat /proc/net/bonding/bond0

      Copy to Clipboard Toggle word wrap

      Example output

      [root@client-bond-tlb /]# cat /proc/net/bonding/bond0
      ...
      
      Bonding Mode: transmit load balancing
      Transmit Hash Policy: layer2 (0)
      Primary Slave: None
      Currently Active Slave: net1
      MII Status: up
      MII Polling Interval (ms): 100
      Up Delay (ms): 0
      Down Delay (ms): 0
      Peer Notification Delay (ms): 0
      
      Slave Interface: net1
      MII Status: up
      Speed: 25000 Mbps
      Duplex: full
      Link Failure Count: 0
      Permanent HW addr: AA:BB:CC:DD:EE:FF
      Slave queue ID: 0
      
      Slave Interface: net2
      MII Status: up
      Speed: 25000 Mbps
      Duplex: full
      Link Failure Count: 0
      Permanent HW addr: BB:CC:DD:EE:FF:GG

      Copy to Clipboard Toggle word wrap

      • Both net1 and net2 interfaces are up.
   3. Exit the pod shell.
   4. Simulate an LACP failure on your upstream physical switch. To simulate this scenario, you can filter LACP traffic on the switch port that you want to test the failure on. This ensures that the physical link remains up while the LACP pollings fails. The command to do this is vendor-dependent.

   5. Verify the failover inside the pod by logging back into the client-bond pod and checking the bond status again:

      sh-4.4# cat /proc/net/bonding/bond0

      Copy to Clipboard Toggle word wrap

      Example output

      ...
      
      Bonding Mode: transmit load balancing
      Transmit Hash Policy: layer2 (0)
      Primary Slave: None
      Currently Active Slave: net2
      MII Status: up
      MII Polling Interval (ms): 100
      Up Delay (ms): 0
      Down Delay (ms): 0
      Peer Notification Delay (ms): 0
      
      Slave Interface: net1
      MII Status: down
      Speed: Unknown
      Duplex: Unknown
      Link Failure Count: 1
      Permanent HW addr: AA:BB:CC:DD:EE:FF
      Slave queue ID: 0
      
      Slave Interface: net2
      MII Status: up
      Speed: 25000 Mbps
      Duplex: full
      Link Failure Count: 0
      Permanent HW addr: BB:CC:DD:EE:FF:GG
      Slave queue ID: 0

      Copy to Clipboard Toggle word wrap

      • The net1 interface is down, and the net2 interface is now the active interface.

        The client-bond pod detects the link state change and switches to the backup network path.