Chapter 12. NVIDIA DPF Operator

1. Management cluster setup: Install and configure a standard OpenShift Container Platform cluster on x86 servers with control-plane nodes only.
2. Management cluster configuration: Configure nodes and cluster-level settings, then install and configure the required Operators on the management cluster.

3. DPF installation: Deploy the DPF Operator, controllers, DPF resources, and DPU service definitions on the management cluster.

   Important

   You must install the DPF Operator before the DPF HCP Provisioner Operator because that Operator requires DPF custom resource definitions (CRDs) such as DPUCluster, DPUFlavor, DPUDeployment, and DPFOperatorConfig.

4. Hosted cluster creation: The DPF HCP Provisioner Operator automates the creation of a hosted DPU cluster by using hosted control planes. The Operator references the DPUDeployment resource during ignition generation.
5. Worker node scale-out and DPU provisioning: When worker nodes with DPUs are added to the cluster, the DPF Operator flashes the DPUs with a Red Hat Enterprise Linux CoreOS (RHCOS) image and configures them to join the hosted cluster as worker nodes.
6. Worker node integration: Approve DPU worker node certificate signing requests (CSRs) and configure security context constraint (SCC) bindings on the hosted cluster.
7. Service deployment: After the DPU hosted cluster is operational, data plane DPU services and chains are deployed by DPF.
8. Verification: Validate end-to-end connectivity through the DPU data plane by running ping and nc traffic tests between workload pods and services.

Procedure

1. Go to the Red Hat Hybrid Cloud Console cluster creation page and create a cluster with control-plane nodes only. Select Data center Assisted Installer.

2. Optional: Configure jumbo MTU for each control plane node.

   1. Under Hosts' network configuration in the Assisted Installer wizard, select Static IP, bridges, and bonds.

   2. Set the Static network configurations section per node according to the following template, using the relevant MAC address and interface name for each node:

      interfaces:
        - ipv4:
            dhcp: true
            enabled: true
          mac-address: <xx:xx:xx:xx:xx:xx>
          mtu: 1500 # Set to 1500 for standard MTU or 9000 for jumbo frames
          name: <interface-name>
          state: up
          type: ethernet

      Note
      • You can alternatively configure MTU allocation on the DHCP server that allocates IPs to the control plane nodes.
      • If virtual machines are used for control-plane nodes, the MTU must be set on the bridge of the hypervisor used by the VMs.
      • When using MTU 9000, ensure the switch ports that connect the cluster’s control-plane nodes are set to handle jumbo frames.

3. Select the following operators to install with the cluster:

   • Storage Logical Volume Manager Storage
   • Platform Operations & Lifecycle MultiCluster Engine
   • Scheduling Node Feature Discovery
4. Click Add hosts to add hosts to the cluster. Only control plane nodes are required at this stage.
5. After the installation completes, download the KUBECONFIG file and save it as mgmt-kubeconfig.

Verification

1. Set the KUBECONFIG environment variable:

   $ export KUBECONFIG="$(pwd)/mgmt-kubeconfig"

2. Verify that all nodes are in a Ready state:

   $ oc get nodes

Procedure

1. Set the OPENSHIFT_PULL_SECRET environment variable to the path of your pull secret file:

   $ export OPENSHIFT_PULL_SECRET="/root/pull-secret.txt"

2. Deploy the dpu-worker-config Helm chart to create the worker node MachineConfig:

   $ helm upgrade --install dpu-worker-config \
       oci://registry.redhat.io/dpu-kit-for-nvidia/dpu-worker-config-chart \
       --version 4.22.0 \
       --registry-config "${OPENSHIFT_PULL_SECRET}" \
       --namespace dpf-hcp-provisioner-system \
       --create-namespace \
       --disable-openapi-validation

Procedure

1. Define the cluster variables used by Node Feature Discovery:

   $ export CLUSTER_NAME="doca-mgmt"
   $ export BASE_DOMAIN="example.com"
   $ export HOST_CLUSTER_API="api.${CLUSTER_NAME}.${BASE_DOMAIN}"

   where:

   CLUSTER_NAME

   Specifies the management cluster name.

   BASE_DOMAIN

   Specifies the management cluster base domain.

   HOST_CLUSTER_API

   Specifies the management cluster API endpoint.

2. Create a file named nfd-instance.yaml with the following NodeFeatureDiscovery resource definition:

   apiVersion: nfd.openshift.io/v1
   kind: NodeFeatureDiscovery
   metadata:
     name: nfd-instance
     namespace: openshift-nfd
   spec:
     operand:
       workerEnvs:
         - name: KUBERNETES_SERVICE_HOST
           value: $HOST_CLUSTER_API
         - name: KUBERNETES_SERVICE_PORT
           value: "6443"
     workerConfig:
       configData: |
         sources:
           pci:
             deviceClassWhitelist:
               - "0200"
               - "03"
               - "12"
               - "0207"
             deviceLabelFields:
               - "vendor"
               - "device"
               - "class"

3. Apply the file by using envsubst to substitute the environment variables:

   $ envsubst < nfd-instance.yaml | oc apply -f -

4. Create a file named nfd-rule.yaml with the following NodeFeatureRule resource definition to detect worker nodes with DPUs and label them with a dpu-enabled label:

   apiVersion: nfd.openshift.io/v1alpha1
   kind: NodeFeatureRule
   metadata:
     name: dpu-detection-rule
     namespace: openshift-nfd
   spec:
     rules:
       - labels:
           dpu-enabled: ""
         matchFeatures:
           - feature: pci.device
             matchExpressions:
               device:
                 op: In
                 value:
                   - a2d6
                   - a2dc
               vendor:
                 op: In
                 value:
                   - 15b3
         name: DPU-detection-rule

5. Apply the file:

   $ oc apply -f nfd-rule.yaml

6. Ensure that the MetalLB Operator Subscription schedules Operator pods on control-plane nodes. When you install the MetalLB Operator, include the following spec.config settings, or patch an existing Subscription to add them:

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: metallb-operator
     namespace: openshift-operators
   spec:
     channel: "stable"
     name: metallb-operator
     source: redhat-operators
     sourceNamespace: openshift-marketplace
     installPlanApproval: Automatic
     config:
       tolerations:
       - key: "node-role.kubernetes.io/control-plane"
         operator: "Exists"
         effect: "NoSchedule"
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: "node-role.kubernetes.io/control-plane"
                 operator: "Exists"

7. Create a file named metallb-config.yaml with the following MetalLB resource definition:

   apiVersion: metallb.io/v1beta1
   kind: MetalLB
   metadata:
     name: metallb
     namespace: openshift-operators
   spec:
     nodeSelector:
       node-role.kubernetes.io/control-plane: ""
     speakerTolerations:
       - key: node-role.kubernetes.io/control-plane
         operator: Exists
         effect: NoSchedule

8. Apply the MetalLB resource file:

   $ oc apply -f metallb-config.yaml

9. Ensure that the Red Hat OpenShift GitOps Subscription includes the DPF-required environment variables. When you install the Operator, set the following spec.config.env values, or patch an existing Subscription to add them so that Argo CD can manage the dpf-operator-system namespace:

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: openshift-gitops-operator
     namespace: openshift-gitops-operator
   spec:
     channel: gitops-1.21
     config:
       env:
       - name: ARGOCD_CLUSTER_CONFIG_NAMESPACES
         value: "openshift-gitops,dpf-operator-system"
       - name: CONTROLLER_CLUSTER_ROLE
         value: "cluster-admin"
       - name: SERVER_CLUSTER_ROLE
         value: "cluster-admin"
     installPlanApproval: Automatic
     name: openshift-gitops-operator
     source: redhat-operators
     sourceNamespace: openshift-marketplace
     startingCSV: openshift-gitops-operator.v1.21.3

10. Create a file named argocd-instance.yaml with the following ArgoCD resource definition:

    apiVersion: argoproj.io/v1beta1
    kind: ArgoCD
    metadata:
      name: argocd
      namespace: dpf-operator-system
    spec:
      nodePlacement:
        nodeSelector:
          node-role.kubernetes.io/control-plane: ""
        tolerations:
        - key: node-role.kubernetes.io/master
          operator: Exists
          effect: NoSchedule
        - key: node-role.kubernetes.io/control-plane
          operator: Exists
          effect: NoSchedule
      server:
        route:
          enabled: true
      controller: {}
      repo: {}
      applicationSet:
        enabled: false
      resourceExclusions: |
        - apiGroups:
          - packages.operators.coreos.com
          kinds:
          - PackageManifest
      sso:
        provider: dex
        dex:
          openShiftOAuth: true
      notifications:
        enabled: false

11. Apply the Argo CD file:

    $ oc apply -f argocd-instance.yaml

12. Wait for the ArgoCD Redis deployment to be ready:

    $ oc wait deployment argocd-redis -n dpf-operator-system \
      --for=condition=Available --timeout=120s

13. Enable global IP forwarding on the OVN-Kubernetes configuration:

    This command enables IP packet forwarding between different networks managed by OVN-Kubernetes.

    $ oc patch network.operator.openshift.io cluster --type=merge -p \
      '{"spec":{"defaultNetwork":{"ovnKubernetesConfig":{"gatewayConfig":{"ipForwarding":"Global"}}}}}'

Procedure

1. Add the DPF Helm repository and update the local cache:

   $ helm repo add --force-update dpf-repository ${REGISTRY}

   $ helm repo update

2. Install the DPF Operator by using Helm:

   $ helm upgrade --install dpf-operator dpf-repository/dpf-operator \
       --namespace dpf-operator-system \
       --version "${TAG}" \
       --set kamajiEtcdDefrag.enabled=false \
       --set isOpenshift=true \
       --set enableNodeFeatureRules=false \
       --wait

Verification

1. Verify that the Operator controller manager deployment has rolled out successfully:

   $ oc rollout status deployment --namespace dpf-operator-system dpf-operator-controller-manager

   Example output

   deployment "dpf-operator-controller-manager" successfully rolled out

2. Verify that all pods in the dpf-operator-system namespace are ready:

   $ oc wait --for=condition=ready --namespace dpf-operator-system pods --all

   Example output

   pod/argocd-application-controller-0 condition met
   pod/argocd-dex-server-6dd56c8469-bhsq4 condition met
   pod/argocd-redis-b4f94bb8d-wr86b condition met
   pod/argocd-repo-server-96765f997-79k9q condition met
   pod/argocd-server-648c7ff85f-7frtg condition met
   pod/dpf-operator-controller-manager-7bf9744c5f-cwrgc condition met
   pod/maintenance-operator-585767f779-8k2lx condition met

Procedure

1. Create a file named dpfoperatorconfig.yaml with the following content:

   Note

   MTU Configuration: Set NODES_MTU to 1500 for standard MTU environments or 9000 for jumbo frame environments. This value must be consistent across:

   • Assisted Installer host configuration
   • DPFOperatorConfig networking section (this step)
   • DPUServiceNAD configuration

   Choose based on your network infrastructure capabilities.

   apiVersion: operator.dpu.nvidia.com/v1alpha1
   kind: DPFOperatorConfig
   metadata:
     name: dpfoperatorconfig
     namespace: dpf-operator-system
   spec:
     kamajiClusterManager:
       disable: true
     multus:
       disable: true
     cniInstaller:
       disable: true
     networking:
       controlPlaneMTU: $NODES_MTU      # management/OOB MTU (= NODES_MTU: 1500 standard, 9000 jumbo)
       highSpeedMTU: $NODES_MTU         # high-speed fabric MTU (= NODES_MTU; must match controlPlaneMTU)
       dpuNodeOOBBridgeName: br-ex      # OOB bridge for DPU provisioning; br-ex on OpenShift
     overrides:
       dpuCNIBinPath: /var/lib/cni/bin/
       dpuCNIPath: /run/multus/cni/net.d/
       dpuOpenvSwitchSystemSharedLib64Path: /lib64
       flannelSkipCNIConfigInstallation: false
       kubernetesAPIServerPort: $TARGETCLUSTER_API_SERVER_PORT
       kubernetesAPIServerVIP: $HOST_CLUSTER_API
       dpuLinkerCachePath: /etc/ld.so.cache
       dpuOptLibraryPath: /usr/opt
     provisioningController:
       enableDynamicBFCFGTemplates: true
       hostAgentDNSPolicy: Default
       dmsTimeout: 900
     nodeSRIOVDevicePluginController:
       devicePlugin:
         defaultResourcePrefix: openshift.io
       disable: false
       replicas: 1
     staticClusterManager:
       disable: false
     dpuServiceController:
       disableHostNetworkReadyNoExecuteTaints: false
     flannel:
       podCIDR: $FLANNEL_POD_CIDR

2. Apply the resource file:

   $ envsubst < dpfoperatorconfig.yaml | oc apply -f -

Verification

1. Verify that the provisioning controller manager deployment has rolled out:

   $ oc rollout status deployment --namespace dpf-operator-system dpf-provisioning-controller-manager

   Example output

   deployment "dpf-provisioning-controller-manager" successfully rolled out

2. Verify that the DPU service controller manager deployment has rolled out:

   $ oc rollout status deployment --namespace dpf-operator-system dpuservice-controller-manager

   Example output

   deployment "dpuservice-controller-manager" successfully rolled out

3. Verify that all Operator deployments in the dpf-operator-system namespace have rolled out:

   Note

   You might need to run this command more than once, because the deployments become available at different times as the Operator reconciles its resources.

   $ oc rollout status deployment --namespace dpf-operator-system

4. Optional: List the pods in the dpf-operator-system namespace to review their status:

   $ oc get pods -n dpf-operator-system

   Example output

   NAME                                                    READY   STATUS    RESTARTS   AGE
   argocd-application-controller-0                          1/1     Running   0          64m
   argocd-dex-server-9dd99cc6c-pkl6j                       1/1     Running   0          64m
   argocd-redis-6749d85f98-mw9rc                           1/1     Running   0          64m
   argocd-repo-server-5fd79d655c-9rl26                     1/1     Running   0          64m
   argocd-server-5cc4f69c45-hzgm9                          1/1     Running   0          64m
   bfb-registry                                            1/1     Running   0          98s
   dpf-nodesriovdeviceplugin-controller-7c7bb889f6-xk6nm   1/1     Running   0          107s
   dpf-operator-controller-manager-6969c6d8bc-4njxd        1/1     Running   0          4m54s
   dpf-provisioning-controller-manager-54b6bdc57f-69lr7    1/1     Running   0          109s
   dpf-provisioning-controller-manager-54b6bdc57f-fg9f6    1/1     Running   0          109s
   dpuservice-controller-manager-5d9c4f6f67-wsdzt          1/1     Running   0          108s
   maintenance-operator-68c794b549-pp2dp                   1/1     Running   0          74m
   static-cm-controller-manager-745c8fd7d5-n8d2z           1/1     Running   0          110s

Procedure

1. Create a file named nodesriovdevicepluginconfig.yaml with the following content:

   apiVersion: noderesources.dpu.nvidia.com/v1alpha1
   kind: NodeSRIOVDevicePluginConfig
   metadata:
     name: bf3-vfs
     namespace: dpf-operator-system
   spec:
     devicePluginResources:
       - name: bf3-p0-vfs-mgmt
         type: vf
         ranges:
           - pfIndex: 0
             start: 1
             end: 1
       - name: bf3_vfs
         type: vf
         options:
           isRdma: true
         ranges:
           - pfIndex: 0
             start: 2
             end: 45
           - pfIndex: 1
             start: 0
             end: 45

   where:

   bf3-p0-vfs-mgmt

   Reserves VF index 1 on PF0 for DPU management connectivity.

   bf3_vfs

   Allocates VF indices 2-45 on PF0 and VF indices 0-45 on PF1 for workload traffic with RDMA enabled.

   pfIndex

   The pfIndex values refer to the first (0) and second (1) physical functions of the BlueField-3 DPU.

2. Apply the resource file:

   $ oc apply -f nodesriovdevicepluginconfig.yaml

Procedure

1. Create a file named dpuflavor.yaml for your MTU configuration and apply it.

   • For Standard MTU (1500):

     apiVersion: provisioning.dpu.nvidia.com/v1alpha1
     kind: DPUFlavor
     metadata:
       name: hbn-ovnk
       namespace: dpf-operator-system
       annotations:
         provisioning.dpu.nvidia.com/skip-bfcfg-size-check: ""
     spec:
       grub:
         kernelParameters:
           - console=hvc0
           - console=ttyAMA0
           - earlycon=pl011,0x13010000
           - iommu.passthrough=1
           - cgroup_no_v1=net_prio,net_cls
           - hugepagesz=2048kB
           - hugepages=250
       nvconfig:
         - device: '*'
           parameters:
             - PF_BAR2_ENABLE=0
             - PER_PF_NUM_SF=1
             - PF_TOTAL_SF=20
             - PF_SF_BAR_SIZE=10
             - NUM_PF_MSIX_VALID=0
             - PF_NUM_PF_MSIX_VALID=1
             - PF_NUM_PF_MSIX=228
             - INTERNAL_CPU_MODEL=1
             - INTERNAL_CPU_OFFLOAD_ENGINE=0
             - SRIOV_EN=1
             - NUM_OF_VFS=$NUM_VFS
             - LAG_RESOURCE_ALLOCATION=1
             - LINK_TYPE_P1=ETH
             - LINK_TYPE_P2=ETH
       ovs:
         rawConfigScript: |
           #!/bin/bash
           set -e
     
           _ovs-vsctl() {
             ovs-vsctl --timeout 15 "$@"
           }
     
           restart_ovs=false
     
           _ovs-get-other-config() {
             _ovs-vsctl --if-exists get Open_vSwitch . "other_config:$1" 2>/dev/null | tr -d '"'
           }
     
           _ovs-set-other-config() {
             if [ "$(_ovs-get-other-config "$1")" != "$2" ]; then
               _ovs-vsctl set Open_vSwitch . "other_config:$1=$2"
               restart_ovs=true
             fi
           }
     
           _ovs-remove-other-config() {
             if [ -n "$(_ovs-get-other-config "$1")" ]; then
               _ovs-vsctl remove Open_vSwitch . other_config "$1"
               restart_ovs=true
             fi
           }
     
           _ovs-set-other-config doca-init true
           _ovs-set-other-config dpdk-max-memzones 50000
           _ovs-set-other-config hw-offload true
           _ovs-set-other-config pmd-quiet-idle true
           _ovs-set-other-config max-idle 20000
           _ovs-set-other-config max-revalidator 5000
           _ovs-set-other-config doca-congestion-threshold 60
           _ovs-set-other-config flow-limit 500000
           _ovs-set-other-config hw-offload-ct-unidir-udp-enabled true
           _ovs-remove-other-config default-datapath-type
     
           if [ "$restart_ovs" = true ]; then
             if systemctl list-unit-files openvswitch-switch.service &>/dev/null; then
               systemctl restart openvswitch-switch
             elif systemctl list-unit-files openvswitch.service &>/dev/null; then
               systemctl restart openvswitch
             fi
           fi
     
           _ovs-vsctl --may-exist add-br br-sfc
           _ovs-vsctl set bridge br-sfc datapath_type=netdev
           _ovs-vsctl set bridge br-sfc fail_mode=secure
           _ovs-vsctl --if-exists del-br br-hbn
           _ovs-vsctl --may-exist add-br br-hbn
           _ovs-vsctl set bridge br-hbn datapath_type=netdev
           _ovs-vsctl set bridge br-hbn fail_mode=secure
           _ovs-vsctl --may-exist add-port br-sfc p0
           _ovs-vsctl set Interface p0 type=dpdk
           _ovs-vsctl set Interface p0 mtu_request=9216
           _ovs-vsctl set Port p0 external_ids:dpf-type=physical
     
           # Activate DOCA for OVNK
           _ovs-vsctl set Open_vSwitch . external-ids:ovn-bridge-datapath-type=netdev
           # setup ovnkube managed bridge, br-dpu (this corresponds to br-ex on ovnk docs)
           _ovs-vsctl --may-exist add-br br-dpu
           _ovs-vsctl br-set-external-id br-dpu bridge-id br-dpu
           _ovs-vsctl br-set-external-id br-dpu bridge-uplink pbrdputobrovn
           _ovs-vsctl set bridge br-dpu datapath_type=netdev
           _ovs-vsctl --may-exist add-port br-dpu pf0hpf
           _ovs-vsctl set Interface pf0hpf type=dpdk
     
           # Create OVS bridge (br-ovn) in between the SC managed bridge and OVNK
           _ovs-vsctl --may-exist add-br br-ovn
           _ovs-vsctl set bridge br-ovn datapath_type=netdev
           _ovs-vsctl --may-exist add-port br-ovn pbrovntobrdpu
           _ovs-vsctl --may-exist add-port br-dpu pbrdputobrovn
     
           # Patch br-ovn and br-dpu together
           _ovs-vsctl set Interface pbrovntobrdpu type=patch options:peer=pbrdputobrovn
           _ovs-vsctl set Interface pbrdputobrovn type=patch options:peer=pbrovntobrdpu

   • For Jumbo frames (MTU 9000):

     apiVersion: provisioning.dpu.nvidia.com/v1alpha1
     kind: DPUFlavor
     metadata:
       name: hbn-ovnk
       namespace: dpf-operator-system
       annotations:
         provisioning.dpu.nvidia.com/skip-bfcfg-size-check: ""
     spec:
       grub:
         kernelParameters:
           - console=hvc0
           - console=ttyAMA0
           - earlycon=pl011,0x13010000
           - iommu.passthrough=1
           - cgroup_no_v1=net_prio,net_cls
           - hugepagesz=2048kB
           - hugepages=250
       nvconfig:
         - device: '*'
           parameters:
             - PF_BAR2_ENABLE=0
             - PER_PF_NUM_SF=1
             - PF_TOTAL_SF=20
             - PF_SF_BAR_SIZE=10
             - NUM_PF_MSIX_VALID=0
             - PF_NUM_PF_MSIX_VALID=1
             - PF_NUM_PF_MSIX=228
             - INTERNAL_CPU_MODEL=1
             - INTERNAL_CPU_OFFLOAD_ENGINE=0
             - SRIOV_EN=1
             - NUM_OF_VFS=$NUM_VFS
             - LAG_RESOURCE_ALLOCATION=1
             - NUM_VF_MSIX=30
             - LINK_TYPE_P1=ETH
             - LINK_TYPE_P2=ETH
       ovs:
         rawConfigScript: |
           #!/bin/bash
           set -e
     
           _ovs-vsctl() {
             ovs-vsctl --timeout 15 "$@"
           }
     
           restart_ovs=false
     
           _ovs-get-other-config() {
             _ovs-vsctl --if-exists get Open_vSwitch . "other_config:$1" 2>/dev/null | tr -d '"'
           }
     
           _ovs-set-other-config() {
             if [ "$(_ovs-get-other-config "$1")" != "$2" ]; then
               _ovs-vsctl set Open_vSwitch . "other_config:$1=$2"
               restart_ovs=true
             fi
           }
     
           _ovs-remove-other-config() {
             if [ -n "$(_ovs-get-other-config "$1")" ]; then
               _ovs-vsctl remove Open_vSwitch . other_config "$1"
               restart_ovs=true
             fi
           }
     
           _ovs-set-other-config doca-init true
           _ovs-set-other-config dpdk-max-memzones 50000
           _ovs-set-other-config hw-offload true
           _ovs-set-other-config pmd-quiet-idle true
           _ovs-set-other-config max-idle 20000
           _ovs-set-other-config max-revalidator 5000
           _ovs-set-other-config doca-congestion-threshold 60
           _ovs-set-other-config flow-limit 500000
           _ovs-set-other-config hw-offload-ct-unidir-udp-enabled true
           _ovs-remove-other-config default-datapath-type
     
           if [ "$restart_ovs" = true ]; then
             if systemctl list-unit-files openvswitch-switch.service &>/dev/null; then
               systemctl restart openvswitch-switch
             elif systemctl list-unit-files openvswitch.service &>/dev/null; then
               systemctl restart openvswitch
             fi
           fi
     
           _ovs-vsctl --may-exist add-br br-sfc
           _ovs-vsctl set bridge br-sfc datapath_type=netdev
           _ovs-vsctl set bridge br-sfc fail_mode=secure
           _ovs-vsctl --if-exists del-br br-hbn
           _ovs-vsctl --may-exist add-br br-hbn
           _ovs-vsctl set bridge br-hbn datapath_type=netdev
           _ovs-vsctl set bridge br-hbn fail_mode=secure
           _ovs-vsctl --may-exist add-port br-sfc p0
           _ovs-vsctl set Interface p0 type=dpdk
           _ovs-vsctl set Interface p0 mtu_request=9216
           _ovs-vsctl set Port p0 external_ids:dpf-type=physical
     
           # Activate DOCA for OVNK
           _ovs-vsctl set Open_vSwitch . external-ids:ovn-bridge-datapath-type=netdev
           # setup ovnkube managed bridge, br-dpu (this corresponds to br-ex on ovnk docs)
           _ovs-vsctl --may-exist add-br br-dpu
           _ovs-vsctl br-set-external-id br-dpu bridge-id br-dpu
           _ovs-vsctl br-set-external-id br-dpu bridge-uplink pbrdputobrovn
           _ovs-vsctl set bridge br-dpu datapath_type=netdev
           _ovs-vsctl set Interface br-dpu mtu_request=9000
           _ovs-vsctl --may-exist add-port br-dpu pf0hpf
           _ovs-vsctl set Interface pf0hpf type=dpdk
     
           # Create OVS bridge (br-ovn) in between the SC managed bridge and OVNK
           _ovs-vsctl --may-exist add-br br-ovn
           _ovs-vsctl set bridge br-ovn datapath_type=netdev
           _ovs-vsctl set Interface br-ovn mtu_request=9000
           _ovs-vsctl --may-exist add-port br-ovn pbrovntobrdpu
           _ovs-vsctl --may-exist add-port br-dpu pbrdputobrovn
     
           # Patch br-ovn and br-dpu together
           _ovs-vsctl set Interface pbrovntobrdpu type=patch options:peer=pbrdputobrovn
           _ovs-vsctl set Interface pbrdputobrovn type=patch options:peer=pbrovntobrdpu

2. Apply the resource file:

   $ envsubst < dpuflavor.yaml | oc apply -f -

Procedure

1. Create a file named bfb.yaml with the following content:

   apiVersion: provisioning.dpu.nvidia.com/v1alpha1
   kind: BFB
   metadata:
     name: bf-bundle
     namespace: dpf-operator-system
   spec:
     fileName: $BFB_FILENAME
     url: $BFB_URL
     versions:
       atf: 4.15.0-4-g419fbf393
       bsp: 4.15.0.13998
       doca: 3.4.1
       uefi: 4.15.0-19-g37c6f5adb2

2. Set the BFB_FILENAME environment variable to the file name of the BFB image, which is the base name of BFB_URL:

   $ export BFB_FILENAME=$(basename "$BFB_URL")

3. Apply the resource file:

   $ envsubst < bfb.yaml | oc apply -f -

Procedure

1. Create a file named dpudeployment.yaml with the following content:

   apiVersion: svc.dpu.nvidia.com/v1alpha1
   kind: DPUDeployment
   metadata:
     name: dpudeployment
     namespace: dpf-operator-system
   spec:
     dpus:
       nodeEffect:
         drain: true
       dpuSetStrategy:
         type: RollingUpdate
       bfb: bf-bundle
       flavor: hbn-ovnk
       dpuSets:
         - nameSuffix: "dpuset1"
           dpuNodeSelector:
             matchLabels:
               feature.node.kubernetes.io/dpu-enabled: ""
           dpuAnnotations:
             noderesources.dpu.nvidia.com/nodesriovdevicepluginconfig: bf3-vfs
     services:
       hbn:
         serviceTemplate: hbn
         serviceConfiguration: hbn
       ovn:
         serviceTemplate: ovn
         serviceConfiguration: ovn
       doca-telemetry-service:
         serviceTemplate: doca-telemetry-service
         serviceConfiguration: doca-telemetry-service
     serviceChains:
       switches:
         - ports:
             - serviceInterface:
                 matchLabels:
                   uplink: p0
             - service:
                 name: hbn
                 interface: p0_if
         - ports:
             - serviceInterface:
                 matchLabels:
                   uplink: p1
             - service:
                 name: hbn
                 interface: p1_if
         - ports:
             - serviceInterface:
                 matchLabels:
                   port: ovn
             - service:
                 name: hbn
                 interface: pf2dpu2_if

2. Apply the resource file:

   $ oc apply -f dpudeployment.yaml

Procedure

1. Create a file named hbn.yaml with the following content:

   apiVersion: svc.dpu.nvidia.com/v1alpha1
   kind: DPUServiceConfiguration
   metadata:
     name: hbn
     namespace: dpf-operator-system
   spec:
     deploymentServiceName: "hbn"
     serviceConfiguration:
       serviceDaemonSet:
         annotations:
           k8s.v1.cni.cncf.io/networks: |-
             [
             {"name": "iprequest", "interface": "ip_lo", "cni-args": {"poolNames": ["loopback"], "poolType": "cidrpool"}},
             {"name": "iprequest", "interface": "ip_pf2dpu2", "cni-args": {"poolNames": ["pool1"], "poolType": "cidrpool", "allocateDefaultGateway": true}}
             ]
       helmChart:
         values:
           configuration:
             perDPUValuesYAML: |
               - hostnamePattern: "*"
                 values:
                   bgp_peer_group: hbn
             startupYAMLJ2: |
               - header:
                   model: BLUEFIELD
                   nvue-api-version: nvue_v1
                   rev-id: 1.0
                   version: HBN 2.4.0
               - set:
                   interface:
                     lo:
                       ip:
                         address:
                           {{ ipaddresses.ip_lo.ip }}/32: {}
                       type: loopback
                     p0_if,p1_if:
                       type: swp
                       link:
                         mtu: 9216
                     pf2dpu2_if:
                       ip:
                         address:
                           {{ ipaddresses.ip_pf2dpu2.cidr }}: {}
                       type: swp
                       link:
                         mtu: 9216
                   router:
                     bgp:
                       autonomous-system: {{ ( ipaddresses.ip_lo.ip.split(".")[3] | int ) + 65101 }}
                       enable: on
                       graceful-restart:
                         mode: full
                       router-id: {{ ipaddresses.ip_lo.ip }}
                   vrf:
                     default:
                       router:
                         bgp:
                           address-family:
                             ipv4-unicast:
                               enable: on
                               redistribute:
                                 connected:
                                   enable: on
                             ipv6-unicast:
                               enable: on
                               redistribute:
                                 connected:
                                   enable: on
                           enable: on
                           neighbor:
                             p0_if:
                               peer-group: {{ config.bgp_peer_group }}
                               type: unnumbered
                             p1_if:
                               peer-group: {{ config.bgp_peer_group }}
                               type: unnumbered
                           path-selection:
                             multipath:
                               aspath-ignore: on
                           peer-group:
                             {{ config.bgp_peer_group }}:
                               remote-as: external
     interfaces:
       - name: p0_if
         network: mybrhbn
       - name: p1_if
         network: mybrhbn
       - name: pf2dpu2_if
         network: mybrhbn

2. Apply the resource file:

   $ oc apply -f hbn.yaml

   Example output

   dpuserviceconfiguration.svc.dpu.nvidia.com/hbn created

Procedure

1. Create a file named ovn-k.yaml with the following content:

   apiVersion: svc.dpu.nvidia.com/v1alpha1
   kind: DPUServiceConfiguration
   metadata:
     name: ovn
     namespace: dpf-operator-system
   spec:
     deploymentServiceName: "ovn"
     serviceConfiguration:
       helmChart:
         values:
           global:
             enableOvnKubeIdentity: false
           k8sAPIServer: https://$HOST_CLUSTER_API:6443
           podNetwork: 10.128.0.0/14/23
           serviceNetwork: 172.30.0.0/16
           hostNetworkNamespace: "openshift-host-network"
           mtu: $OVN_MTU
           dpuManifests:
             kubernetesSecretName: "ovn-dpu"
             vtepCIDR: $VTEP_CIDR
             hostCIDR: $DPU_HOST_CIDR
             ipamPool: "pool1"
             ipamPoolType: "cidrpool"
             ipamVTEPIPIndex: 0
             ipamPFIPIndex: 1
             cniBinDir: "/var/lib/cni/bin/"
             cniConfDir: "/run/multus/cni/net.d"

2. Apply the resource file:

   $ envsubst < ovn-k.yaml | oc apply -f -

Procedure

1. Create a file named dts.yaml with the following content:

   apiVersion: svc.dpu.nvidia.com/v1alpha1
   kind: DPUServiceConfiguration
   metadata:
     name: doca-telemetry-service
     namespace: dpf-operator-system
   spec:
     deploymentServiceName: "doca-telemetry-service"
     serviceConfiguration:
       configPorts:
         ports:
           - name: httpserverport
             port: 9189
             protocol: TCP
         serviceType: None

2. Apply the resource file:

   $ oc apply -f dts.yaml

Procedure

1. Create a file named dpucredentialreq.yaml with the following content:

   apiVersion: svc.dpu.nvidia.com/v1alpha1
   kind: DPUServiceCredentialRequest
   metadata:
     name: ovn-dpu
     namespace: dpf-operator-system
   spec:
     serviceAccount:
       name: ovn-kubernetes-node-dpu-service
       namespace: openshift-ovn-kubernetes
     duration: 24h
     type: tokenFile
     secret:
       name: ovn-dpu
       namespace: dpf-operator-system
     metadata:
       labels:
         dpu.nvidia.com/image-pull-secret: ""
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: RoleBinding
   metadata:
     name: openshift-ovn-kubernetes-node-limited-dpu-service
     namespace: openshift-ovn-kubernetes
   roleRef:
     apiGroup: rbac.authorization.k8s.io
     kind: Role
     name: openshift-ovn-kubernetes-node-limited
   subjects:
   - kind: ServiceAccount
     name: ovn-kubernetes-node-dpu-service
     namespace: openshift-ovn-kubernetes
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRoleBinding
   metadata:
     name: ovn-kubernetes-node-limited-binding
   roleRef:
     apiGroup: rbac.authorization.k8s.io
     kind: ClusterRole
     name: openshift-ovn-kubernetes-node-limited
   subjects:
   - kind: ServiceAccount
     name: ovn-kubernetes-node-dpu-service
     namespace: openshift-ovn-kubernetes

2. Apply the resource file:

   $ oc apply -f dpucredentialreq.yaml

Procedure

1. Create a file named physical-if.yaml with the following content to define the physical DPU port interfaces:

   apiVersion: svc.dpu.nvidia.com/v1alpha1
   kind: DPUServiceInterface
   metadata:
     name: p0
     namespace: dpf-operator-system
   spec:
     template:
       spec:
         template:
           metadata:
             labels:
               uplink: "p0"
           spec:
             interfaceType: physical
             physical:
               interfaceName: p0
   ---
   apiVersion: svc.dpu.nvidia.com/v1alpha1
   kind: DPUServiceInterface
   metadata:
     name: p1
     namespace: dpf-operator-system
   spec:
     template:
       spec:
         template:
           metadata:
             labels:
               uplink: "p1"
           spec:
             interfaceType: physical
             physical:
               interfaceName: p1

2. Apply the physical interface resource file:

   $ oc apply -f physical-if.yaml

3. Create a file named ovnk-if.yaml with the following content to define the OVN-Kubernetes interface:

   apiVersion: svc.dpu.nvidia.com/v1alpha1
   kind: DPUServiceInterface
   metadata:
     name: ovn
     namespace: dpf-operator-system
   spec:
     template:
       spec:
         template:
           metadata:
             labels:
               port: ovn
           spec:
             interfaceType: ovn

4. Apply the OVN-Kubernetes interface resource file:

   $ oc apply -f ovnk-if.yaml

Procedure

1. Create a file named dpuservice-nad.yaml with the following content:

   apiVersion: svc.dpu.nvidia.com/v1alpha1
   kind: DPUServiceNAD
   metadata:
     name: mybrhbn
     namespace: dpf-operator-system
   spec:
     resourceType: sf
     ipam: false
     bridge: "br-hbn"
     serviceMTU: $NODES_MTU

2. Apply the resource file:

   $ envsubst < dpuservice-nad.yaml | oc apply -f -

Procedure

1. Create a file named dpuservice-ipam.yaml with the following content:

   ---
   apiVersion: svc.dpu.nvidia.com/v1alpha1
   kind: DPUServiceIPAM
   metadata:
     name: pool1
     namespace: dpf-operator-system
   spec:
     ipv4Network:
       network: $VTEP_CIDR
       gatewayIndex: 3
       prefixSize: 29
   ---
   apiVersion: svc.dpu.nvidia.com/v1alpha1
   kind: DPUServiceIPAM
   metadata:
     name: loopback
     namespace: dpf-operator-system
   spec:
     ipv4Network:
       network: "11.0.0.0/24"
       prefixSize: 32

2. Apply the resource file:

   $ envsubst < dpuservice-ipam.yaml | oc apply -f -

Procedure

1. Set the OPENSHIFT_PULL_SECRET environment variable to the path of your pull secret file:

   $ export OPENSHIFT_PULL_SECRET="/root/pull-secret.txt"

2. Install the operator by using Helm:

   $ helm upgrade --install dpf-hcp-provisioner-operator \
       oci://registry.redhat.io/dpu-kit-for-nvidia/dpf-hcp-provisioner-chart \
       --registry-config "${OPENSHIFT_PULL_SECRET}" \
       --version 4.22.0 \
       --namespace dpf-hcp-provisioner-system \
       --create-namespace \
       --set provisionerConfig.manageDPUServiceTemplates=true

   Example output

   NAME: dpf-hcp-provisioner-operator
   LAST DEPLOYED: ...
   NAMESPACE: dpf-hcp-provisioner-system
   STATUS: deployed

   Note

   The Helm chart creates a DPFHCPProvisionerConfig singleton custom resource named default that defines the Operator-wide configuration. This resource controls settings such as the BlueField OpenShift Container Platform layer image repository, MetalLB integration, and DPUServiceTemplate management. To customize these settings, modify the provisionerConfig section in your Helm values.yaml file before running the helm upgrade --install command.

Procedure

1. Create the clusters namespace:

   $ oc create namespace $CLUSTERS_NAMESPACE

2. Create the pull secret:

   $ oc create secret generic $PULL_SECRET_NAME \
       --from-file=.dockerconfigjson=$OPENSHIFT_PULL_SECRET \
       --type=Opaque \
       -n $CLUSTERS_NAMESPACE

3. Create the SSH key secret:

   $ oc create secret generic $SSH_KEY_SECRET_NAME \
       --from-file=id_rsa.pub=$SSH_KEY \
       --type=Opaque \
       -n $CLUSTERS_NAMESPACE

   Note

   The id_rsa.pub secret data key is a fixed name that the provisioner expects and it does not require an RSA key. The SSH_KEY variable can point to any supported public key file, such as an Ed25519 key.

Procedure

1. Create a file named dpucluster.yaml with the following content:

   apiVersion: provisioning.dpu.nvidia.com/v1alpha1
   kind: DPUCluster
   metadata:
     name: $HOSTED_CLUSTER_NAME
     namespace: dpf-operator-system
   spec:
     type: static
     maxNodes: 10

2. Apply the resource with variable substitution:

   $ envsubst < dpucluster.yaml | oc apply -f -

Procedure

1. Create a file named dpfhcpprovisioner.yaml with the following content:

   apiVersion: provisioning.dpu.hcp.io/v1alpha1
   kind: DPFHCPProvisioner
   metadata:
     name: $HOSTED_CLUSTER_NAME
     namespace: $CLUSTERS_NAMESPACE
   spec:
     baseDomain: $BASE_DOMAIN
     dpuClusterRef:
       name: $HOSTED_CLUSTER_NAME
       namespace: dpf-operator-system
     dpuDeploymentRef:
       name: dpudeployment
       namespace: dpf-operator-system
     etcdStorageClass: $ETCD_STORAGE_CLASS
     ocpReleaseImage: $OCP_RELEASE_IMAGE
     pullSecretRef:
       name: $PULL_SECRET_NAME
     sshKeySecretRef:
       name: $SSH_KEY_SECRET_NAME
     virtualIP: $HOSTED_CLUSTER_VIP

   where:

   baseDomain

   Specifies the base DNS domain for the hosted cluster.

   dpuClusterRef

   Specifies a reference to the DPUCluster resource that represents the DPU hosted cluster.

   dpuDeploymentRef

   Specifies a reference to the DPUDeployment resource in the dpf-operator-system namespace.

   etcdStorageClass

   Specifies the storage class for etcd persistent volume claims.

   ocpReleaseImage

   Specifies the OpenShift Container Platform release image for the hosted cluster.

   pullSecretRef

   Specifies the name of the pull secret in the same namespace.

   sshKeySecretRef

   Specifies the name of the SSH key secret in the same namespace.

   virtualIP

   Specifies the virtual IP address for the hosted cluster API server, allocated from the management cluster subnet.

   The following optional fields use working defaults and do not need to be specified unless you want to override them:

   controlPlaneAvailabilityPolicy

   Specifies the availability policy for the hosted cluster control plane. This optional parameter defaults to HighlyAvailable. Set it to SingleReplica for a single-node control plane, in which case virtualIP is not required.

   flannelEnabled

   Specifies whether to enable Flannel networking in the hosted cluster. This optional parameter defaults to true.

   clusterNetwork

   Specifies the pod network CIDR for the hosted cluster. This optional parameter defaults to 10.128.0.0/14.

   serviceNetwork

   Specifies the service network CIDR for the hosted cluster. This optional parameter defaults to 172.30.0.0/16.

   machineNetwork

   Specifies the machine network CIDR for the hosted cluster. This optional parameter uses the management cluster network by default.

   nodeSelector

   Specifies a node selector for scheduling the hosted control plane pods. This optional parameter uses default scheduling by default.

2. Apply the resource with variable substitution:

   $ envsubst < dpfhcpprovisioner.yaml | oc apply -f -

Procedure

1. Monitor the DPFHCPProvisioner status:

   $ oc get dpfhcpprovisioner -n ${CLUSTERS_NAMESPACE}

   Example output

   NAME         PHASE          READY   HOSTEDCLUSTER   AGE
   dpf-hosted   Provisioning   False   dpf-hosted      2m

2. Wait for the DPFHCPProvisioner to reach the Ready phase:

   $ oc wait dpfhcpprovisioner ${HOSTED_CLUSTER_NAME} -n ${CLUSTERS_NAMESPACE} \
       --for=jsonpath='{.status.phase}'=Ready --timeout=30m

   Example output

   dpfhcpprovisioner.provisioning.dpu.hcp.io/dpf-hosted condition met

Procedure

1. Verify that the DPUService resources are created and reconciled:

   $ oc wait --for=condition=ApplicationsReconciled \
     --namespace dpf-operator-system dpuservices \
     -l svc.dpu.nvidia.com/owned-by-dpudeployment=dpf-operator-system_dpudeployment

2. Verify that the DPUServiceIPAM resources are reconciled:

   $ oc wait --for=condition=DPUIPAMObjectReconciled \
     --namespace dpf-operator-system dpuserviceipam --all

3. Verify that the DPUServiceInterface resources are reconciled:

   $ oc wait --for=condition=ServiceInterfaceSetReconciled \
     --namespace dpf-operator-system dpuserviceinterface --all

4. Verify that the DPUServiceChain resources are reconciled:

   $ oc wait --for=condition=ServiceChainSetReconciled \
     --namespace dpf-operator-system dpuservicechain --all

Procedure

1. Install the OVN-Kubernetes resource injector by using Helm:

   $ helm upgrade --install -n openshift-ovn-kubernetes ovn-kubernetes \
     "$OVN_TEMPLATE_CHART_URL/ovn-kubernetes-chart" \
     --version "${OVN_CHART_VERSION}" \
     --skip-crds \
     --set ovn-kubernetes-resource-injector.enabled=true \
     --set ovn-kubernetes-resource-injector.resourceName="openshift.io/bf3_vfs" \
     --set ovn-kubernetes-resource-injector.prioritizeOffloading=false \
     --set ovn-kubernetes-resource-injector.controllerManager.hostNetwork=true \
     --set ovn-kubernetes-resource-injector.controllerManager.webhookPort="19443" \
     --set ovn-kubernetes-resource-injector.controllerManager.healthProbeBindAddress=":18081" \
     --set ovn-kubernetes-resource-injector.controllerManager.webhook.image.pullPolicy=IfNotPresent \
     --set "ovn-kubernetes-resource-injector.controllerManager.webhook.args={--leader-elect,--metrics-bind-address=:29091}" \
     --set nodeWithDPUManifests.enabled=false \
     --set nodeWithoutDPUManifests.enabled=false \
     --set dpuManifests.enabled=false \
     --set controlPlaneManifests.enabled=false \
     --set commonManifests.enabled=false

   Example output

   NAME: ovn-kubernetes
   LAST DEPLOYED: Sun Nov  2 17:10:29 2025
   NAMESPACE: openshift-ovn-kubernetes
   STATUS: deployed
   REVISION: 1
   DESCRIPTION: Install complete
   TEST SUITE: None

Procedure

1. Set the following environment variables for the worker node:

   $ export BMC_IP=<bmc_ip_address>
   $ export BMC_USER=<bmc_username>
   $ export BMC_PASSWORD=<bmc_password>
   $ export WORKER_NAME=<worker_name>
   $ export BOOT_MAC=<management_interface_mac>
   $ export ROOT_DEVICE=<root_device_path>

   where:

   <bmc_ip_address>

   Specifies the IP address of the worker node BMC interface.

   <bmc_username>

   Specifies the username for BMC access.

   <bmc_password>

   Specifies the password for BMC access.

   <worker_name>

   Specifies a name for the worker node, such as worker-01.

   <management_interface_mac>

   Specifies the MAC address of the out-of-band management interface, such as 00:00:5E:00:53:01.

   <root_device_path>

   Specifies the path to the root disk device, such as /dev/nvme0n1.

2. Verify BMC connectivity from one of the control plane nodes:

   $ ping $BMC_IP

   $ curl -k https://$BMC_IP/redfish/v1/

   $ curl -k -u $BMC_USER:$BMC_PASSWORD https://$BMC_IP/redfish/v1/Systems

3. Verify that the Bare Metal Operator is available:

   $ oc get clusteroperator baremetal

4. Create a file named provisioning.yaml with the following content to disable the provisioning network:

   apiVersion: metal3.io/v1alpha1
   kind: Provisioning
   metadata:
     name: provisioning-configuration
   spec:
     provisioningNetwork: "Disabled"
     watchAllNamespaces: false

   Important

   When provisioningNetwork is set to Disabled, servers boot by using Redfish virtual media instead of PXE.

5. Apply the Provisioning resource:

   $ oc apply -f provisioning.yaml

6. Create a file named bmc-secret.yaml with the following content to store the BMC credentials:

   apiVersion: v1
   kind: Secret
   metadata:
     name: ${WORKER_NAME}-bmc-secret
     namespace: openshift-machine-api
   type: Opaque
   stringData:
     username: ${BMC_USER}
     password: ${BMC_PASSWORD}

7. Apply the BMC credentials secret:

   $ envsubst < bmc-secret.yaml | oc apply -f -

8. Create a file named baremetalhost.yaml. The userData secret determines the node type:

   • For a DPU-equipped worker node, reference the worker-dpu-user-data-managed secret:

     apiVersion: metal3.io/v1alpha1
     kind: BareMetalHost
     metadata:
       name: $WORKER_NAME
       namespace: openshift-machine-api
     spec:
       online: true
       bootMACAddress: $BOOT_MAC
       rootDeviceHints:
         deviceName: $ROOT_DEVICE
       bmc:
         address: redfish-virtualmedia+https://$BMC_IP
         credentialsName: $WORKER_NAME-bmc-secret
         disableCertificateVerification: true
       customDeploy:
         method: install_coreos
       userData:
         name: worker-dpu-user-data-managed
         namespace: openshift-machine-api

   • For a regular worker node without a DPU, reference the worker-user-data-managed secret instead:

     apiVersion: metal3.io/v1alpha1
     kind: BareMetalHost
     metadata:
       name: $WORKER_NAME
       namespace: openshift-machine-api
     spec:
       online: true
       bootMACAddress: $BOOT_MAC
       rootDeviceHints:
         deviceName: $ROOT_DEVICE
       bmc:
         address: redfish-virtualmedia+https://$BMC_IP
         credentialsName: $WORKER_NAME-bmc-secret
         disableCertificateVerification: true
       customDeploy:
         method: install_coreos
       userData:
         name: worker-user-data-managed
         namespace: openshift-machine-api

     Important

     Adding a regular worker node without a DPU is a Technology Preview feature.

9. Apply the BareMetalHost resource:

   $ envsubst < baremetalhost.yaml | oc apply -f -

Procedure

1. Watch for pending CSRs:

   $ oc get csr -w

2. Approve all pending CSRs:

   $ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs oc adm certificate approve

   Example output

   certificatesigningrequest.certificates.k8s.io/csr-27bgq approved
   certificatesigningrequest.certificates.k8s.io/csr-69g65 approved
   certificatesigningrequest.certificates.k8s.io/csr-7r862 approved
   certificatesigningrequest.certificates.k8s.io/csr-f5vk7 approved

   Repeat this step until no pending CSRs remain. Each node typically generates multiple CSRs.

3. Verify that the worker nodes joined the cluster:

   $ oc get nodes

   Example output

   NAME               STATUS     ROLES                         AGE     VERSION
   host-worker1       NotReady   worker                        68s     v1.35.6
   host-worker2       NotReady   worker                        75s     v1.35.6
   master-0           Ready      control-plane,master,worker   4d22h   v1.35.6
   master-1           Ready      control-plane,master,worker   4d21h   v1.35.6
   master-2           Ready      control-plane,master,worker   4d22h   v1.35.6

   Note

   The worker nodes show a status of NotReady until the DPU provisioning process is fully completed and all OVN-Kubernetes CNI components on the host and the DPU are running. Do not proceed to the next steps until all pending CSRs are approved.

Procedure

1. Watch for DPU object creation:

   $ oc get dpu -n dpf-operator-system -w

   Example output

   NAME                     READY   OPERATIONAL   PHASE                        AGE
   <node-name>-<dpu-id>             Unknown       Node Effect                  25s
   <node-name>-<dpu-id>             Unknown       Initialize Interface         26s
   <node-name>-<dpu-id>             Unknown       Config FW Parameters         28s
   <node-name>-<dpu-id>             Unknown       Prepare BFB                  28s
   <node-name>-<dpu-id>             Unknown       OS Installing                5m28s
   <node-name>-<dpu-id>             Unknown       DPU Config                   15m
   <node-name>-<dpu-id>             Unknown       Rebooting                    26m
   <node-name>-<dpu-id>             Unknown       Host Network Configuration   27m
   <node-name>-<dpu-id>             False         DPU Cluster Config           64m
   <node-name>-<dpu-id>             Unknown       Node Effect Removal          71m
   <node-name>-<dpu-id>     True    True          Ready                        71m

   The DPU objects progress through the following provisioning stages:

   Initializing

   The DPU object is created.

   OS Installing

   The BFB installation is in progress.

   Rebooting

   The host and DPU are resetting.

   DPU Cluster Config

   The DPU Kubernetes node join procedure is in progress. Manual CSR approval is required during this stage.

   Host Network Configuration

   Networking configuration adjustments are applied on the host.

   Ready

   The DPU is successfully provisioned and ready to use.

   Error

   Provisioning failed. Check events and conditions for details.

   Important

   When the provisioning stage reaches DPU Cluster Config, proceed to "Configure authorization for the hosted cluster" and "Approve DPU node CSRs" to complete the DPU node join process.

2. Monitor detailed provisioning progress:

   $ oc -n dpf-operator-system exec deploy/dpf-operator-controller-manager -- /dpfctl describe dpudeployments

3. Optional: View detailed status for a specific DPU object:

   In the following command, replace <dpu_name> with the name of the DPU resource:

   $ oc describe dpu -n dpf-operator-system <dpu_name>

4. Optional: Follow the provisioning controller logs for a specific DPU:

   In the following command, replace <dpu_name> with the name of the DPU resource:

   $ oc logs -n dpf-operator-system -l dpu.nvidia.com/component=dpf-provisioning-controller-manager --tail=-1 -f | grep <dpu_name>

Procedure

1. Get the hosted cluster kubeconfig:

   $ oc get secret $HOSTED_CLUSTER_NAME-admin-kubeconfig -n $CLUSTERS_NAMESPACE -o jsonpath='{.data.kubeconfig}' | base64 -d > $HOSTED_CLUSTER_NAME.kubeconfig

2. Switch to the hosted cluster context:

   $ export KUBECONFIG=$HOSTED_CLUSTER_NAME.kubeconfig

3. Create a file named dpu-cluster-scc.yaml with the following content:

   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRoleBinding
   metadata:
     name: dpf-system-scc-privileged
     labels:
       app.kubernetes.io/component: rbac
       app.kubernetes.io/part-of: dpu-services
   roleRef:
     apiGroup: rbac.authorization.k8s.io
     kind: ClusterRole
     name: system:openshift:scc:privileged
   subjects:
   - kind: Group
     apiGroup: rbac.authorization.k8s.io
     name: system:serviceaccounts:dpf-operator-system

4. Apply the resource file on the hosted cluster:

   $ oc apply -f dpu-cluster-scc.yaml

Procedure

1. Watch for pending CSRs from the DPU nodes:

   $ oc get csr -w

   The DPU node name typically follows the pattern <host_worker_node_name>-<dpu_serial_number>.

2. Approve all pending CSRs:

   $ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs oc adm certificate approve

   Example output

   certificatesigningrequest.certificates.k8s.io/csr-6jx22 approved
   certificatesigningrequest.certificates.k8s.io/csr-tb6nd approved

   Repeat this step until no pending CSRs remain.

Procedure

1. Switch back to the management cluster context:

   $ export KUBECONFIG="$(pwd)/mgmt-kubeconfig"

2. Verify that all worker nodes are in a Ready state:

   $ oc get node

   Example output

   NAME               STATUS   ROLES                         AGE     VERSION
   host-worker1       Ready    worker                        57m     v1.35.6
   host-worker2       Ready    worker                        57m     v1.35.6
   master-0           Ready    control-plane,master,worker   4d23h   v1.35.6
   master-1           Ready    control-plane,master,worker   4d22h   v1.35.6
   master-2           Ready    control-plane,master,worker   4d23h   v1.35.6

3. Verify that SR-IOV virtual functions are registered as Kubernetes node resources on the worker nodes:

   $ oc get nodes -l 'node-role.kubernetes.io/worker,!node-role.kubernetes.io/control-plane' -o json | \
     jq '.items[] | {name: .metadata.name, capacity: .status.capacity."openshift.io/bf3_vfs", allocatable: .status.allocatable."openshift.io/bf3_vfs"}'

   Example output

   {
     "name": "host-worker1",
     "capacity": "90",
     "allocatable": "90"
   }
   {
     "name": "host-worker2",
     "capacity": "90",
     "allocatable": "90"
   }

4. Verify that all DPU services are in a Success phase:

   $ oc get dpuservices -n dpf-operator-system

   Example output

   NAME                            READY   PHASE     AGE
   doca-telemetry-service-7s8pb    True    Success   42m
   flannel                         True    Success   26h
   hbn-gffmv                       True    Success   25m
   kube-state-metrics-rbac         True    Success   4h10m
   node-problem-detector           True    Success   4h10m
   nvidia-k8s-ipam-node            True    Success   4h10m
   ovn-f49zx                       True    Success   17m
   ovs-cni                         True    Success   26h
   servicechainset-rbac-and-crds   True    Success   138m
   sfc-controller                  True    Success   26h
   sriov-device-plugin             True    Success   26h

5. Optional: View detailed DPU service status:

   $ oc -n dpf-operator-system exec deploy/dpf-operator-controller-manager -- /dpfctl describe all --show-resources=dpuservice --grouping=false

Procedure

1. Create a file named traffic-pods.yaml with the following content:

   # 1. Namespace
   ---
   apiVersion: v1
   kind: Namespace
   metadata:
     name: workload
   
   # 2. SCC RoleBinding (grants 'default' ServiceAccount in 'workload' NS access to 'privileged' SCC)
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: RoleBinding
   metadata:
     name: privileged-scc-default-sa
     namespace: workload
   subjects:
   - kind: ServiceAccount
     name: default
     namespace: workload
   roleRef:
     kind: ClusterRole
     name: system:openshift:scc:privileged
     apiGroup: rbac.authorization.k8s.io
   
   # 3. Deployments and Services
   # Deployment: traffic-test-master
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: traffic-test-master
     namespace: workload
     labels:
       app: traffic-test-master
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: traffic-test-master
     template:
       metadata:
         labels:
           app: traffic-test-master
       spec:
         topologySpreadConstraints:
         - maxSkew: 1
           topologyKey: kubernetes.io/hostname
           whenUnsatisfiable: DoNotSchedule
           labelSelector:
             matchLabels:
               app: traffic-test-master
         nodeSelector:
           node-role.kubernetes.io/control-plane: ""
         tolerations:
         - key: node-role.kubernetes.io/master
           operator: Exists
           effect: NoSchedule
         - key: node-role.kubernetes.io/control-plane
           operator: Exists
           effect: NoSchedule
         containers:
         - name: nginx
           securityContext:
             privileged: true
             capabilities:
               add:
               - NET_ADMIN
           image: nicolaka/netshoot
           command: ["nc", "-kl", "5000"]
           ports:
           - containerPort: 5000
             name: tcp-server
           resources:
             requests:
               cpu: 1
               memory: 1Gi
             limits:
               cpu: 1
               memory: 1Gi
   ---
   # Service: traffic-test-master
   apiVersion: v1
   kind: Service
   metadata:
     name: traffic-test-master
     namespace: workload
     labels:
       app: traffic-test-master
   spec:
     selector:
       app: traffic-test-master
     ports:
     - protocol: TCP
       port: 5000
       targetPort: 5000
   ---
   # Service: traffic-test-master-nodeport
   apiVersion: v1
   kind: Service
   metadata:
     name: traffic-test-master-nodeport
     namespace: workload
     labels:
       app: traffic-test-master
   spec:
     type: NodePort
     selector:
       app: traffic-test-master
     ports:
     - protocol: TCP
       port: 5000
       targetPort: 5000
   ---
   # Deployment: traffic-test-worker
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: traffic-test-worker
     namespace: workload
     labels:
       app: traffic-test-worker
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: traffic-test-worker
     template:
       metadata:
         labels:
           app: traffic-test-worker
       spec:
         topologySpreadConstraints:
         - maxSkew: 1
           topologyKey: kubernetes.io/hostname
           whenUnsatisfiable: DoNotSchedule
           labelSelector:
             matchLabels:
               app: traffic-test-worker
         nodeSelector:
           feature.node.kubernetes.io/dpu-enabled: ""
         containers:
         - name: nginx
           securityContext:
             privileged: true
             capabilities:
               add:
               - NET_ADMIN
           image: nicolaka/netshoot
           command: ["nc", "-kl", "5000"]
           ports:
           - containerPort: 5000
             name: tcp-server
           resources:
             requests:
               cpu: 16
               memory: 6Gi
             limits:
               cpu: 16
               memory: 6Gi
   ---
   # Service: traffic-test-worker
   apiVersion: v1
   kind: Service
   metadata:
     name: traffic-test-worker
     namespace: workload
     labels:
       app: traffic-test-worker
   spec:
     selector:
       app: traffic-test-worker
     ports:
     - protocol: TCP
       port: 5000
       targetPort: 5000
   ---
   # Service: traffic-test-worker-nodeport
   apiVersion: v1
   kind: Service
   metadata:
     name: traffic-test-worker-nodeport
     namespace: workload
     labels:
       app: traffic-test-worker
   spec:
     type: NodePort
     selector:
       app: traffic-test-worker
     ports:
     - protocol: TCP
       port: 5000
       targetPort: 5000
   ---
   # Deployment: traffic-test-worker-hostnetwork
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: traffic-test-worker-hostnetwork
     namespace: workload
     labels:
       app: traffic-test-worker-hostnetwork
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: traffic-test-worker-hostnetwork
     template:
       metadata:
         labels:
           app: traffic-test-worker-hostnetwork
       spec:
         topologySpreadConstraints:
         - maxSkew: 1
           topologyKey: kubernetes.io/hostname
           whenUnsatisfiable: DoNotSchedule
           labelSelector:
             matchLabels:
               app: traffic-test-worker-hostnetwork
         nodeSelector:
           feature.node.kubernetes.io/dpu-enabled: ""
         hostNetwork: true
         containers:
         - name: nginx
           securityContext:
             privileged: true
             capabilities:
               add:
               - NET_ADMIN
           image: nicolaka/netshoot
           command: ["nc", "-kl", "5000"]
           ports:
           - containerPort: 5000
             name: tcp-server
           resources:
             requests:
               cpu: 1
               memory: 1Gi
             limits:
               cpu: 1
               memory: 1Gi
   ---
   # Service: traffic-test-worker-hostnetwork
   apiVersion: v1
   kind: Service
   metadata:
     name: traffic-test-worker-hostnetwork
     namespace: workload
     labels:
       app: traffic-test-worker-hostnetwork
   spec:
     selector:
       app: traffic-test-worker-hostnetwork
     ports:
     - protocol: TCP
       port: 5000
       targetPort: 5000
   ---
   # Service: traffic-test-worker-hostnetwork-nodeport
   apiVersion: v1
   kind: Service
   metadata:
     name: traffic-test-worker-hostnetwork-nodeport
     namespace: workload
     labels:
       app: traffic-test-worker-hostnetwork
   spec:
     type: NodePort
     selector:
       app: traffic-test-worker-hostnetwork
     ports:
     - protocol: TCP
       port: 5000
       targetPort: 5000

   The manifest creates the following resources:

   • A workload namespace for the test pods.
   • A RoleBinding resource that grants the default service account in the workload namespace access to the privileged security context constraint.
   • A traffic-test-master deployment and ClusterIP and NodePort services on a control plane node.
   • A traffic-test-worker deployment and ClusterIP and NodePort services on DPU-enabled worker nodes.

   • A traffic-test-worker-hostnetwork deployment that uses host networking on DPU-enabled worker nodes, with ClusterIP and NodePort services.

     Note

     The worker deployments set replicas: 1 for a single DPU worker node. Set the replica count of the traffic-test-worker and traffic-test-worker-hostnetwork deployments to the number of DPU-enabled worker nodes so that the topologySpreadConstraints place one pod on each node.

2. Apply the manifest:

   $ oc apply -f traffic-pods.yaml