Chapter 10. Creating the data plane for SR-IOV and DPDK environments

Procedure

1. For unprovisioned nodes, create the SSH key pair for Ansible:

   $ ssh-keygen -f <key_file_name> -N "" -t rsa -b 4096

   • Replace <key_file_name> with the name to use for the key pair.

2. Create the Secret CR for Ansible and apply it to the cluster:

   $ oc create secret generic dataplane-ansible-ssh-private-key-secret \
   --save-config \
   --dry-run=client \
   --from-file=ssh-privatekey=<key_file_name> \
   --from-file=ssh-publickey=<key_file_name>.pub \
   [--from-file=authorized_keys=<key_file_name>.pub] -n openstack \
   -o yaml | oc apply -f -

   • Replace <key_file_name> with the name and location of your SSH key pair file.
   • Optional: Only include the --from-file=authorized_keys option for bare-metal nodes that must be provisioned when creating the data plane.

3. If you are creating Compute nodes, create a secret for migration.

   1. Create the SSH key pair for instance migration:

      $ ssh-keygen -f ./nova-migration-ssh-key -t ecdsa-sha2-nistp521 -N ''

   2. Create the Secret CR for migration and apply it to the cluster:

      $ oc create secret generic nova-migration-ssh-key \
      --save-config \
      --from-file=ssh-privatekey=nova-migration-ssh-key \
      --from-file=ssh-publickey=nova-migration-ssh-key.pub \
      -n openstack \
      -o yaml | oc apply -f -

4. For nodes that have not been registered to the Red Hat Customer Portal, create the Secret CR for subscription-manager credentials to register the nodes:

   $ oc create secret generic subscription-manager \
   --from-literal rhc_auth='{"login": {"username": "<subscription_manager_username>", "password": "<subscription_manager_password>"}}'

   • Replace <subscription_manager_username> with the username you set for subscription-manager.
   • Replace <subscription_manager_password> with the password you set for subscription-manager.

5. Create a Secret CR that contains the Red Hat registry credentials:

   $ oc create secret generic redhat-registry --from-literal edpm_container_registry_logins='{"registry.redhat.io": {"<username>": "<password>"}}'

   • Replace <username> and <password> with your Red Hat registry username and password credentials.

     For information about how to create your registry service account, see the Knowledge Base article Creating Registry Service Accounts.

6. If you are creating Compute nodes, create a secret for libvirt.

   1. Create a file on your workstation named secret_libvirt.yaml to define the libvirt secret:

      apiVersion: v1
      kind: Secret
      metadata:
       name: libvirt-secret
       namespace: openstack
      type: Opaque
      data:
       LibvirtPassword: <base64_password>

      • Replace <base64_password> with a base64-encoded string with maximum length 63 characters. You can use the following command to generate a base64-encoded password:

        $ echo -n <password> | base64

        Tip

        If you do not want to base64-encode the username and password, you can use the stringData field instead of the data field to set the username and password.

   2. Create the Secret CR:

      $ oc apply -f secret_libvirt.yaml -n openstack

7. Verify that the Secret CRs are created:

   $ oc describe secret dataplane-ansible-ssh-private-key-secret
   $ oc describe secret nova-migration-ssh-key
   $ oc describe secret subscription-manager
   $ oc describe secret redhat-registry
   $ oc describe secret libvirt-secret

Procedure

1. Create a ConfigMap CR that defines configurations for CPU pinning and PCI passthrough, and save it to a YAML file on your workstation, for example, pinning-passthrough.yaml.

   Change the values (in boldface) as appropriate for your environment:

   ---
   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cpu-pinning-nova
   data:
     25-cpu-pinning-nova.conf: |
       [DEFAULT]
       reserved_host_memory_mb = 4096
       [compute]
       cpu_shared_set = 0-3,24-27
       cpu_dedicated_set = 8-23,32-47
       [neutron]
       physnets = <network_name1>, <network_name2>
       [neutron_physnet_<network_name1>]
       numa_nodes = <ID list>
       [neutron_physnet_<network_name2>]
       numa_nodes = <ID list>
       [neutron_tunnel]
       numa_nodes = <ID list>
   ---
   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: sriov-nova
   data:
     26-sriov-nova.conf: |
       [libvirt]
       cpu_power_management=false
       [pci]
       passthrough_whitelist = {"address": "0000:05:00.2", "physical_network":"sriov-1", "trusted":"true"}
       passthrough_whitelist = {"address": "0000:05:00.3", "physical_network":"sriov-2", "trusted":"true"}
   ---

   • cpu_shared_set: enter a comma-separated list or range of physical host CPU numbers used to provide vCPU inventory, determine the host CPUs that unpinned instances can be scheduled to, and determine the host CPUs that instance emulator threads should be offloaded to for instances configured with the share emulator thread policy.
   • cpu_dedicated_set: enter a comma-separated list or range of physical host CPU numbers to which processes for pinned instance CPUs can be scheduled. For example, 4-12,^8,15 reserves cores from 4-12 and 15, excluding 8.
   • <network_name_n_>: replace <network_name1> and <network_name2> with the names of the physical networks that your gateways are on. (This network is set in the neutron network provider:*name field.)

   • numa_nodes = <ID list>: replace <ID list> with a comma-separated list of IDs of the NUMA nodes associated with this physnet. For example: 0,1. For example:

     [neutron]
     physnets = foo,bar
     
     [neutron_physnet_foo]
     numa_nodes = 0
     
     [neutron_physnet_bar]
     numa_nodes = 2, 3

     This configuration ensures that instances using one or more L2-type networks with provider:physical_network=foo must be scheduled on host cores from NUMA node 0, while instances using one or more networks with provider:physical_network=bar must be scheduled on host cores from both NUMA nodes 2 and 3. For the latter case, it will be necessary to split the guest across two or more host NUMA nodes using the hw:numa_nodes extra spec.

   • passthrough_whitelist: specify valid NIC addresses and names for "address" and "physical_network".

2. Create the ConfigMap object, using the ConfigMap CR file:

   Example

   $ oc create -f sriov-pinning-passthru.yaml -n openstack

3. Create an OpenStackDataPlaneService CR that defines the SR-IOV custom service, and save it to a YAML file on your workstation, for example nova-custom-sriov.yaml:

   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneService
   metadata:
     name: nova-custom-sriov

4. Add the ConfigMap CRs to the custom service, and specify the Secret CR for the cell that the node set that runs this service connects to:

   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneService
   metadata:
     name: nova-custom-sriov
   spec:
     label: dataplane-deployment-nova-custom-sriov
     dataSources:
       - configMapRef:
           name: cpu-pinning-nova
       - configMapRef:
           name: sriov-nova
       - secretRef:
           name: nova-cell1-compute-config
       - secretRef:
           name: nova-migration-ssh-key
     tlsCerts:
       default:
         contents:
           - dnsnames
           - ips
         networks:
           - ctlplane
         issuer: osp-rootca-issuer-internal
     caCerts: combined-ca-bundle

5. Specify the Ansible commands to create the custom service, by referencing an Ansible Playbook or by including the Ansible play in the playbookContents field:

   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneService
   metadata:
     name: nova-custom-sriov
   spec:
     label: dataplane-deployment-nova-custom-sriov
     edpmServiceType: nova
     dataSources:
       - configMapRef:
           name: cpu-pinning-nova
       - configMapRef:
           name: sriov-nova
       - secretRef:
           name: nova-cell1-compute-config
       - secretRef:
           name: nova-migration-ssh-key
     playbook: osp.edpm.nova
     tlsCerts:
       default:
         contents:
           - dnsnames
           - ips
         networks:
           - ctlplane
         issuer: osp-rootca-issuer-internal
     caCerts: combined-ca-bundle

   • playbook: identifies the default playbook available for your service.

     In this case, it is the Compute service (nova). To see the listing of default playbooks, see https://openstack-k8s-operators.github.io/edpm-ansible/playbooks.html.

6. Create the custom-nova-sriov service:

   $ oc apply -f nova-custom-sriov.yaml -n openstack

7. Verify that the custom service is created:

   $ oc get openstackdataplaneservice nova-custom-sriov -o yaml -n openstack

Procedure

1. Create a ConfigMap CR that defines a configuration for the parameters, and save it to a YAML file on your workstation, for example, dpdk-custom.yaml.

   Change the values (in boldface) as appropriate for your environment:

   ---
   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: dpdk-custom-nova
     namespace: openstack
   data:
     25-dpdk-custom-nova.conf: |
       [DEFAULT]
       reserved_host_memory_mb = 4096
       [compute]
       cpu_shared_set = 0-3,24-27
       cpu_dedicated_set = 8-23,32-47
       [neutron]
       physnets = <network_name1>, <network_name2>
       [neutron_physnet_<network_name1>]
       numa_nodes = <ID list>
       [neutron_physnet_<network_name2>]
       numa_nodes = <ID list>
       [neutron_tunnel]
       numa_nodes = <ID list>
       [libvirt]
       live_migration_permit_post_copy=false
   ---

   • cpu_shared_set: enter a comma-separated list or range of physical host CPU numbers used to provide vCPU inventory, determine the host CPUs that unpinned instances can be scheduled to, and determine the host CPUs that instance emulator threads should be offloaded to for instances configured with the share emulator thread policy.
   • cpu_dedicated_set: enter a comma-separated list or range of physical host CPU numbers to which processes for pinned instance CPUs can be scheduled. For example, 4-12,^8,15 reserves cores from 4-12 and 15, excluding 8.
   • <network_name_n_>: replace <network_name1> and <network_name2> with the names of the physical networks that your gateways are on, for which you need to configure NUMA affinity. (This network is set in the neutron network provider:*name field.)

   • <ID list>: replace <ID list> with a comma-separated list of IDs of the NUMA nodes associated with this physnet. For example: 0,1. For example:

     [neutron]
     physnets = foo,bar
     
     [neutron_physnet_foo]
     numa_nodes = 0
     
     [neutron_physnet_bar]
     numa_nodes = 2, 3

     This configuration ensures that instances using one or more L2-type networks with provider:physical_network=foo` must be scheduled on host cores from NUMA node 0, while instances using one or more networks with provider:physical_network=bar` must be scheduled on host cores from both NUMA nodes 2 and 3. For the latter case, it will be necessary to split the guest across two or more host NUMA nodes using the hw:numa_nodes extra` spec.

   • live_migration_permit_post_copy=false: necessary for successful block live migration of instances attached to a Geneve network with DPDK.

2. Create the ConfigMap object, using the ConfigMap CR file:

   Example

   $ oc create -f dpdk-custom.yaml -n openstack

3. Create an OpenStackDataPlaneService CR that defines the OVS-DPDK custom service, and save it to a YAML file on your workstation, for example nova-custom-ovsdpdk.yaml:

   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneService
   metadata:
     name: nova-custom-ovsdpdk
     namespace: openstack

4. Add the ConfigMap CR to the custom service, and specify the Secret CR for the cell that the node set that runs this service connects to:

   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneService
   metadata:
     name: nova-custom-ovsdpdk
     namespace: openstack
   spec:
     label: dataplane-deployment-nova-custom-ovsdpdk
     edpmServiceType: nova
     dataSources:
       - configMapRef:
           name: dpdk-custom-nova
       - secretRef:
           name: nova-cell1-compute-config
       - secretRef:
           name: nova-migration-ssh-key
     tlsCerts:
       default:
         contents:
           - dnsnames
           - ips
         networks:
           - ctlplane
         issuer: osp-rootca-issuer-internal
     caCerts: combined-ca-bundle

5. Specify the Ansible commands to create the custom service, by referencing an Ansible Playbook or by including the Ansible play in the playbookContents field:

   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneService
   metadata:
     name: nova-custom-ovsdpdk
     namespace: openstack
   spec:
     label: dataplane-deployment-nova-custom-ovsdpdk
     edpmServiceType: nova
     dataSources:
       - configMapRef:
           name: dpdk-custom-nova
       - secretRef:
           name: nova-cell1-compute-config
       - secretRef:
           name: nova-migration-ssh-key
     playbook: osp.edpm.nova
     tlsCerts:
       default:
         contents:
           - dnsnames
           - ips
         networks:
           - ctlplane
         issuer: osp-rootca-issuer-internal
     caCerts: combined-ca-bundle

   • playbook: identifies the default playbook available for your service.

     In this case, it is the Compute service (nova). To see the listing of default playbooks, see https://openstack-k8s-operators.github.io/edpm-ansible/playbooks.html.

6. Create the nova-custom-ovsdpdk service:

   $ oc apply -f nova-custom-ovsdpdk.yaml -n openstack

7. Verify that the custom service is created:

   $ oc get openstackdataplaneservice nova-custom-ovsdpdk -o yaml -n openstack

Procedure

1. Create a file on your workstation named openstack_preprovisioned_node_set.yaml to define the OpenStackDataPlaneNodeSet CR:

   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneNodeSet
   metadata:
     name: openstack-data-plane
     namespace: openstack
   spec:
     env:
       - name: ANSIBLE_FORCE_COLOR
         value: "True"

   • name - The OpenStackDataPlaneNodeSet CR name must be unique, contain only lower case alphanumeric characters and - (hyphens) or . (periods), start and end with an alphanumeric character, and have a maximum length of 53 characters. Update the name in this example to a name that reflects the nodes in the set.
   • env - Optional: a list of environment variables to pass to the pod.

2. Specify that the nodes in this set are pre-provisioned:

     preProvisioned: true

3. Add the SSH key secret that you created to enable Ansible to connect to the data plane nodes:

     nodeTemplate:
       ansibleSSHPrivateKeySecret: <secret-key>

   • Replace <secret-key> with the name of the SSH key Secret CR you created for this node set in Creating the data plane secrets, for example, dataplane-ansible-ssh-private-key-secret.
4. Create a Persistent Volume Claim (PVC) in the openstack namespace on your Red Hat OpenShift Container Platform (RHOCP) cluster to store logs. Set the volumeMode to Filesystem and accessModes to ReadWriteOnce. Do not request storage for logs from a PersistentVolume (PV) that uses the NFS volume plugin. NFS is incompatible with FIFO and the ansible-runner creates a FIFO file to write to store logs. For information about PVCs, see Understanding persistent storage in the RHOCP Storage guide and Red Hat OpenShift Container Platform cluster requirements in Planning your deployment.

5. Enable persistent logging for the data plane nodes:

     nodeTemplate:
       ansibleSSHPrivateKeySecret: <secret-key>
       extraMounts:
         - extraVolType: Logs
           volumes:
           - name: ansible-logs
             persistentVolumeClaim:
               claimName: <pvc_name>
           mounts:
           - name: ansible-logs
             mountPath: "/runner/artifacts"

   • Replace <pvc_name> with the name of the PVC storage on your RHOCP cluster.
6. Add the common configuration for the set of nodes in this group under the nodeTemplate section. Each node in this OpenStackDataPlaneNodeSet inherits this configuration. For information about the properties you can use to configure common node attributes, see OpenStackDataPlaneNodeSet CR spec properties.

7. Register the operating system of the nodes that are not registered to the Red Hat Customer Portal, and enable repositories for your nodes. The following steps demonstrate how to register your nodes to CDN. For details on how to register your nodes with Red Hat Satellite 6.13, see Managing Hosts.

   1. Create a Secret CR that contains the subscription-manager credentials:

      apiVersion: v1
      kind: Secret
      metadata:
        name: subscription-manager
      data:
        username: <base64_encoded_username>
        password: <base64_encoded_password>

   2. Create a Secret CR that contains the Red Hat registry credentials:

      $ oc create secret generic redhat-registry --from-literal edpm_container_registry_logins='{"registry.redhat.io": {"<username>": "<password>"}}'

      • Replace <username> and <password> with your Red Hat registry username and password credentials.

   For information about how to create your registry service account, see the Red Hat Knowledgebase article Creating Registry Service Accounts.

   1. Specify the Secret CRs to use to source the usernames and passwords:

        nodeTemplate:
          ansible:
            ...
            ansibleVarsFrom:
              - prefix: subscription_manager_
                secretRef:
                  name: subscription-manager
              - secretRef:
                  name: redhat-registry
            ansibleVars:
             rhc_release: 9.4
             rhc_repositories:
               - {name: "*", state: disabled}
               - {name: "<repository_name1>", state: enabled}
               - {name: "<repository_name2>", state: enabled}
               - {name: "<repository_name3>", state: enabled}
               - {name: "<repository_name4>", state: enabled}
               - {name: "<repository_name5>", state: enabled}
               - {name: "<repository_name6>", state: enabled}

      where:

      <repository_name1>, <repository_name2>, <repository_name3>, <repository_name4>, <repository_name5>, <repository_name6>

      Specifies the name of the repository to enable. For more information about which repositories are required, see Required repositories for Red Hat OpenStack Services on OpenShift 18.0.

8. Define each node in this node set:

     nodes:
       edpm-compute-0:
         hostName: edpm-compute-0
         networks:
           - name: ctlplane
             subnetName: subnet1
             defaultRoute: true
             fixedIP: 192.168.122.100
           - name: internalapi
             subnetName: subnet1
             fixedIP: 172.17.0.100
           - name: storage
             subnetName: subnet1
             fixedIP: 172.18.0.100
           - name: tenant
             subnetName: subnet1
             fixedIP: 172.19.0.100
         ansible:
           ansibleHost: 192.168.122.100
           ansibleUser: cloud-admin
           ansibleVars:
             fqdn_internal_api: edpm-compute-0.example.com
             edpm_network_config_nmstate: true
       edpm-compute-1:
         hostName: edpm-compute-1
         networks:
           - name: ctlplane
             subnetName: subnet1
             defaultRoute: true
             fixedIP: 192.168.122.101
           - name: internalapi
             subnetName: subnet1
             fixedIP: 172.17.0.101
           - name: storage
             subnetName: subnet1
             fixedIP: 172.18.0.101
           - name: tenant
             subnetName: subnet1
             fixedIP: 172.19.0.101
         ansible:
           ansibleHost: 192.168.122.101
           ansibleUser: cloud-admin
           ansibleVars:
             fqdn_internal_api: edpm-compute-1.example.com

   • edpm-compute-0 - The node definition reference, for example, edpm-compute-0. Each node in the node set must have a node definition.
   • networks - Defines the IPAM and the DNS records for the node.
   • fixedIP - Specifies a predictable IP address for the network that must be in the allocation range defined for the network in the NetConfig CR.

   • ansibleVars - Node-specific Ansible variables that customize the node.

     Note
     • Nodes defined within the nodes section can configure the same Ansible variables that are configured in the nodeTemplate section. Where an Ansible variable is configured for both a specific node and within the nodeTemplate section, the node-specific values override those from the nodeTemplate section.
     • You do not need to replicate all the nodeTemplate Ansible variables for a node to override the default and set some node-specific values. You only need to configure the Ansible variables you want to override for the node.
     • When you define the networkData Secret for an individual node (such as edpm-compute-0), it acts as a complete override rather than a supplemental configuration. Because node-specific configurations override the inherited default values from the nodeTemplate section, you must ensure that your node-specific networkData Secret contains the full set of required network configurations for that node, not just the unique values.
     • Many ansibleVars include edpm in the name, which stands for "External Data Plane Management".

   • edpm_network_config_nmstate - Sets the os-net-config provider to nmstate. The default value is true. Change it to false only if a specific limitation of the nmstate provider requires you to use the ifcfg provider. For more information on advantages and limitations of the nmstate provider, see https://docs.redhat.com/en/documentation/red_hat_openstack_services_on_openshift/18.0/html/planning_your_deployment/plan-networks_planning#plan-os-net-config_plan-network in Planning your deployment.

     For more information, see:

   • OpenStackDataPlaneNodeSet CR properties
   • Network interface configuration options
   • Example custom network interfaces for NFV

9. In the services section, add the list of services that will run on the data plane node. Ensure that you replace nova with nova-custom-sriov, or nova-custom-ovsdpdk, or both:

   ...
     services:
     - bootstrap
     - download-cache
     - reboot-os
     - configure-ovs-dpdk
     - configure-network
     - validate-network
     - install-os
     - configure-os
     - ssh-known-hosts
     - run-os
     - install-certs
     - ovn
     - neutron-ovn
     - neutron-ovn-igmp
     - neutron-metadata
     - libvirt
     - nova-custom-sriov
     - nova-custom-ovsdpdk
     - telemetry
   ...

10. Save the openstack_preprovisioned_node_set.yaml definition file.

11. Create the data plane resources:

    $ oc create -f openstack_preprovisioned_node_set.yaml -n openstack

12. Verify that the data plane resources have been created:

    $ oc get openstackdataplanenodeset -n openstack

    Sample output

    NAME           		STATUS MESSAGE
    openstack-data-plane 	False  Deployment not started

    For information about the meaning of the returned status, see Data plane conditions and states.

13. Verify that the Secret resource was created for the node set:

    $ oc get secret | grep openstack-data-plane

    Sample output

    dataplanenodeset-openstack-data-plane Opaque 1 3m50s

14. Verify the services were created:

    $ oc get openstackdataplaneservice -n openstack

    Sample output

    NAME                AGE
    configure-network   6d7h
    configure-os        6d6h
    install-os          6d6h
    run-os              6d6h
    validate-network    6d6h
    ovn                 6d6h
    libvirt             6d6h
    nova                6d6h
    telemetry           6d6h

Procedure

1. Create a file on your workstation named openstack_unprovisioned_node_set.yaml to define the OpenStackDataPlaneNodeSet CR:

   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneNodeSet
   metadata:
     name: openstack-data-plane
     namespace: openstack
   spec:
     tlsEnabled: true
     env:
       - name: ANSIBLE_FORCE_COLOR
         value: "True"

   • name - The OpenStackDataPlaneNodeSet CR name must be unique, contain only lower case alphanumeric characters and - (hyphens) or . (periods), start and end with an alphanumeric character, and have a maximum length of 53 characters. Update the name in this example to a name that reflects the nodes in the set.
   • env - Optional: a list of environment variables to pass to the pod.

2. Define the baremetalSetTemplate field to describe the configuration of the bare-metal nodes:

     preProvisioned: false
     baremetalSetTemplate:
       deploymentSSHSecret: dataplane-ansible-ssh-private-key-secret
       bmhNamespace: <bmh_namespace>
       cloudUserName: <ansible_ssh_user>
       bmhLabelSelector:
         app: <bmh_label>
       ctlplaneInterface: <interface>

   • Replace <bmh_namespace> with the namespace defined in the corresponding BareMetalHost CR for the node.
   • Replace <ansible_ssh_user> with the username of the Ansible SSH user.
   • Replace <bmh_label> with the metadata label defined in the corresponding BareMetalHost CR for the node, for example, openstack. Metadata labels, such as app, workload, and nodeName are key-value pairs for labelling nodes. Set the bmhLabelSelector field to select data plane nodes based on one or more labels that match the labels in the corresponding BareMetalHost CR.
   • Replace <interface> with the control plane interface the node connects to, for example, enp6s0.

3. The BMO manages BareMetalHost CRs in the openshift-machine-api namespace by default. You must update the Provisioning CR to watch all namespaces:

   $ oc patch provisioning provisioning-configuration --type merge -p '{"spec":{"watchAllNamespaces": true }}'

4. Add the SSH key secret that you created to enable Ansible to connect to the data plane nodes:

     nodeTemplate:
       ansibleSSHPrivateKeySecret: <secret-key>

   • Replace <secret-key> with the name of the SSH key Secret CR you created in Creating the data plane secrets, for example, dataplane-ansible-ssh-private-key-secret.
5. Create a Persistent Volume Claim (PVC) in the openstack namespace on your RHOCP cluster to store logs. Set the volumeMode to Filesystem and accessModes to ReadWriteOnce. Do not request storage for logs from a PersistentVolume (PV) that uses the NFS volume plugin. NFS is incompatible with FIFO and the ansible-runner creates a FIFO file to write to store logs. For information about PVCs, see Understanding persistent storage in the RHOCP Storage guide and Red Hat OpenShift Container Platform cluster requirements in Planning your deployment.

6. Enable persistent logging for the data plane nodes:

     nodeTemplate:
       ansibleSSHPrivateKeySecret: <secret-key>
       extraMounts:
         - extraVolType: Logs
           volumes:
           - name: ansible-logs
             persistentVolumeClaim:
               claimName: <pvc_name>
           mounts:
           - name: ansible-logs
             mountPath: "/runner/artifacts"

   • Replace <pvc_name> with the name of the PVC storage on your RHOCP cluster.

7. Add the common configuration for the set of nodes in this group under the nodeTemplate section. Each node in this OpenStackDataPlaneNodeSet inherits this configuration.

   For more information, see:

   • OpenStackDataPlaneNodeSet CR properties
   • Network interface configuration options
   • Example custom network interfaces for NFV

8. Define each node in this node set:

     nodes:
       edpm-compute-0:
         hostName: edpm-compute-0
         networks:
           - name: ctlplane
             subnetName: subnet1
             defaultRoute: true
             fixedIP: 192.168.122.100
           - name: internalapi
             subnetName: subnet1
           - name: storage
             subnetName: subnet1
           - name: tenant
             subnetName: subnet1
         ansible:
           ansibleHost: 192.168.122.100
           ansibleUser: cloud-admin
           ansibleVars:
             fqdn_internal_api: edpm-compute-0.example.com
             edpm_network_config_nmstate: true
       edpm-compute-1:
         hostName: edpm-compute-1
         networks:
           - name: ctlplane
             subnetName: subnet1
             defaultRoute: true
             fixedIP: 192.168.122.101
           - name: internalapi
             subnetName: subnet1
           - name: storage
             subnetName: subnet1
           - name: tenant
             subnetName: subnet1
         ansible:
           ansibleHost: 192.168.122.101
           ansibleUser: cloud-admin
           ansibleVars:
             fqdn_internal_api: edpm-compute-1.example.com

   • edpm-compute-0 - The node definition reference, for example, edpm-compute-0. Each node in the node set must have a node definition.
   • networks - Defines the IPAM and the DNS records for the node.
   • fixedIP - Specifies a predictable IP address for the network that must be in the allocation range defined for the network in the NetConfig CR.

   • ansibleVars - Node-specific Ansible variables that customize the node.

     Note
     • Nodes defined within the nodes section can configure the same Ansible variables that are configured in the nodeTemplate section. Where an Ansible variable is configured for both a specific node and within the nodeTemplate section, the node-specific values override those from the nodeTemplate section.
     • You do not need to replicate all the nodeTemplate Ansible variables for a node to override the default and set some node-specific values. You only need to configure the Ansible variables you want to override for the node.
     • When you define the networkData Secret for an individual node (such as edpm-compute-0), it acts as a complete override rather than a supplemental configuration. Because node-specific configurations override the inherited default values from the nodeTemplate section, you must ensure that your node-specific networkData Secret contains the full set of required network configurations for that node, not just the unique values.
     • Many ansibleVars include edpm in the name, which stands for "External Data Plane Management".

   • edpm_network_config_nmstate - Sets the os-net-config provider to nmstate. The default value is true. Change it to false only if a specific limitation of the nmstate provider requires you to use the ifcfg provider. For more information on advantages and limitations of the nmstate provider, see https://docs.redhat.com/en/documentation/red_hat_openstack_services_on_openshift/18.0/html/planning_your_deployment/plan-networks_planning#plan-os-net-config_plan-network in Planning your deployment.

     For information about the properties you can use to configure node attributes, see OpenStackDataPlaneNodeSet CR properties.

9. In the services section, add the list of services that will run on the data plane node. Ensure that you replace nova with nova-custom-sriov, or nova-custom-ovsdpdk, or both:

   ...
     services:
     - bootstrap
     - download-cache
     - reboot-os
     - configure-ovs-dpdk
     - configure-network
     - validate-network
     - install-os
     - configure-os
     - ssh-known-hosts
     - run-os
     - install-certs
     - ovn
     - neutron-ovn
     - neutron-ovn-igmp
     - neutron-metadata
     - libvirt
     - nova-custom-sriov
     - nova-custom-ovsdpdk
     - telemetry
   ...

10. Save the openstack_unprovisioned_node_set.yaml definition file.

11. Create the data plane resources:

    $ oc create -f openstack_unprovisioned_node_set.yaml -n openstack

12. Verify that the data plane resources have been created:

    $ oc get openstackdataplanenodeset -n openstack
    NAME           		STATUS MESSAGE
    openstack-data-plane 	False  Deployment not started

    For information on the meaning of the returned status, see Data plane conditions and states.

13. Verify that the Secret resource was created for the node set:

    $ oc get secret -n openstack | grep openstack-data-plane
    dataplanenodeset-openstack-data-plane Opaque 1 3m50s

14. Verify the services were created:

    $ oc get openstackdataplaneservice -n openstack
    NAME                AGE
    configure-network   6d7h
    configure-os        6d6h
    install-os          6d6h
    run-os              6d6h
    validate-network    6d6h
    ovn                 6d6h
    libvirt             6d6h
    nova                6d6h
    telemetry           6d6h

Procedure

1. Create a file on your workstation named openstack_data_plane_deploy.yaml to define the OpenStackDataPlaneDeployment CR:

   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneDeployment
   metadata:
     name: openstack-data-plane

   • name - The OpenStackDataPlaneDeployment CR name must be unique, must consist of lower case alphanumeric characters, - (hyphen) or . (period), and must start and end with an alphanumeric character. Update the name in this example to a name that reflects the node sets in the deployment.

2. Add all the OpenStackDataPlaneNodeSet CRs that you want to deploy.

   spec:
     nodeSets:
       - openstack-data-plane
       - <nodeSet_name>
       - ...
       - <nodeSet_name>
     services:
     ...

   • Replace <nodeSet_name> with the names of the OpenStackDataPlaneNodeSet CRs that you want to include in your data plane deployment, for example, openstack_preprovisioned_node_set.yaml or openstack_unprovisioned_node_set.yaml.
3. Save the openstack_data_plane_deploy.yaml deployment file.

4. Deploy the data plane:

   $ oc create -f openstack_data_plane_deploy.yaml -n openstack

   You can view the Ansible logs while the deployment executes:

   $ oc get pod -l app=openstackansibleee -w
   $ oc logs -l app=openstackansibleee -f --max-log-requests 10

5. Confirm that the data plane is deployed:

   $ oc get openstackdataplanedeployment -n openstack

   Sample output

   NAME                   STATUS   MESSAGE
   openstack-data-plane   True     Setup Complete

6. Repeat the oc get command until you see the NodeSet Ready message:

   $ oc get openstackdataplanenodeset -n openstack

   Sample output

   NAME                   STATUS   MESSAGE
   openstack-data-plane   True     NodeSet Ready

   For information about the meaning of the returned status, see Data plane conditions and states.

   If the status indicates that the data plane has not been deployed, then troubleshoot the deployment. For information, see Troubleshooting the data plane creation and deployment.

7. Map the Compute nodes to the Compute cell that they are connected to:

   $ oc rsh nova-cell0-conductor-0 nova-manage cell_v2 discover_hosts --verbose

   If you did not create additional cells, this command maps the Compute nodes to cell1.