Chapter 7. Creating the data plane for dynamic routing

Procedure

1. Create a file on your workstation named openstack_compute_node_set.yaml to define the OpenStackDataPlaneNodeSet CR:

   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneNodeSet
   metadata:
     name: openstack-compute-nodes
     namespace: openstack
   spec:
     env:
       - name: ANSIBLE_FORCE_COLOR
         value: "True"

   • name - The OpenStackDataPlaneNodeSet CR name must be unique, must consist of lower case alphanumeric characters, - (hyphen) or . (period), and must start and end with an alphanumeric character. Update the name in this example to a name that reflects the nodes in the set.
   • env - Optional: a list of environment variables to pass to the pod.

2. Connect the Compute nodes on the data plane to the control plane network:

   spec:
     ...
     networkAttachments:
       - ctlplane

3. Specify that the nodes in this set are pre-provisioned:

     preProvisioned: true

4. Add the SSH key secret that you created to enable Ansible to connect to the Compute nodes on the data plane:

     nodeTemplate:
       ansibleSSHPrivateKeySecret: <secret-key>

   • Replace <secret-key> with the name of the SSH key Secret CR you created for this node set in Creating the data plane secrets, for example, dataplane-ansible-ssh-private-key-secret.
5. Create a Persistent Volume Claim (PVC) in the openstack namespace on your Red Hat OpenShift Container Platform (RHOCP) cluster to store logs. Set the volumeMode to Filesystem and accessModes to ReadWriteOnce. Do not request storage for logs from a PersistentVolume (PV) that uses the NFS volume plugin. NFS is incompatible with FIFO and the ansible-runner creates a FIFO file to write to store logs. For information about PVCs, see Understanding persistent storage in the RHOCP Storage guide and Red Hat OpenShift Container Platform cluster requirements in Planning your deployment.

6. Enable persistent logging for the data plane nodes:

     nodeTemplate:
       ...
       extraMounts:
         - extraVolType: Logs
           volumes:
           - name: ansible-logs
             persistentVolumeClaim:
               claimName: <pvc_name>
           mounts:
           - name: ansible-logs
             mountPath: "/runner/artifacts"

   • Replace <pvc_name> with the name of the PVC storage on your RHOCP cluster.

7. Specify the management network:

     nodeTemplate:
       ...
       managementNetwork: ctlplane

8. Specify the Secret CRs used to source the usernames and passwords to register the operating system of your nodes and to enable repositories. The following example demonstrates how to register your nodes to CDN. For details on how to register your nodes with Red Hat Satellite 6.13, see Managing Hosts.

     nodeTemplate:
       ...
       ansible:
         ansibleUser: cloud-admin
         ansiblePort: 22
         ansibleVarsFrom:
           - prefix: subscription_manager_
             secretRef:
               name: subscription-manager
           - prefix: registry_
             secretRef:
               name: redhat-registry
         ansibleVars:
           edpm_bootstrap_command: |
             set -e
             subscription-manager register --username {{ subscription_manager_username }} --password {{ subscription_manager_password }}
             subscription-manager release --set=9.4
             subscription-manager repos --disable=* subscription-manager repos --enable=rhel-9-for-x86_64-baseos-eus-rpms --enable=rhel-9-for-x86_64-appstream-eus-rpms --enable=rhel-9-for-x86_64-highavailability-eus-rpms --enable=fast-datapath-for-rhel-9-x86_64-rpms --enable=rhoso-18.0-for-rhel-9-x86_64-rpms --enable=rhceph-7-tools-for-rhel-9-x86_64-rpms edpm_bootstrap_release_version_package: []**

   • ansibleUser - The user associated with the secret you created in Creating the data plane secrets.

   • ansibleVars - The Ansible variables that customize the set of nodes. For a list of Ansible variables that you can use, see https://openstack-k8s-operators.github.io/edpm-ansible/.

     For a complete list of the Red Hat Customer Portal registration commands, see https://access.redhat.com/solutions/253273. For information about how to log in to registry.redhat.io, see https://access.redhat.com/RegistryAuthentication#creating-registry-service-accounts-6.

9. Add the network configuration template to apply to your Compute nodes. The following example applies the single NIC VLANs network configuration to the data plane nodes:

     nodeTemplate:
       ...
       ansible:
         ...
         ansibleVars:
           ...
           edpm_network_config_os_net_config_mappings:
             edpm-compute-0:
               nic1: 52:54:04:60:55:22
           neutron_physical_bridge_name: br-ex
           edpm_network_config_template: |
             ---
             {% set mtu_list = [ctlplane_mtu] %}
             {% for network in nodeset_networks %}
             {{ mtu_list.append(lookup(vars, networks_lower[network] ~ _mtu)) }}
             {%- endfor %}
             {% set min_viable_mtu = mtu_list | max %}
             network_config:
             - type: ovs_bridge
               name: {{ neutron_physical_bridge_name }}
               use_dhcp: false
               use_dhcpv6: true  # needed to enable IPv6 on bridges
             - type: interface
               name: nic1
               use_dhcp: true
               defroute: false
             - type: interface
               name: nic2
               use_dhcp: false
               defroute: false
               dns_servers: {{ ctlplane_dns_nameservers }}
               domain: {{ dns_search_domains }}
               addresses:
               - ip_netmask: {{ ctlplane_ip }}/{{ ctlplane_cidr }}
             - type: interface
               name: nic3
               use_dhcp: false
               addresses:
               - ip_netmask: {{ lookup(vars, bgpnet0_ip) }}/30
               - ip_netmask: {{ lookup(vars, bgpnet0v6_ip) }}/126
             - type: interface
               name: nic4
               use_dhcp: false
               addresses:
               - ip_netmask: {{ lookup(vars, bgpnet1_ip) }}/30
               - ip_netmask: {{ lookup(vars, bgpnet1v6_ip) }}/126
             - type: interface
               name: lo
               addresses:
               - ip_netmask: {{ lookup(vars, bgpmainnet_ip) }}/32
               - ip_netmask: {{ lookup(vars, bgpmainnetv6_ip) }}/128
               - ip_netmask: {{ lookup(vars, octavia_ip) }}/32      # /128 in case of IPv6

   • nic1 - Update to the MAC address assigned to the NIC to use for network configuration on the Compute node.

     For alternative templates, see roles/edpm_network_config/templates. For more information about data plane network configuration, see Customizing data plane networks in Configuring networking services.

10. Add the common configuration for the set of Compute nodes in this group under the nodeTemplate section. Each node in this OpenStackDataPlaneNodeSet inherits this configuration:

    Example

    edpm_frr_bgp_ipv6: true
    edpm_frr_bgp_ipv6_src_network: bgpmainnetv6
    edpm_frr_bgp_ipv4_src_network: bgpmainnet
    edpm_frr_bgp_neighbor_password: f00barZ
    edpm_frr_bgp_uplinks:
    - nic3
    - nic4
    edpm_ovn_encap_ip: '{{ lookup(''vars'', ''bgpmainnet_ip'') }}' # edpm_ovn_encap_ip can use either bgpmainnet_ip or bgpmainnetv6_ip.

    For information about the properties you can use to configure common node attributes, see OpenStackDataPlaneNodeSet CR spec properties for dynamic routing.

11. Define each node in this node set:

      nodes:
        edpm-compute-0:
          hostName: edpm-compute-0
          networks:
          - name: ctlplane
            subnetName: subnet1
            defaultRoute: true
            fixedIP: 192.168.122.100
          - name: BgpNet0
            subnetName: subnet0
            fixedIP: 100.64.0.2
          - name: BgpNet1
            subnetName: subnet0
            fixedIP: 100.65.0.2
          - name: BgpMainNet
            subnetName: subnet0
            fixedIP: 172.30.0.2
          - name: BgpNet0v6
            subnetName: subnet0
            fixedIP: 2620:cf::100:64:0:2
          - name: BgpNet1v6
            subnetName: subnet0
            fixedIP: 2620:cf::100:65:0:2
          - name: BgpMainNetv6
            subnetName: subnet0
            fixedIP: f00d:f00d:f00d:f00d:99:99:0:2
          ansible:
            ansibleHost: 192.168.122.100
            ansibleUser: cloud-admin
            ansibleVars:
              fqdn_internal_api: edpm-compute-0.example.com
        edpm-compute-1:
          hostName: edpm-compute-1
          networks:
          - name: ctlplane
            subnetName: subnet1
            defaultRoute: true
            fixedIP: 192.168.122.101
          - name: BgpNet0
            subnetName: subnet0
            fixedIP: 100.64.1.2
          - name: BgpNet1
            subnetName: subnet0
            fixedIP: 100.65.1.2
          - name: BgpMainNet
            subnetName: subnet0
            fixedIP: 172.30.1.2
          ansible:
            ansibleHost: 192.168.122.101
            ansibleUser: cloud-admin
            ansibleVars:
              fqdn_internal_api: edpm-compute-1.example.com

    • edpm-compute-0 - The node definition reference, for example, edpm-compute-0. Each node in the node set must have a node definition.
    • networks - Defines the IPAM and the DNS records for the node.
    • fixedIP - Specifies a predictable IP address for the network that must be in the allocation range defined for the network in the NetConfig CR.

    • ansibleVar - Node-specific Ansible variables that customize the node.

      Note
      • Nodes defined within the nodes section can configure the same Ansible variables that are configured in the nodeTemplate section. Where an Ansible variable is configured for both a specific node and within the nodeTemplate section, the node-specific values override those from the nodeTemplate section.
      • You do not need to replicate all the nodeTemplate Ansible variables for a node to override the default and set some node-specific values. You only need to configure the Ansible variables you want to override for the node.
      • When you define the networkData Secret for an individual node (such as edpm-compute-0), it acts as a complete override rather than a supplemental configuration. Because node-specific configurations override the inherited default values from the nodeTemplate section, you must ensure that your node-specific networkData Secret contains the full set of required network configurations for that node, not just the unique values.
      • Many ansibleVars include edpm in the name, which stands for "External Data Plane Management".

    For information about the properties you can use to configure node attributes, see OpenStackDataPlaneNodeSet CR properties.

12. In the services section, ensure that the frr and ovn-bgp-agent services are included:

    Example

    services:
    - download-cache
    - redhat
    - bootstrap
    - install-os
    - configure-os
    - configure-network
    - frr
    - validate-network
    - ssh-known-hosts
    - run-os
    - reboot-os
    - install-certs
    - ovn
    - neutron-metadata
    - ovn-bgp-agent
    - libvirt
    - nova

13. Save the openstack_compute_node_set.yaml definition file.

14. Create the data plane resources:

    $ oc create --save-config -f openstack_compute_node_set.yaml -n openstack

15. Verify that the data plane resources have been created by confirming that the status is SetupReady:

    $ oc wait openstackdataplanenodeset openstack-compute-nodes \
    --for condition=SetupReady --timeout=10m

    When the status is SetupReady the command returns a condition met message, otherwise it returns a timeout error.

    For information about the data plane conditions and states, see Data plane conditions and states.

16. Verify that the Secret resource was created for the node set:

    $ oc get secret | grep openstack-compute-nodes
    dataplanenodeset-openstack-compute-nodes Opaque 1 3m50s

17. Verify the services were created:

    $ oc get openstackdataplaneservice -n openstack
    NAME                AGE
    download-cache      46m
    bootstrap           46m
    configure-network   46m
    validate-network    46m
    frr                 46m
    install-os          46m
    ...

Procedure

1. Create a file on your workstation named openstack_networker_node_set.yaml to define the OpenStackDataPlaneNodeSet CR:

   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneNodeSet
   metadata:
     name: openstack-networker-nodes
     namespace: openstack
   spec:
     env:
       - name: ANSIBLE_FORCE_COLOR
         value: "True"

   • name - The OpenStackDataPlaneNodeSet CR name must be unique, must consist of lower case alphanumeric characters, - (hyphen) or . (period), and must start and end with an alphanumeric character. Update the name in this example to a name that reflects the nodes in the set.
   • env - Optional: a list of environment variables to pass to the pod.

2. Connect the Networker nodes on the data plane to the control plane network:

   spec:
     ...
     networkAttachments:
       - ctlplane

3. Specify that the nodes in this set are pre-provisioned:

     preProvisioned: true

4. Add the SSH key secret that you created to enable Ansible to connect to the Networker nodes on the data plane:

     nodeTemplate:
       ansibleSSHPrivateKeySecret: <secret-key>

   • Replace <secret-key> with the name of the SSH key Secret CR you created for this node set in Creating the data plane secrets, for example, dataplane-ansible-ssh-private-key-secret.
5. Create a Persistent Volume Claim (PVC) in the openstack namespace on your Red Hat OpenShift Container Platform (RHOCP) cluster to store logs. Set the volumeMode to Filesystem and accessModes to ReadWriteOnce. Do not request storage for logs from a PersistentVolume (PV) that uses the NFS volume plugin. NFS is incompatible with FIFO and the ansible-runner creates a FIFO file to write to store logs. For information about PVCs, see Understanding persistent storage in the RHOCP Storage guide and Red Hat OpenShift Container Platform cluster requirements in Planning your deployment.

6. Enable persistent logging for the data plane nodes:

     nodeTemplate:
       ...
       extraMounts:
         - extraVolType: Logs
           volumes:
           - name: ansible-logs
             persistentVolumeClaim:
               claimName: <pvc_name>
           mounts:
           - name: ansible-logs
             mountPath: "/runner/artifacts"

   • Replace <pvc_name> with the name of the PVC storage on your RHOCP cluster.

7. Specify the management network:

     nodeTemplate:
       ...
       managementNetwork: ctlplane

8. Specify the Secret CRs used to source the usernames and passwords to register the operating system of your nodes and to enable repositories. The following example demonstrates how to register your nodes to CDN. For details on how to register your nodes with Red Hat Satellite 6.13, see Managing Hosts.

     nodeTemplate:
       ...
       ansible:
         ansibleUser: cloud-admin
         ansiblePort: 22
         ansibleVarsFrom:
           - prefix: subscription_manager_
             secretRef:
               name: subscription-manager
           - prefix: registry_
             secretRef:
               name: redhat-registry
         ansibleVars:
           edpm_bootstrap_command: |
             set -e
             subscription-manager register --username {{ subscription_manager_username }} --password {{ subscription_manager_password }}
             subscription-manager release --set=9.4
             subscription-manager repos --disable=* 
             subscription-manager repos --enable=rhel-9-for-x86_64-baseos-eus-rpms --enable=rhel-9-for-x86_64-appstream-eus-rpms --enable=rhel-9-for-x86_64-highavailability-eus-rpms --enable=fast-datapath-for-rhel-9-x86_64-rpms --enable=rhoso-18.0-for-rhel-9-x86_64-rpms --enable=rhceph-7-tools-for-rhel-9-x86_64-rpms
           edpm_bootstrap_release_version_package: []

   • ansibleUser - The user associated with the secret you created in Creating the data plane secrets.
   • ansibleVars - The Ansible variables that customize the set of nodes. For a list of Ansible variables that you can use, see https://openstack-k8s-operators.github.io/edpm-ansible/.

   For a complete list of the Red Hat Customer Portal registration commands, see https://access.redhat.com/solutions/253273. For information about how to log in to registry.redhat.io, see https://access.redhat.com/RegistryAuthentication#creating-registry-service-accounts-6.

9. Add the network configuration template to apply to your Networker nodes. The following example applies the single NIC VLANs network configuration to the data plane nodes:

     nodeTemplate:
       ...
       ansible:
         ...
         ansibleVars:
           ...
           edpm_network_config_os_net_config_mappings:
             edpm-networker-0:
               nic1: 52:54:04:60:55:22
           neutron_physical_bridge_name: br-ex
           edpm_network_config_template: |
             ---
             {% set mtu_list = [ctlplane_mtu] %}
             {% for network in nodeset_networks %}
             {{ mtu_list.append(lookup(vars, networks_lower[network] ~ _mtu)) }}
             {%- endfor %}
             {% set min_viable_mtu = mtu_list | max %}
             network_config:
             - type: ovs_bridge
               name: {{ neutron_physical_bridge_name }}
               use_dhcp: false
               use_dhcpv6: true
             - type: interface
               name: nic1
               use_dhcp: true
               defroute: false
             - type: interface
               name: nic2
               use_dhcp: false
               defroute: false
               dns_servers: {{ ctlplane_dns_nameservers }}
               domain: {{ dns_search_domains }}
               addresses:
               - ip_netmask: {{ ctlplane_ip }}/{{ ctlplane_cidr }}
             - type: interface
               name: nic3
               use_dhcp: false
               addresses:
               - ip_netmask: {{ lookup(vars, bgpnet0_ip) }}/30
               - ip_netmask: {{ lookup(vars, bgpnet0v6_ip) }}/126
             - type: interface
               name: nic4
               use_dhcp: false
               addresses:
               - ip_netmask: {{ lookup(vars, bgpnet1_ip) }}/30
               - ip_netmask: {{ lookup(vars, bgpnet1v6_ip) }}/126
             - type: interface
               name: lo
               addresses:
               - ip_netmask: {{ lookup(vars, bgpmainnet_ip) }}/32
               - ip_netmask: {{ lookup(vars, bgpmainnetv6_ip) }}/128
               - ip_netmask: {{ lookup(vars, octavia_ip) }}/32      # /128 in case of IPv6

   • nic1 - Update the nic1 to the MAC address assigned to the NIC to use for network configuration on the Compute node.

     For alternative templates, see roles/edpm_network_config/templates. For more information about data plane network configuration, see Customizing data plane networks in the Configuring network services guide.

10. Add the common configuration for the set of Networker nodes in this group under the nodeTemplate section. Each node in this OpenStackDataPlaneNodeSet inherits this configuration:

    Example

    edpm_frr_bgp_ipv6: true
    edpm_frr_bgp_ipv6_src_network: bgpmainnetv6
    edpm_frr_bgp_ipv4_src_network: bgpmainnet
    edpm_frr_bgp_neighbor_password: f00barZ
    edpm_frr_bgp_uplinks:
    - nic3
    - nic4
    edpm_ovn_encap_ip: '{{ lookup(''vars'', ''bgpmainnet_ip'') }}' # edpm_ovn_encap_ip can use either bgpmainnet_ip or bgpmainnetv6_ip.

    For information about the properties you can use to configure common node attributes, see OpenStackDataPlaneNodeSet CR spec properties for dynamic routing.

11. Define each node in this node set:

      nodes:
        edpm-networker-0:
          hostName: edpm-networker-0
          networks:
          - name: ctlplane
            subnetName: subnet1
            defaultRoute: true
            fixedIP: 192.168.122.100
          - name: BgpNet0
            subnetName: subnet0
            fixedIP: 100.64.0.2
          - name: BgpNet1
            subnetName: subnet0
            fixedIP: 100.65.0.2
          - name: BgpMainNet
            subnetName: subnet0
            fixedIP: 172.30.0.2
          ansible:
            ansibleHost: 192.168.122.100
            ansibleUser: cloud-admin
            ansibleVars:
              fqdn_internal_api: edpm-networker-0.example.com
        edpm-networker-1:
          hostName: edpm-networker-1
          networks:
          - name: ctlplane
            subnetName: subnet1
            defaultRoute: true
            fixedIP: 192.168.122.101
          - name: BgpNet0
            subnetName: subnet0
            fixedIP: 100.64.0.2
          - name: BgpNet1
            subnetName: subnet0
            fixedIP: 100.65.0.2
          - name: BgpMainNet
            subnetName: subnet0
            fixedIP: 172.30.0.2
          - name: BgpNet0v6
            subnetName: subnet0
            fixedIP: 2620:cf::100:64:0:2
          - name: BgpNet1v6
            subnetName: subnet0
            fixedIP: 2620:cf::100:65:0:2
          - name: BgpMainNetv6
            subnetName: subnet0
            fixedIP: f00d:f00d:f00d:f00d:99:99:0:2
          ansible:
            ansibleHost: 192.168.122.101
            ansibleUser: cloud-admin
            ansibleVars:
              fqdn_internal_api: edpm-networker-1.example.com

    • edpm-networker-0 - The node definition reference, for example, edpm-networker-0. Each node in the node set must have a node definition.
    • networks - Defines the IPAM and the DNS records for the node.
    • fixedIP - Specifies a predictable IP address for the network that must be in the allocation range defined for the network in the NetConfig CR.

    • ansibleVars - Node-specific Ansible variables that customize the node.

      Note
      • Nodes defined within the nodes section can configure the same Ansible variables that are configured in the nodeTemplate section. Where an Ansible variable is configured for both a specific node and within the nodeTemplate section, the node-specific values override those from the nodeTemplate section.
      • You do not need to replicate all the nodeTemplate Ansible variables for a node to override the default and set some node-specific values. You only need to configure the Ansible variables you want to override for the node.
      • When you define the networkData Secret for an individual node (such as edpm-compute-0), it acts as a complete override rather than a supplemental configuration. Because node-specific configurations override the inherited default values from the nodeTemplate section, you must ensure that your node-specific networkData Secret contains the full set of required network configurations for that node, not just the unique values.
      • Many ansibleVars include edpm in the name, which stands for "External Data Plane Management".

    For information about the properties you can use to configure node attributes, see OpenStackDataPlaneNodeSet CR properties.

12. In the services section, ensure that the frr and ovn-bgp-agent services are included.

    Example

    services:
    - download-cache
    - bootstrap
    - install-os
    - configure-os
    - configure-network
    - frr
    - validate-network
    - ssh-known-hosts
    - run-os
    - reboot-os
    - install-certs
    - ovn
    - neutron-metadata
    - ovn-bgp-agent

13. Save the openstack_networker_node_set.yaml definition file.

14. Create the Networker node resources for the data plane:

    $ oc create --save-config -f openstack_networker_node_set.yaml -n openstack

15. Verify that the data plane resources have been created by confirming that the status is SetupReady:

    $ oc wait openstackdataplanenodeset openstack-networker-nodes --for condition=SetupReady --timeout=10m

    When the status is SetupReady the command returns a condition met message, otherwise it returns a timeout error.

    For information about the data plane conditions and states, see Data plane conditions and states.

16. Verify that the Secret resource was created for the node set:

    $ oc get secret | grep openstack-networker-nodes
    dataplanenodeset-openstack-networker-nodes Opaque 1 3m50s

17. Verify the services were created:

    $ oc get openstackdataplaneservice -n openstack
    NAME                AGE
    download-cache      46m
    bootstrap           46m
    configure-network   46m
    validate-network    46m
    frr                 46m
    install-os          46m
    ...

Procedure

1. The Bare Metal Operator (BMO) manages BareMetalHost custom resources (CRs) in the openshift-machine-api namespace by default. Update the Provisioning CR to watch all namespaces:

   $ oc patch provisioning provisioning-configuration --type merge -p '{"spec":{"watchAllNamespaces": true }}'

2. If you are using virtual media boot for bare-metal data plane nodes and the nodes are not connected to a provisioning network, you must update the Provisioning CR to enable virtualMediaViaExternalNetwork, which enables bare-metal connectivity through the external network:

   $ oc patch provisioning provisioning-configuration --type merge -p '{"spec":{"virtualMediaViaExternalNetwork": true }}'

3. Create a file on your workstation that defines the Secret CR with the credentials for accessing the Baseboard Management Controller (BMC) of each bare-metal data plane node in the node set:

   apiVersion: v1
   kind: Secret
   metadata:
     name: edpm-compute-0-bmc-secret
     namespace: openstack
   type: Opaque
   data:
     username: <base64_username>
     password: <base64_password>

   • Replace <base64_username> and <base64_password> with strings that are base64-encoded. You can use the following command to generate a base64-encoded string:

     $ echo -n <string> | base64

     Tip

     If you don’t want to base64-encode the username and password, you can use the stringData field instead of the data field to set the username and password.

4. Create a file named bmh_nodes.yaml on your workstation, that defines the BareMetalHost CR for each bare-metal data plane node. The following example creates a BareMetalHost CR with the provisioning method Redfish virtual media:

   apiVersion: metal3.io/v1alpha1
   kind: BareMetalHost
   metadata:
     name: edpm-compute-0
     namespace: openstack
     labels:
       app: openstack
       workload: compute
   spec:
     bmc:
       address: redfish-virtualmedia+http://192.168.111.1:8000/redfish/v1/Systems/e8efd888-f844-4fe0-9e2e-498f4ab7806d
       credentialsName: edpm-compute-0-bmc-secret
     bootMACAddress: 00:c7:e4:a7:e7:f3
     bootMode: UEFI
     online: false
    [preprovisioningNetworkDataName: <network_config_secret_name>]

   • labels - Metadata labels, such as app, workload, and nodeName are key-value pairs that provide varying levels of granularity for labelling nodes. You can use these labels when you create an OpenStackDataPlaneNodeSet CR to describe the configuration of bare-metal nodes to be provisioned or to define nodes in a node set.
   • address - The URL for communicating with the BMC controller of the node. For information about BMC addressing for other provisioning methods, see BMC addressing in the RHOCP Installing on bare metal guide.
   • credentialsName - The name of the Secret CR you created in the previous step for accessing the BMC of the node.

   • preprovisioningNetworkDataName - Optional: The name of the network configuration secret in the local namespace to pass to the pre-provisioning image. The network configuration must be in nmstate format.

     For more information about how to create a BareMetalHost CR, see About the BareMetalHost resource in the RHOCP Installing on bare metal guide.

5. Create the BareMetalHost resources:

   $ oc create -f bmh_nodes.yaml

6. Verify that the BareMetalHost resources have been created and are in the Available state:

   $ oc get bmh
   NAME         STATE            CONSUMER              ONLINE   ERROR   AGE
   edpm-compute-0   Available      openstack-edpm        true             2d21h
   edpm-compute-1   Available      openstack-edpm        true             2d21h
   ...

Procedure

1. Create a file on your workstation named openstack_unprovisioned_compute_node_set.yaml to define the OpenStackDataPlaneNodeSet CR:

   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneNodeSet
   metadata:
     name: openstack-compute-nodes
     namespace: openstack
   spec:
     tlsEnabled: true
     env:
       - name: ANSIBLE_FORCE_COLOR
         value: "True"

   • name - The OpenStackDataPlaneNodeSet CR name must be unique, must consist of lower case alphanumeric characters, - (hyphen) or . (period), must start and end with an alphanumeric character, and must have a maximum length of 20 characters. Update the name in this example to a name that reflects the nodes in the set.
   • env - Optional: a list of environment variables to pass to the pod.

2. Connect the Compute nodes data plane to the control plane network:

   spec:
     ...
     networkAttachments:
       - ctlplane

3. Specify that the nodes in this set are unprovisioned and must be provisioned when creating the resource:

     preProvisioned: false

4. Define the baremetalSetTemplate field to describe the configuration of the bare-metal nodes that must be provisioned when creating the resource:

     baremetalSetTemplate:
       deploymentSSHSecret: dataplane-ansible-ssh-private-key-secret
       bmhNamespace: <bmh_namespace>
       cloudUserName: <ansible_ssh_user>
       bmhLabelSelector:
         app: <bmh_label>
       ctlplaneInterface: <interface>

   • Replace <bmh_namespace> with the namespace defined in the corresponding BareMetalHost CR for the node, for example, openstack.
   • Replace <ansible_ssh_user> with the username of the Ansible SSH user, for example, cloud-admin.
   • Replace <bmh_label> with the metadata label defined in the corresponding BareMetalHost CR for the node, for example, openstack. Metadata labels, such as app, workload, and nodeName are key-value pairs for labelling nodes. Set the bmhLabelSelector field to select data plane nodes based on one or more labels that match the labels in the corresponding BareMetalHost CR.
   • Replace <interface> with the control plane interface the node connects to, for example, enp6s0.

5. Add the SSH key secret that you created to enable Ansible to connect to the data plane nodes:

     nodeTemplate:
       ansibleSSHPrivateKeySecret: <secret-key>

   • Replace <secret-key> with the name of the SSH key Secret CR you created in Creating the data plane secrets, for example, dataplane-ansible-ssh-private-key-secret.
6. Create a Persistent Volume Claim (PVC) in the openstack namespace on your Red Hat OpenShift Container Platform (RHOCP) cluster to store logs. Set the volumeMode to Filesystem and accessModes to ReadWriteOnce. Do not request storage for logs from a PersistentVolume (PV) that uses the NFS volume plugin. NFS is incompatible with FIFO and the ansible-runner creates a FIFO file to write to store logs. For information about PVCs, see Understanding persistent storage in the RHOCP Storage guide and Red Hat OpenShift Container Platform cluster requirements in Planning your deployment.

7. Enable persistent logging for the data plane nodes:

     nodeTemplate:
       ...
       extraMounts:
         - extraVolType: Logs
           volumes:
           - name: ansible-logs
             persistentVolumeClaim:
               claimName: <pvc_name>
           mounts:
           - name: ansible-logs
             mountPath: "/runner/artifacts"

   • Replace <pvc_name> with the name of the PVC storage on your RHOCP cluster.

8. Specify the management network:

     nodeTemplate:
       ...
       managementNetwork: ctlplane

9. Specify the Secret CRs used to source the usernames and passwords to register the operating system of your nodes and to enable repositories. The following example demonstrates how to register your nodes to CDN. For details on how to register your nodes with Red Hat Satellite 6.13, see Managing Hosts.

     nodeTemplate:
       ansible:
         ansibleUser: cloud-admin
         ansiblePort: 22
         ansibleVarsFrom:
           - prefix: subscription_manager_
             secretRef:
               name: subscription-manager
           - secretRef:
               name: redhat-registry
         ansibleVars:
           edpm_bootstrap_command: |
             subscription-manager register --username {{ subscription_manager_username }} --password {{ subscription_manager_password }}
             subscription-manager release --set=9.4
             subscription-manager repos --disable=* 
             subscription-manager repos --enable=rhel-9-for-x86_64-baseos-eus-rpms --enable=rhel-9-for-x86_64-appstream-eus-rpms --enable=rhel-9-for-x86_64-highavailability-eus-rpms --enable=fast-datapath-for-rhel-9-x86_64-rpms --enable=rhoso-18.0-for-rhel-9-x86_64-rpms --enable=rhceph-7-tools-for-rhel-9-x86_64-rpms
           edpm_bootstrap_release_version_package: []

   • ansibleUser - The user associated with the secret you created in Creating the data plane secrets.
   • ansibleVars - The Ansible variables that customize the set of nodes. For a list of Ansible variables that you can use, see https://openstack-k8s-operators.github.io/edpm-ansible/.

   For a complete list of the Red Hat Customer Portal registration commands, see https://access.redhat.com/solutions/253273. For information about how to log in to registry.redhat.io, see https://access.redhat.com/RegistryAuthentication#creating-registry-service-accounts-6.

10. Add the network configuration template to apply to your Compute nodes. The following example applies the single NIC VLANs network configuration to the data plane nodes:

      nodeTemplate:
        ...
        ansible:
          ...
          ansibleVars:
            ...
            edpm_network_config_os_net_config_mappings:
              edpm-compute-0:
                nic1: 52:54:04:60:55:22
              edpm-compute-1:
                nic1: 52:54:04:60:55:22
            neutron_physical_bridge_name: br-ex
            edpm_network_config_update: false
            edpm_network_config_template: |
              ---
              {% set mtu_list = [ctlplane_mtu] %}
              {% for network in nodeset_networks %}
              {{ mtu_list.append(lookup(vars, networks_lower[network] ~ _mtu)) }}
              {%- endfor %}
              {% set min_viable_mtu = mtu_list | max %}
              network_config:
              - type: ovs_bridge
                name: {{ neutron_physical_bridge_name }}
                use_dhcp: false
                use_dhcpv6: true
              - type: interface
                name: nic1
                use_dhcp: true
                defroute: false
              - type: interface
                name: nic2
                use_dhcp: false
                defroute: false
                dns_servers: {{ ctlplane_dns_nameservers }}
                domain: {{ dns_search_domains }}
                addresses:
                - ip_netmask: {{ ctlplane_ip }}/{{ ctlplane_cidr }}
              - type: interface
                name: nic3
                use_dhcp: false
                addresses:
                - ip_netmask: {{ lookup(vars, bgpnet0_ip) }}/30
                - ip_netmask: {{ lookup(vars, bgpnet0v6_ip) }}/126
              - type: interface
                name: nic4
                use_dhcp: false
                addresses:
                - ip_netmask: {{ lookup(vars, bgpnet1_ip) }}/30
                - ip_netmask: {{ lookup(vars, bgpnet1v6_ip) }}/126
              - type: interface
                name: lo
                addresses:
                - ip_netmask: {{ lookup(vars, bgpmainnet_ip) }}/32
                - ip_netmask: {{ lookup(vars, bgpmainnetv6_ip) }}/128
                - ip_netmask: {{ lookup(vars, octavia_ip) }}/32      # /128 in case of IPv6

    • nic1 - Update the nic1 to the MAC address assigned to the NIC to use for network configuration on the Compute node.

    • edpm_network_config_update - When deploying a node set for the first time, ensure that the edpm_network_config_update variable is set to false. If you later modify edpm_network_config_template, first set edpm_network_config_update to true. Reset it to false after the update.

      Important

      After an edpm_network_config_template update, you must reset edpm_network_config_update to false. Otherwise, the nodes could lose network access. Whenever edpm_network_config_update is true, the updated network configuration is reapplied every time an OpenStackDataPlaneDeployment CR is created that includes the configure-network service that is a member of the servicesOverride list.

11. Add the common configuration for the set of Compute nodes in this group under the nodeTemplate section. Each node in this OpenStackDataPlaneNodeSet inherits this configuration:

    Example

    edpm_frr_bgp_ipv6: true
    edpm_frr_bgp_ipv6_src_network: bgpmainnetv6
    edpm_frr_bgp_ipv4_src_network: bgpmainnet
    edpm_frr_bgp_neighbor_password: f00barZ
    edpm_frr_bgp_uplinks:
    - nic3
    - nic4
    edpm_ovn_encap_ip: '{{ lookup(''vars'', ''bgpmainnet_ip'') }}' # edpm_ovn_encap_ip can use either bgpmainnet_ip or bgpmainnetv6_ip.

    For information about the properties you can use to configure common node attributes, see OpenStackDataPlaneNodeSet CR spec properties for dynamic routing.

12. Define each node in this node set:

      nodes:
        edpm-compute-0:
          hostName: edpm-compute-0
          networks:
          - name: ctlplane
            subnetName: subnet1
            defaultRoute: true
            fixedIP: 192.168.122.100
          - name: BgpNet0
            subnetName: subnet0
            fixedIP: 100.64.0.2
          - name: BgpNet1
            subnetName: subnet0
            fixedIP: 100.65.0.2
          - name: BgpMainNet
            subnetName: subnet0
            fixedIP: 172.30.0.2
          - name: BgpNet0v6
            subnetName: subnet0
            fixedIP: 2620:cf::100:64:0:2
          - name: BgpNet1v6
            subnetName: subnet0
            fixedIP: 2620:cf::100:65:0:2
          - name: BgpMainNetv6
            subnetName: subnet0
            fixedIP: f00d:f00d:f00d:f00d:99:99:0:2
          ansible:
            ansibleHost: 192.168.122.100
            ansibleUser: cloud-admin
            ansibleVars:
              fqdn_internal_api: edpm-compute-0.example.com
          bmhLabelSelector:
            nodeName: edpm-compute-0
        edpm-compute-1:
          hostName: edpm-compute-1
          networks:
          - name: ctlplane
            subnetName: subnet1
            defaultRoute: true
            fixedIP: 192.168.122.101
          - name: BgpNet0
            subnetName: subnet0
            fixedIP: 100.64.1.2
          - name: BgpNet1
            subnetName: subnet0
            fixedIP: 100.65.1.2
          - name: BgpMainNet
            subnetName: subnet0
            fixedIP: 172.30.1.2
          ansible:
            ansibleHost: 192.168.122.101
            ansibleUser: cloud-admin
            ansibleVars:
              fqdn_internal_api: edpm-compute-1.example.com
          bmhLabelSelector:
            nodeName: edpm-compute-1

    • edpm-compute-0 - The node definition reference, for example, edpm-compute-0. Each node in the node set must have a node definition.
    • networks - Defines the IPAM and the DNS records for the node.
    • fixedIP - Specifies a predictable IP address for the network that must be in the allocation range defined for the network in the NetConfig CR.
    • ansibleVars - Node-specific Ansible variables that customize the node.

    • bmhLabelSelector - Optional: metadata labels, such as app, workload, and nodeName are key-value pairs for labelling nodes. Set the bmhLabelSelector field to select data plane nodes based on one or more labels that match the labels in the corresponding BareMetalHost CR.

      Note
      • Nodes defined within the nodes section can configure the same Ansible variables that are configured in the nodeTemplate section. Where an Ansible variable is configured for both a specific node and within the nodeTemplate section, the node-specific values override those from the nodeTemplate section.
      • You do not need to replicate all the nodeTemplate Ansible variables for a node to override the default and set some node-specific values. You only need to configure the Ansible variables you want to override for the node.
      • When you define the networkData Secret for an individual node (such as edpm-compute-0), it acts as a complete override rather than a supplemental configuration. Because node-specific configurations override the inherited default values from the nodeTemplate section, you must ensure that your node-specific networkData Secret contains the full set of required network configurations for that node, not just the unique values.
      • Many ansibleVars include edpm in the name, which stands for "External Data Plane Management".

    For information about the properties you can use to configure node attributes, see OpenStackDataPlaneNodeSet CR properties.

13. In the services section, ensure that the frr and ovn-bgp-agent services are included:

    Example

    services:
    - download-cache
    - redhat
    - bootstrap
    - install-os
    - configure-os
    - configure-network
    - frr
    - validate-network
    - ssh-known-hosts
    - run-os
    - reboot-os
    - install-certs
    - ovn
    - neutron-metadata
    - ovn-bgp-agent
    - libvirt
    - nova

14. Save the openstack_unprovisioned_compute_node_set.yaml definition file.

15. Create the data plane resources:

    $ oc create --save-config -f openstack_unprovisioned_compute_node_set.yaml -n openstack

16. Verify that the data plane resources have been created by confirming that the status is SetupReady:

    $ oc wait openstackdataplanenodeset openstack-compute-nodes --for condition=SetupReady --timeout=10m

    When the status is SetupReady the command returns a condition met message, otherwise it returns a timeout error.

    For information about the data plane conditions and states, see Data plane conditions and states.

17. Verify that the Secret resource was created for the node set:

    $ oc get secret -n openstack | grep openstack-compute-nodes
    dataplanenodeset-openstack-compute-nodes Opaque 1 3m50s

18. Verify that the nodes have transitioned to the provisioned state:

    $ oc get bmh
    NAME            STATE         CONSUMER                  ONLINE   ERROR   AGE
    edpm-compute-0  provisioned   openstack-compute-nodes   true             3d21h

19. Verify that the services were created:

    $ oc get openstackdataplaneservice -n openstack
    NAME                AGE
    download-cache      8m40s
    bootstrap           8m40s
    configure-network   8m40s
    validate-network    8m40s
    frr                 8m40s
    install-os          8m40s
    ...

Procedure

1. Create a file on your workstation named openstack_unprovisioned_networker_node_set.yaml to define the OpenStackDataPlaneNodeSet CR:

   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneNodeSet
   metadata:
     name: openstack-networker-nodes
     namespace: openstack
   spec:
     tlsEnabled: true
     env:
       - name: ANSIBLE_FORCE_COLOR
         value: "True"

   • name - The OpenStackDataPlaneNodeSet CR name must be unique, must consist of lower case alphanumeric characters, - (hyphen) or . (period), must start and end with an alphanumeric character, and must have a maximum length of 20 characters. Update the name in this example to a name that reflects the nodes in the set.
   • env - Optional: a list of environment variables to pass to the pod.

2. Connect the Networker nodes on the data plane to the control plane network:

   spec:
     ...
     networkAttachments:
       - ctlplane

3. Specify that the nodes in this set are unprovisioned and must be provisioned when creating the resource:

     preProvisioned: false

4. Define the baremetalSetTemplate field to describe the configuration of the bare-metal nodes that must be provisioned when creating the resource:

     baremetalSetTemplate:
       deploymentSSHSecret: dataplane-ansible-ssh-private-key-secret
       bmhNamespace: <bmh_namespace>
       cloudUserName: <ansible_ssh_user>
       bmhLabelSelector:
         app: <bmh_label>
       ctlplaneInterface: <interface>

   • Replace <bmh_namespace> with the namespace defined in the corresponding BareMetalHost CR for the node, for example, openstack.
   • Replace <ansible_ssh_user> with the username of the Ansible SSH user, for example, cloud-admin.
   • Replace <bmh_label> with the metadata label defined in the corresponding BareMetalHost CR for the node, for example, openstack. Metadata labels, such as app, workload, and nodeName are key-value pairs for labelling nodes. Set the bmhLabelSelector field to select data plane nodes based on one or more labels that match the labels in the corresponding BareMetalHost CR.
   • Replace <interface> with the control plane interface the node connects to, for example, enp6s0.

5. Add the SSH key secret that you created to enable Ansible to connect to the data plane nodes:

     nodeTemplate:
       ansibleSSHPrivateKeySecret: <secret-key>

   • Replace <secret-key> with the name of the SSH key Secret CR you created in Creating the data plane secrets, for example, dataplane-ansible-ssh-private-key-secret.
6. Create a Persistent Volume Claim (PVC) in the openstack namespace on your RHOCP cluster to store logs. Set the volumeMode to Filesystem and accessModes to ReadWriteOnce. For information about PVCs, see Understanding persistent storage in the RHOCP Storage guide and Red Hat OpenShift Container Platform cluster requirements in Planning your deployment.

7. Enable persistent logging for the data plane nodes:

     nodeTemplate:
       ...
       extraMounts:
         - extraVolType: Logs
           volumes:
           - name: ansible-logs
             persistentVolumeClaim:
               claimName: <pvc_name>
           mounts:
           - name: ansible-logs
             mountPath: "/runner/artifacts"

   • Replace <pvc_name> with the name of the PVC storage on your RHOCP cluster.

8. Specify the management network:

     nodeTemplate:
       ...
       managementNetwork: ctlplane

9. Specify the Secret CRs used to source the usernames and passwords to register the operating system of your nodes and to enable repositories. The following example demonstrates how to register your nodes to CDN. For details on how to register your nodes with Red Hat Satellite 6.13, see Managing Hosts.

     nodeTemplate:
       ansible:
         ansibleUser: cloud-admin
         ansiblePort: 22
         ansibleVarsFrom:
           - prefix: subscription_manager_
             secretRef:
               name: subscription-manager
           - secretRef:
               name: redhat-registry
         ansibleVars:
           edpm_bootstrap_command: |
             set -e
             subscription-manager register --username {{ subscription_manager_username }} --password {{ subscription_manager_password }}
             subscription-manager release --set=9.4
             subscription-manager repos --disable=* 
             subscription-manager repos --enable=rhel-9-for-x86_64-baseos-eus-rpms --enable=rhel-9-for-x86_64-appstream-eus-rpms --enable=rhel-9-for-x86_64-highavailability-eus-rpms --enable=fast-datapath-for-rhel-9-x86_64-rpms --enable=rhoso-18.0-for-rhel-9-x86_64-rpms --enable=rhceph-7-tools-for-rhel-9-x86_64-rpms
           edpm_bootstrap_release_version_package: []

   • ansibleUser - The user associated with the secret you created in Creating the data plane secrets.

   • ansibleVars - The Ansible variables that customize the set of nodes. For a list of Ansible variables that you can use, see https://openstack-k8s-operators.github.io/edpm-ansible/.

     For a complete list of the Red Hat Customer Portal registration commands, see https://access.redhat.com/solutions/253273. For information about how to log in to registry.redhat.io, see https://access.redhat.com/RegistryAuthentication#creating-registry-service-accounts-6.

10. Add the network configuration template to apply to your Networker nodes. The following example applies the single NIC VLANs network configuration to the data plane nodes:

      nodeTemplate:
        ...
        ansible:
          ...
          ansibleVars:
            ...
            edpm_network_config_os_net_config_mappings:
              edpm-compute-0:
                nic1: 52:54:04:60:55:22
              edpm-compute-1:
                nic1: 52:54:04:60:55:22
            neutron_physical_bridge_name: br-ex
            edpm_network_config_update: false
            edpm_network_config_template: |
              ---
              {% set mtu_list = [ctlplane_mtu] %}
              {% for network in nodeset_networks %}
              {{ mtu_list.append(lookup(vars, networks_lower[network] ~ _mtu)) }}
              {%- endfor %}
              {% set min_viable_mtu = mtu_list | max %}
              network_config:
              - type: ovs_bridge
                name: {{ neutron_physical_bridge_name }}
                use_dhcp: false
                use_dhcpv6: true
              - type: interface
                name: nic1
                use_dhcp: true
                defroute: false
              - type: interface
                name: nic2
                use_dhcp: false
                defroute: false
                dns_servers: {{ ctlplane_dns_nameservers }}
                domain: {{ dns_search_domains }}
                addresses:
                - ip_netmask: {{ ctlplane_ip }}/{{ ctlplane_cidr }}
              - type: interface
                name: nic3
                use_dhcp: false
                addresses:
                - ip_netmask: {{ lookup(vars, bgpnet0_ip) }}/30
                - ip_netmask: {{ lookup(vars, bgpnet0v6_ip) }}/126
              - type: interface
                name: nic4
                use_dhcp: false
                addresses:
                - ip_netmask: {{ lookup(vars, bgpnet1_ip) }}/30
                - ip_netmask: {{ lookup(vars, bgpnet1v6_ip) }}/126
              - type: interface
                name: lo
                addresses:
                - ip_netmask: {{ lookup(vars, bgpmainnet_ip) }}/32
                - ip_netmask: {{ lookup(vars, bgpmainnetv6_ip) }}/128
                - ip_netmask: {{ lookup(vars, octavia_ip) }}/32      # /128 in case of IPv6

    • nic1 - Update the nic1 to the MAC address assigned to the NIC to use for network configuration on the Networker node.

    • edpm_network_config_update - When deploying a node set for the first time, ensure that the edpm_network_config_update variable is set to false. If you later modify edpm_network_config_template, first set edpm_network_config_update to true. Reset it to false after the update.

      Important

      After an edpm_network_config_template update, you must reset edpm_network_config_update to false. Otherwise, the nodes could lose network access. Whenever edpm_network_config_update is true, the updated network configuration is reapplied every time an OpenStackDataPlaneDeployment CR is created that includes the configure-network service that is a member of the servicesOverride list.

11. Add the common configuration for the set of Networker nodes in this group under the nodeTemplate section. Each node in this OpenStackDataPlaneNodeSet inherits this configuration:

    Example

    edpm_frr_bgp_ipv6: true
    edpm_frr_bgp_ipv6_src_network: bgpmainnetv6
    edpm_frr_bgp_ipv4_src_network: bgpmainnet
    edpm_frr_bgp_neighbor_password: f00barZ
    edpm_frr_bgp_uplinks:
    - nic3
    - nic4
    edpm_ovn_encap_ip: '{{ lookup(''vars'', ''bgpmainnet_ip'') }}' # edpm_ovn_encap_ip can use either bgpmainnet_ip or bgpmainnetv6_ip.

    For more information about data plane network configuration, see Customizing data plane networks in Configuring network services.

12. Define each node in this node set:

      nodes:
        edpm-networker-0:
          hostName: edpm-networker-0
          networks:
          - name: ctlplane
            subnetName: subnet1
            defaultRoute: true
            fixedIP: 192.168.122.100
          - name: BgpNet0
            subnetName: subnet0
            fixedIP: 100.64.0.2
          - name: BgpNet1
            subnetName: subnet0
            fixedIP: 100.65.0.2
          - name: BgpMainNet
            subnetName: subnet0
            fixedIP: 172.30.0.2
          - name: BgpNet0v6
            subnetName: subnet0
            fixedIP: 2620:cf::100:64:0:2
          - name: BgpNet1v6
            subnetName: subnet0
            fixedIP: 2620:cf::100:65:0:2
          - name: BgpMainNetv6
            subnetName: subnet0
            fixedIP: f00d:f00d:f00d:f00d:99:99:0:2
          ansible:
            ansibleHost: 192.168.122.100
            ansibleUser: cloud-admin
            ansibleVars:
              fqdn_internal_api: edpm-networker-0.example.com
          bmhLabelSelector:
            nodeName: edpm-networker-0
        edpm-networker-1:
          hostName: edpm-networker-1
          networks:
          - name: ctlplane
            subnetName: subnet1
            defaultRoute: true
            fixedIP: 192.168.122.101
          - name: BgpNet0
            subnetName: subnet0
            fixedIP: 100.64.0.2
          - name: BgpNet1
            subnetName: subnet0
            fixedIP: 100.65.0.2
          - name: BgpMainNet
            subnetName: subnet0
            fixedIP: 172.30.0.2
          ansible:
            ansibleHost: 192.168.122.101
            ansibleUser: cloud-admin
            ansibleVars:
              fqdn_internal_api: edpm-networker-1.example.com
          bmhLabelSelector:
            nodeName: edpm-networker-1

    • edpm-networker-0 - The node definition reference, for example, edpm-compute-0. Each node in the node set must have a node definition.
    • networks - Defines the IPAM and the DNS records for the node.
    • fixedIP - Specifies a predictable IP address for the network that must be in the allocation range defined for the network in the NetConfig CR.
    • ansibleVars - Node-specific Ansible variables that customize the node.

    • bmhLabelSelector - Optional: metadata labels, such as app, workload, and nodeName are key-value pairs for labelling nodes. Set the bmhLabelSelector field to select data plane nodes based on one or more labels that match the labels in the corresponding BareMetalHost CR.

      Note
      • Nodes defined within the nodes section can configure the same Ansible variables that are configured in the nodeTemplate section. Where an Ansible variable is configured for both a specific node and within the nodeTemplate section, the node-specific values override those from the nodeTemplate section.
      • You do not need to replicate all the nodeTemplate Ansible variables for a node to override the default and set some node-specific values. You only need to configure the Ansible variables you want to override for the node.
      • When you define the networkData Secret for an individual node (such as edpm-compute-0), it acts as a complete override rather than a supplemental configuration. Because node-specific configurations override the inherited default values from the nodeTemplate section, you must ensure that your node-specific networkData Secret contains the full set of required network configurations for that node, not just the unique values.
      • Many ansibleVars include edpm in the name, which stands for "External Data Plane Management".

    For information about the properties you can use to configure node attributes, see OpenStackDataPlaneNodeSet CR properties.

13. In the services section, ensure that the frr and ovn-bgp-agent services are included.

    Example

    services:
    - download-cache
    - redhat
    - bootstrap
    - install-os
    - configure-os
    - configure-network
    - frr
    - validate-network
    - ssh-known-hosts
    - run-os
    - reboot-os
    - install-certs
    - ovn
    - neutron-metadata
    - ovn-bgp-agent

14. Save the openstack_unprovisioned_networker_node_set.yaml definition file.

15. Create the data plane resources:

    $ oc create --save-config -f openstack_unprovisioned_networker_node_set.yaml -n openstack

16. Verify that the data plane resources have been created by confirming that the status is SetupReady:

    $ oc wait openstackdataplanenodeset openstack-networker-nodes --for condition=SetupReady --timeout=10m

    When the status is SetupReady the command returns a condition met message, otherwise it returns a timeout error.

    For information about the data plane conditions and states, see Data plane conditions and states.

17. Verify that the Secret resource was created for the node set:

    $ oc get secret -n openstack | grep openstack-networker-nodes
    dataplanenodeset-openstack-networker-nodes Opaque 1 3m50s

18. Verify that the nodes have transitioned to the provisioned state:

    $ oc get bmh
    NAME            STATE         CONSUMER                    ONLINE   ERROR   AGE
    edpm-compute-0  provisioned   openstack-networker-nodes   true             3d21h

19. Verify that the services were created:

    $ oc get openstackdataplaneservice -n openstack
    NAME                AGE
    download-cache      9m17s
    bootstrap           9m17s
    configure-network   9m17s
    validate-network    9m17s
    frr                 9m17s
    install-os          9m17s
    ...

Procedure

1. Determine the name and status of all deployments:

   $ oc get openstackdataplanedeployment

   The following example output shows two deployments currently in progress:

   $ oc get openstackdataplanedeployment
   
   NAME              NODESETS                       STATUS  MESSAGE
   data-plane-deploy ["openstack-compute-nodes"]    False   Deployment in progress
   data-plane-deploy ["openstack-networker-nodes"]  False   Deployment in progress

2. Determine the name and status of all services and their job condition:

   $ oc get openstackansibleee

   The following example output shows all services and their job condition for all current deployments:

   $ oc get openstackansibleee
   
   NAME                             NETWORKATTACHMENTS   STATUS   MESSAGE
   bootstrap-openstack-edpm         ["ctlplane"]         True     Job complete
   download-cache-openstack-edpm    ["ctlplane"]         False    Job is running
   repo-setup-openstack-edpm        ["ctlplane"]         True     Job complete
   validate-network-another-osdpd   ["ctlplane"]         False    Job is running

   For information on the job condition messages, see Job condition messages.

3. Filter for the name and service for a specific deployment:

   $ oc get openstackansibleee -l \
   openstackdataplanedeployment=<deployment_name>

   • Replace <deployment_name> with the name of the deployment to use to filter the services list.

     The following example filters the list to only show services and their job condition for the data-plane-deploy deployment:

     $ oc get openstackansibleee -l \
     openstackdataplanedeployment=data-plane-deploy
     
     NAME                            NETWORKATTACHMENTS   STATUS   MESSAGE
     bootstrap-openstack-edpm        ["ctlplane"]         True     Job complete
     download-cache-openstack-edpm   ["ctlplane"]         False    Job is running
     repo-setup-openstack-edpm       ["ctlplane"]         True     Job complete

Procedure

1. Retrieve pods with the OpenStackAnsibleEE label:

   $ oc get pods -l app=openstackansibleee
   configure-network-edpm-compute-j6r4l   0/1     Completed           0          3m36s
   validate-network-edpm-compute-6g7n9    0/1     Pending             0          0s
   validate-network-edpm-compute-6g7n9    0/1     ContainerCreating   0          11s
   validate-network-edpm-compute-6g7n9    1/1     Running             0          13s

2. SSH into the pod you want to check:

   1. Pod that is running:

      $ oc rsh validate-network-edpm-compute-6g7n9

   2. Pod that is not running:

      $ oc debug configure-network-edpm-compute-j6r4l

3. List the directories in the /runner/artifacts mount:

   $ ls /runner/artifacts
   configure-network-edpm-compute
   validate-network-edpm-compute

4. View the stdout for the required artifact:

   $ cat /runner/artifacts/configure-network-edpm-compute/stdout