Chapter 5. Creating the data plane

Procedure

1. The Bare Metal Operator (BMO) manages BareMetalHost custom resources (CRs) in the openshift-machine-api namespace by default. Update the Provisioning CR to watch all namespaces:

   $ oc patch provisioning provisioning-configuration --type merge -p '{"spec":{"watchAllNamespaces": true }}'

   Copy to Clipboard Toggle word wrap

2. If you are using virtual media boot for bare-metal data plane nodes and the nodes are not connected to a provisioning network, you must update the Provisioning CR to enable virtualMediaViaExternalNetwork, which enables bare-metal connectivity through the external network:

   $ oc patch provisioning provisioning-configuration --type merge -p '{"spec":{"virtualMediaViaExternalNetwork": true }}'

   Copy to Clipboard Toggle word wrap

3. Create a file on your workstation that defines the Secret CR with the credentials for accessing the Baseboard Management Controller (BMC) of each bare-metal data plane node in the node set:

   apiVersion: v1
   kind: Secret
   metadata:
     name: edpm-compute-0-bmc-secret
     namespace: openstack
   type: Opaque
   data:
     username: <base64_username>
     password: <base64_password>

   Copy to Clipboard Toggle word wrap

   • Replace <base64_username> and <base64_password> with strings that are base64-encoded. You can use the following command to generate a base64-encoded string:

     $ echo -n <string> | base64

     Copy to Clipboard Toggle word wrap
     Tip

     If you do not want to base64-encode the username and password, you can use the stringData field instead of the data field to set the username and password.

4. Create a file named bmh_nodes.yaml on your workstation, that defines the BareMetalHost CR for each bare-metal data plane node. The following example creates a BareMetalHost CR with the provisioning method Redfish virtual media:

   apiVersion: metal3.io/v1alpha1
   kind: BareMetalHost
   metadata:
     name: edpm-compute-0
     namespace: openstack
     labels:
       app: openstack
       workload: compute
   spec:
     bmc:
       address: redfish-virtualmedia+http://192.168.111.1:8000/redfish/v1/Systems/e8efd888-f844-4fe0-9e2e-498f4ab7806d
       credentialsName: edpm-compute-0-bmc-secret
     bootMACAddress: 00:c7:e4:a7:e7:f3
     bootMode: UEFI
     online: false
    [preprovisioningNetworkDataName: <network_config_secret_name>]

   Copy to Clipboard Toggle word wrap
   • metadata.labels: Key-value pairs, such as app, workload, and nodeName, that provide varying levels of granularity for labelling nodes. You can use these labels when you create an OpenStackDataPlaneNodeSet CR to describe the configuration of bare-metal nodes to be provisioned or to define nodes in a node set.
   • spec.bmc.address: The URL for communicating with the BMC controller of the node. For information about BMC addressing for other provisioning methods, see BMC addressing in the RHOCP Installing on bare metal guide.
   • spec.bmc.credentialsName: The name of the Secret CR you created in the previous step for accessing the BMC of the node.
   • preprovisioningNetworkDataName: An optional field that specifies the name of the network configuration secret in the local namespace to pass to the pre-provisioning image. The network configuration must be in nmstate format.

   For more information about how to create a BareMetalHost CR, see About the BareMetalHost resource in the RHOCP Installing on bare metal guide.

5. Create the BareMetalHost resources:

   $ oc create -f bmh_nodes.yaml

   Copy to Clipboard Toggle word wrap

6. Verify that the BareMetalHost resources have been created and are in the Available state:

   $ oc wait --for=jsonpath='{.status.provisioning.state}'=available bmh/edpm-compute-baremetal-00 --timeout=<timeout_value>

   Copy to Clipboard Toggle word wrap
   • Replace <timeout_value> with a value in minutes you want the command to wait for completion of the task. For example, if you want the command to wait 60 minutes, you would use the value 60m. Use a value that is appropriate to the size of your deployment. Give large deployments more time to complete deployment tasks.

Procedure

1. Create a file on your workstation named openstack_unprovisioned_node_set.yaml to define the OpenStackDataPlaneNodeSet CR:

   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneNodeSet
   metadata:
     name: openstack-data-plane
     namespace: openstack
   spec:
     tlsEnabled: true
     env:
       - name: ANSIBLE_FORCE_COLOR
         value: "True"

   Copy to Clipboard Toggle word wrap
   • metadata.name: The OpenStackDataPlaneNodeSet CR name must be unique, contain only lower case alphanumeric characters and - (hyphens) or . (periods), start and end with an alphanumeric character, and have a maximum length of 53 characters. Update the name in this example to a name that reflects the nodes in the set.
   • spec.env: An optional list of environment variables to pass to the pod.

2. Connect the data plane to the control plane network:

   spec:
     ...
     networkAttachments:
       - ctlplane

   Copy to Clipboard Toggle word wrap

3. Specify that the nodes in this set are unprovisioned and must be provisioned when creating the resource:

     preProvisioned: false

   Copy to Clipboard Toggle word wrap

4. Define the baremetalSetTemplate field to describe the configuration of the bare-metal nodes that must be provisioned when creating the resource:

     baremetalSetTemplate:
       deploymentSSHSecret: dataplane-ansible-ssh-private-key-secret
       bmhNamespace: <bmh_namespace>
       cloudUserName: <ansible_ssh_user>
       bmhLabelSelector:
         app: <bmh_label>
       ctlplaneInterface: <interface>

   Copy to Clipboard Toggle word wrap
   • Replace <bmh_namespace> with the namespace defined in the corresponding BareMetalHost CR for the node, for example, openstack.
   • Replace <ansible_ssh_user> with the username of the Ansible SSH user, for example, cloud-admin.
   • Replace <bmh_label> with the metadata label defined in the corresponding BareMetalHost CR for the node, for example, openstack. Metadata labels, such as app, workload, and nodeName are key-value pairs for labelling nodes. Set the bmhLabelSelector field to select data plane nodes based on one or more labels that match the labels in the corresponding BareMetalHost CR.
   • Replace <interface> with the control plane interface the node connects to, for example, enp6s0.

5. If you created a custom OpenStackProvisionServer CR, add it to your baremetalSetTemplate definition:

     baremetalSetTemplate:
       ...
       provisionServerName: my-os-provision-server

   Copy to Clipboard Toggle word wrap

6. Add the SSH key secret that you created to enable Ansible to connect to the data plane nodes:

     nodeTemplate:
       ansibleSSHPrivateKeySecret: <secret-key>

   Copy to Clipboard Toggle word wrap
   • Replace <secret-key> with the name of the SSH key Secret CR you created in Creating the data plane secrets, for example, dataplane-ansible-ssh-private-key-secret.
7. Create a Persistent Volume Claim (PVC) in the openstack namespace on your Red Hat OpenShift Container Platform (RHOCP) cluster to store logs. Set the volumeMode to Filesystem and accessModes to ReadWriteOnce. Do not request storage for logs from a PersistentVolume (PV) that uses the NFS volume plugin. NFS is incompatible with FIFO and the ansible-runner creates a FIFO file to write to store logs. For information about PVCs, see Understanding persistent storage in the RHOCP Storage guide and Red Hat OpenShift Container Platform cluster requirements in Planning your deployment.

8. Enable persistent logging for the data plane nodes:

     nodeTemplate:
       ...
       extraMounts:
         - extraVolType: Logs
           volumes:
           - name: ansible-logs
             persistentVolumeClaim:
               claimName: <pvc_name>
           mounts:
           - name: ansible-logs
             mountPath: "/runner/artifacts"

   Copy to Clipboard Toggle word wrap
   • Replace <pvc_name> with the name of the PVC storage on your RHOCP cluster.

9. Specify the management network:

     nodeTemplate:
       ...
       managementNetwork: ctlplane

   Copy to Clipboard Toggle word wrap

10. Specify the Secret CRs used to source the usernames and passwords to register the operating system of your nodes and to enable repositories. The following example demonstrates how to register your nodes to Red Hat Content Delivery Network (CDN). For information about how to register your nodes with Red Hat Satellite 6.13, see Managing Hosts.

      nodeTemplate:
        ansible:
          ansibleUser: cloud-admin
          ansiblePort: 22
          ansibleVarsFrom:
            - secretRef:
                name: subscription-manager
            - secretRef:
                name: redhat-registry
          ansibleVars:
            rhc_release: 9.4
            rhc_repositories:
                - {name: "*", state: disabled}
                - {name: "rhel-9-for-x86_64-baseos-eus-rpms", state: enabled}
                - {name: "rhel-9-for-x86_64-appstream-eus-rpms", state: enabled}
                - {name: "rhel-9-for-x86_64-highavailability-eus-rpms", state: enabled}
                - {name: "fast-datapath-for-rhel-9-x86_64-rpms", state: enabled}
                - {name: "rhoso-18.0-for-rhel-9-x86_64-rpms", state: enabled}
                - {name: "rhceph-7-tools-for-rhel-9-x86_64-rpms", state: enabled}
            edpm_bootstrap_release_version_package: []

    Copy to Clipboard Toggle word wrap
    • ansibleUser: The user associated with the secret you created in Creating the data plane secrets.
    • ansibleVars: The Ansible variables that customize the set of nodes. For a list of Ansible variables that you can use, see https://openstack-k8s-operators.github.io/edpm-ansible/. For a complete list of the Red Hat Customer Portal registration commands, see https://access.redhat.com/solutions/253273. For information about how to log into registry.redhat.io, see https://access.redhat.com/RegistryAuthentication#creating-registry-service-accounts-6.

11. Add the network configuration template to apply to your data plane nodes. The following example applies the single NIC VLANs network configuration to the data plane nodes:

      nodeTemplate:
        ...
        ansible:
          ...
          ansibleVars:
            ...
            edpm_network_config_os_net_config_mappings:
              edpm-compute-0:
                nic1: 52:54:04:60:55:22
              edpm-compute-1:
                nic1: 52:54:04:60:55:22
            neutron_physical_bridge_name: br-ex
            neutron_public_interface_name: eth0
            edpm_network_config_nmstate: true
            edpm_network_config_update: false
            edpm_network_config_template: |
              ---
              {% set mtu_list = [ctlplane_mtu] %}
              {% for network in nodeset_networks %}
              {{ mtu_list.append(lookup('vars', networks_lower[network] ~ '_mtu')) }}
              {%- endfor %}
              {% set min_viable_mtu = mtu_list | max %}
              network_config:
              - type: ovs_bridge
                name: {{ neutron_physical_bridge_name }}
                mtu: {{ min_viable_mtu }}
                use_dhcp: false
                dns_servers: {{ ctlplane_dns_nameservers }}
                domain: {{ dns_search_domains }}
                addresses:
                - ip_netmask: {{ ctlplane_ip }}/{{ ctlplane_cidr }}
                routes: {{ ctlplane_host_routes }}
                members:
                - type: interface
                  name: nic1
                  mtu: {{ min_viable_mtu }}
                  # force the MAC address of the bridge to this interface
                  primary: true
              {% for network in nodeset_networks %}
                - type: vlan
                  mtu: {{ lookup('vars', networks_lower[network] ~ '_mtu') }}
                  vlan_id: {{ lookup('vars', networks_lower[network] ~ '_vlan_id') }}
                  addresses:
                  - ip_netmask:
                      {{ lookup('vars', networks_lower[network] ~ '_ip') }}/{{ lookup('vars', networks_lower[network] ~ '_cidr') }}
                  routes: {{ lookup('vars', networks_lower[network] ~ '_host_routes') }}
              {% endfor %}

    Copy to Clipboard Toggle word wrap
    • nic1: The MAC address assigned to the NIC to use for network configuration on the Compute node.
    • edpm_network_config_nmstate: Sets the os-net-config provider to nmstate. The default value is true. Change it to false only if a specific limitation of the nmstate provider requires you to use the ifcfg provider now. In a future release after nmstate limitations are resolved, the ifcfg provider will be deprecated and removed. In this RHOSO release, adoption of a RHOSP 17.1 deployment with the nmstate provider is not supported. For this and other limitations of RHOSO nmstate support, see https://issues.redhat.com/browse/OSPRH-11309.

    • edpm_network_config_update: When deploying a node set for the first time, set the edpm_network_config_update variable to false. When updating or adopting a node set, set edpm_network_config_update to true.

      Important

      After an update or an adoption, you must reset edpm_network_config_update to false. Otherwise, the nodes could lose network access. Whenever edpm_network_config_update is true, the updated network configuration is reapplied every time an OpenStackDataPlaneDeployment CR is created that includes the configure-network service that is a member of the servicesOverride list.

    • dns_servers: Autogenerated from IPAM and DNS and no user input is required.
    • domain: Autogenerated from IPAM and DNS and no user input is required.
    • routes: Autogenerated from IPAM and DNS and no user input is required.
12. Add the common configuration for the set of nodes in this group under the nodeTemplate section. Each node in this OpenStackDataPlaneNodeSet inherits this configuration. For information about the properties you can use to configure common node attributes, see OpenStackDataPlaneNodeSet CR properties.

13. Define each node in this node set:

      nodes:
        edpm-compute-0:
          hostName: edpm-compute-0
          networks:
          - name: ctlplane
            subnetName: subnet1
            defaultRoute: true
            fixedIP: 192.168.122.100
          - name: internalapi
            subnetName: subnet1
          - name: storage
            subnetName: subnet1
          - name: tenant
            subnetName: subnet1
          networkData:
            name: edpm-compute-0-network-data
            namespace: openstack
          userData:
            name: edpm-compute-0-user-data
            namespace: openstack
          ansible:
            ansibleHost: 192.168.122.100
            ansibleUser: cloud-admin
            ansibleVars:
              fqdn_internal_api: edpm-compute-0.example.com
          bmhLabelSelector:
            nodeName: edpm-compute-0
        edpm-compute-1:
          hostName: edpm-compute-1
          networks:
          - name: ctlplane
            subnetName: subnet1
            defaultRoute: true
            fixedIP: 192.168.122.101
          - name: internalapi
            subnetName: subnet1
          - name: storage
            subnetName: subnet1
          - name: tenant
            subnetName: subnet1
          networkData:
            name: edpm-compute-1-network-data
            namespace: openstack
          userData:
            name: edpm-compute-1-user-data
            namespace: openstack
          ansible:
            ansibleHost: 192.168.122.101
            ansibleUser: cloud-admin
            ansibleVars:
              fqdn_internal_api: edpm-compute-1.example.com
          bmhLabelSelector:
            nodeName: edpm-compute-1

    Copy to Clipboard Toggle word wrap
    • edpm-compute-0: The node definition reference, for example, edpm-compute-0. Each node in the node set must have a node definition.
    • networks: Defines the IPAM and the DNS records for the node.
    • fixedIP: Specifies a predictable IP address for the network that must be in the allocation range defined for the network in the NetConfig CR.
    • networkData: The Secret that contains network configuration that is node-specific.
    • userData.name: The Secret that contains user data that is node-specific.
    • ansibleHost: Overrides the hostname or IP address that Ansible uses to connect to the node. The default value is the value set for the hostName field for the node or the node definition reference, for example, edpm-compute-0.
    • ansibleVars: Node-specific Ansible variables that customize the node.

    • bmhLabelSelector: Metadata labels, such as app, workload, and nodeName are key-value pairs for labelling nodes. Set the bmhLabelSelector field to select data plane nodes based on one or more labels that match the labels in the corresponding BareMetalHost CR.

      Note
      • Nodes defined within the nodes section can configure the same Ansible variables that are configured in the nodeTemplate section. Where an Ansible variable is configured for both a specific node and within the nodeTemplate section, the node-specific values override those from the nodeTemplate section.
      • You do not need to replicate all the nodeTemplate Ansible variables for a node to override the default and set some node-specific values. You only need to configure the Ansible variables you want to override for the node.

      • Many ansibleVars include edpm in the name, which stands for "External Data Plane Management".

        For information about the properties you can use to configure node attributes, see OpenStackDataPlaneNodeSet CR properties.

14. Save the openstack_unprovisioned_node_set.yaml definition file.

15. Create the data plane resources:

    $ oc create --save-config -f openstack_unprovisioned_node_set.yaml -n openstack

    Copy to Clipboard Toggle word wrap

Verification

1. Verify that the data plane resources have been created by confirming that the status is SetupReady:

   $ oc wait openstackdataplanenodeset openstack-data-plane --for condition=SetupReady --timeout=10m

   Copy to Clipboard Toggle word wrap

   When the status is SetupReady the command returns a condition met message, otherwise it returns a timeout error.

   For information about the data plane conditions and states, see Data plane conditions and states.

2. Verify that the Secret resource was created for the node set:

   $ oc get secret -n openstack | grep openstack-data-plane
   dataplanenodeset-openstack-data-plane Opaque 1 3m50s

   Copy to Clipboard Toggle word wrap

3. Verify that the nodes have transitioned to the provisioned state:

   $ oc get bmh
   NAME            STATE         CONSUMER               ONLINE   ERROR   AGE
   edpm-compute-0  provisioned   openstack-data-plane   true             3d21h

   Copy to Clipboard Toggle word wrap

4. Verify that the services were created:

   $ oc get openstackdataplaneservice -n openstack
   NAME                    AGE
   bootstrap               8m40s
   ceph-client             8m40s
   ceph-hci-pre            8m40s
   configure-network       8m40s
   configure-os            8m40s
   ...

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a Secret CR with preprovisioningNetworkData in nmstate format for each BareMetalHost CR:

   apiVersion: v1
   kind: Secret
   metadata:
     name: leaf0-0-preprovision-network-data
     namespace: openstack
   stringData:
     nmstate: |
       interfaces:
         - name: enp5s0
           type: ethernet
           state: up
           ipv4:
             enabled: true
             address:
               - ip: 192.168.130.100
                 prefix-length: 24
       dns-resolver:
         config:
           server:
             - 192.168.122.1
       routes:
         config:
           - destination: 0.0.0.0/0
             next-hop-address: 192.168.130.1
             next-hop-interface: enp5s0
   type: Opaque

   Copy to Clipboard Toggle word wrap

2. Create the Secret resources:

   $ oc create -f secret_leaf0-0.yaml

   Copy to Clipboard Toggle word wrap
3. Open the BareMetalHost CR file, for example, bmh_nodes.yaml.

4. Add the preprovisioningNetworkDataName field to each BareMetalHost CR defined for each node in the bmh_nodes.yaml file:

   apiVersion: metal3.io/v1alpha1
   kind: BareMetalHost
   metadata:
     annotations:
       inspect.metal3.io: disabled
     labels:
       app: openstack
       nodeset: leaf0
     name: leaf0-0
     namespace: openstack
   spec:
     architecture: x86_64
     automatedCleaningMode: metadata
     bmc:
       address: redfish-virtualmedia+http://sushy.utility:8000/redfish/v1/Systems/df2bf92f-3e2c-47e1-b1fa-0d2e06bd1b1d
       credentialsName: bmc-secret
     bootMACAddress: 52:54:04:15:a8:d9
     bootMode: UEFI
     online: false
     preprovisioningNetworkDataName: leaf0-0-preprovision-network-data
     rootDeviceHints:
       deviceName: /dev/sda

   Copy to Clipboard Toggle word wrap

5. Update the BareMetalHost CRs:

   $ oc apply -f bmh_nodes.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Determine the name and status of all deployments:

   $ oc get openstackdataplanedeployment

   Copy to Clipboard Toggle word wrap

   The following example output shows two deployments currently in progress:

   $ oc get openstackdataplanedeployment
   
   NAME                   NODESETS             STATUS   MESSAGE
   edpm-compute   ["openstack-edpm-ipam"]   False    Deployment in progress

   Copy to Clipboard Toggle word wrap

2. Retrieve and inspect Ansible execution jobs.

   The Kubernetes jobs are labelled with the name of the OpenStackDataPlaneDeployment. You can list jobs for each OpenStackDataPlaneDeployment by using the label:

    $ oc get job -l openstackdataplanedeployment=edpm-compute
    NAME                                                 STATUS     COMPLETIONS   DURATION   AGE
    bootstrap-edpm-compute-openstack-edpm-ipam           Complete   1/1           78s        25h
    configure-network-edpm-compute-openstack-edpm-ipam   Complete   1/1           37s        25h
    configure-os-edpm-compute-openstack-edpm-ipam        Complete   1/1           66s        25h
    download-cache-edpm-compute-openstack-edpm-ipam      Complete   1/1           64s        25h
    install-certs-edpm-compute-openstack-edpm-ipam       Complete   1/1           46s        25h
    install-os-edpm-compute-openstack-edpm-ipam          Complete   1/1           57s        25h
    libvirt-edpm-compute-openstack-edpm-ipam             Complete   1/1           2m37s      25h
    neutron-metadata-edpm-compute-openstack-edpm-ipam    Complete   1/1           61s        25h
    nova-edpm-compute-openstack-edpm-ipam                Complete   1/1           3m20s      25h
    ovn-edpm-compute-openstack-edpm-ipam                 Complete   1/1           78s        25h
    run-os-edpm-compute-openstack-edpm-ipam              Complete   1/1           33s        25h
    ssh-known-hosts-edpm-compute                         Complete   1/1           19s        25h
    telemetry-edpm-compute-openstack-edpm-ipam           Complete   1/1           2m5s       25h
    validate-network-edpm-compute-openstack-edpm-ipam    Complete   1/1           16s        25h

   Copy to Clipboard Toggle word wrap

   You can check logs by using oc logs -f job/<job-name>, for example, if you want to check the logs from the configure-network job:

    $ oc logs -f jobs/configure-network-edpm-compute-openstack-edpm-ipam | tail -n2
    PLAY RECAP *********************************************************************
    edpm-compute-0             : ok=22   changed=0    unreachable=0    failed=0    skipped=17   rescued=0    ignored=0

   Copy to Clipboard Toggle word wrap

Procedure

1. Retrieve pods with the OpenStackAnsibleEE label:

   $ oc get pods -l app=openstackansibleee
   configure-network-edpm-compute-j6r4l   0/1     Completed           0          3m36s
   validate-network-edpm-compute-6g7n9    0/1     Pending             0          0s
   validate-network-edpm-compute-6g7n9    0/1     ContainerCreating   0          11s
   validate-network-edpm-compute-6g7n9    1/1     Running             0          13s

   Copy to Clipboard Toggle word wrap

2. SSH into the pod you want to check:

   1. Pod that is running:

      $ oc rsh validate-network-edpm-compute-6g7n9

      Copy to Clipboard Toggle word wrap

   2. Pod that is not running:

      $ oc debug configure-network-edpm-compute-j6r4l

      Copy to Clipboard Toggle word wrap

3. List the directories in the /runner/artifacts mount:

   $ ls /runner/artifacts
   configure-network-edpm-compute
   validate-network-edpm-compute

   Copy to Clipboard Toggle word wrap

4. View the stdout for the required artifact:

   $ cat /runner/artifacts/configure-network-edpm-compute/stdout

   Copy to Clipboard Toggle word wrap