Chapter 5. Creating the data plane

Procedure

1. The Bare Metal Operator (BMO) manages BareMetalHost custom resources (CRs) in the openshift-machine-api namespace by default. Update the Provisioning CR to watch all namespaces:

   $ oc patch provisioning provisioning-configuration --type merge -p '{"spec":{"watchAllNamespaces": true }}'

2. If you are using virtual media boot for bare-metal data plane nodes and the nodes are not connected to a provisioning network, you must update the Provisioning CR to enable virtualMediaViaExternalNetwork, which enables bare-metal connectivity through the external network:

   $ oc patch provisioning provisioning-configuration --type merge -p '{"spec":{"virtualMediaViaExternalNetwork": true }}'

3. Create a file on your workstation that defines the Secret CR with the credentials for accessing the Baseboard Management Controller (BMC) of each bare-metal data plane node in the node set:

   apiVersion: v1
   kind: Secret
   metadata:
     name: edpm-compute-0-bmc-secret
     namespace: openstack
   type: Opaque
   data:
     username: <base64_username>
     password: <base64_password>

   • Replace <base64_username> and <base64_password> with strings that are base64-encoded. You can use the following command to generate a base64-encoded string:

     $ echo -n <string> | base64

     Tip

     If you don’t want to base64-encode the username and password, you can use the stringData field instead of the data field to set the username and password.

4. Create a file named bmh_nodes.yaml on your workstation, that defines the BareMetalHost CR for each bare-metal data plane node. The following example creates a BareMetalHost CR with the provisioning method Redfish virtual media:

   apiVersion: metal3.io/v1alpha1
   kind: BareMetalHost
   metadata:
     name: edpm-compute-0
     namespace: openstack
     labels:
       app: openstack
       workload: compute
   spec:
     bmc:
       address: redfish-virtualmedia+http://192.168.111.1:8000/redfish/v1/Systems/e8efd888-f844-4fe0-9e2e-498f4ab7806d 1
       credentialsName: edpm-compute-0-bmc-secret 2
     bootMACAddress: 00:c7:e4:a7:e7:f3
     bootMode: UEFI
     online: false

   1

   The URL for communicating with the node’s BMC controller. For information on BMC addressing for other boot methods, see BMC addressing in the RHOCP Deploying installer-provisioned clusters on bare metal guide.

   2

   The name of the Secret CR you created in the previous step for accessing the BMC of the node.

   For more information about how to create a BareMetalHost CR, see About the BareMetalHost resource in the RHOCP Postinstallation configuration guide.

5. Create the BareMetalHost resources:

   $ oc create -f bmh_nodes.yaml

6. Verify that the BareMetalHost resources have been created and are in the Available state:

   $ oc get bmh
   NAME         STATE            CONSUMER              ONLINE   ERROR   AGE
   edpm-compute-0   Available      openstack-edpm        true             2d21h
   edpm-compute-1   Available      openstack-edpm        true             2d21h
   ...

Procedure

1. Create a file on your workstation named openstack_unprovisioned_node_set.yaml to define the OpenStackDataPlaneNodeSet CR:

   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneNodeSet
   metadata:
     name: openstack-data-plane 1
     namespace: openstack
   spec:
     tlsEnabled: true
     env: 2
       - name: ANSIBLE_FORCE_COLOR
         value: "True"

   1

   The OpenStackDataPlaneNodeSet CR name must be unique, must consist of lower case alphanumeric characters, - (hyphen) or . (period), must start and end with an alphanumeric character, and must have a maximum length of 20 characters. Update the name in this example to a name that reflects the nodes in the set.

   2

   Optional: A list of environment variables to pass to the pod.

2. Connect the data plane to the control plane network:

   spec:
     ...
     networkAttachments:
       - ctlplane

3. Specify that the nodes in this set are unprovisioned and must be provisioned when creating the resource:

     preProvisioned: false

4. Define the baremetalSetTemplate field to describe the configuration of the bare-metal nodes that must be provisioned when creating the resource:

     baremetalSetTemplate:
       deploymentSSHSecret: dataplane-ansible-ssh-private-key-secret
       bmhNamespace: <bmh_namespace>
       cloudUserName: <ansible_ssh_user>
       bmhLabelSelector:
         app: <bmh_label>
       ctlplaneInterface: <interface>
       dnsSearchDomains:
       - osptest.openstack.org

   • Replace <bmh_namespace> with the namespace defined in the corresponding BareMetalHost CR for the node, for example, openshift-machine-api.
   • Replace <ansible_ssh_user> with the username of the Ansible SSH user, for example, cloud-admin.
   • Replace <bmh_label> with the label defined in the corresponding BareMetalHost CR for the node, for example, openstack.
   • Replace <interface> with the control plane interface the node connects to, for example, enp6s0.

5. Add the SSH key secret that you created to enable Ansible to connect to the data plane nodes:

     nodeTemplate:
       ansibleSSHPrivateKeySecret: <secret-key>

   • Replace <secret-key> with the name of the SSH key Secret CR you created in Creating the data plane secrets, for example, dataplane-ansible-ssh-private-key-secret.
6. Create a Persistent Volume Claim (PVC) on your RHOCP cluster to store logs. Set the volumeType to Filesystem and accessModes to ReadWriteOnce. For information on how to create a PVC, see Understanding persistent storage in the RHOCP Storage guide.

7. Enable persistent logging for the data plane nodes:

     nodeTemplate:
       ...
       extraMounts:
         - extraVolType: Logs
           volumes:
           - name: ansible-logs
             persistentVolumeClaim:
               claimName: <pvc_name>
           mounts:
           - name: ansible-logs
             mountPath: "/runner/artifacts"

   • Replace <pvc_name> with the name of the Persistent Volume Claim (PVC) storage on your RHOCP cluster.

8. Specify the management network:

     nodeTemplate:
       ...
       managementNetwork: ctlplane

9. Specify the Secret CRs used to source the usernames and passwords to register the operating system of the nodes that are not registered to the Red Hat Customer Portal, and enable repositories for your nodes. The following example demonstrates how to register your nodes to CDN. For details on how to register your nodes with Red Hat Satellite 6.13, see Managing Hosts.

     nodeTemplate:
       ansible:
         ansibleUser: cloud-admin 1
         ansiblePort: 22
         ansibleVarsFrom:
           - prefix: subscription_manager_
             secretRef:
               name: subscription-manager
           - secretRef:
               name: redhat-registry
         ansibleVars: 2
           edpm_bootstrap_command: |
             subscription-manager register --username {{ subscription_manager_username }} --password {{ subscription_manager_password }}
             subscription-manager release --set=9.4
             subscription-manager repos --disable=*
             subscription-manager repos --enable=rhel-9-for-x86_64-baseos-eus-rpms --enable=rhel-9-for-x86_64-appstream-eus-rpms --enable=rhel-9-for-x86_64-highavailability-eus-rpms --enable=fast-datapath-for-rhel-9-x86_64-rpms --enable=rhoso-18.0-for-rhel-9-x86_64-rpms --enable=rhceph-7-tools-for-rhel-9-x86_64-rpms
           edpm_bootstrap_release_version_package: []

   1

   The user associated with the secret you created in Creating the data plane secrets.

   2

   The Ansible variables that customize the set of nodes. For a list of Ansible variables that you can use, see https://openstack-k8s-operators.github.io/edpm-ansible/.

   For a complete list of the Red Hat Customer Portal registration commands, see https://access.redhat.com/solutions/253273. For information about how to log into registry.redhat.io, see https://access.redhat.com/RegistryAuthentication#creating-registry-service-accounts-6.

10. Add the network configuration template to apply to your data plane nodes. The following example applies the single NIC VLANs network configuration to the data plane nodes:

      nodeTemplate:
        ...
        ansible:
          ...
          ansibleVars:
            ...
            edpm_network_config_os_net_config_mappings:
              edpm-compute-0:
                nic1: 52:54:04:60:55:22 1
              edpm-compute-1:
                nic1: 52:54:04:60:55:22
            neutron_physical_bridge_name: br-ex
            neutron_public_interface_name: eth0
            edpm_network_config_update: false 2
            edpm_network_config_template: |
              ---
              {% set mtu_list = [ctlplane_mtu] %}
              {% for network in nodeset_networks %}
              {{ mtu_list.append(lookup('vars', networks_lower[network] ~ '_mtu')) }}
              {%- endfor %}
              {% set min_viable_mtu = mtu_list | max %}
              network_config:
              - type: ovs_bridge
                name: {{ neutron_physical_bridge_name }}
                mtu: {{ min_viable_mtu }}
                use_dhcp: false
                dns_servers: {{ ctlplane_dns_nameservers }}
                domain: {{ dns_search_domains }}
                addresses:
                - ip_netmask: {{ ctlplane_ip }}/{{ ctlplane_cidr }}
                routes: {{ ctlplane_host_routes }}
                members:
                - type: interface
                  name: nic1
                  mtu: {{ min_viable_mtu }}
                  # force the MAC address of the bridge to this interface
                  primary: true
              {% for network in nodeset_networks %}
                - type: vlan
                  mtu: {{ lookup('vars', networks_lower[network] ~ '_mtu') }}
                  vlan_id: {{ lookup('vars', networks_lower[network] ~ '_vlan_id') }}
                  addresses:
                  - ip_netmask:
                      {{ lookup('vars', networks_lower[network] ~ '_ip') }}/{{ lookup('vars', networks_lower[network] ~ '_cidr') }}
                  routes: {{ lookup('vars', networks_lower[network] ~ '_host_routes') }}
              {% endfor %}

    1

    Update the nic1 to the MAC address assigned to the NIC to use for network configuration on the Compute node.

    2

    Set the edpm_network_config_update variable to true to apply any updates you make to the network configuration after the node set is deployed.

    Note

    You must reset the edpm_network_config_update variable to false after the updated network configuration is applied in a new OpenStackDataPlaneDeployment CR, otherwise the updated network configuration is reapplied every time an OpenStackDataPlaneDeployment CR is created that includes the configure-network service.

    For more information about data plane network configuration, see Customizing data plane networks in Configuring network services.

11. Add the common configuration for the set of nodes in this group under the nodeTemplate section. Each node in this OpenStackDataPlaneNodeSet inherits this configuration. For information about the properties you can use to configure common node attributes, see OpenStackDataPlaneNodeSet CR properties.

12. Define each node in this node set:

      nodes:
        edpm-compute-0: 1
          hostName: edpm-compute-0
          networks: 2
          - name: ctlplane
            subnetName: subnet1
            defaultRoute: true
            fixedIP: 192.168.122.100 3
          - name: internalapi
            subnetName: subnet1
          - name: storage
            subnetName: subnet1
          - name: tenant
            subnetName: subnet1
          ansible:
            ansibleHost: 192.168.122.100
            ansibleUser: cloud-admin
            ansibleVars: 4
              fqdn_internal_api: edpm-compute-0.example.com
          bmhLabelSelector: 5
            nodeName: edpm-compute-0
        edpm-compute-1:
          hostName: edpm-compute-1
          networks:
          - name: ctlplane
            subnetName: subnet1
            defaultRoute: true
            fixedIP: 192.168.122.101
          - name: internalapi
            subnetName: subnet1
          - name: storage
            subnetName: subnet1
          - name: tenant
            subnetName: subnet1
          ansible:
            ansibleHost: 192.168.122.101
            ansibleUser: cloud-admin
            ansibleVars:
              fqdn_internal_api: edpm-compute-1.example.com
          bmhLabelSelector:
            nodeName: edpm-compute-1

    1

    The node definition reference, for example, edpm-compute-0. Each node in the node set must have a node definition.

    2

    Defines the IPAM and the DNS records for the node.

    3

    Defines the predictable IP addresses for each network.

    4

    Node-specific Ansible variables that customize the node.

    Note
    • Nodes defined within the nodes section can configure the same Ansible variables that are configured in the nodeTemplate section. Where an Ansible variable is configured for both a specific node and within the nodeTemplate section, the node-specific values override those from the nodeTemplate section.
    • You do not need to replicate all the nodeTemplate Ansible variables for a node to override the default and set some node-specific values. You only need to configure the Ansible variables you want to override for the node.
    • Many ansibleVars include edpm in the name, which stands for "External Data Plane Management".

    Optional: The BareMetalHost CR label that selects the BareMetalHost CR for the data plane node. The label can be any label that is defined for the BareMetalHost CR. The label is used with the bmhLabelSelector label configured in the baremetalSetTemplate definition to select the BareMetalHost for the node.

    For information about the properties you can use to configure node attributes, see OpenStackDataPlaneNodeSet CR properties.

13. Save the openstack_unprovisioned_node_set.yaml definition file.

14. Create the data plane resources:

    $ oc create --save-config -f openstack_unprovisioned_node_set.yaml -n openstack

15. Verify that the data plane resources have been created by confirming that the status is SetupReady:

    $ oc wait openstackdataplanenodeset openstack-data-plane --for condition=SetupReady --timeout=10m

    When the status is SetupReady the command returns a condition met message, otherwise it returns a timeout error.

    For information about the data plane conditions and states, see Data plane conditions and states.

16. Verify that the Secret resource was created for the node set:

    $ oc get secret -n openstack | grep openstack-data-plane
    dataplanenodeset-openstack-data-plane Opaque 1 3m50s

17. Verify that the nodes have transitioned to the provisioned state:

    $ oc get bmh
    NAME            STATE         CONSUMER               ONLINE   ERROR   AGE
    edpm-compute-0  provisioned   openstack-data-plane   true             3d21h

18. Verify that the services were created:

    $ oc get openstackdataplaneservice -n openstack
    NAME                    AGE
    bootstrap               8m40s
    ceph-client             8m40s
    ceph-hci-pre            8m40s
    configure-network       8m40s
    configure-os            8m40s
    ...

Procedure

1. Determine the name and status of all deployments:

   $ oc get openstackdataplanedeployment

   The following example output shows two deployments currently in progress:

   $ oc get openstackdataplanedeployment
   
   NAME                   NODESETS             STATUS   MESSAGE
   data-plane-deploy   ["openstack-data-plane-1"]   False    Deployment in progress
   data-plane-deploy   ["openstack-data-plane-2"]   False    Deployment in progress

2. Determine the name and status of all services and their job condition:

   $ oc get openstackansibleee

   The following example output shows all services and their job condition for all current deployments:

   $ oc get openstackansibleee
   
   NAME                             NETWORKATTACHMENTS   STATUS   MESSAGE
   bootstrap-openstack-edpm         ["ctlplane"]         True     Job complete
   download-cache-openstack-edpm    ["ctlplane"]         False    Job is running
   repo-setup-openstack-edpm        ["ctlplane"]         True     Job complete
   validate-network-another-osdpd   ["ctlplane"]         False    Job is running

   For information on the job condition messages, see Job condition messages.

3. Filter for the name and service for a specific deployment:

   $ oc get openstackansibleee -l \
   openstackdataplanedeployment=<deployment_name>

   • Replace <deployment_name> with the name of the deployment to use to filter the services list.

     The following example filters the list to only show services and their job condition for the data-plane-deploy deployment:

     $ oc get openstackansibleee -l \
     openstackdataplanedeployment=data-plane-deploy
     
     NAME                            NETWORKATTACHMENTS   STATUS   MESSAGE
     bootstrap-openstack-edpm        ["ctlplane"]         True     Job complete
     download-cache-openstack-edpm   ["ctlplane"]         False    Job is running
     repo-setup-openstack-edpm       ["ctlplane"]         True     Job complete

Procedure

1. Retrieve pods with the OpenStackAnsibleEE label:

   $ oc get pods -l app=openstackansibleee
   configure-network-edpm-compute-j6r4l   0/1     Completed           0          3m36s
   validate-network-edpm-compute-6g7n9    0/1     Pending             0          0s
   validate-network-edpm-compute-6g7n9    0/1     ContainerCreating   0          11s
   validate-network-edpm-compute-6g7n9    1/1     Running             0          13s

2. SSH into the pod you want to check:

   1. Pod that is running:

      $ oc rsh validate-network-edpm-compute-6g7n9

   2. Pod that is not running:

      $ oc debug configure-network-edpm-compute-j6r4l

3. List the directories in the /runner/artifacts mount:

   $ ls /runner/artifacts
   configure-network-edpm-compute
   validate-network-edpm-compute

4. View the stdout for the required artifact:

   $ cat /runner/artifacts/configure-network-edpm-compute/stdout