Deploying a RHOSO environment with a routed spine-leaf network topology

Procedure

1. Log in to the RHOCP web console as a user with cluster-admin permissions.
2. Select Operators → OperatorHub.
3. In the Filter by keyword field, type OpenStack.
4. Click the OpenStack Operator tile with the Red Hat source label.
5. Read the information about the Operator and click Install.
6. On the Install Operator page, select "Operator recommended Namespace: openstack-operators" from the Installed Namespace list.
7. On the Install Operator page, select "Manual" from the Update approval list. For information about how to manually approve a pending Operator update, see Manually approving a pending Operator update in the RHOCP Operators guide.
8. Click Install to make the Operator available to the openstack-operators namespace. The OpenStack Operator is installed when the Status is Succeeded.
9. Click Create OpenStack to open the Create OpenStack page.
10. On the Create OpenStack page, click Create to create an instance of the OpenStack Operator initialization resource. The OpenStack Operator is ready to use when the Status of the openstack instance is Conditions: Ready.

Procedure

1. Create the openstack-operators namespace for the RHOSP operators:

   $ cat << EOF | oc apply -f -
   apiVersion: v1
   kind: Namespace
   metadata:
     name: openstack-operators
   spec:
     finalizers:
     - kubernetes
   EOF

2. Create the OperatorGroup CR in the openstack-operators namespace:

   $ cat << EOF | oc apply -f -
   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: openstack
     namespace: openstack-operators
   EOF

3. Create the Subscription CR that subscribes to openstack-operator:

   $ cat << EOF| oc apply -f -
   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: openstack-operator
     namespace: openstack-operators
   spec:
     name: openstack-operator
     channel: stable-v1.0
     source: redhat-operators
     sourceNamespace: openshift-marketplace
     installPlanApproval: Manual
   EOF

4. Wait for the install plan to be created:

   $ oc get installplan -n openstack-operators -o json | jq -r '.items[] | select(.spec.approval=="Manual" and .spec.approved==false) | .metadata.name' | head -n1

5. Approve the install plan:

   $ oc patch installplan <install_plan_name> -n openstack-operators --type merge -p '{"spec":{"approved":true}}'

6. Verify that the OpenStack Operator is installed:

   $ oc wait csv -n openstack-operators \
    -l operators.coreos.com/openstack-operator.openstack-operators="" \
    --for jsonpath='{.status.phase}'=Succeeded

7. Create an instance of the openstack-operator:

   $ cat << EOF | oc apply -f -
   apiVersion: operator.openstack.org/v1beta1
   kind: OpenStack
   metadata:
     name: openstack
     namespace: openstack-operators
   EOF

8. Confirm that the Openstack Operator is deployed:

   $ oc wait openstack/openstack -n openstack-operators --for condition=Ready --timeout=500s

Procedure

1. Create the openstack project for the deployed RHOSO environment:

   $ oc new-project openstack

2. Ensure the openstack namespace is labeled to enable privileged pod creation by the OpenStack Operators:

   $ oc get namespace openstack -ojsonpath='{.metadata.labels}' | jq
   {
     "kubernetes.io/metadata.name": "openstack",
     "pod-security.kubernetes.io/enforce": "privileged",
     "security.openshift.io/scc.podSecurityLabelSync": "false"
   }

   If the security context constraint (SCC) is not "privileged", use the following commands to change it:

   $ oc label ns openstack security.openshift.io/scc.podSecurityLabelSync=false --overwrite
   $ oc label ns openstack pod-security.kubernetes.io/enforce=privileged --overwrite

3. Optional: To remove the need to specify the namespace when executing commands on the openstack namespace, set the default namespace to openstack:

   $ oc project openstack

Procedure

1. Create a Secret CR on your workstation, for example, openstack_service_secret.yaml.

2. Add the following initial configuration to openstack_service_secret.yaml:

   apiVersion: v1
   data:
     AdminPassword: <base64_password>
     AodhPassword: <base64_password>
     BarbicanPassword: <base64_password>
     BarbicanSimpleCryptoKEK: <base64_fernet_key>
     CeilometerPassword: <base64_password>
     CinderPassword: <base64_password>
     DbRootPassword: <base64_password>
     DesignatePassword: <base64_password>
     GlancePassword: <base64_password>
     HeatAuthEncryptionKey: <base64_password>
     HeatPassword: <base64_password>
     IronicInspectorPassword: <base64_password>
     IronicPassword: <base64_password>
     ManilaPassword: <base64_password>
     MetadataSecret: <base64_password>
     NeutronPassword: <base64_password>
     NovaPassword: <base64_password>
     OctaviaPassword: <base64_password>
     PlacementPassword: <base64_password>
     SwiftPassword: <base64_password>
   kind: Secret
   metadata:
     name: osp-secret
     namespace: openstack
   type: Opaque

   • Replace <base64_password> with a 32-character key that is base64 encoded.

     Note

     The HeatAuthEncryptionKey password must be a 32-character key for Orchestration service (heat) encryption. If you increase the length of the passwords for all other services, ensure that the HeatAuthEncryptionKey password remains at length 32.

     You can use the following command to manually generate a base64 encoded password:

     $ echo -n <password> | base64

     Alternatively, if you are using a Linux workstation and you are generating the Secret CR by using a Bash command such as cat, you can replace <base64_password> with the following command to auto-generate random passwords for each service:

     $(tr -dc 'A-Za-z0-9' < /dev/urandom | head -c 32 | base64)

   • Replace the <base64_fernet_key> with a base64 encoded fernet key. You can use the following command to manually generate it:

     $(python3 -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode('UTF-8'))" | base64)

3. Create the Secret CR in the cluster:

   $ oc create -f openstack_service_secret.yaml -n openstack

4. Verify that the Secret CR is created:

   $ oc describe secret osp-secret -n openstack

Procedure

1. Create a NodeNetworkConfigurationPolicy (nncp) CR file on your workstation, for example, openstack-nncp.yaml.

2. Retrieve the names of the worker nodes in the RHOCP cluster:

   $ oc get nodes -l node-role.kubernetes.io/worker -o jsonpath="{.items[*].metadata.name}"

3. Discover the network configuration:

   $ oc get nns/<worker_node> -o yaml | more

   • Replace <worker_node> with the name of a worker node retrieved in step 2, for example, worker-1. Repeat this step for each worker node.

4. In the nncp CR file, configure the interfaces for each isolated network on each worker node in the RHOCP cluster. For information about the default physical data center networks that must be configured with network isolation, see Default Red Hat OpenStack Services on OpenShift networks.

   In the following example, the nncp CR configures the enp6s0 interface for worker node 1, osp-enp6s0-worker-1, to use VLAN interfaces with IPv4 addresses for network isolation:

   apiVersion: nmstate.io/v1
   kind: NodeNetworkConfigurationPolicy
   metadata:
     name: osp-enp6s0-worker-1
   spec:
     desiredState:
       interfaces:
       - description: internalapi vlan interface
         ipv4:
           address:
           - ip: 172.17.0.10
             prefix-length: 24
           enabled: true
           dhcp: false
         ipv6:
           enabled: false
         name: internalapi
         state: up
         type: vlan
         vlan:
           base-iface: enp6s0
           id: 20
           reorder-headers: true
       - description: storage vlan interface
         ipv4:
           address:
           - ip: 172.18.0.10
             prefix-length: 24
           enabled: true
           dhcp: false
         ipv6:
           enabled: false
         name: storage
         state: up
         type: vlan
         vlan:
           base-iface: enp6s0
           id: 21
           reorder-headers: true
       - description: tenant vlan interface
         ipv4:
           address:
           - ip: 172.19.0.10
             prefix-length: 24
           enabled: true
           dhcp: false
         ipv6:
           enabled: false
         name: tenant
         state: up
         type: vlan
         vlan:
           base-iface: enp6s0
           id: 22
           reorder-headers: true
       - description: Configuring enp6s0
         ipv4:
           address:
           - ip: 192.168.122.10
             prefix-length: 24
           enabled: true
           dhcp: false
         ipv6:
           enabled: false
         mtu: 1500
         name: enp6s0
         state: up
         type: ethernet
     nodeSelector:
       kubernetes.io/hostname: worker-1
       node-role.kubernetes.io/worker: ""

5. In the nncp CR file, add routing configuration for each isolated network on each worker node in the RHOCP cluster:

   apiVersion: nmstate.io/v1
   kind: NodeNetworkConfigurationPolicy
   metadata:
     name: osp-enp6s0-worker-1
     namespace: openstack
   spec:
     desiredState:
       interfaces:
         <....>
       routes:
         config:
         - destination: 192.168.123.0/24
           metric: 150
           next-hop-address: 192.168.122.1
           next-hop-interface: enp6s0
         - destination: 192.168.124.0/24
           metric: 150
           next-hop-address: 192.168.122.1
           next-hop-interface: enp6s0
         - destination: 172.17.1.0/24
           metric: 150
           next-hop-address: 172.17.0.1
           next-hop-interface: internalapi
         - destination: 172.17.2.0/24
           metric: 150
           next-hop-address: 172.17.0.1
           next-hop-interface: internalapi
         - destination: 172.18.1.0/24
           metric: 150
           next-hop-address: 172.18.0.1
           next-hop-interface: storage
         - destination: 172.18.2.0/24
           metric: 150
           next-hop-address: 172.18.0.1
           next-hop-interface: storage
         - destination: 172.19.1.0/24
           metric: 150
           next-hop-address: 172.19.0.1
           next-hop-interface: tenant
         - destination: 172.19.2.0/24
           metric: 150
           next-hop-address: 172.19.0.1
           next-hop-interface: tenant

6. Create the nncp CR in the cluster:

   $ oc apply -f openstack-nncp.yaml

7. Verify that the nncp CR is created:

   $ oc get nncp -w
   NAME                        STATUS        REASON
   osp-enp6s0-worker-1   Progressing   ConfigurationProgressing
   osp-enp6s0-worker-1   Progressing   ConfigurationProgressing
   osp-enp6s0-worker-1   Available     SuccessfullyConfigured

8. Create a NetworkAttachmentDefinition (net-attach-def) CR file on your workstation, for example, openstack-net-attach-def.yaml.

9. In the NetworkAttachmentDefinition CR file, configure a NetworkAttachmentDefinition resource for each isolated network to attach a service deployment pod to the network that includes routing configuration for each network. The following example creates a NetworkAttachmentDefinition resource for the ctlplane network of type macvlan:

   apiVersion: k8s.cni.cncf.io/v1
   kind: NetworkAttachmentDefinition
   metadata:
     name: ctlplane
     namespace: openstack
   spec:
     config: |
       {
         "cniVersion": "0.3.1",
         "name": "ctlplane",
         "type": "macvlan",
         "master": "enp6s0",
         "ipam": {
           "type": "whereabouts",
           "range": "192.168.122.0/24",
           "range_start": "192.168.122.30",
           "range_end": "192.168.122.70",
           "routes": [
             {
               "dst": "192.168.123.0/24",
               "gw": "192.168.122.1"
             },
             {
               "dst": "192.168.124.0/24",
               "gw": "192.168.122.1"
             }
           ]
         }
       }

   • namespace - The namespace where the services are deployed.
   • master - The node interface name associated with the network, as defined in the nncp CR.
   • ipam - The whereabouts CNI IPAM plugin to assign IPs to the created pods from the range .30 - .70.
   • `` - The IP address pool range must not overlap with the MetalLB IPAddressPool range and the NetConfig allocationRange.
   • dst - The destination network in CIDR notation.
   • gw - The gateway or router IP address.

10. Create the NetworkAttachmentDefinition CR in the cluster:

    $ oc apply -f openstack-net-attach-def.yaml

11. Verify that the NetworkAttachmentDefinition CR is created:

    $ oc get net-attach-def -n openstack

Procedure

1. Create an IPAddressPool CR file on your workstation, for example, openstack-ipaddresspools.yaml.

2. In the IPAddressPool CR file, configure an IPAddressPool resource on the isolated network to specify the IP address ranges over which MetalLB has authority:

   apiVersion: metallb.io/v1beta1
   kind: IPAddressPool
   metadata:
     name: internalapi
     namespace: metallb-system
   spec:
     addresses:
       - 172.17.0.80-172.17.0.90
     autoAssign: true
     avoidBuggyIPs: false
   ---
   apiVersion: metallb.io/v1beta1
   kind: IPAddressPool
   metadata:
     namespace: metallb-system
     name: ctlplane
   spec:
     addresses:
       - 192.168.122.80-192.168.122.90
     autoAssign: true
     avoidBuggyIPs: false
   ---
   apiVersion: metallb.io/v1beta1
   kind: IPAddressPool
   metadata:
     namespace: metallb-system
     name: storage
   spec:
     addresses:
       - 172.18.0.80-172.18.0.90
     autoAssign: true
     avoidBuggyIPs: false
   ---
   apiVersion: metallb.io/v1beta1
   kind: IPAddressPool
   metadata:
     namespace: metallb-system
     name: tenant
   spec:
     addresses:
       - 172.19.0.80-172.19.0.90
     autoAssign: true
     avoidBuggyIPs: false

   • addresses - The IPAddressPool range must not overlap with the whereabouts IPAM range and the NetConfig allocationRange.

   For information about how to configure the other IPAddressPool resource parameters, see Configuring MetalLB address pools in the RHOCP Networking guide.

3. Create the IPAddressPool CR in the cluster:

   $ oc apply -f openstack-ipaddresspools.yaml

4. Verify that the IPAddressPool CR is created:

   $ oc describe -n metallb-system IPAddressPool

5. Create a L2Advertisement CR file on your workstation, for example, openstack-l2advertisement.yaml.

6. In the L2Advertisement CR file, configure L2Advertisement CRs to define which node advertises a service to the local network. Create one L2Advertisement resource for each network.

   In the following example, each L2Advertisement CR specifies that the VIPs requested from the network address pools are announced on the interface that is attached to the VLAN:

   apiVersion: metallb.io/v1beta1
   kind: L2Advertisement
   metadata:
     name: internalapi
     namespace: metallb-system
   spec:
     ipAddressPools:
     - internalapi
     interfaces:
     - internalapi
   ---
   apiVersion: metallb.io/v1beta1
   kind: L2Advertisement
   metadata:
     name: ctlplane
     namespace: metallb-system
   spec:
     ipAddressPools:
     - ctlplane
     interfaces:
     - enp6s0
   ---
   apiVersion: metallb.io/v1beta1
   kind: L2Advertisement
   metadata:
     name: storage
     namespace: metallb-system
   spec:
     ipAddressPools:
     - storage
     interfaces:
     - storage
   ---
   apiVersion: metallb.io/v1beta1
   kind: L2Advertisement
   metadata:
     name: tenant
     namespace: metallb-system
   spec:
     ipAddressPools:
     - tenant
     interfaces:
     - tenant

   • internalapi - The interface where the VIPs requested from the VLAN address pool are announced.

   For information about how to configure the other L2Advertisement resource parameters, see Configuring MetalLB with a L2 advertisement and label in the RHOCP Networking guide.

7. Create the L2Advertisement CRs in the cluster:

   $ oc apply -f openstack-l2advertisement.yaml

8. Verify that the L2Advertisement CRs are created:

   $ oc get -n metallb-system L2Advertisement
   NAME          IPADDRESSPOOLS    IPADDRESSPOOL SELECTORS   INTERFACES
   ctlplane      ["ctlplane"]                                ["enp6s0"]
   internalapi   ["internalapi"]                             ["internalapi"]
   storage       ["storage"]                                 ["storage"]
   tenant        ["tenant"]                                  ["tenant"]

9. If your cluster has OVNKubernetes as the network back end, then you must enable global forwarding so that MetalLB can work on a secondary network interface.

   1. Check the network back end used by your cluster:

      $ oc get network.operator cluster --output=jsonpath='{.spec.defaultNetwork.type}'

   2. If the back end is OVNKubernetes, then run the following command to enable global IP forwarding:

      $ oc patch network.operator cluster -p '{"spec":{"defaultNetwork":{"ovnKubernetesConfig":{"gatewayConfig":{"ipForwarding": "Global"}}}}}' --type=merge

Procedure

1. Create a file named openstack_netconfig.yaml on your workstation.

2. Add the following configuration to openstack_netconfig.yaml to create the NetConfig CR:

   apiVersion: network.openstack.org/v1beta1
   kind: NetConfig
   metadata:
     name: openstacknetconfig
     namespace: openstack

3. In the openstack_netconfig.yaml file, define the topology for each isolated network on the data plane by defining a specification for each isolated network, a subnet for each leaf for each isolated network, and routing configuration for each subnet. For information about the default RHOSO networks, see Default Red Hat OpenStack Services on OpenShift networks. The following example creates the ctlplane isolated network for the data plane, with three subnets and routing configuration for each subnet:

   spec:
     networks:
     - name: ctlplane
       dnsDomain: ctlplane.example.com
       subnets:
       - name: subnet1
         allocationRanges:
         - end: 192.168.122.120
           start: 192.168.122.100
         - end: 192.168.122.200
           start: 192.168.122.150
         cidr: 192.168.122.0/24
         gateway: 192.168.122.1
         routes:
         - destination: 192.168.123.0/24
           nexthop: 192.168.122.1
         - destination: 192.168.124.0/24
           nexthop: 192.168.122.1
       - name: subnet2
         allocationRanges:
         - end: 192.168.123.120
           start: 192.168.123.100
         - end: 192.168.123.200
           start: 192.168.123.150
         cidr: 192.168.123.0/24
         gateway: 192.168.123.1
         routes:
         - destination: 192.168.122.0/24
           nexthop: 192.168.123.1
         - destination: 192.168.124.0/24
           nexthop: 192.168.123.1
       - name: subnet3
         allocationRanges:
         - end: 192.168.124.120
           start: 192.168.124.100
         - end: 192.168.124.200
           start: 192.168.124.150
         cidr: 192.168.124.0/24
         gateway: 192.168.124.1
         routes:
         - destination: 192.168.122.0/24
           nexthop: 192.168.124.1
         - destination: 192.168.123.0/24
           nexthop: 192.168.124.1

   • name - The name of the network.
   • subnets - The IPv4 subnet specification.
   • name - The name of the subnet, for example, subnet1.
   • allocationRanges - The NetConfig allocationRange. The allocationRange must not overlap with the MetalLB IPAddressPool range and the IP address pool range.
   • routes - Routing configuration for the subnet.
4. Save the openstack_netconfig.yaml definition file.

5. Create the data plane network:

   $ oc create -f openstack_netconfig.yaml -n openstack

6. To verify that the data plane network is created, view the openstacknetconfig resource:

   $ oc get netconfig/openstacknetconfig -n openstack

   If you see errors, check the underlying network-attach-definition and node network configuration policies:

   $ oc get network-attachment-definitions -n openstack
   $ oc get nncp

Procedure

1. Create a file on your workstation named openstack_control_plane.yaml to define the OpenStackControlPlane CR:

   apiVersion: core.openstack.org/v1beta1
   kind: OpenStackControlPlane
   metadata:
     name: openstack-control-plane
     namespace: openstack

2. Specify the Secret CR you created to provide secure access to the RHOSO service pods in Providing secure access to the Red Hat OpenStack Services on OpenShift services:

   apiVersion: core.openstack.org/v1beta1
   kind: OpenStackControlPlane
   metadata:
     name: openstack-control-plane
     namespace: openstack
   spec:
     secret: osp-secret

3. Specify the storageClass you created for your Red Hat OpenShift Container Platform (RHOCP) cluster storage back end:

   spec:
     secret: osp-secret
     storageClass: <RHOCP_storage_class>

   • Replace <RHOCP_storage_class> with the storage class you created for your RHOCP cluster storage back end. For information about storage classes, see Creating a storage class.

4. Add the global RabbitMQ settings messagingBus and notificationsBus to specify the default RabbitMQ cluster for RHOSO services:

   apiVersion: core.openstack.org/v1beta1
   kind: OpenStackControlPlane
   metadata:
     name: openstack
   spec:
     messagingBus:
       cluster: <rabbitmq-cluster>
   
     notificationsBus:
       cluster: <rabbitmq-cluster>

   • Replace <rabbitmq-cluster> with the default RabbitMQ cluster that all the RHOSO services use, in this example, rabbitmq. You can customize the RabbitMQ interface for OpenStack services. For more information, see Understand the RabbitMQ interface for OpenStack services in the Monitoring high availability services guide.

5. Add the following service configurations:

   Note
   • The following service examples use IP addresses from the default RHOSO MetalLB IPAddressPool range for the loadBalancerIPs field. Update the loadBalancerIPs field with the IP address from the MetalLB IPAddressPool range that you created.
   • You cannot override the default public service endpoint. The public service endpoints are exposed as RHOCP routes by default, because only routes are supported for public endpoints.

   • Block Storage service (cinder):

       cinder:
         apiOverride:
           route: {}
         template:
           databaseInstance: openstack
           secret: osp-secret
           cinderAPI:
             replicas: 3
             override:
               service:
                 internal:
                   metadata:
                     annotations:
                       metallb.universe.tf/address-pool: internalapi
                       metallb.universe.tf/allow-shared-ip: internalapi
                       metallb.universe.tf/loadBalancerIPs: 172.17.0.80
                   spec:
                     type: LoadBalancer
           cinderScheduler:
             replicas: 1
           cinderBackup:
             networkAttachments:
             - storage
             replicas: 0
           cinderVolumes:
             volume1:
               networkAttachments:
               - storage
               replicas: 0

     • cinderBackup.replicas: You can deploy the initial control plane without activating the cinderBackup service. To deploy the service, you must set the number of replicas for the service and configure the back end for the service. For information about the recommended replicas for each service and how to configure a back end for the Block Storage service and the backup service, see Configuring the Block Storage backup service in Configuring persistent storage.
     • cinderVolumes.replicas: You can deploy the initial control plane without activating the cinderVolumes service. To deploy the service, you must set the number of replicas for the service and configure the back end for the service. For information about the recommended replicas for the cinderVolumes service and how to configure a back end for the service, see Configuring the Block Storage volume service component in Configuring persistent storage.

   • Compute service (nova):

       nova:
         apiOverride:
           route: {}
         template:
           apiServiceTemplate:
             replicas: 3
             override:
               service:
                 internal:
                   metadata:
                     annotations:
                       metallb.universe.tf/address-pool: internalapi
                       metallb.universe.tf/allow-shared-ip: internalapi
                       metallb.universe.tf/loadBalancerIPs: 172.17.0.80
                   spec:
                     type: LoadBalancer
           metadataServiceTemplate:
             replicas: 3
             override:
               service:
                 metadata:
                   annotations:
                     metallb.universe.tf/address-pool: internalapi
                     metallb.universe.tf/allow-shared-ip: internalapi
                     metallb.universe.tf/loadBalancerIPs: 172.17.0.80
                 spec:
                   type: LoadBalancer
           schedulerServiceTemplate:
             replicas: 3
           cellTemplates:
             cell0:
               cellDatabaseAccount: nova-cell0
               cellDatabaseInstance: openstack
               messagingBus:
                 cluster: rabbitmq
               hasAPIAccess: true
             cell1:
               cellDatabaseAccount: nova-cell1
               cellDatabaseInstance: openstack-cell1
               messagingBus:
                 cluster: rabbitmq-cell1
               noVNCProxyServiceTemplate:
                 enabled: true
                 networkAttachments:
                 - ctlplane
               hasAPIAccess: true
           secret: osp-secret

     Note

     A full set of Compute services (nova) are deployed by default for each of the default cells, cell0 and cell1: nova-api, nova-metadata, nova-scheduler, and nova-conductor. The novncproxy service is also enabled for cell1 by default.

   • DNS service for the data plane:

       dns:
         template:
           options:
           - key: server
             values:
             - <IP address for DNS server reachable from dnsmasq pod>
           override:
             service:
               metadata:
                 annotations:
                   metallb.universe.tf/address-pool: ctlplane
                   metallb.universe.tf/allow-shared-ip: ctlplane
                   metallb.universe.tf/loadBalancerIPs: 192.168.122.80
               spec:
                 type: LoadBalancer
           replicas: 2

     • options: Defines the dnsmasq instances required for each DNS server by using key-value pairs. In this example, there is one key-value pair defined because there is only one DNS server configured to forward requests to.

     • key: Specifies the dnsmasq parameter to customize for the deployed dnsmasq instance. Set to one of the following valid values:

       • server
       • rev-server
       • srv-host
       • txt-record
       • ptr-record
       • rebind-domain-ok
       • naptr-record
       • cname
       • host-record
       • caa-record
       • dns-rr
       • auth-zone
       • synth-domain
       • no-negcache
       • local

     • values: Specifies the value for the DNS server reachable from the dnsmasq pod on the RHOCP cluster network. You can specify a generic DNS server as the value, for example, 1.1.1.1, or a DNS server for a specific domain, for example, /google.com/8.8.8.8.

       Note

       This DNS service, dnsmasq, provides DNS services for nodes on the RHOSO data plane. dnsmasq is different from the RHOSO DNS service (designate) that provides DNS as a service for cloud tenants.

   • Identity service (keystone)

       keystone:
         apiOverride:
           route: {}
         template:
           override:
             service:
               internal:
                 metadata:
                   annotations:
                     metallb.universe.tf/address-pool: internalapi
                     metallb.universe.tf/allow-shared-ip: internalapi
                     metallb.universe.tf/loadBalancerIPs: 172.17.0.80
                 spec:
                   type: LoadBalancer
           databaseInstance: openstack
           secret: osp-secret
           replicas: 3

   • Image service (glance):

       glance:
         apiOverrides:
           default:
             route: {}
         template:
           databaseInstance: openstack
           storage:
             storageRequest: 10G
           secret: osp-secret
           keystoneEndpoint: default
           glanceAPIs:
             default:
               replicas: 0 # Configure back end; set to 3 when deploying service
               override:
                 service:
                   internal:
                     metadata:
                       annotations:
                         metallb.universe.tf/address-pool: internalapi
                         metallb.universe.tf/allow-shared-ip: internalapi
                         metallb.universe.tf/loadBalancerIPs: 172.17.0.80
                     spec:
                       type: LoadBalancer
               networkAttachments:
               - storage

     • glanceAPIs.default.replicas: You can deploy the initial control plane without activating the Image service (glance). To deploy the Image service, you must set the number of replicas for the service and configure the back end for the service. For information about the recommended replicas for the Image service and how to configure a back end for the service, see Configuring the Image service (glance) in Configuring persistent storage. If you do not deploy the Image service, you cannot upload images to the cloud or start an instance.

   • Key Management service (barbican):

       barbican:
         apiOverride:
           route: {}
         template:
           databaseInstance: openstack
           secret: osp-secret
           barbicanAPI:
             replicas: 3
             override:
               service:
                 internal:
                   metadata:
                     annotations:
                       metallb.universe.tf/address-pool: internalapi
                       metallb.universe.tf/allow-shared-ip: internalapi
                       metallb.universe.tf/loadBalancerIPs: 172.17.0.80
                   spec:
                     type: LoadBalancer
           barbicanWorker:
             replicas: 3
           barbicanKeystoneListener:
             replicas: 1

   • Networking service (neutron):

       neutron:
         apiOverride:
           route: {}
         template:
           replicas: 3
           override:
             service:
               internal:
                 metadata:
                   annotations:
                     metallb.universe.tf/address-pool: internalapi
                     metallb.universe.tf/allow-shared-ip: internalapi
                     metallb.universe.tf/loadBalancerIPs: 172.17.0.80
                 spec:
                   type: LoadBalancer
           databaseInstance: openstack
           secret: osp-secret
           networkAttachments:
           - internalapi

     Important

     Because of a known issue [1], you must set mac_binding_age_threshold`to `0. The default value is 300. After the issue is resolved in a future RHOSO 18.0 release, you must reset mac_binding_age_threshold to 300.

     The issue is caused by a bug in core OVN. The bug causes OVN to send address resolution protocol (ARP) and neighbor discovery protocol (ND) messages for all ports connected to a local network. Anytime a Logical_Router port in another chassis detects the gateway IP, it creates a MAC_Binding register and sends an ARP/ND back with the incorrect MAC address. That confuses the external switches/routers.

     [1] For more information on the known issue, which affects RHOSO 18.0.18 and earlier versions, see OVN MAC_Binding aging must be 0 until core OVN is fixed.

   • Object Storage service (swift):

       swift:
         enabled: true
         proxyOverride:
           route: {}
         template:
           swiftProxy:
             networkAttachments:
             - storage
             override:
               service:
                 internal:
                   metadata:
                     annotations:
                       metallb.universe.tf/address-pool: internalapi
                       metallb.universe.tf/allow-shared-ip: internalapi
                       metallb.universe.tf/loadBalancerIPs: 172.17.0.80
                   spec:
                     type: LoadBalancer
             replicas: 2
             secret: osp-secret
           swiftRing:
             ringReplicas: 3
           swiftStorage:
             networkAttachments:
             - storage
             replicas: 3
             storageRequest: 10Gi

   • Optimize service (watcher):

       watcher:
         enabled: true

   • OVN:

       ovn:
         template:
           ovnDBCluster:
             ovndbcluster-nb:
               replicas: 3
               dbType: NB
               storageRequest: 10G
               networkAttachment: internalapi
             ovndbcluster-sb:
               replicas: 3
               dbType: SB
               storageRequest: 10G
               networkAttachment: internalapi
           ovnNorthd: {}

   • Placement service (placement):

       placement:
         apiOverride:
           route: {}
         template:
           override:
             service:
               internal:
                 metadata:
                   annotations:
                     metallb.universe.tf/address-pool: internalapi
                     metallb.universe.tf/allow-shared-ip: internalapi
                     metallb.universe.tf/loadBalancerIPs: 172.17.0.80
                 spec:
                   type: LoadBalancer
           databaseInstance: openstack
           replicas: 3
           secret: osp-secret

   • Optional: Telemetry service (ceilometer, prometheus):

       telemetry:
         enabled: true
         template:
           metricStorage:
             enabled: true
             dashboardsEnabled: true
             dataplaneNetwork: ctlplane
             networkAttachments:
               - ctlplane
             monitoringStack:
               alertingEnabled: true
               scrapeInterval: 30s
               storage:
                 strategy: persistent
                 retention: 24h
                 persistent:
                   pvcStorageRequest: 20G
           autoscaling:
             enabled: false
             aodh:
               databaseAccount: aodh
               databaseInstance: openstack
               passwordSelector:
                 aodhService: AodhPassword
               serviceUser: aodh
               secret: osp-secret
             heatInstance: heat
           ceilometer:
             enabled: true
             secret: osp-secret
           logging:
             enabled: false

     • telemetry.enabled: Set to false if your RHOSO environment does not require the Telemetry service. When true, you must have installed the Cluster Observability Operator on your RHOCP cluster.
     • telemetry.template.metricStorage.dataplaneNetwork: Defines the network that you use to scrape dataplane node_exporter endpoints.
     • telemetry.template.metricStorage.networkAttachments: Lists the networks that each service pod is attached to by using the NetworkAttachmentDefinition resource names. You configure a NIC for the service for each network attachment that you specify. If you do not configure the isolated networks that each service pod is attached to, then the default pod network is used. You must create a networkAttachment that matches the network that you specify as the dataplaneNetwork, so that Prometheus can scrape data from the dataplane nodes.
     • telemetry.template.autoscaling: You must have the autoscaling field present, even if autoscaling is disabled. For more information about autoscaling, see Autoscaling for Instances.

6. Add the following service configurations to implement high availability (HA):

   • A MariaDB Galera cluster for use by all RHOSO services (openstack), and a MariaDB Galera cluster for use by the Compute service for cell1 (openstack-cell1):

       galera:
         templates:
           openstack:
             storageRequest: 5000M
             secret: osp-secret
             replicas: 3
           openstack-cell1:
             storageRequest: 5000M
             secret: osp-secret
             replicas: 3

   • A single memcached cluster that contains three memcached servers:

       memcached:
         templates:
           memcached:
              replicas: 3

   • A RabbitMQ cluster for use by all RHOSO services (rabbitmq), and a RabbitMQ cluster for use by the Compute service for cell1 (rabbitmq-cell1):

       rabbitmq:
         templates:
           rabbitmq:
             persistence:
               storage: <rabbitmq_cluster_storage>
             replicas: 3
             override:
               service:
                 metadata:
                   annotations:
                     metallb.universe.tf/address-pool: internalapi
                     metallb.universe.tf/loadBalancerIPs: 172.17.0.85
                 spec:
                   type: LoadBalancer
           rabbitmq-cell1:
             persistence:
               storage: <rabbitmq_cluster_storage>
             replicas: 3
             override:
               service:
                 metadata:
                   annotations:
                     metallb.universe.tf/address-pool: internalapi
                     metallb.universe.tf/loadBalancerIPs: 172.17.0.86
                 spec:
                   type: LoadBalancer

   • Replace <rabbitmq_cluster_storage> with sufficient storage for each RabbitMQ cluster, for example, 10Gi.

     Note

     You cannot configure multiple RabbitMQ instances on the same virtual IP (VIP) address because all RabbitMQ instances use the same port. If you need to expose multiple RabbitMQ instances to the same network, then you must use distinct IP addresses.

     By default, RabbitMQ uses Quorum queues to provide increased data safety and high availability at the expense of a slight increase in latency.

     Warning

     You must not configure the RabbitMQ clusters of an existing RHOSO deployment to use Quorum queues! If you do so then your existing RHOSO services will not start or work properly.

     The RabbitMQ Quorum queue is a durable replicated queue based on the Raft consensus algorithm.

     Note

     The RabbitMQ Quorum queues must be saved to disk to ensure their durability. Quorum queues can occupy significant disk space. Depending on the workload and settings the time taken to free up space after messages are consumed or expire can vary substantially. Ensure that sufficient storage is assigned to each RabbitMQ cluster.

     Improved RabbitMQ failover behavior

     The RabbitMQ service can be configured to provide an improved failover behavior that bypasses the default availability checks of OpenShift, which can wait up to five minutes to declare a service dead. By implementing this improved failover behavior, each OpenStack service checks to see if a RabbitMQ pod is alive and moves to the next pod if it is not.

     Note

     To implement this improved failover behavior for a RabbitMQ cluster, you must reserve three free IP addresses from the default RHOSO MetalLB IPAddressPool range already used for RabbitMQ.

     For example, if you want to improve the failover behavior of the rabbitmq-cell1 RabbitMQ cluster add the following lines to the end of the rabbitmq.templates.rabbitmq-cell1 section of the OpenStackControlPlane CR:

         podOverride:
           services:
             - metadata:
                 annotations:
                   metallb.universe.tf/address-pool: "internalapi"
                   metallb.universe.tf/loadBalancerIPs: "172.17.0.87"
               spec:
                 type: LoadBalancer
             - metadata:
                 annotations:
                   metallb.universe.tf/address-pool: "internalapi"
                   metallb.universe.tf/loadBalancerIPs: "172.17.0.88"
               spec:
                 type: LoadBalancer
             - metadata:
                 annotations:
                   metallb.universe.tf/address-pool: "internalapi"
                   metallb.universe.tf/loadBalancerIPs: "172.17.0.89"
               spec:
                 type: LoadBalancer

7. Create the control plane:

   $ oc create -f openstack_control_plane.yaml -n openstack

8. Wait until RHOCP creates the resources related to the OpenStackControlPlane CR. Run the following command to check the status:

   $ oc get openstackcontrolplane -n openstack
   NAME 						STATUS 	MESSAGE
   openstack-control-plane 	Unknown 	Setup started

   The OpenStackControlPlane resources are created when the status is "Setup complete".

   Tip

   Append the -w option to the end of the get command to track deployment progress.

   Note

   Creating the control plane also creates an OpenStackClient pod that you can access through a remote shell (rsh) to run OpenStack CLI commands.

   $ oc rsh -n openstack openstackclient

9. Optional: Confirm that the control plane is deployed by reviewing the pods in the openstack namespace:

   $ oc get pods -n openstack

   The control plane is deployed when all the pods are either completed or running.

Verification

1. Open a remote shell connection to the OpenStackClient pod:

   $ oc rsh -n openstack openstackclient

2. Confirm that the internal service endpoints are registered with each service:

   $ openstack endpoint list -c 'Service Name' -c Interface -c URL --service glance
   +--------------+-----------+---------------------------------------------------------------+
   | Service Name | Interface | URL                                                           |
   +--------------+-----------+---------------------------------------------------------------+
   | glance       | internal  | https://glance-internal.openstack.svc                     |
   | glance       | public    | https://glance-default-public-openstack.apps.ostest.test.metalkube.org |
   +--------------+-----------+---------------------------------------------------------------+

3. Exit the OpenStackClient pod:

   $ exit

Procedure

1. Open the OpenStackControlPlane CR file on your workstation.

2. Locate the service you want to remove from the control plane and disable it:

     cinder:
       enabled: false
       apiOverride:
         route: {}
         ...

3. Update the control plane:

   $ oc apply -f openstack_control_plane.yaml -n openstack

4. Wait until RHOCP removes the resource related to the disabled service. Run the following command to check the status:

   $ oc get openstackcontrolplane -n openstack
   NAME 							STATUS 	MESSAGE
   openstack-control-plane 	Unknown 	Setup started

   The OpenStackControlPlane resource is updated with the disabled service when the status is "Setup complete".

   Tip

   Append the -w option to the end of the get command to track deployment progress.

5. Optional: Confirm that the pods from the disabled service are no longer listed by reviewing the pods in the openstack namespace:

   $ oc get pods -n openstack

6. Check that the service is removed:

   $ oc get cinder -n openstack

   This command returns the following message when the service is successfully removed:

   No resources found in openstack namespace.

7. Check that the API endpoints for the service are removed from the Identity service (keystone):

   $ oc rsh -n openstack openstackclient
   $ openstack endpoint list --service volumev3

   This command returns the following message when the API endpoints for the service are successfully removed:

   No service with a type, name or ID of 'volumev3' exists.

Procedure

1. For unprovisioned nodes, create the SSH key pair for Ansible:

   $ ssh-keygen -f <key_file_name> -N "" -t rsa -b 4096

   • Replace <key_file_name> with the name to use for the key pair.

2. Create the Secret CR for Ansible and apply it to the cluster:

   $ oc create secret generic dataplane-ansible-ssh-private-key-secret \
   --save-config \
   --dry-run=client \
   --from-file=ssh-privatekey=<key_file_name> \
   --from-file=ssh-publickey=<key_file_name>.pub \
   [--from-file=authorized_keys=<key_file_name>.pub] -n openstack \
   -o yaml | oc apply -f -

   • Replace <key_file_name> with the name and location of your SSH key pair file.
   • Optional: Only include the --from-file=authorized_keys option for bare-metal nodes that must be provisioned when creating the data plane.

3. If you are creating Compute nodes, create a secret for migration.

   1. Create the SSH key pair for instance migration:

      $ ssh-keygen -f ./nova-migration-ssh-key -t ecdsa-sha2-nistp521 -N ''

   2. Create the Secret CR for migration and apply it to the cluster:

      $ oc create secret generic nova-migration-ssh-key \
      --save-config \
      --from-file=ssh-privatekey=nova-migration-ssh-key \
      --from-file=ssh-publickey=nova-migration-ssh-key.pub \
      -n openstack \
      -o yaml | oc apply -f -

4. For nodes that have not been registered to the Red Hat Customer Portal, create the Secret CR for subscription-manager credentials to register the nodes:

   $ oc create secret generic subscription-manager \
   --from-literal rhc_auth='{"login": {"username": "<subscription_manager_username>", "password": "<subscription_manager_password>"}}'

   • Replace <subscription_manager_username> with the username you set for subscription-manager.
   • Replace <subscription_manager_password> with the password you set for subscription-manager.

5. Create a Secret CR that contains the Red Hat registry credentials:

   $ oc create secret generic redhat-registry --from-literal edpm_container_registry_logins='{"registry.redhat.io": {"<username>": "<password>"}}'

   • Replace <username> and <password> with your Red Hat registry username and password credentials.

     For information about how to create your registry service account, see the Knowledge Base article Creating Registry Service Accounts.

6. If you are creating Compute nodes, create a secret for libvirt.

   1. Create a file on your workstation named secret_libvirt.yaml to define the libvirt secret:

      apiVersion: v1
      kind: Secret
      metadata:
       name: libvirt-secret
       namespace: openstack
      type: Opaque
      data:
       LibvirtPassword: <base64_password>

      • Replace <base64_password> with a base64-encoded string with maximum length 63 characters. You can use the following command to generate a base64-encoded password:

        $ echo -n <password> | base64

        Tip

        If you do not want to base64-encode the username and password, you can use the stringData field instead of the data field to set the username and password.

   2. Create the Secret CR:

      $ oc apply -f secret_libvirt.yaml -n openstack

7. Verify that the Secret CRs are created:

   $ oc describe secret dataplane-ansible-ssh-private-key-secret
   $ oc describe secret nova-migration-ssh-key
   $ oc describe secret subscription-manager
   $ oc describe secret redhat-registry
   $ oc describe secret libvirt-secret

Procedure

1. The Bare Metal Operator (BMO) manages BareMetalHost custom resources (CRs) in the openshift-machine-api namespace by default. Update the Provisioning CR to watch all namespaces:

   $ oc patch provisioning provisioning-configuration --type merge -p '{"spec":{"watchAllNamespaces": true }}'

2. Update the Provisioning CR to enable virtualMediaViaExternalNetwork, which enables bare-metal connectivity through the external network:

   $ oc patch provisioning provisioning-configuration --type merge -p '{"spec":{"virtualMediaViaExternalNetwork": true }}'

3. Create a file on your workstation named bmh_leaf1_nodes.yaml that defines the Secret CR with the credentials for accessing the BMC of each bare-metal data plane node in the node set:

   apiVersion: v1
   kind: Secret
   metadata:
     name: edpm-compute-0-bmc-secret
     namespace: openstack
   type: Opaque
   data:
     username: <base64_username>
     password: <base64_password>

   • Replace <base64_username> and <base64_password> with strings that are base64-encoded. You can use the following command to generate a base64-encoded string:

     $ echo -n <string> | base64

     Tip

     If you don’t want to base64-encode the username and password, you can use the stringData field instead of the data field to set the username and password.

4. Create a file on your workstation that defines the BareMetalHost CR for each bare-metal data plane node, with virtual media as the boot method:

   apiVersion: metal3.io/v1alpha1
   kind: BareMetalHost
   metadata:
     name: edpm-compute-0
     namespace: openstack
   labels:
     app: openstack
     workload: compute
   spec:
     bmc:
       address: redfish-virtualmedia+http://192.168.111.1:8000/redfish/v1/Systems/e8efd888-f844-4fe0-9e2e-498f4ab7806d
       credentialsName: edpm-compute-0-bmc-secret
     bootMACAddress: 00:c7:e4:a7:e7:f3
     bootMode: UEFI
     online: false

   • address - The URL for communicating with the node’s Baseboard Management Controller (BMC) controller. For information on BMC addressing for other boot methods, see BMC addressing in the RHOCP Installing on bare metal guide.
   • credentialsName - The name of the Secret CR you created in the previous step for accessing the BMC of the node.

5. Create the BareMetalHost resources:

   $ oc create -f bmh_leaf1_nodes.yaml

6. Verify that the BareMetalHost resources have been created and are in the Available state:

   $ oc get bmh
   NAME         STATE            CONSUMER              ONLINE   ERROR   AGE
   edpm-compute-0   Available      openstack-edpm        true             2d21h

Procedure

1. Create a Secret CR for each bare-metal data plane node in the node set, that defines the pre-provisioning network configuration data for the ramdisk in nmstate format:

   apiVersion: v1
   kind: Secret
   metadata:
     name: <bmh-name>-preprovision-network-data
     namespace: openstack
   stringData:
     nmstate: |
       interfaces:
         - name: enp5s0
           type: ethernet
           state: up
           ipv4:
             enabled: true
             address:
             - ip: 192.168.130.100
               prefix-length: 24
       dns-resolver:
         config:
           server:
             - 192.168.122.1
           routes:
             config:
             - destination: 0.0.0.0/0
               next-hop-address: 192.168.130.1
               next-hop-interface: enp5s0
       type: Opaque

   • Replace <bmh-name> with the name of the BareMetalHost CR the secret is for, for example, edpm-compute-0-preprovision-network-data.

   For more information about the nmstate schema, see https://nmstate.io/devel/yaml_api.html.

2. The Bare Metal Operator (BMO) manages BareMetalHost custom resources (CRs) in the openshift-machine-api namespace by default. Update the Provisioning CR to watch all namespaces:

   $ oc patch provisioning provisioning-configuration --type merge -p '{"spec":{"watchAllNamespaces": true }}'

3. Update the Provisioning CR to enable virtualMediaViaExternalNetwork, which enables bare-metal connectivity through the external network:

   $ oc patch provisioning provisioning-configuration --type merge -p '{"spec":{"virtualMediaViaExternalNetwork": true }}'

4. Create a file on your workstation that defines the Secret CR with the credentials for accessing the BMC of each bare-metal data plane node in the node set:

   apiVersion: v1
   kind: Secret
   metadata:
     name: edpm-compute-0-bmc-secret
     namespace: openstack
   type: Opaque
   data:
     username: <base64_username>
     password: <base64_password>

   • Replace <base64_username> and <base64_password> with strings that are base64-encoded. You can use the following command to generate a base64-encoded string:

     $ echo -n <string> | base64

     Tip

     If you don’t want to base64-encode the username and password, you can use the stringData field instead of the data field to set the username and password.

5. Create a file on your workstation named bmh_leaf1_nodes.yaml that defines the BareMetalHost CR for each bare-metal data plane node, with virtual media as the boot method:

   apiVersion: metal3.io/v1alpha1
   kind: BareMetalHost
   metadata:
     name: edpm-compute-0
     namespace: openstack
   labels:
     app: openstack
     workload: compute
   spec:
     bmc:
       address: redfish-virtualmedia+http://192.168.111.1:8000/redfish/v1/Systems/e8efd888-f844-4fe0-9e2e-498f4ab7806d
       credentialsName: edpm-compute-0-bmc-secret
     bootMACAddress: 00:c7:e4:a7:e7:f3
     bootMode: UEFI
     online: false

   • address - The URL for communicating with the node’s Baseboard Management Controller (BMC) controller. For information on BMC addressing for other boot methods, see BMC addressing in the RHOCP Installing on bare metal guide
   • credentialsName - The name of the Secret CR you created in the previous step for accessing the BMC of the node.

6. Add the preprovisioningNetworkDataName field to each BareMetalHost CR to specify the pre-provisioning network configuration data Secret CR:

   apiVersion: metal3.io/v1alpha1
   kind: BareMetalHost
   metadata:
   name: edpm-compute-0
   namespace: openstack
   ...
   spec:
     bmc:
       address: redfish-virtualmedia+http://192.168.111.1:8000/redfish/v1/Systems/e8efd888-f844-4fe0-9e2e-498f4ab7806d
     ...
     preprovisioningNetworkDataName: <pre_provision_network_secret>

   • Replace <pre_provision_network_secret> with the Secret CR you created in step 1 for the pre-provisioning network configuration data.

7. Create the BareMetalHost resources:

   $ oc create -f bmh_leaf1_nodes.yaml

8. Verify that the BareMetalHost resources have been created and are in the Available state:

   $ oc get bmh
   NAME         STATE            CONSUMER              ONLINE   ERROR   AGE
   edpm-compute-0   Available      openstack-edpm        true             2d21h

Procedure

1. Create YAML files on your workstation that define the OpenStackDataPlaneNodeSet CRs for each leaf in the spine-leaf topology:

   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneNodeSet
   metadata:
     name: data-plane-leaf1
     namespace: openstack
   spec:
     tlsEnabled: true
     env:
       - name: ANSIBLE_FORCE_COLOR
         value: "True"

   • name - The OpenStackDataPlaneNodeSet CR name must be unique, contain only lower case alphanumeric characters and - (hyphens) or . (periods), start and end with an alphanumeric character, and have a maximum length of 53 characters. Update the name in this example to a name that reflects the nodes in the set.
   • env - Optional: A list of environment variables to pass to the pod.

2. Connect the data plane to the control plane network:

   spec:
     ...
     networkAttachments:
       - ctlplane

3. Specify that the nodes in this set are unprovisioned and must be provisioned when creating the resource:

     preProvisioned: false

4. Use the baremetalSetTemplate field to describe the configuration of the bare-metal nodes that are provisioned when the data plane is deployed:

     baremetalSetTemplate:
       deploymentSSHSecret: dataplane-ansible-ssh-private-key-secret
       bmhNamespace: <bmh_namespace>
       cloudUserName: <ansible_ssh_user>
       bmhLabelSelector:
         app: <bmh_label>
       ctlplaneInterface: <interface>

   • Replace <bmh_namespace> with the namespace defined in the corresponding BareMetalHost CR for the node, for example, openstack.
   • Replace <ansible_ssh_user> with the username of the Ansible SSH user, for example, cloud-admin.
   • Replace <bmh_label> with the label defined in the corresponding BareMetalHost CR for the node, for example, openstack.
   • Replace <interface> with the control plane interface the node connects to, for example, enp6s0.

5. Add the SSH key secret that you created to enable Ansible to connect to the data plane nodes:

     nodeTemplate:
       ansibleSSHPrivateKeySecret: <secret-key>

   • Replace <secret-key> with the name of the SSH key Secret CR you created in Creating the data plane secrets, for example, dataplane-ansible-ssh-private-key-secret.
6. Create a Persistent Volume Claim (PVC) in the openstack namespace on your RHOCP cluster to store logs. Set the volumeMode to Filesystem and accessModes to ReadWriteOnce. Do not request storage for logs from a PersistentVolume (PV) that uses the NFS volume plugin. NFS is incompatible with FIFO and the ansible-runner creates a FIFO file to write to store logs. For information about PVCs, see Understanding persistent storage in the RHOCP Storage guide and Red Hat OpenShift Container Platform cluster requirements in Planning your deployment.

7. Enable persistent logging for the data plane nodes:

     nodeTemplate:
       ...
       extraMounts:
         - extraVolType: Logs
           volumes:
           - name: ansible-logs
             persistentVolumeClaim:
               claimName: <pvc_name>
           mounts:
           - name: ansible-logs
             mountPath: "/runner/artifacts"

   • Replace <pvc_name> with the name of the PVC storage on your RHOCP cluster.

8. Specify the management network:

     nodeTemplate:
       ...
       managementNetwork: ctlplane

9. Specify the Secret CRs that Ansible uses to source the usernames and passwords to register the operating system of the nodes to the Red Hat Customer Portal, and enable repositories for your nodes. The following example demonstrates how to register your nodes to CDN. For details on how to register your nodes with Red Hat Satellite 6.13, see Managing Hosts.

     nodeTemplate:
       ansible:
         ansibleUser: cloud-admin
         ansiblePort: 22
         ansibleVarsFrom:
           - secretRef:
               name: subscription-manager
           - secretRef:
               name: redhat-registry
         ansibleVars:
           rhc_release: 9.4
           rhc_repositories:
               - {name: "*", state: disabled}
               - {name: "rhel-9-for-x86_64-baseos-eus-rpms", state: enabled}
               - {name: "rhel-9-for-x86_64-appstream-eus-rpms", state: enabled}
               - {name: "rhel-9-for-x86_64-highavailability-eus-rpms", state: enabled}
               - {name: "fast-datapath-for-rhel-9-x86_64-rpms", state: enabled}
               - {name: "rhoso-18.0-for-rhel-9-x86_64-rpms", state: enabled}
               - {name: "rhceph-7-tools-for-rhel-9-x86_64-rpms", state: enabled}
           edpm_bootstrap_release_version_package: []

   • ansibleUser - The user associated with the secret you created in Creating the data plane secrets.
   • ansibleVars - The Ansible variables that customize the set of nodes. For a list of Ansible variables that you can use, see https://openstack-k8s-operators.github.io/edpm-ansible/.

   For a complete list of the Red Hat Customer Portal registration commands, see https://access.redhat.com/solutions/253273. For information about how to log into registry.redhat.io, see https://access.redhat.com/RegistryAuthentication#creating-registry-service-accounts-6.

10. Add the network configuration template to apply to your data plane nodes. The following example applies the single NIC VLANs network configuration to the data plane nodes:

      nodeTemplate:
        ...
        ansible:
          ...
          ansibleVars:
            ...
            neutron_physical_bridge_name: br-ex
            neutron_public_interface_name: eth0
            edpm_network_config_template: |
              ---
              {% set mtu_list = [ctlplane_mtu] %}
              {% for network in nodeset_networks %}
              {{ mtu_list.append(lookup('vars', networks_lower[network] ~ '_mtu')) }}
              {%- endfor %}
              {% set min_viable_mtu = mtu_list | max %}
              network_config:
              - type: ovs_bridge
                name: {{ neutron_physical_bridge_name }}
                mtu: {{ min_viable_mtu }}
                use_dhcp: false
                dns_servers: {{ ctlplane_dns_nameservers }}
                domain: {{ dns_search_domains }}
                addresses:
                - ip_netmask: {{ ctlplane_ip }}/{{ ctlplane_cidr }}
                routes: {{ ctlplane_host_routes }}
                members:
                - type: interface
                  name: nic1
                  mtu: {{ min_viable_mtu }}
                  # force the MAC address of the bridge to this interface
                  primary: true
              {% for network in nodeset_networks %}
                - type: vlan
                  mtu: {{ lookup('vars', networks_lower[network] ~ '_mtu') }}
                  vlan_id: {{ lookup('vars', networks_lower[network] ~ '_vlan_id') }}
                  addresses:
                  - ip_netmask:
                      {{ lookup('vars', networks_lower[network] ~ '_ip') }}/{{ lookup('vars', networks_lower[network] ~ '_cidr') }}
                  routes: {{ lookup('vars', networks_lower[network] ~ '_host_routes') }}
              {% endfor %}

    For more information about data plane network configuration, see Customizing data plane networks in Configuring network services.

11. Add the common configuration for the set of nodes in this group under the nodeTemplate section. Each node in this OpenStackDataPlaneNodeSet inherits this configuration. For information about the properties you can use to configure common node attributes, see OpenStackDataPlaneNodeSet CR properties.

12. Define each node in this node set:

      nodes:
        edpm-compute-0:
          hostName: edpm-compute-0
          networks:
          - name: ctlplane
            subnetName: subnet1
            defaultRoute: true
            fixedIP: 192.168.122.100
          - name: internalapi
            subnetName: subnet1
          - name: storage
            subnetName: subnet1
          - name: tenant
            subnetName: subnet1
          ansible:
            ansibleHost: 192.168.122.100
            ansibleUser: cloud-admin
            ansibleVars:
              fqdn_internal_api: edpm-compute-0.example.com
        edpm-compute-1:
          hostName: edpm-compute-1
          networks:
          - name: ctlplane
            subnetName: subnet1
            defaultRoute: true
            fixedIP: 192.168.122.101
          - name: internalapi
            subnetName: subnet1
          - name: storage
            subnetName: subnet1
          - name: tenant
            subnetName: subnet1
          ansible:
            ansibleHost: 192.168.122.101
            ansibleUser: cloud-admin
            ansibleVars:
              fqdn_internal_api: edpm-compute-1.example.com

    • edpm-compute-0 - The node definition reference, for example, edpm-compute-0. Each node in the node set must have a node definition.
    • networks - Defines the IPAM and the DNS records for the node.
    • fixedIP - Specifies a predictable IP address for the network that must be in the allocation range defined for the network in the NetConfig CR.

    • ansibleVars - Node-specific Ansible variables that customize the node.

      Note
      • Nodes defined within the nodes section can configure the same Ansible variables that are configured in the nodeTemplate section. Where an Ansible variable is configured for both a specific node and within the nodeTemplate section, the node-specific values override those from the nodeTemplate section.
      • You do not need to replicate all the nodeTemplate Ansible variables for a node to override the default and set some node-specific values. You only need to configure the Ansible variables you want to override for the node.
      • When you define the networkData Secret for an individual node (such as edpm-compute-0), it acts as a complete override rather than a supplemental configuration. Because node-specific configurations override the inherited default values from the nodeTemplate section, you must ensure that your node-specific networkData Secret contains the full set of required network configurations for that node, not just the unique values.
      • Many ansibleVars include edpm in the name, which stands for "External Data Plane Management".

    For information about the properties you can use to configure node attributes, see OpenStackDataPlaneNodeSet CR properties.

13. Save the openstack_unprovisioned_node_set.yaml definition file.

14. Create the data plane resources:

    $ oc create --save-config -f openstack_unprovisioned_node_set.yaml -n openstack

15. Verify that the data plane resources have been created by confirming that the status is SetupReady:

    $ oc wait openstackdataplanenodeset openstack-data-plane --for condition=SetupReady --timeout=10m

    When the status is SetupReady, the command returns a condition met message, otherwise it returns a timeout error.

    For information about the data plane conditions and states, see Data plane conditions and states.

16. Verify that the Secret resource was created for the node set:

    $ oc get secret -n openstack | grep openstack-data-plane
    dataplanenodeset-openstack-data-plane Opaque 1 3m50s

17. Verify that the nodes have transitioned to the provisioned state:

    $ oc get bmh
    NAME            STATE         CONSUMER               ONLINE   ERROR   AGE
    edpm-compute-0  provisioned   openstack-data-plane   true             3d21h

18. Verify that the services were created:

    $ oc get openstackdataplaneservice -n openstack
    NAME                    AGE
    bootstrap               8m40s
    ceph-client             8m40s
    ceph-hci-pre            8m40s
    configure-network       8m40s
    configure-os            8m40s
    ...

Procedure

1. Create a file on your workstation named openstack_data_plane_deploy.yaml to define the OpenStackDataPlaneDeployment CR:

   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneDeployment
   metadata:
     name: data-plane-deploy
     namespace: openstack

   • metadata.name: The OpenStackDataPlaneDeployment CR name must be unique, must consist of lower case alphanumeric characters, - (hyphen) or . (period), and must start and end with an alphanumeric character. Update the name in this example to a name that reflects the node sets in the deployment.

2. Add all the OpenStackDataPlaneNodeSet CRs that you want to deploy:

   spec:
     nodeSets:
       - openstack-data-plane
       - <nodeSet_name>
       - ...
       - <nodeSet_name>

   • Replace <nodeSet_name> with the names of the OpenStackDataPlaneNodeSet CRs that you want to include in your data plane deployment.
3. Save the openstack_data_plane_deploy.yaml deployment file.

4. Deploy the data plane:

   $ oc create -f openstack_data_plane_deploy.yaml -n openstack

   You can view the Ansible logs while the deployment executes:

   $ oc get pod -l app=openstackansibleee -w
   $ oc logs -l app=openstackansibleee -f --max-log-requests 10

   If the oc logs command returns an error similar to the following error, increase the --max-log-requests value:

   error: you are attempting to follow 19 log streams, but maximum allowed concurrency is 10, use --max-log-requests to increase the limit

5. Verify that the data plane is deployed:

   $ oc wait openstackdataplanedeployment data-plane-deploy --for=condition=Ready --timeout=<timeout_value>
   $ oc wait openstackdataplanenodeset openstack-data-plane --for=condition=Ready --timeout=<timeout_value>

   • Replace <timeout_value> with a value in minutes you want the command to wait for completion of the task. For example, if you want the command to wait 60 minutes, you would use the value 60m. If the completion status of SetupReady for oc wait openstackdataplanedeployment or NodeSetReady for oc wait openstackdataplanenodeset is not returned in this time frame, the command returns a timeout error. Use a value that is appropriate to the size of your deployment. Give larger deployments more time to complete deployment tasks.

     For information about the data plane conditions and states, see Data plane conditions and states in Deploying Red Hat OpenStack Services on OpenShift.

6. Map the Compute nodes to the Compute cell that they are connected to:

   $ oc rsh nova-cell0-conductor-0 nova-manage cell_v2 discover_hosts --verbose

   If you did not create additional cells, this command maps the Compute nodes to cell1.

7. Access the remote shell for the openstackclient pod and verify that the deployed Compute nodes are visible on the control plane:

   $ oc rsh -n openstack openstackclient
   $ openstack hypervisor list

   If some Compute nodes are missing from the hypervisor list, retry the previous step. If the Compute nodes are still missing from the list, check the status and health of the nova-compute services on the deployed data plane nodes.

8. Verify that the hypervisor hostname is a fully qualified domain name (FQDN):

   $ hostname -f

   If the hypervisor hostname is not an FQDN, for example, if it was registered as a short name or full name instead, contact Red Hat Support.

Procedure

1. Access the remote shell for the OpenStackClient pod:

   $ oc rsh -n openstack openstackclient

2. Run your openstack commands. For example, you can create a default network with the following command:

   $ openstack network create default

3. Exit the OpenStackClient pod:

   $ exit

Procedure

1. Retrieve the admin password from the AdminPassword parameter in the osp-secret secret:

   $ oc get secret osp-secret -o jsonpath='{.data.AdminPassword}' | base64 -d

2. Retrieve the Dashboard service endpoint URL:

   $ oc get horizons horizon -o jsonpath='{.status.endpoint}'

3. Open a browser.
4. Enter the Dashboard endpoint URL.
5. Log in to the Dashboard by providing the username of admin and the admin password.