Chapter 4. Creating the control plane

Procedure

1. Create a file on your workstation named openstack_control_plane.yaml to define the OpenStackControlPlane CR:

   apiVersion: core.openstack.org/v1beta1
   kind: OpenStackControlPlane
   metadata:
     name: openstack-control-plane
     namespace: openstack

2. Specify the Secret CR you created to provide secure access to the RHOSO service pods in Providing secure access to the Red Hat OpenStack Services on OpenShift services:

   apiVersion: core.openstack.org/v1beta1
   kind: OpenStackControlPlane
   metadata:
     name: openstack-control-plane
     namespace: openstack
   spec:
     secret: osp-secret

3. Specify the storageClass you created for your Red Hat OpenShift Container Platform (RHOCP) cluster storage back end:

   spec:
     secret: osp-secret
     storageClass: <RHOCP_storage_class>

   • Replace <RHOCP_storage_class> with the storage class you created for your RHOCP cluster storage back end. For information about storage classes, see Creating a storage class.

4. Add the global RabbitMQ settings messagingBus and notificationsBus to specify the default RabbitMQ cluster for RHOSO services:

   apiVersion: core.openstack.org/v1beta1
   kind: OpenStackControlPlane
   metadata:
     name: openstack
   spec:
     messagingBus:
       cluster: <rabbitmq-cluster>
   
     notificationsBus:
       cluster: <rabbitmq-cluster>

   • Replace <rabbitmq-cluster> with the default RabbitMQ cluster that all the RHOSO services use, in this example, rabbitmq. You can customize the RabbitMQ interface for OpenStack services. For more information, see Understand the RabbitMQ interface for OpenStack services in the Monitoring high availability services guide.

5. Add the following service configurations:

   Note
   • The following service examples use IP addresses from the default RHOSO MetalLB IPAddressPool range for the loadBalancerIPs field. Update the loadBalancerIPs field with the IP address from the MetalLB IPAddressPool range that you created.
   • You cannot override the default public service endpoint. The public service endpoints are exposed as RHOCP routes by default, because only routes are supported for public endpoints.

   • Block Storage service (cinder):

       cinder:
         apiOverride:
           route: {}
         template:
           databaseInstance: openstack
           secret: osp-secret
           cinderAPI:
             replicas: 3
             override:
               service:
                 internal:
                   metadata:
                     annotations:
                       metallb.universe.tf/address-pool: internalapi
                       metallb.universe.tf/allow-shared-ip: internalapi
                       metallb.universe.tf/loadBalancerIPs: 172.17.0.80
                   spec:
                     type: LoadBalancer
           cinderScheduler:
             replicas: 1
           cinderBackup:
             networkAttachments:
             - storage
             replicas: 0
           cinderVolumes:
             volume1:
               networkAttachments:
               - storage
               replicas: 0

     • cinderBackup.replicas: You can deploy the initial control plane without activating the cinderBackup service. To deploy the service, you must set the number of replicas for the service and configure the back end for the service. For information about the recommended replicas for each service and how to configure a back end for the Block Storage service and the backup service, see Configuring the Block Storage backup service in Configuring persistent storage.
     • cinderVolumes.replicas: You can deploy the initial control plane without activating the cinderVolumes service. To deploy the service, you must set the number of replicas for the service and configure the back end for the service. For information about the recommended replicas for the cinderVolumes service and how to configure a back end for the service, see Configuring the Block Storage volume service component in Configuring persistent storage.

   • Compute service (nova):

       nova:
         apiOverride:
           route: {}
         template:
           apiServiceTemplate:
             replicas: 3
             override:
               service:
                 internal:
                   metadata:
                     annotations:
                       metallb.universe.tf/address-pool: internalapi
                       metallb.universe.tf/allow-shared-ip: internalapi
                       metallb.universe.tf/loadBalancerIPs: 172.17.0.80
                   spec:
                     type: LoadBalancer
           metadataServiceTemplate:
             replicas: 3
             override:
               service:
                 metadata:
                   annotations:
                     metallb.universe.tf/address-pool: internalapi
                     metallb.universe.tf/allow-shared-ip: internalapi
                     metallb.universe.tf/loadBalancerIPs: 172.17.0.80
                 spec:
                   type: LoadBalancer
           schedulerServiceTemplate:
             replicas: 3
           cellTemplates:
             cell0:
               cellDatabaseAccount: nova-cell0
               cellDatabaseInstance: openstack
               messagingBus:
                 cluster: rabbitmq
               hasAPIAccess: true
             cell1:
               cellDatabaseAccount: nova-cell1
               cellDatabaseInstance: openstack-cell1
               messagingBus:
                 cluster: rabbitmq-cell1
               noVNCProxyServiceTemplate:
                 enabled: true
                 networkAttachments:
                 - ctlplane
               hasAPIAccess: true
           secret: osp-secret

     Note

     A full set of Compute services (nova) are deployed by default for each of the default cells, cell0 and cell1: nova-api, nova-metadata, nova-scheduler, and nova-conductor. The novncproxy service is also enabled for cell1 by default.

   • DNS service for the data plane:

       dns:
         template:
           options:
           - key: server
             values:
             - <IP address for DNS server reachable from dnsmasq pod>
           override:
             service:
               metadata:
                 annotations:
                   metallb.universe.tf/address-pool: ctlplane
                   metallb.universe.tf/allow-shared-ip: ctlplane
                   metallb.universe.tf/loadBalancerIPs: 192.168.122.80
               spec:
                 type: LoadBalancer
           replicas: 2

     • options: Defines the dnsmasq instances required for each DNS server by using key-value pairs. In this example, there is one key-value pair defined because there is only one DNS server configured to forward requests to.

     • key: Specifies the dnsmasq parameter to customize for the deployed dnsmasq instance. Set to one of the following valid values:

       • server
       • rev-server
       • srv-host
       • txt-record
       • ptr-record
       • rebind-domain-ok
       • naptr-record
       • cname
       • host-record
       • caa-record
       • dns-rr
       • auth-zone
       • synth-domain
       • no-negcache
       • local

     • values: Specifies the value for the DNS server reachable from the dnsmasq pod on the RHOCP cluster network. You can specify a generic DNS server as the value, for example, 1.1.1.1, or a DNS server for a specific domain, for example, /google.com/8.8.8.8.

       Note

       This DNS service, dnsmasq, provides DNS services for nodes on the RHOSO data plane. dnsmasq is different from the RHOSO DNS service (designate) that provides DNS as a service for cloud tenants.

   • Identity service (keystone)

       keystone:
         apiOverride:
           route: {}
         template:
           override:
             service:
               internal:
                 metadata:
                   annotations:
                     metallb.universe.tf/address-pool: internalapi
                     metallb.universe.tf/allow-shared-ip: internalapi
                     metallb.universe.tf/loadBalancerIPs: 172.17.0.80
                 spec:
                   type: LoadBalancer
           databaseInstance: openstack
           secret: osp-secret
           replicas: 3

   • Image service (glance):

       glance:
         apiOverrides:
           default:
             route: {}
         template:
           databaseInstance: openstack
           storage:
             storageRequest: 10G
           secret: osp-secret
           keystoneEndpoint: default
           glanceAPIs:
             default:
               replicas: 0 # Configure back end; set to 3 when deploying service
               override:
                 service:
                   internal:
                     metadata:
                       annotations:
                         metallb.universe.tf/address-pool: internalapi
                         metallb.universe.tf/allow-shared-ip: internalapi
                         metallb.universe.tf/loadBalancerIPs: 172.17.0.80
                     spec:
                       type: LoadBalancer
               networkAttachments:
               - storage

     • glanceAPIs.default.replicas: You can deploy the initial control plane without activating the Image service (glance). To deploy the Image service, you must set the number of replicas for the service and configure the back end for the service. For information about the recommended replicas for the Image service and how to configure a back end for the service, see Configuring the Image service (glance) in Configuring persistent storage. If you do not deploy the Image service, you cannot upload images to the cloud or start an instance.

   • Key Management service (barbican):

       barbican:
         apiOverride:
           route: {}
         template:
           databaseInstance: openstack
           secret: osp-secret
           barbicanAPI:
             replicas: 3
             override:
               service:
                 internal:
                   metadata:
                     annotations:
                       metallb.universe.tf/address-pool: internalapi
                       metallb.universe.tf/allow-shared-ip: internalapi
                       metallb.universe.tf/loadBalancerIPs: 172.17.0.80
                   spec:
                     type: LoadBalancer
           barbicanWorker:
             replicas: 3
           barbicanKeystoneListener:
             replicas: 1

   • Networking service (neutron):

       neutron:
         apiOverride:
           route: {}
         template:
           replicas: 3
           override:
             service:
               internal:
                 metadata:
                   annotations:
                     metallb.universe.tf/address-pool: internalapi
                     metallb.universe.tf/allow-shared-ip: internalapi
                     metallb.universe.tf/loadBalancerIPs: 172.17.0.80
                 spec:
                   type: LoadBalancer
           databaseInstance: openstack
           secret: osp-secret
           networkAttachments:
           - internalapi

   • Object Storage service (swift):

       swift:
         enabled: true
         proxyOverride:
           route: {}
         template:
           swiftProxy:
             networkAttachments:
             - storage
             override:
               service:
                 internal:
                   metadata:
                     annotations:
                       metallb.universe.tf/address-pool: internalapi
                       metallb.universe.tf/allow-shared-ip: internalapi
                       metallb.universe.tf/loadBalancerIPs: 172.17.0.80
                   spec:
                     type: LoadBalancer
             replicas: 2
             secret: osp-secret
           swiftRing:
             ringReplicas: 3
           swiftStorage:
             networkAttachments:
             - storage
             replicas: 3
             storageRequest: 10Gi

   • Optimize service (watcher):

       watcher:
         enabled: true

   • OVN:

       ovn:
         template:
           ovnDBCluster:
             ovndbcluster-nb:
               replicas: 3
               dbType: NB
               storageRequest: 10G
               networkAttachment: internalapi
             ovndbcluster-sb:
               replicas: 3
               dbType: SB
               storageRequest: 10G
               networkAttachment: internalapi
           ovnNorthd: {}

   • Placement service (placement):

       placement:
         apiOverride:
           route: {}
         template:
           override:
             service:
               internal:
                 metadata:
                   annotations:
                     metallb.universe.tf/address-pool: internalapi
                     metallb.universe.tf/allow-shared-ip: internalapi
                     metallb.universe.tf/loadBalancerIPs: 172.17.0.80
                 spec:
                   type: LoadBalancer
           databaseInstance: openstack
           replicas: 3
           secret: osp-secret

   • Optional: Telemetry service (ceilometer, prometheus):

       telemetry:
         enabled: true
         template:
           metricStorage:
             enabled: true
             dashboardsEnabled: true
             dataplaneNetwork: ctlplane
             networkAttachments:
               - ctlplane
             monitoringStack:
               alertingEnabled: true
               scrapeInterval: 30s
               storage:
                 strategy: persistent
                 retention: 24h
                 persistent:
                   pvcStorageRequest: 20G
           autoscaling:
             enabled: false
             aodh:
               databaseAccount: aodh
               databaseInstance: openstack
               passwordSelector:
                 aodhService: AodhPassword
               serviceUser: aodh
               secret: osp-secret
             heatInstance: heat
           ceilometer:
             enabled: true
             secret: osp-secret
           logging:
             enabled: false

     • telemetry.enabled: Set to false if your RHOSO environment does not require the Telemetry service. When true, you must have installed the Cluster Observability Operator on your RHOCP cluster.
     • telemetry.template.metricStorage.dataplaneNetwork: Defines the network that you use to scrape dataplane node_exporter endpoints.
     • telemetry.template.metricStorage.networkAttachments: Lists the networks that each service pod is attached to by using the NetworkAttachmentDefinition resource names. You configure a NIC for the service for each network attachment that you specify. If you do not configure the isolated networks that each service pod is attached to, then the default pod network is used. You must create a networkAttachment that matches the network that you specify as the dataplaneNetwork, so that Prometheus can scrape data from the dataplane nodes.
     • telemetry.template.autoscaling: You must have the autoscaling field present, even if autoscaling is disabled. For more information about autoscaling, see Autoscaling for Instances.

6. Add the following service configurations to implement high availability (HA):

   • A MariaDB Galera cluster for use by all RHOSO services (openstack), and a MariaDB Galera cluster for use by the Compute service for cell1 (openstack-cell1):

       galera:
         templates:
           openstack:
             storageRequest: 5000M
             secret: osp-secret
             replicas: 3
           openstack-cell1:
             storageRequest: 5000M
             secret: osp-secret
             replicas: 3

   • A single memcached cluster that contains three memcached servers:

       memcached:
         templates:
           memcached:
              replicas: 3

   • A RabbitMQ cluster for use by all RHOSO services (rabbitmq), and a RabbitMQ cluster for use by the Compute service for cell1 (rabbitmq-cell1):

       rabbitmq:
         templates:
           rabbitmq:
             persistence:
               storage: <rabbitmq_cluster_storage>
             replicas: 3
             override:
               service:
                 metadata:
                   annotations:
                     metallb.universe.tf/address-pool: internalapi
                     metallb.universe.tf/loadBalancerIPs: 172.17.0.85
                 spec:
                   type: LoadBalancer
           rabbitmq-cell1:
             persistence:
               storage: <rabbitmq_cluster_storage>
             replicas: 3
             override:
               service:
                 metadata:
                   annotations:
                     metallb.universe.tf/address-pool: internalapi
                     metallb.universe.tf/loadBalancerIPs: 172.17.0.86
                 spec:
                   type: LoadBalancer

   • Replace <rabbitmq_cluster_storage> with sufficient storage for each RabbitMQ cluster, for example, 10Gi.

     Note

     You cannot configure multiple RabbitMQ instances on the same virtual IP (VIP) address because all RabbitMQ instances use the same port. If you need to expose multiple RabbitMQ instances to the same network, then you must use distinct IP addresses.

     By default, RabbitMQ uses Quorum queues to provide increased data safety and high availability at the expense of a slight increase in latency.

     Warning

     You must not configure the RabbitMQ clusters of an existing RHOSO deployment to use Quorum queues! If you do so then your existing RHOSO services will not start or work properly.

     The RabbitMQ Quorum queue is a durable replicated queue based on the Raft consensus algorithm.

     Note

     The RabbitMQ Quorum queues must be saved to disk to ensure their durability. Quorum queues can occupy significant disk space. Depending on the workload and settings the time taken to free up space after messages are consumed or expire can vary substantially. Ensure that sufficient storage is assigned to each RabbitMQ cluster.

     Improved RabbitMQ failover behavior

     The RabbitMQ service can be configured to provide an improved failover behavior that bypasses the default availability checks of OpenShift, which can wait up to five minutes to declare a service dead. By implementing this improved failover behavior, each OpenStack service checks to see if a RabbitMQ pod is alive and moves to the next pod if it is not.

     Note

     To implement this improved failover behavior for a RabbitMQ cluster, you must reserve three free IP addresses from the default RHOSO MetalLB IPAddressPool range already used for RabbitMQ.

     For example, if you want to improve the failover behavior of the rabbitmq-cell1 RabbitMQ cluster add the following lines to the end of the rabbitmq.templates.rabbitmq-cell1 section of the OpenStackControlPlane CR:

         podOverride:
           services:
             - metadata:
                 annotations:
                   metallb.universe.tf/address-pool: "internalapi"
                   metallb.universe.tf/loadBalancerIPs: "172.17.0.87"
               spec:
                 type: LoadBalancer
             - metadata:
                 annotations:
                   metallb.universe.tf/address-pool: "internalapi"
                   metallb.universe.tf/loadBalancerIPs: "172.17.0.88"
               spec:
                 type: LoadBalancer  !
             - metadata:
                 annotations:
                   metallb.universe.tf/address-pool: "internalapi"
                   metallb.universe.tf/loadBalancerIPs: "172.17.0.89"
               spec:
                 type: LoadBalancer

7. Create the control plane:

   $ oc create -f openstack_control_plane.yaml -n openstack

8. Wait until RHOCP creates the resources related to the OpenStackControlPlane CR. Run the following command to check the status:

   $ oc get openstackcontrolplane -n openstack
   NAME 						STATUS 	MESSAGE
   openstack-control-plane 	Unknown 	Setup started

   The OpenStackControlPlane resources are created when the status is "Setup complete".

   Tip

   Append the -w option to the end of the get command to track deployment progress.

   Note

   Creating the control plane also creates an OpenStackClient pod that you can access through a remote shell (rsh) to run OpenStack CLI commands.

   $ oc rsh -n openstack openstackclient

9. Optional: Confirm that the control plane is deployed by reviewing the pods in the openstack namespace:

   $ oc get pods -n openstack

   The control plane is deployed when all the pods are either completed or running.

Verification

1. Open a remote shell connection to the OpenStackClient pod:

   $ oc rsh -n openstack openstackclient

2. Confirm that the internal service endpoints are registered with each service:

   $ openstack endpoint list -c 'Service Name' -c Interface -c URL --service glance
   +--------------+-----------+---------------------------------------------------------------+
   | Service Name | Interface | URL                                                           |
   +--------------+-----------+---------------------------------------------------------------+
   | glance       | internal  | https://glance-internal.openstack.svc                     |
   | glance       | public    | https://glance-default-public-openstack.apps.ostest.test.metalkube.org |
   +--------------+-----------+---------------------------------------------------------------+

3. Exit the OpenStackClient pod:

   $ exit

Procedure

1. Open the OpenStackControlPlane CR file on your workstation.

2. Locate the service you want to remove from the control plane and disable it:

     cinder:
       enabled: false
       apiOverride:
         route: {}
         ...

3. Update the control plane:

   $ oc apply -f openstack_control_plane.yaml -n openstack

4. Wait until RHOCP removes the resource related to the disabled service. Run the following command to check the status:

   $ oc get openstackcontrolplane -n openstack
   NAME 							STATUS 	MESSAGE
   openstack-control-plane 	Unknown 	Setup started

   The OpenStackControlPlane resource is updated with the disabled service when the status is "Setup complete".

   Tip

   Append the -w option to the end of the get command to track deployment progress.

5. Optional: Confirm that the pods from the disabled service are no longer listed by reviewing the pods in the openstack namespace:

   $ oc get pods -n openstack

6. Check that the service is removed:

   $ oc get cinder -n openstack

   This command returns the following message when the service is successfully removed:

   No resources found in openstack namespace.

7. Check that the API endpoints for the service are removed from the Identity service (keystone):

   $ oc rsh -n openstack openstackclient
   $ openstack endpoint list --service volumev3

   This command returns the following message when the API endpoints for the service are successfully removed:

   No service with a type, name or ID of 'volumev3' exists.