Deploying Red Hat OpenStack Services on OpenShift

Procedure

1. Log in to the RHOCP web console as a user with cluster-admin permissions.
2. Select Operators → OperatorHub.
3. In the Filter by keyword field, type OpenStack.
4. Click the OpenStack Operator tile with the Red Hat source label.
5. Read the information about the Operator and click Install.
6. On the Install Operator page, select "Operator recommended Namespace: openstack-operators" from the Installed Namespace list.
7. On the Install Operator page, select "Manual" from the Update approval list. For information about how to manually approve a pending Operator update, see Manually approving a pending Operator update in the RHOCP Operators guide.
8. Click Install to make the Operator available to the openstack-operators namespace. The OpenStack Operator is installed when the Status is Succeeded.
9. Click Create OpenStack to open the Create OpenStack page.
10. On the Create OpenStack page, click Create to create an instance of the OpenStack Operator initialization resource. The OpenStack Operator is ready to use when the Status of the openstack instance is Conditions: Ready.

Procedure

1. Create the openstack-operators namespace for the RHOSP operators:

   $ cat << EOF | oc apply -f -
   apiVersion: v1
   kind: Namespace
   metadata:
     name: openstack-operators
   spec:
     finalizers:
     - kubernetes
   EOF

2. Create the OperatorGroup CR in the openstack-operators namespace:

   $ cat << EOF | oc apply -f -
   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: openstack
     namespace: openstack-operators
   EOF

3. Create the Subscription CR that subscribes to openstack-operator:

   $ cat << EOF| oc apply -f -
   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: openstack-operator
     namespace: openstack-operators
   spec:
     name: openstack-operator
     channel: stable-v1.0
     source: redhat-operators
     sourceNamespace: openshift-marketplace
     installPlanApproval: Manual
   EOF

4. Wait for the install plan to be created:

   $ oc get installplan -n openstack-operators -o json | jq -r '.items[] | select(.spec.approval=="Manual" and .spec.approved==false) | .metadata.name' | head -n1

5. Approve the install plan:

   $ oc patch installplan <install_plan_name> -n openstack-operators --type merge -p '{"spec":{"approved":true}}'

6. Verify that the OpenStack Operator is installed:

   $ oc wait csv -n openstack-operators \
    -l operators.coreos.com/openstack-operator.openstack-operators="" \
    --for jsonpath='{.status.phase}'=Succeeded

7. Create an instance of the openstack-operator:

   $ cat << EOF | oc apply -f -
   apiVersion: operator.openstack.org/v1beta1
   kind: OpenStack
   metadata:
     name: openstack
     namespace: openstack-operators
   EOF

8. Confirm that the Openstack Operator is deployed:

   $ oc wait openstack/openstack -n openstack-operators --for condition=Ready --timeout=500s

Procedure

1. Create the openstack project for the deployed RHOSO environment:

   $ oc new-project openstack

2. Ensure the openstack namespace is labeled to enable privileged pod creation by the OpenStack Operators:

   $ oc get namespace openstack -ojsonpath='{.metadata.labels}' | jq
   {
     "kubernetes.io/metadata.name": "openstack",
     "pod-security.kubernetes.io/enforce": "privileged",
     "security.openshift.io/scc.podSecurityLabelSync": "false"
   }

   If the security context constraint (SCC) is not "privileged", use the following commands to change it:

   $ oc label ns openstack security.openshift.io/scc.podSecurityLabelSync=false --overwrite
   $ oc label ns openstack pod-security.kubernetes.io/enforce=privileged --overwrite

3. Optional: To remove the need to specify the namespace when executing commands on the openstack namespace, set the default namespace to openstack:

   $ oc project openstack

Procedure

1. Create a Secret CR on your workstation, for example, openstack_service_secret.yaml.

2. Add the following initial configuration to openstack_service_secret.yaml:

   apiVersion: v1
   data:
     AdminPassword: <base64_password>
     AodhPassword: <base64_password>
     BarbicanPassword: <base64_password>
     BarbicanSimpleCryptoKEK: <base64_fernet_key>
     CeilometerPassword: <base64_password>
     CinderPassword: <base64_password>
     DbRootPassword: <base64_password>
     DesignatePassword: <base64_password>
     GlancePassword: <base64_password>
     HeatAuthEncryptionKey: <base64_password>
     HeatPassword: <base64_password>
     IronicInspectorPassword: <base64_password>
     IronicPassword: <base64_password>
     ManilaPassword: <base64_password>
     MetadataSecret: <base64_password>
     NeutronPassword: <base64_password>
     NovaPassword: <base64_password>
     OctaviaPassword: <base64_password>
     PlacementPassword: <base64_password>
     SwiftPassword: <base64_password>
   kind: Secret
   metadata:
     name: osp-secret
     namespace: openstack
   type: Opaque

   • Replace <base64_password> with a 32-character key that is base64 encoded.

     Note

     The HeatAuthEncryptionKey password must be a 32-character key for Orchestration service (heat) encryption. If you increase the length of the passwords for all other services, ensure that the HeatAuthEncryptionKey password remains at length 32.

     You can use the following command to manually generate a base64 encoded password:

     $ echo -n <password> | base64

     Alternatively, if you are using a Linux workstation and you are generating the Secret CR by using a Bash command such as cat, you can replace <base64_password> with the following command to auto-generate random passwords for each service:

     $(tr -dc 'A-Za-z0-9' < /dev/urandom | head -c 32 | base64)

   • Replace the <base64_fernet_key> with a base64 encoded fernet key. You can use the following command to manually generate it:

     $(python3 -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode('UTF-8'))" | base64)

3. Create the Secret CR in the cluster:

   $ oc create -f openstack_service_secret.yaml -n openstack

4. Verify that the Secret CR is created:

   $ oc describe secret osp-secret -n openstack

Procedure

1. Create a file named openstack_netconfig.yaml on your workstation.

2. Add the following configuration to openstack_netconfig.yaml to create the NetConfig CR:

   apiVersion: network.openstack.org/v1beta1
   kind: NetConfig
   metadata:
     name: openstacknetconfig
     namespace: openstack

3. In the openstack_netconfig.yaml file, define the topology for each data plane network. To use the default Red Hat OpenStack Services on OpenShift (RHOSO) networks, you must define a specification for each network. For information about the default RHOSO networks, see Networks for Red Hat OpenStack Services on OpenShift.

   Note

   If you are using pre-provisioned data plane nodes, then the control plane network and IP address must match the pre-provisioned data plane nodes. If the ctlplane network uses tagged VLANs, then the VLAN ID must also match the VLAN ID on the pre-provisioned data plane node.

   The following example creates isolated networks for the data plane:

   spec:
     networks:
     - name: ctlplane
       dnsDomain: ctlplane.example.com
       subnets:
       - name: subnet1
         allocationRanges:
         - end: 192.168.122.120
           start: 192.168.122.100
         - end: 192.168.122.200
           start: 192.168.122.150
         cidr: 192.168.122.0/24
         gateway: 192.168.122.1
     - name: internalapi
       dnsDomain: internalapi.example.com
       subnets:
       - name: subnet1
         allocationRanges:
         - end: 172.17.0.250
           start: 172.17.0.100
         excludeAddresses:
         - 172.17.0.10
         - 172.17.0.12
         cidr: 172.17.0.0/24
         vlan: 20
     - name: external
       dnsDomain: external.example.com
       subnets:
       - name: subnet1
         allocationRanges:
         - end: 10.0.0.250
           start: 10.0.0.100
         cidr: 10.0.0.0/24
         gateway: 10.0.0.1
     - name: storage
       dnsDomain: storage.example.com
       subnets:
       - name: subnet1
         allocationRanges:
         - end: 172.18.0.250
           start: 172.18.0.100
         cidr: 172.18.0.0/24
         vlan: 21
     - name: tenant
       dnsDomain: tenant.example.com
       subnets:
       - name: subnet1
         allocationRanges:
         - end: 172.19.0.250
           start: 172.19.0.100
         cidr: 172.19.0.0/24
         vlan: 22

   • spec.networks.name: The name of the network, for example, ctlplane.
   • spec.networks.subnets: The IPv4 subnet specification.
   • spec.networks.subnets.name: The name of the subnet, for example, subnet1.
   • spec.networks.subnets.allocationRanges: The NetConfig allocationRange. The allocationRange must not overlap with the MetalLB IPAddressPool range and the IP address pool range.
   • spec.networks.subnets.excludeAddresses: Optional: List of IP addresses from the allocation range that must not be used by data plane nodes.
   • spec.networks.subnets.vlan: The network VLAN. For information about the default RHOSO networks, see Networks for Red Hat OpenStack Services on OpenShift.
4. Save the openstack_netconfig.yaml definition file.

5. Create the data plane network:

   $ oc create -f openstack_netconfig.yaml -n openstack

6. To verify that the data plane network is created, view the openstacknetconfig resource:

   $ oc get netconfig/openstacknetconfig -n openstack

   If you see errors, check the underlying network-attach-definition and node network configuration policies:

   $ oc get network-attachment-definitions -n openstack
   $ oc get nncp

Procedure

1. Create a file on your workstation named openstack_control_plane.yaml to define the OpenStackControlPlane CR:

   apiVersion: core.openstack.org/v1beta1
   kind: OpenStackControlPlane
   metadata:
     name: openstack-control-plane
     namespace: openstack

2. Specify the Secret CR you created to provide secure access to the RHOSO service pods in Providing secure access to the Red Hat OpenStack Services on OpenShift services:

   apiVersion: core.openstack.org/v1beta1
   kind: OpenStackControlPlane
   metadata:
     name: openstack-control-plane
     namespace: openstack
   spec:
     secret: osp-secret

3. Specify the storageClass you created for your Red Hat OpenShift Container Platform (RHOCP) cluster storage back end:

   spec:
     secret: osp-secret
     storageClass: <RHOCP_storage_class>

   • Replace <RHOCP_storage_class> with the storage class you created for your RHOCP cluster storage back end. For information about storage classes, see Creating a storage class.

4. Add the global RabbitMQ settings messagingBus and notificationsBus to specify the default RabbitMQ cluster for RHOSO services:

   apiVersion: core.openstack.org/v1beta1
   kind: OpenStackControlPlane
   metadata:
     name: openstack
   spec:
     messagingBus:
       cluster: <rabbitmq-cluster>
   
     notificationsBus:
       cluster: <rabbitmq-cluster>

   • Replace <rabbitmq-cluster> with the default RabbitMQ cluster that all the RHOSO services use, in this example, rabbitmq. You can customize the RabbitMQ interface for OpenStack services. For more information, see Understand the RabbitMQ interface for OpenStack services in the Monitoring high availability services guide.

5. Add the following service configurations:

   Note
   • The following service examples use IP addresses from the default RHOSO MetalLB IPAddressPool range for the loadBalancerIPs field. Update the loadBalancerIPs field with the IP address from the MetalLB IPAddressPool range that you created.
   • You cannot override the default public service endpoint. The public service endpoints are exposed as RHOCP routes by default, because only routes are supported for public endpoints.

   • Block Storage service (cinder):

       cinder:
         apiOverride:
           route: {}
         template:
           databaseInstance: openstack
           secret: osp-secret
           cinderAPI:
             replicas: 3
             override:
               service:
                 internal:
                   metadata:
                     annotations:
                       metallb.universe.tf/address-pool: internalapi
                       metallb.universe.tf/allow-shared-ip: internalapi
                       metallb.universe.tf/loadBalancerIPs: 172.17.0.80
                   spec:
                     type: LoadBalancer
           cinderScheduler:
             replicas: 1
           cinderBackup:
             networkAttachments:
             - storage
             replicas: 0
           cinderVolumes:
             volume1:
               networkAttachments:
               - storage
               replicas: 0

     • cinderBackup.replicas: You can deploy the initial control plane without activating the cinderBackup service. To deploy the service, you must set the number of replicas for the service and configure the back end for the service. For information about the recommended replicas for each service and how to configure a back end for the Block Storage service and the backup service, see Configuring the Block Storage backup service in Configuring persistent storage.
     • cinderVolumes.replicas: You can deploy the initial control plane without activating the cinderVolumes service. To deploy the service, you must set the number of replicas for the service and configure the back end for the service. For information about the recommended replicas for the cinderVolumes service and how to configure a back end for the service, see Configuring the Block Storage volume service component in Configuring persistent storage.

   • Compute service (nova):

       nova:
         apiOverride:
           route: {}
         template:
           apiServiceTemplate:
             replicas: 3
             override:
               service:
                 internal:
                   metadata:
                     annotations:
                       metallb.universe.tf/address-pool: internalapi
                       metallb.universe.tf/allow-shared-ip: internalapi
                       metallb.universe.tf/loadBalancerIPs: 172.17.0.80
                   spec:
                     type: LoadBalancer
           metadataServiceTemplate:
             replicas: 3
             override:
               service:
                 metadata:
                   annotations:
                     metallb.universe.tf/address-pool: internalapi
                     metallb.universe.tf/allow-shared-ip: internalapi
                     metallb.universe.tf/loadBalancerIPs: 172.17.0.80
                 spec:
                   type: LoadBalancer
           schedulerServiceTemplate:
             replicas: 3
           cellTemplates:
             cell0:
               cellDatabaseAccount: nova-cell0
               cellDatabaseInstance: openstack
               messagingBus:
                 cluster: rabbitmq
               hasAPIAccess: true
             cell1:
               cellDatabaseAccount: nova-cell1
               cellDatabaseInstance: openstack-cell1
               messagingBus:
                 cluster: rabbitmq-cell1
               noVNCProxyServiceTemplate:
                 enabled: true
                 networkAttachments:
                 - ctlplane
               hasAPIAccess: true
           secret: osp-secret

     Note

     A full set of Compute services (nova) are deployed by default for each of the default cells, cell0 and cell1: nova-api, nova-metadata, nova-scheduler, and nova-conductor. The novncproxy service is also enabled for cell1 by default.

   • DNS service for the data plane:

       dns:
         template:
           options:
           - key: server
             values:
             - <IP address for DNS server reachable from dnsmasq pod>
           override:
             service:
               metadata:
                 annotations:
                   metallb.universe.tf/address-pool: ctlplane
                   metallb.universe.tf/allow-shared-ip: ctlplane
                   metallb.universe.tf/loadBalancerIPs: 192.168.122.80
               spec:
                 type: LoadBalancer
           replicas: 2

     • options: Defines the dnsmasq instances required for each DNS server by using key-value pairs. In this example, there is one key-value pair defined because there is only one DNS server configured to forward requests to.

     • key: Specifies the dnsmasq parameter to customize for the deployed dnsmasq instance. Set to one of the following valid values:

       • server
       • rev-server
       • srv-host
       • txt-record
       • ptr-record
       • rebind-domain-ok
       • naptr-record
       • cname
       • host-record
       • caa-record
       • dns-rr
       • auth-zone
       • synth-domain
       • no-negcache
       • local

     • values: Specifies the value for the DNS server reachable from the dnsmasq pod on the RHOCP cluster network. You can specify a generic DNS server as the value, for example, 1.1.1.1, or a DNS server for a specific domain, for example, /google.com/8.8.8.8.

       Note

       This DNS service, dnsmasq, provides DNS services for nodes on the RHOSO data plane. dnsmasq is different from the RHOSO DNS service (designate) that provides DNS as a service for cloud tenants.

   • Identity service (keystone)

       keystone:
         apiOverride:
           route: {}
         template:
           override:
             service:
               internal:
                 metadata:
                   annotations:
                     metallb.universe.tf/address-pool: internalapi
                     metallb.universe.tf/allow-shared-ip: internalapi
                     metallb.universe.tf/loadBalancerIPs: 172.17.0.80
                 spec:
                   type: LoadBalancer
           databaseInstance: openstack
           secret: osp-secret
           replicas: 3

   • Image service (glance):

       glance:
         apiOverrides:
           default:
             route: {}
         template:
           databaseInstance: openstack
           storage:
             storageRequest: 10G
           secret: osp-secret
           keystoneEndpoint: default
           glanceAPIs:
             default:
               replicas: 0 # Configure back end; set to 3 when deploying service
               override:
                 service:
                   internal:
                     metadata:
                       annotations:
                         metallb.universe.tf/address-pool: internalapi
                         metallb.universe.tf/allow-shared-ip: internalapi
                         metallb.universe.tf/loadBalancerIPs: 172.17.0.80
                     spec:
                       type: LoadBalancer
               networkAttachments:
               - storage

     • glanceAPIs.default.replicas: You can deploy the initial control plane without activating the Image service (glance). To deploy the Image service, you must set the number of replicas for the service and configure the back end for the service. For information about the recommended replicas for the Image service and how to configure a back end for the service, see Configuring the Image service (glance) in Configuring persistent storage. If you do not deploy the Image service, you cannot upload images to the cloud or start an instance.

   • Key Management service (barbican):

       barbican:
         apiOverride:
           route: {}
         template:
           databaseInstance: openstack
           secret: osp-secret
           barbicanAPI:
             replicas: 3
             override:
               service:
                 internal:
                   metadata:
                     annotations:
                       metallb.universe.tf/address-pool: internalapi
                       metallb.universe.tf/allow-shared-ip: internalapi
                       metallb.universe.tf/loadBalancerIPs: 172.17.0.80
                   spec:
                     type: LoadBalancer
           barbicanWorker:
             replicas: 3
           barbicanKeystoneListener:
             replicas: 1

   • Networking service (neutron):

       neutron:
         apiOverride:
           route: {}
         template:
           replicas: 3
           override:
             service:
               internal:
                 metadata:
                   annotations:
                     metallb.universe.tf/address-pool: internalapi
                     metallb.universe.tf/allow-shared-ip: internalapi
                     metallb.universe.tf/loadBalancerIPs: 172.17.0.80
                 spec:
                   type: LoadBalancer
           databaseInstance: openstack
           secret: osp-secret
           networkAttachments:
           - internalapi

   • Object Storage service (swift):

       swift:
         enabled: true
         proxyOverride:
           route: {}
         template:
           swiftProxy:
             networkAttachments:
             - storage
             override:
               service:
                 internal:
                   metadata:
                     annotations:
                       metallb.universe.tf/address-pool: internalapi
                       metallb.universe.tf/allow-shared-ip: internalapi
                       metallb.universe.tf/loadBalancerIPs: 172.17.0.80
                   spec:
                     type: LoadBalancer
             replicas: 2
             secret: osp-secret
           swiftRing:
             ringReplicas: 3
           swiftStorage:
             networkAttachments:
             - storage
             replicas: 3
             storageRequest: 10Gi

   • Optimize service (watcher):

       watcher:
         enabled: true

   • OVN:

       ovn:
         template:
           ovnDBCluster:
             ovndbcluster-nb:
               replicas: 3
               dbType: NB
               storageRequest: 10G
               networkAttachment: internalapi
             ovndbcluster-sb:
               replicas: 3
               dbType: SB
               storageRequest: 10G
               networkAttachment: internalapi
           ovnNorthd: {}

   • Placement service (placement):

       placement:
         apiOverride:
           route: {}
         template:
           override:
             service:
               internal:
                 metadata:
                   annotations:
                     metallb.universe.tf/address-pool: internalapi
                     metallb.universe.tf/allow-shared-ip: internalapi
                     metallb.universe.tf/loadBalancerIPs: 172.17.0.80
                 spec:
                   type: LoadBalancer
           databaseInstance: openstack
           replicas: 3
           secret: osp-secret

   • Telemetry service (ceilometer, prometheus):

       telemetry:
         enabled: true
         template:
           metricStorage:
             enabled: true
             dashboardsEnabled: true
             dataplaneNetwork: ctlplane
             networkAttachments:
               - ctlplane
             monitoringStack:
               alertingEnabled: true
               scrapeInterval: 30s
               storage:
                 strategy: persistent
                 retention: 24h
                 persistent:
                   pvcStorageRequest: 20G
           autoscaling:
             enabled: false
             aodh:
               databaseAccount: aodh
               databaseInstance: openstack
               passwordSelector:
                 aodhService: AodhPassword
               serviceUser: aodh
               secret: osp-secret
             heatInstance: heat
           ceilometer:
             enabled: true
             secret: osp-secret
           logging:
             enabled: false

     • telemetry.template.metricStorage.dataplaneNetwork: Defines the network that you use to scrape dataplane node_exporter endpoints.
     • telemetry.template.metricStorage.networkAttachments: Lists the networks that each service pod is attached to by using the NetworkAttachmentDefinition resource names. You configure a NIC for the service for each network attachment that you specify. If you do not configure the isolated networks that each service pod is attached to, then the default pod network is used. You must create a networkAttachment that matches the network that you specify as the dataplaneNetwork, so that Prometheus can scrape data from the dataplane nodes.
     • telemetry.template.autoscaling: You must have the autoscaling field present, even if autoscaling is disabled. For more information about autoscaling, see Autoscaling for Instances.

6. Add the following service configurations to implement high availability (HA):

   • A MariaDB Galera cluster for use by all RHOSO services (openstack), and a MariaDB Galera cluster for use by the Compute service for cell1 (openstack-cell1):

       galera:
         templates:
           openstack:
             storageRequest: 5000M
             secret: osp-secret
             replicas: 3
           openstack-cell1:
             storageRequest: 5000M
             secret: osp-secret
             replicas: 3

   • A single memcached cluster that contains three memcached servers:

       memcached:
         templates:
           memcached:
              replicas: 3

   • A RabbitMQ cluster for use by all RHOSO services (rabbitmq), and a RabbitMQ cluster for use by the Compute service for cell1 (rabbitmq-cell1):

       rabbitmq:
         templates:
           rabbitmq:
             persistence:
               storage: <rabbitmq_cluster_storage>
             replicas: 3
             override:
               service:
                 metadata:
                   annotations:
                     metallb.universe.tf/address-pool: internalapi
                     metallb.universe.tf/loadBalancerIPs: 172.17.0.85
                 spec:
                   type: LoadBalancer
           rabbitmq-cell1:
             persistence:
               storage: <rabbitmq_cluster_storage>
             replicas: 3
             override:
               service:
                 metadata:
                   annotations:
                     metallb.universe.tf/address-pool: internalapi
                     metallb.universe.tf/loadBalancerIPs: 172.17.0.86
                 spec:
                   type: LoadBalancer

   • Replace <rabbitmq_cluster_storage> with sufficient storage for each RabbitMQ cluster, for example, 10Gi.

     Note

     You cannot configure multiple RabbitMQ instances on the same virtual IP (VIP) address because all RabbitMQ instances use the same port. If you need to expose multiple RabbitMQ instances to the same network, then you must use distinct IP addresses.

     By default, RabbitMQ uses Quorum queues to provide increased data safety and high availability at the expense of a slight increase in latency.

     Warning

     You must not configure the RabbitMQ clusters of an existing RHOSO deployment to use Quorum queues! If you do so then your existing RHOSO services will not start or work properly.

     The RabbitMQ Quorum queue is a durable replicated queue based on the Raft consensus algorithm.

     Note

     The RabbitMQ Quorum queues must be saved to disk to ensure their durability. Quorum queues can occupy significant disk space. Depending on the workload and settings the time taken to free up space after messages are consumed or expire can vary substantially. Ensure that sufficient storage is assigned to each RabbitMQ cluster.

     Improved RabbitMQ failover behavior

     The RabbitMQ service can be configured to provide an improved failover behavior that bypasses the default availability checks of OpenShift, which can wait up to five minutes to declare a service dead. By implementing this improved failover behavior, each OpenStack service checks to see if a RabbitMQ pod is alive and moves to the next pod if it is not.

     Note

     To implement this improved failover behavior for a RabbitMQ cluster, you must reserve three free IP addresses from the default RHOSO MetalLB IPAddressPool range already used for RabbitMQ.

     For example, if you want to improve the failover behavior of the rabbitmq-cell1 RabbitMQ cluster add the following lines to the end of the rabbitmq.templates.rabbitmq-cell1 section of the OpenStackControlPlane CR:

         podOverride:
           services:
             - metadata:
                 annotations:
                   metallb.universe.tf/address-pool: "internalapi"
                   metallb.universe.tf/loadBalancerIPs: "172.17.0.87"
               spec:
                 type: LoadBalancer
             - metadata:
                 annotations:
                   metallb.universe.tf/address-pool: "internalapi"
                   metallb.universe.tf/loadBalancerIPs: "172.17.0.88"
               spec:
                 type: LoadBalancer  !
             - metadata:
                 annotations:
                   metallb.universe.tf/address-pool: "internalapi"
                   metallb.universe.tf/loadBalancerIPs: "172.17.0.89"
               spec:
                 type: LoadBalancer

7. Create the control plane:

   $ oc create -f openstack_control_plane.yaml -n openstack

8. Wait until RHOCP creates the resources related to the OpenStackControlPlane CR. Run the following command to check the status:

   $ oc get openstackcontrolplane -n openstack
   NAME 						STATUS 	MESSAGE
   openstack-control-plane 	Unknown 	Setup started

   The OpenStackControlPlane resources are created when the status is "Setup complete".

   Tip

   Append the -w option to the end of the get command to track deployment progress.

   Note

   Creating the control plane also creates an OpenStackClient pod that you can access through a remote shell (rsh) to run OpenStack CLI commands.

   $ oc rsh -n openstack openstackclient

9. Optional: Confirm that the control plane is deployed by reviewing the pods in the openstack namespace:

   $ oc get pods -n openstack

   The control plane is deployed when all the pods are either completed or running.

Verification

1. Open a remote shell connection to the OpenStackClient pod:

   $ oc rsh -n openstack openstackclient

2. Confirm that the internal service endpoints are registered with each service:

   $ openstack endpoint list -c 'Service Name' -c Interface -c URL --service glance
   +--------------+-----------+---------------------------------------------------------------+
   | Service Name | Interface | URL                                                           |
   +--------------+-----------+---------------------------------------------------------------+
   | glance       | internal  | https://glance-internal.openstack.svc                     |
   | glance       | public    | https://glance-default-public-openstack.apps.ostest.test.metalkube.org |
   +--------------+-----------+---------------------------------------------------------------+

3. Exit the OpenStackClient pod:

   $ exit

Procedure

1. Open the OpenStackControlPlane CR file on your workstation.

2. Locate the service you want to remove from the control plane and disable it:

     cinder:
       enabled: false
       apiOverride:
         route: {}
         ...

3. Update the control plane:

   $ oc apply -f openstack_control_plane.yaml -n openstack

4. Wait until RHOCP removes the resource related to the disabled service. Run the following command to check the status:

   $ oc get openstackcontrolplane -n openstack
   NAME 							STATUS 	MESSAGE
   openstack-control-plane 	Unknown 	Setup started

   The OpenStackControlPlane resource is updated with the disabled service when the status is "Setup complete".

   Tip

   Append the -w option to the end of the get command to track deployment progress.

5. Optional: Confirm that the pods from the disabled service are no longer listed by reviewing the pods in the openstack namespace:

   $ oc get pods -n openstack

6. Check that the service is removed:

   $ oc get cinder -n openstack

   This command returns the following message when the service is successfully removed:

   No resources found in openstack namespace.

7. Check that the API endpoints for the service are removed from the Identity service (keystone):

   $ oc rsh -n openstack openstackclient
   $ openstack endpoint list --service volumev3

   This command returns the following message when the API endpoints for the service are successfully removed:

   No service with a type, name or ID of 'volumev3' exists.

Procedure

1. For unprovisioned nodes, create the SSH key pair for Ansible:

   $ ssh-keygen -f <key_file_name> -N "" -t rsa -b 4096

   • Replace <key_file_name> with the name to use for the key pair.

2. Create the Secret CR for Ansible and apply it to the cluster:

   $ oc create secret generic dataplane-ansible-ssh-private-key-secret \
   --save-config \
   --dry-run=client \
   --from-file=ssh-privatekey=<key_file_name> \
   --from-file=ssh-publickey=<key_file_name>.pub \
   [--from-file=authorized_keys=<key_file_name>.pub] -n openstack \
   -o yaml | oc apply -f -

   • Replace <key_file_name> with the name and location of your SSH key pair file.
   • Optional: Only include the --from-file=authorized_keys option for bare-metal nodes that must be provisioned when creating the data plane.

3. If you are creating Compute nodes, create a secret for migration.

   1. Create the SSH key pair for instance migration:

      $ ssh-keygen -f ./nova-migration-ssh-key -t ecdsa-sha2-nistp521 -N ''

   2. Create the Secret CR for migration and apply it to the cluster:

      $ oc create secret generic nova-migration-ssh-key \
      --save-config \
      --from-file=ssh-privatekey=nova-migration-ssh-key \
      --from-file=ssh-publickey=nova-migration-ssh-key.pub \
      -n openstack \
      -o yaml | oc apply -f -

4. For nodes that have not been registered to the Red Hat Customer Portal, create the Secret CR for subscription-manager credentials to register the nodes:

   $ oc create secret generic subscription-manager \
   --from-literal rhc_auth='{"login": {"username": "<subscription_manager_username>", "password": "<subscription_manager_password>"}}'

   • Replace <subscription_manager_username> with the username you set for subscription-manager.
   • Replace <subscription_manager_password> with the password you set for subscription-manager.

5. Create a Secret CR that contains the Red Hat registry credentials:

   $ oc create secret generic redhat-registry --from-literal edpm_container_registry_logins='{"registry.redhat.io": {"<username>": "<password>"}}'

   • Replace <username> and <password> with your Red Hat registry username and password credentials.

     For information about how to create your registry service account, see the Knowledge Base article Creating Registry Service Accounts.

6. If you are creating Compute nodes, create a secret for libvirt.

   1. Create a file on your workstation named secret_libvirt.yaml to define the libvirt secret:

      apiVersion: v1
      kind: Secret
      metadata:
       name: libvirt-secret
       namespace: openstack
      type: Opaque
      data:
       LibvirtPassword: <base64_password>

      • Replace <base64_password> with a base64-encoded string with maximum length 63 characters. You can use the following command to generate a base64-encoded password:

        $ echo -n <password> | base64

        Tip

        If you do not want to base64-encode the username and password, you can use the stringData field instead of the data field to set the username and password.

   2. Create the Secret CR:

      $ oc apply -f secret_libvirt.yaml -n openstack

7. Verify that the Secret CRs are created:

   $ oc describe secret dataplane-ansible-ssh-private-key-secret
   $ oc describe secret nova-migration-ssh-key
   $ oc describe secret subscription-manager
   $ oc describe secret redhat-registry
   $ oc describe secret libvirt-secret

Procedure

1. Create a file on your workstation named openstack_preprovisioned_node_set.yaml to define the OpenStackDataPlaneNodeSet CR:

   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneNodeSet
   metadata:
     name: openstack-data-plane
     namespace: openstack
   spec:
     env:
       - name: ANSIBLE_FORCE_COLOR
         value: "True"

   • metadata.name: The OpenStackDataPlaneNodeSet CR name must be unique, contain only lower case alphanumeric characters and - (hyphens) or . (periods), and start and end with an alphanumeric character. Update the name in this example to a name that reflects the nodes in the set.

     Important

     The OpenStackDataPlaneNodeSet CR name you provide should be a maximum of 28 characters. Although 63 characters are allotted for the CR name, this character count includes system identifiers that are appended to the name after it is saved. Additional characters above the 63 character limit are truncated when the CR is saved and this causes errors in system processes.

   • spec.env: An optional list of environment variables to pass to the pod.

2. Connect the data plane to the control plane network:

   spec:
     ...
     networkAttachments:
       - ctlplane

3. Specify that the nodes in this set are pre-provisioned:

     preProvisioned: true

4. Add the SSH key secret that you created to enable Ansible to connect to the data plane nodes:

     nodeTemplate:
       ansibleSSHPrivateKeySecret: <secret-key>

   • Replace <secret-key> with the name of the SSH key Secret CR you created for this node set in Creating the data plane secrets, for example, dataplane-ansible-ssh-private-key-secret.
5. Create a Persistent Volume Claim (PVC) in the openstack namespace on your Red Hat OpenShift Container Platform (RHOCP) cluster to store logs. Set the volumeMode to Filesystem and accessModes to ReadWriteOnce. Do not request storage for logs from a PersistentVolume (PV) that uses the NFS volume plugin. NFS is incompatible with FIFO and the ansible-runner creates a FIFO file to write to store logs. For information about PVCs, see Understanding persistent storage in the RHOCP Storage guide and Red Hat OpenShift Container Platform cluster requirements in Planning your deployment.

6. Enable persistent logging for the data plane nodes:

     nodeTemplate:
       ...
       extraMounts:
         - extraVolType: Logs
           volumes:
           - name: ansible-logs
             persistentVolumeClaim:
               claimName: <pvc_name>
           mounts:
           - name: ansible-logs
             mountPath: "/runner/artifacts"

   • Replace <pvc_name> with the name of the PVC storage on your RHOCP cluster.

7. Specify the management network:

     nodeTemplate:
       ...
       managementNetwork: ctlplane

8. Specify the Secret CRs used to source the usernames and passwords to register the operating system of your nodes and to enable repositories. The following example demonstrates how to register your nodes to Red Hat Content Delivery Network (CDN). For information about how to register your nodes with Red Hat Satellite 6.13, see Managing Hosts.

     nodeTemplate:
       ...
       ansible:
         ansibleUser: cloud-admin
         ansiblePort: 22
         ansibleVarsFrom:
           - secretRef:
               name: subscription-manager
           - secretRef:
               name: redhat-registry
         ansibleVars:
           rhc_release: 9.4
           rhc_repositories:
               - {name: "*", state: disabled}
               - {name: "rhel-9-for-x86_64-baseos-eus-rpms", state: enabled}
               - {name: "rhel-9-for-x86_64-appstream-eus-rpms", state: enabled}
               - {name: "rhel-9-for-x86_64-highavailability-eus-rpms", state: enabled}
               - {name: "fast-datapath-for-rhel-9-x86_64-rpms", state: enabled}
               - {name: "rhoso-18.0-for-rhel-9-x86_64-rpms", state: enabled}
               - {name: "rhceph-7-tools-for-rhel-9-x86_64-rpms", state: enabled}
           edpm_bootstrap_release_version_package: []

   • ansibleUser: The user associated with the secret you created in Creating the data plane secrets.
   • ansibleVars: The Ansible variables that customize the set of nodes. For a list of Ansible variables that you can use, see https://openstack-k8s-operators.github.io/edpm-ansible/. For a complete list of the Red Hat Customer Portal registration commands, see https://access.redhat.com/solutions/253273. For information about how to log into registry.redhat.io, see https://access.redhat.com/RegistryAuthentication#creating-registry-service-accounts-6.

9. Add the network configuration template to apply to your data plane nodes. The following example applies a network configuration to the data plane nodes:

     nodeTemplate:
       ...
       ansible:
         ...
         ansibleVars:
           ...
           edpm_network_config_os_net_config_mappings:
             edpm-compute-0:
               nic1: 52:54:04:60:55:22
               nic2: 53:13:10:33:24:61
           neutron_physical_bridge_name: br-ex
           edpm_network_config_nmstate: true
           edpm_network_config_update: false
           edpm_network_config_template: |
             ---
             {% set mtu_list = [ctlplane_mtu] %}
             {% for network in nodeset_networks %}
             {{ mtu_list.append(lookup('vars', networks_lower[network] ~ '_mtu')) }}
             {%- endfor %}
             {% set min_viable_mtu = mtu_list | max %}
             network_config:
             - type: ovs_bridge
               name: {{ neutron_physical_bridge_name }}
               mtu: {{ min_viable_mtu }}
               use_dhcp: false
               dns_servers: {{ ctlplane_dns_nameservers }}
               domain: {{ dns_search_domains }}
               addresses:
               - ip_netmask: {{ ctlplane_ip }}/{{ ctlplane_cidr }}
               routes: {{ ctlplane_host_routes }}
               members:
               - type: linux_bond
                 name: bond0
                 mtu: {{ min_viable_mtu }}
                 bonding_options: "mode=802.3ad lacp_rate=fast updelay=1000 miimon=100 xmit_hash_policy=layer3+4"
                 members:
                 - type: interface
                   name: nic1
                   mtu: {{ min_viable_mtu }}
                   primary: true
                 - type: interface
                   name: nic2
                   mtu: {{ min_viable_mtu }}
             {% for network in nodeset_networks %}
               - type: vlan
                 mtu: {{ lookup('vars', networks_lower[network] ~ '_mtu') }}
                 vlan_id: {{ lookup('vars', networks_lower[network] ~ '_vlan_id') }}
                 addresses:
                 - ip_netmask:
                     {{ lookup('vars', networks_lower[network] ~ '_ip') }}/{{ lookup('vars', networks_lower[network] ~ '_cidr') }}
                 routes: {{ lookup('vars', networks_lower[network] ~ '_host_routes') }}
             {% endfor %}

   • nic1 and nic2: The MAC addresses assigned to the NIC to use for network configuration on the Compute node.
   • neutron_physical_bridge_name: The name of the OVS bridge to setup on the Compute node.
   • edpm_network_config_nmstate: Sets the os-net-config provider to nmstate. The default value is true. Change it to false only if a specific limitation of the nmstate provider requires you to use the ifcfg provider now. In a future release after nmstate limitations are resolved, the ifcfg provider will be deprecated and removed. In this RHOSO release, adoption of a RHOSP 17.1 deployment with the nmstate provider is not supported. For this and other limitations of RHOSO nmstate support, see https://issues.redhat.com/browse/OSPRH-11309.

   • edpm_network_config_update: When deploying a node set for the first time, ensure that the edpm_network_config_update variable is set to false. If you later update a edpm_network_config_template, first set edpm_network_config_update to true. After you complete the update, reset it to false.

     Important

     After an edpm_network_config_template update, you must reset edpm_network_config_update to false. Otherwise, the nodes could lose network access. Whenever edpm_network_config_update is true, the updated network configuration is reapplied every time an OpenStackDataPlaneDeployment CR is created that includes the configure-network service that is a member of the servicesOverride list.

   • dns_servers: Autogenerated from IPAM and DNS and no user input is required.
   • domain: Autogenerated from IPAM and DNS and no user input is required.
   • routes: Autogenerated from IPAM and DNS and no user input is required.
10. Add the common configuration for the set of nodes in this group under the nodeTemplate section. Each node in this OpenStackDataPlaneNodeSet inherits this configuration. For information about the properties you can use to configure common node attributes, see OpenStackDataPlaneNodeSet CR spec properties.

11. Add the time synchronization configuration to align the node time with a central source. Multiple NTP servers can be defined in this configuration. Time synchronization is provided by the chrony service. The following example creates a time synchronization configuration for all nodes in the nodeset use:

      nodeTemplate:
            ...
        ansible:
            ...
          ansibleVars:
            timesync_ntp_servers:
              - hostname: <ntp_server>
                iburst: <burst_value>

    • Replace <ntp_server> with the hostname or IP address of the NTP server.

    • Replace <burst_value> with either true or false. This configures fast initial synchronization with the NTP server. The default is false.

      Note

      Configure time synchronization at the node level by adding it in the ansibleVars section of the individual node.

12. Define each node in this node set:

      nodes:
        edpm-compute-0:
          hostName: edpm-compute-0
          networks:
          - name: ctlplane
            subnetName: subnet1
            defaultRoute: true
            fixedIP: 192.168.122.100
          - name: internalapi
            subnetName: subnet1
            fixedIP: 172.17.0.100
          - name: storage
            subnetName: subnet1
            fixedIP: 172.18.0.100
          - name: tenant
            subnetName: subnet1
            fixedIP: 172.19.0.100
          ansible:
            ansibleHost: 192.168.122.100
            ansibleUser: cloud-admin
            ansibleVars:
              fqdn_internal_api: edpm-compute-0.example.com
        edpm-compute-1:
          hostName: edpm-compute-1
          networks:
          - name: ctlplane
            subnetName: subnet1
            defaultRoute: true
            fixedIP: 192.168.122.101
          - name: internalapi
            subnetName: subnet1
            fixedIP: 172.17.0.101
          - name: storage
            subnetName: subnet1
            fixedIP: 172.18.0.101
          - name: tenant
            subnetName: subnet1
            fixedIP: 172.19.0.101
          ansible:
            ansibleHost: 192.168.122.101
            ansibleUser: cloud-admin
            ansibleVars:
              fqdn_internal_api: edpm-compute-1.example.com

    • edpm-compute-0: The node definition reference, for example, edpm-compute-0. Each node in the node set must have a node definition.
    • networks: Defines the IPAM and the DNS records for the node.
    • fixedIP: Specifies a predictable IP address for the network that must be in the allocation range defined for the network in the NetConfig CR.
    • networkData: The Secret that contains network configuration that is node-specific.
    • userData.name: The Secret that contains user data that is node-specific.
    • ansibleHost: Overrides the hostname or IP address that Ansible uses to connect to the node. The default value is the value set for the hostName field for the node or the node definition reference, for example, edpm-compute-0. Set to the same IP address as the IP address configured on the pre-provisioned node.

    • ansibleVars: Node-specific Ansible variables that customize the node.

      Note
      • Nodes defined within the nodes section can configure the same Ansible variables that are configured in the nodeTemplate section. Where an Ansible variable is configured for both a specific node and within the nodeTemplate section, the node-specific values override those from the nodeTemplate section.
      • You do not need to replicate all the nodeTemplate Ansible variables for a node to override the default and set some node-specific values. You only need to configure the Ansible variables you want to override for the node.

      • Many ansibleVars include edpm in the name, which stands for "External Data Plane Management".

        For information about the properties you can use to configure node attributes, see OpenStackDataPlaneNodeSet CR properties.

13. Save the openstack_preprovisioned_node_set.yaml definition file.

14. Create the data plane resources:

    $ oc create --save-config -f openstack_preprovisioned_node_set.yaml -n openstack

Verification

1. Verify that the data plane resources have been created by confirming that the status is SetupReady:

   $ oc wait openstackdataplanenodeset openstack-data-plane --for condition=SetupReady --timeout=10m

   When the status is SetupReady the command returns a condition met message, otherwise it returns a timeout error.

   For information about the data plane conditions and states, see Data plane conditions and states.

2. Verify that the Secret resource was created for the node set:

   $ oc get secret | grep openstack-data-plane
   dataplanenodeset-openstack-data-plane Opaque 1 3m50s

3. Verify the services were created:

   $ oc get openstackdataplaneservice -n openstack
   NAME                AGE
   bootstrap           46m
   ceph-client         46m
   ceph-hci-pre        46m
   configure-network   46m
   configure-os        46m
   ...

1. Create a BareMetalHost custom resource (CR) for each bare-metal data plane node.
2. Define an OpenStackDataPlaneNodeSet CR for each logical grouping of unprovisioned nodes in your data plane, for example, nodes grouped by hardware, location, or networking.

Procedure

1. Create a file on your workstation named openstack_data_plane_deploy.yaml to define the OpenStackDataPlaneDeployment CR:

   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneDeployment
   metadata:
     name: data-plane-deploy
     namespace: openstack

   • metadata.name: The OpenStackDataPlaneDeployment CR name must be unique, must consist of lower case alphanumeric characters, - (hyphen) or . (period), and must start and end with an alphanumeric character. Update the name in this example to a name that reflects the node sets in the deployment.

2. Add all the OpenStackDataPlaneNodeSet CRs that you want to deploy:

   spec:
     nodeSets:
       - openstack-data-plane
       - <nodeSet_name>
       - ...
       - <nodeSet_name>

   • Replace <nodeSet_name> with the names of the OpenStackDataPlaneNodeSet CRs that you want to include in your data plane deployment.
3. Save the openstack_data_plane_deploy.yaml deployment file.

4. Deploy the data plane:

   $ oc create -f openstack_data_plane_deploy.yaml -n openstack

   You can view the Ansible logs while the deployment executes:

   $ oc get pod -l app=openstackansibleee -w
   $ oc logs -l app=openstackansibleee -f --max-log-requests 10

   If the oc logs command returns an error similar to the following error, increase the --max-log-requests value:

   error: you are attempting to follow 19 log streams, but maximum allowed concurrency is 10, use --max-log-requests to increase the limit

5. Verify that the data plane is deployed:

   $ oc wait openstackdataplanedeployment data-plane-deploy --for=condition=Ready --timeout=<timeout_value>
   $ oc wait openstackdataplanenodeset openstack-data-plane --for=condition=Ready --timeout=<timeout_value>

   • Replace <timeout_value> with a value in minutes you want the command to wait for completion of the task. For example, if you want the command to wait 60 minutes, you would use the value 60m. If the completion status of SetupReady for oc wait openstackdataplanedeployment or NodeSetReady for oc wait openstackdataplanenodeset is not returned in this time frame, the command returns a timeout error. Use a value that is appropriate to the size of your deployment. Give larger deployments more time to complete deployment tasks.

     For information about the data plane conditions and states, see Data plane conditions and states in Deploying Red Hat OpenStack Services on OpenShift.

6. Map the Compute nodes to the Compute cell that they are connected to:

   $ oc rsh nova-cell0-conductor-0 nova-manage cell_v2 discover_hosts --verbose

   If you did not create additional cells, this command maps the Compute nodes to cell1.

7. Access the remote shell for the openstackclient pod and verify that the deployed Compute nodes are visible on the control plane:

   $ oc rsh -n openstack openstackclient
   $ openstack hypervisor list

   If some Compute nodes are missing from the hypervisor list, retry the previous step. If the Compute nodes are still missing from the list, check the status and health of the nova-compute services on the deployed data plane nodes.

8. Verify that the hypervisor hostname is a fully qualified domain name (FQDN):

   $ hostname -f

   If the hypervisor hostname is not an FQDN, for example, if it was registered as a short name or full name instead, contact Red Hat Support.