Deploying a RHOSO environment with distributed zones

Procedure

1. Log in to the RHOCP web console as a user with cluster-admin permissions.
2. Select Operators → OperatorHub.
3. In the Filter by keyword field, type OpenStack.
4. Click the OpenStack Operator tile with the Red Hat source label.
5. Read the information about the Operator and click Install.
6. On the Install Operator page, select "Operator recommended Namespace: openstack-operators" from the Installed Namespace list.
7. On the Install Operator page, select "Manual" from the Update approval list. For information about how to manually approve a pending Operator update, see Manually approving a pending Operator update in the RHOCP Operators guide.
8. Click Install to make the Operator available to the openstack-operators namespace. The OpenStack Operator is installed when the Status is Succeeded.
9. Click Create OpenStack to open the Create OpenStack page.
10. On the Create OpenStack page, click Create to create an instance of the OpenStack Operator initialization resource. The OpenStack Operator is ready to use when the Status of the openstack instance is Conditions: Ready.

Procedure

1. Create the openstack-operators namespace for the RHOSP operators:

   $ cat << EOF | oc apply -f -
   apiVersion: v1
   kind: Namespace
   metadata:
     name: openstack-operators
   spec:
     finalizers:
     - kubernetes
   EOF

2. Create the OperatorGroup CR in the openstack-operators namespace:

   $ cat << EOF | oc apply -f -
   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: openstack
     namespace: openstack-operators
   EOF

3. Create the Subscription CR that subscribes to openstack-operator:

   $ cat << EOF| oc apply -f -
   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: openstack-operator
     namespace: openstack-operators
   spec:
     name: openstack-operator
     channel: stable-v1.0
     source: redhat-operators
     sourceNamespace: openshift-marketplace
     installPlanApproval: Manual
   EOF

4. Wait for the install plan to be created:

   $ oc get installplan -n openstack-operators -o json | jq -r '.items[] | select(.spec.approval=="Manual" and .spec.approved==false) | .metadata.name' | head -n1

5. Approve the install plan:

   $ oc patch installplan <install_plan_name> -n openstack-operators --type merge -p '{"spec":{"approved":true}}'

6. Verify that the OpenStack Operator is installed:

   $ oc wait csv -n openstack-operators \
    -l operators.coreos.com/openstack-operator.openstack-operators="" \
    --for jsonpath='{.status.phase}'=Succeeded

7. Create an instance of the openstack-operator:

   $ cat << EOF | oc apply -f -
   apiVersion: operator.openstack.org/v1beta1
   kind: OpenStack
   metadata:
     name: openstack
     namespace: openstack-operators
   EOF

8. Confirm that the Openstack Operator is deployed:

   $ oc wait openstack/openstack -n openstack-operators --for condition=Ready --timeout=500s

Procedure

1. Create the openstack project for the deployed RHOSO environment:

   $ oc new-project openstack

2. Ensure the openstack namespace is labeled to enable privileged pod creation by the OpenStack Operators:

   $ oc get namespace openstack -ojsonpath='{.metadata.labels}' | jq
   {
     "kubernetes.io/metadata.name": "openstack",
     "pod-security.kubernetes.io/enforce": "privileged",
     "security.openshift.io/scc.podSecurityLabelSync": "false"
   }

   If the security context constraint (SCC) is not "privileged", use the following commands to change it:

   $ oc label ns openstack security.openshift.io/scc.podSecurityLabelSync=false --overwrite
   $ oc label ns openstack pod-security.kubernetes.io/enforce=privileged --overwrite

3. Optional: To remove the need to specify the namespace when executing commands on the openstack namespace, set the default namespace to openstack:

   $ oc project openstack

Procedure

1. Create a Secret CR on your workstation, for example, openstack_service_secret.yaml.

2. Add the following initial configuration to openstack_service_secret.yaml:

   apiVersion: v1
   data:
     AdminPassword: <base64_password>
     AodhPassword: <base64_password>
     BarbicanPassword: <base64_password>
     BarbicanSimpleCryptoKEK: <base64_fernet_key>
     CeilometerPassword: <base64_password>
     CinderPassword: <base64_password>
     DbRootPassword: <base64_password>
     DesignatePassword: <base64_password>
     GlancePassword: <base64_password>
     HeatAuthEncryptionKey: <base64_password>
     HeatPassword: <base64_password>
     IronicInspectorPassword: <base64_password>
     IronicPassword: <base64_password>
     ManilaPassword: <base64_password>
     MetadataSecret: <base64_password>
     NeutronPassword: <base64_password>
     NovaPassword: <base64_password>
     OctaviaPassword: <base64_password>
     PlacementPassword: <base64_password>
     SwiftPassword: <base64_password>
   kind: Secret
   metadata:
     name: osp-secret
     namespace: openstack
   type: Opaque

   • Replace <base64_password> with a 32-character key that is base64 encoded.

     Note

     The HeatAuthEncryptionKey password must be a 32-character key for Orchestration service (heat) encryption. If you increase the length of the passwords for all other services, ensure that the HeatAuthEncryptionKey password remains at length 32.

     You can use the following command to manually generate a base64 encoded password:

     $ echo -n <password> | base64

     Alternatively, if you are using a Linux workstation and you are generating the Secret CR by using a Bash command such as cat, you can replace <base64_password> with the following command to auto-generate random passwords for each service:

     $(tr -dc 'A-Za-z0-9' < /dev/urandom | head -c 32 | base64)

   • Replace the <base64_fernet_key> with a base64 encoded fernet key. You can use the following command to manually generate it:

     $(python3 -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode('UTF-8'))" | base64)

3. Create the Secret CR in the cluster:

   $ oc create -f openstack_service_secret.yaml -n openstack

4. Verify that the Secret CR is created:

   $ oc describe secret osp-secret -n openstack

Procedure

1. Create a file named netconfig_bgp.yaml on your workstation.

2. Add the following configuration to netconfig_bgp.yaml to create the NetConfig CR:

   apiVersion: network.openstack.org/v1beta1
   kind: NetConfig
   metadata:
     name: bgp-netconfig
     namespace: openstack

3. In the netconfig_bgp.yaml file, define the topology for each data plane network. To use the default Red Hat OpenStack Services on OpenShift (RHOSO) networks, you must define a specification for each network. For information about the default RHOSO networks, see Default Red Hat OpenStack Services on OpenShift networks for BGP. The following example creates isolated networks for the data plane:

   Note

   The example provided in this step and in later steps include only IPv4 addresses. However, RHOSO also supports IPv6 addresses.

   apiVersion: network.openstack.org/v1beta1
   kind: NetConfig
   metadata:
     name: bgp-netconfig
     namespace: openstack
   spec:
     networks:
     - name: ctlplane
       dnsDomain: ctlplane.example.com
       serviceNetwork: ctlplane
       mtu: 1500
       subnets:
       - name: subnet1
         allocationRanges:
         - end: 192.168.122.120
           start: 192.168.122.100
         - end: 192.168.122.200
           start: 192.168.122.150
         cidr: 192.168.122.0/24
         gateway: 192.168.122.1
       - name: subnet2
         allocationRanges:
         - end: 192.168.123.120
           start: 192.168.123.100
         - end: 192.168.123.200
           start: 192.168.123.150
         cidr: 192.168.123.0/24
         gateway: 192.168.123.1
       - name: subnet3
         allocationRanges:
         - end: 192.168.124.120
           start: 192.168.124.100
         - end: 192.168.124.200
           start: 192.168.124.150
         cidr: 192.168.124.0/24
         gateway: 192.168.124.1
     - name: tenant
       dnsDomain: tenant.example.com
       mtu: 1500
       subnets:
       - name: subnet1
         allocationRanges:
         - end: 172.19.0.250
           start: 172.19.0.100
         cidr: 172.19.0.0/24
       vlan: 22

   • spec.networks.name: The name of the network, for example, ctlplane.
   • spec.networks.subnets: The IPv4 subnet specification.
   • spec.networks.subnets.name: The name of the subnet, for example, subnet1.
   • spec.networks.subnets.allocationRanges: The NetConfig allocationRange. The allocationRange must not overlap with the MetalLB IPAddressPool range and the IP address pool range.
   • spec.networks.subnets.vlan: The network VLAN. For information about the default RHOSO networks, see Default Red Hat OpenStack Services on OpenShift networks for BGP.

4. In the netconfig_bgp.yaml file, define the network interfaces that establish connectivity within each zone. The following example defines two network interfaces, bgpnet0 for eth2 and bgpnet1 for `eth3, with a subnet for each zone:

     - name: bgpnet0
       dnsDomain: bgpnet0.example.com
       serviceNetwork: bgpnet0
       mtu: 1500
       subnets:
       - name: subnet0
         allocationRanges:
         - end: 100.64.0.36
           start: 100.64.0.1
         cidr: 100.64.0.0/24
         gateway: 100.64.0.1
         routes:
         - destination: 0.0.0.0/0
           nexthop: 100.64.0.1
       - name: subnet1
         allocationRanges:
         - end: 100.64.1.36
           start: 100.64.1.1
         cidr: 100.64.1.0/24
         gateway: 100.64.1.1
         routes:
         - destination: 0.0.0.0/0
           nexthop: 100.64.1.1
       - name: subnet2
         allocationRanges:
         - end: 100.64.2.36
           start: 100.64.2.1
         cidr: 100.64.2.0/24
         gateway: 100.64.2.1
         routes:
         - destination: 0.0.0.0/0
           nexthop: 100.64.2.1
     - name: bgpnet1
       dnsDomain: bgpnet1.example.com
       serviceNetwork: bgpnet1
       mtu: 1500
       subnets:
       - name: subnet0
         allocationRanges:
         - end: 100.65.0.36
           start: 100.65.0.1
         cidr: 100.65.0.0/24
         gateway: 100.65.0.1
         routes:
         - destination: 0.0.0.0/0
           nexthop: 100.65.0.1
       - name: subnet1
         allocationRanges:
         - end: 100.65.1.36
           start: 100.65.1.1
         cidr: 100.65.1.0/24
         gateway: 100.65.1.1
         routes:
         - destination: 0.0.0.0/0
           nexthop: 100.65.1.1
       - name: subnet2
         allocationRanges:
         - end: 100.65.2.36
           start: 100.65.2.1
         cidr: 100.65.2.0/24
         gateway: 100.65.2.1
         routes:
         - destination: 0.0.0.0/0
           nexthop: 100.65.2.1

   • name: bgpnet0: The network used by the data plane node to communicate with its BGP peer.
   • name: bgpnet1: The network used by the data plane node to communicate with its BGP peer.

5. In the netconfig_bgp.yaml file, configure the IP address of the loopback interface, bgpmainnet, used by each node to communicate with each other:

     - name: bgpmainnet
       dnsDomain: bgpmainnet.example.com
       serviceNetwork: bgpmainnet
       mtu: 1500
       subnets:
       - name: subnet0
         allocationRanges:
         - end: 99.99.0.36
           start: 99.99.0.2
         cidr: 99.99.0.0/24
       - name: subnet1
         allocationRanges:
         - end: 99.99.1.36
           start: 99.99.1.2
         cidr: 99.99.1.0/24
       - name: subnet2
         allocationRanges:
         - end: 99.99.2.36
           start: 99.99.2.2
         cidr: 99.99.2.0/24

6. Save the ` netconfig_bgp.yaml` definition file.

7. Create the data plane network:

   $ oc create -f  netconfig_bgp.yaml -n openstack

8. Create a BGPConfiguration CR file named bgpconfig.yml to announce the IP addresses of the pods over BGP:

   apiVersion: network.openstack.org/v1beta1
   kind: BGPConfiguration
   metadata:
     name: bgpconfiguration
     namespace: openstack
   spec: {}

9. Create the BGPConfiguration CR to create the required FRR configurations for each pod:

   $ oc create -f bgpconfig.yml

Verification

1. Verify that the data plane network is created:

   $ oc get netconfig/openstacknetconfig -n openstack

   If you see errors, check the underlying network-attach-definition and node network configuration policies:

   $ oc get network-attachment-definitions -n openstack
   $ oc get nncp

Procedure

1. Create a Secret CR for the default zone, Zone 1, named ceph-conf-files-az1, that contains the Red Hat Ceph Storage cluster keyrings and conf files for all the zones:

   $ oc create secret generic ceph-conf-files-az1 \
   --from-file=az1.client.openstack.keyring \
   --from-file=az1.conf \
   --from-file=az2.client.openstack.keyring \
   --from-file=az2.conf \
   --from-file=az3.client.openstack.keyring \
   --from-file=az3.conf -n openstack

2. Create a Secret CR for Zone 2 named ceph-conf-files-az2, that contains all the Red Hat Ceph Storage cluster keyrings and conf files for Zone 2 and the keyrings and conf files for the default zone:

   $ oc create secret generic ceph-conf-files-az2 \
   --from-file=az1.client.openstack.keyring \
   --from-file=az1.conf \
   --from-file=az2.client.openstack.keyring \
   --from-file=az2.conf -n openstack

3. Create a Secret CR for Zone 3 named ceph-conf-files-az3, that contains all the Red Hat Ceph Storage cluster keyrings and conf files for Zone 3 and the keyrings and conf files for the default zone:

   $ oc create secret generic ceph-conf-files-az3 \
   --from-file=az1.client.openstack.keyring \
   --from-file=az1.conf \
   --from-file=az3.client.openstack.keyring \
   --from-file=az3.conf -n openstack

Procedure

1. Create a Secret CR to store the credentials to connect to the storage service. The following example is for NetApp iSCSI:

   apiVersion: v1
   kind: Secret
   metadata:
     labels:
       component: cinder-volume
       service: cinder
     name: cinder-volume-ontap-secrets-az1
     namespace: openstack
   stringData:
     cinder-volume-ontap-secrets-az1: |
       [ontap-az1]
       netapp_login = <login>
       netapp_password = <password>
       netapp_vserver = <vserver>
       netapp_pool_name_search_pattern = <pool>

2. Create a separate Secret CR for each zone:

   $ oc create -f cinder-volume-ontap-secrets-az1.yaml
   $ oc create -f cinder-volume-ontap-secrets-az2.yaml
   $ oc create -f cinder-volume-ontap-secrets-az3.yaml

Procedure

1. Create a Secret CR to store the credentials to connect to the storage service. The following example is for NetApp NFS:

   apiVersion: v1
   kind: Secret
   metadata:
     name: osp-secret-manila-az1
     namespace: openstack
   stringData:
     netapp-secrets.conf: |
       [nfs_az1]
       netapp_server_hostname = <hostname>
       netapp_login = <login>
       netapp_password = <password>
       netapp_vserver = <vserver>

2. Create a separate Secret CR for each zone:

   $ oc create -f osp-secret-manila-az1.yaml
   $ oc create -f osp-secret-manila-az2.yaml
   $ oc create -f osp-secret-manila-az3.yaml

Procedure

1. Add a label to the Node object of the RHOCP node that identifies the zone in which the node is located:

   $ oc label nodes <node_name> topology.kubernetes.io/zone=<zone_name> --overwrite

   • Replace <node_name> with the name of the node, for example, worker-0.
   • Replace <zone_name> with the name of the zone, for example, zone1.
2. Repeat step 1 for each RHOCP node in your RHOSO environment.

3. Optional: Review the node zone assignments:

   $ oc get nodes -o json | jq -r '.items[].metadata.labels'

Procedure

1. Create a file on your workstation that defines a Topology CR for a zone, for example, topology_zone1.yaml:

   apiVersion: topology.openstack.org/v1beta1
   kind: Topology
   metadata:
     name: <topology_name>
     namespace: openstack
   spec:
     affinity:
       nodeAffinity:
         requiredDuringSchedulingIgnoredDuringExecution:
           nodeSelectorTerms:
             - matchExpressions:
                 - key: topology.kubernetes.io/zone
                   operator: In
                   values:
                     - <zone_name>

   • Replace <topology_name> with the name for this Topology CR, for example, zone1-node-affinity. The name must be unique, contain only lower case alphanumeric characters and - (hyphens) or . (periods), start and end with an alphanumeric character.
   • Replace <zone_name> with the name of the zone that is associated with this Topology CR, for example, zone1.

2. If you have multiple host nodes in each zone, you can configure host anti-affinity to spread the service pods across hosts within the same zone:

   spec:
     ...
     topologySpreadConstraints:
     - maxSkew: 1
       topologyKey: kubernetes.io/hostname
       whenUnsatisfiable: DoNotSchedule
       matchLabelKeys:
         - pod-template-hash
         - controller-revision-hash

   • whenUnsatisfiable: Specifies how the scheduler handles a pod if it does not satisfy the spread constraint:

     • DoNotSchedule: Instructs the scheduler not to schedule the pod. This is the default behavior. To ensure the deployment has high availability (HA), set the HA services rabbitmq and galera to DoNotSchedule.
     • ScheduleAnyway: Instructs the scheduler to schedule the pod in any location, but to give higher precedence to topologies that minimize the skew. If you set HA services ScheduleAnyway, then when the spread constraint cannot be satisfied, the pod is placed in a different zone. You must then move the pod manually to the correct zone once the zone is operational. For more information about how to manually move pods, see Controlling pod placement onto nodes (scheduling) in RHOCP Nodes.
   • matchLabelKeys: Specifies the label keys to use to group the pods that the affinity rules are applied to, to ensure that the affinity rules are applied only to pods from the same statefulset or deployment resource when scheduling. The matchLabelKeys field enables the resource to be updated with new pods and the spread constraint rules to be applied to only the new set of pods.

3. Create the Topology CR:

   $ oc create -f topology_zone1.yaml

Procedure

1. Create a file on your workstation that defines a Topology CR that spreads the control plane service pods across the zones, for example, spread_pods.yaml:

   apiVersion: topology.openstack.org/v1beta1
   kind: Topology
   metadata:
     name: spread-pods
     namespace: openstack
   spec:
     topologySpreadConstraints:
     - maxSkew: 1
       topologyKey: topology.kubernetes.io/zone
       whenUnsatisfiable: DoNotSchedule
       matchLabelKeys:
         - pod-template-hash
         - controller-revision-hash

   • whenUnsatisfiable: Specifies how to deal with a pod if it does not satisfy the spread constraint:

     • DoNotSchedule: Instructs the scheduler not to schedule the pod. This is the default behavior. To ensure the deployment has high availability (HA), set the HA services rabbitmq and galera to DoNotSchedule.
     • ScheduleAnyway: Instructs the scheduler to schedule the pod in any location, but to give higher precedence to topologies that minimize the skew. If you set HA services to ScheduleAnyway, then when the spread constraint cannot be satisfied, the pod is placed in a different zone. You must then move the pod manually to the correct zone once the zone is operational. For more information about how to manually move pods, see Controlling pod placement onto nodes (scheduling) in RHOCP Nodes.
   • matchLabelKeys: Specifies the label keys to use to group the pods that the affinity rules are applied to. Use this field to ensure that the affinity rules are applied only to pods from the same statefulset or deployment resource when scheduling. The matchLabelKeys field enables the resource to be updated with new pods and the spread constraint rules to be applied to only the new set of pods.

2. If you have multiple host nodes in each zone, you can configure host anti-affinity to spread the service pods across hosts within the same zone:

   spec:
     topologySpreadConstraints:
     ...
     - maxSkew: 1
       topologyKey: kubernetes.io/hostname
       whenUnsatisfiable: DoNotSchedule
       matchLabelKeys:
         - pod-template-hash
         - controller-revision-hash

3. Create the Topology CR:

   $ oc create -f spread_pods.yaml

Procedure

1. Create a file on your workstation named distributed_control_plane.yaml to define the OpenStackControlPlane CR:

   apiVersion: core.openstack.org/v1beta1
   kind: OpenStackControlPlane
   metadata:
     name: distributed-control-plane
     namespace: openstack

2. Specify that the service pods, when created, are spread across the zones in your distributed zone environment:

   apiVersion: core.openstack.org/v1beta1
   kind: OpenStackControlPlane
   metadata:
     name: distributed-control-plane
     namespace: openstack
   spec:
     topologyRef:
       name: spread-pods

3. Specify the Secret CR you created to provide secure access to the RHOSO service pods in Providing secure access to the Red Hat OpenStack Services on OpenShift services:

   spec:
     ...
     secret: osp-secret

4. Specify the storageClass you created for your Red Hat OpenShift Container Platform (RHOCP) cluster storage back end:

   spec:
     ...
     storageClass: <RHOCP_storage_class>

   • Replace <RHOCP_storage_class> with the storage class you created for your RHOCP cluster storage back end. For information about storage classes, see Creating a storage class.

5. If you are using Red Hat Ceph Storage, define the zones and services that require access to the Red Hat Ceph Storage secret:

    extraMounts:
     - extraVol:
       - extraVolType: Ceph
         mounts:
         - mountPath: /etc/ceph
           name: ceph-az1
           readOnly: true
         propagation:
         - az1
         - CinderBackup
         volumes:
         - name: ceph-az1
           projected:
             sources:
             - secret:
                 name: ceph-conf-files-az1
       - extraVolType: Ceph
         mounts:
         - mountPath: /etc/ceph
           name: ceph-az2
           readOnly: true
         propagation:
         - az2
         volumes:
         - name: ceph-az2
           projected:
             sources:
             - secret:
                 name: ceph-conf-files-az2
       - extraVolType: Ceph
         mounts:
         - mountPath: /etc/ceph
           name: ceph-az3
           readOnly: true
         propagation:
         - az3
         volumes:
         - name: ceph-az3
           projected:
             sources:
             - secret:
                 name: ceph-conf-files-az3

6. Configure the Block Storage service (cinder).

   • If you are using the Block Storage service with Red Hat Ceph Storage, add the following configuration:

       cinder:
         template:
           customServiceConfig: |
             [DEFAULT]
             storage_availability_zone = az1,az2,az3
           databaseInstance: openstack
           secret: osp-secret
           cinderAPI:
             replicas: 3
             override:
               service:
                 internal:
                   metadata:
                     annotations:
                       metallb.universe.tf/address-pool: internalapi
                       metallb.universe.tf/allow-shared-ip: internalapi
                       metallb.universe.tf/loadBalancerIPs: 172.17.0.80
                   spec:
                     type: LoadBalancer
           cinderScheduler:
             replicas: 1
           cinderBackup:
             customServiceConfig: |
               [DEFAULT]
               backup_driver = cinder.backup.drivers.ceph.CephBackupDriver
               backup_ceph_conf = /etc/ceph/az1.conf
               backup_ceph_pool = backups
               backup_ceph_user = openstack
             networkAttachments:
             - storage
             replicas: 1
             topologyRef:
               name: zone1-node-affinity
           cinderVolumes:
             az1:
               customServiceConfig: |
                 [DEFAULT]
                 enabled_backends = ceph
                 glance_api_servers = https://glance-az1-internal.openstack.svc:9292
                 [ceph]
                 volume_backend_name = ceph
                 volume_driver = cinder.volume.drivers.rbd.RBDDriver
                 rbd_ceph_conf = /etc/ceph/az1.conf
                 rbd_user = openstack
                 rbd_pool = volumes
                 rbd_flatten_volume_from_snapshot = False
                 rbd_secret_uuid = 7fd06e2a-a3e4-49fa-980c-1ed0865d2886
                 rbd_cluster_name = az1
                 backend_availability_zone = az1
               topologyRef:
                 name: zone1-node-affinity
               networkAttachments:
               - storage
               replicas: 1
             az2:
               customServiceConfig: |
                 [DEFAULT]
                 enabled_backends = ceph
                 glance_api_servers = https://glance-az2-internal.openstack.svc:9292
                 [ceph]
                 volume_backend_name = ceph
                 volume_driver = cinder.volume.drivers.rbd.RBDDriver
                 rbd_ceph_conf = /etc/ceph/az2.conf
                 rbd_user = openstack
                 rbd_pool = volumes
                 rbd_flatten_volume_from_snapshot = False
                 rbd_secret_uuid = d4596afb-43f2-4e5d-a204-bc38a3485dc3
                 rbd_cluster_name = az2
                 backend_availability_zone = az2
               topologyRef:
                 name: zone2-node-affinity
               networkAttachments:
               - storage
               replicas: 1
             az3:
               customServiceConfig: |
                 [DEFAULT]
                 enabled_backends = ceph
                 glance_api_servers = https://glance-az3-internal.openstack.svc:9292
                 [ceph]
                 volume_backend_name = ceph
                 volume_driver = cinder.volume.drivers.rbd.RBDDriver
                 rbd_ceph_conf = /etc/ceph/az3.conf
                 rbd_user = openstack
                 rbd_pool = volumes
                 rbd_flatten_volume_from_snapshot = False
                 rbd_secret_uuid = 1c348616-c493-4369-91f2-a55e4f404fbe
                 rbd_cluster_name = az3
                 backend_availability_zone = az3
               topologyRef:
                 name: zone3-node-affinity
               networkAttachments:
               - storage
               replicas: 1

   • If you are using the Block Storage service with third-party storage, add the following configuration. In this example, you configure cinderVolumes in three availability zones (AZs) with NetApp:

     cinder:
       apiOverride:
         route:
           haproxy.router.openshift.io/timeout: 60s
       template:
         customServiceConfig: |
           [DEFAULT]
           storage_availability_zone = az1
         cinderAPI:
           override:
             service:
               internal:
                 metadata:
                   annotations:
                     metallb.universe.tf/address-pool: internalapi
                     metallb.universe.tf/allow-shared-ip: internalapi
                     metallb.universe.tf/loadBalancerIPs: 172.17.0.80
                 spec:
                   type: LoadBalancer
           replicas: 3
         cinderBackup:
           customServiceConfig: |
             [DEFAULT]
             # TODO:
           networkAttachments:
           - storage
           replicas: 0
           topologyRef:
             name: azone-node-affinity
         cinderScheduler:
           replicas: 1
         cinderVolumes:
           ontap-iscsi-az1:
             customServiceConfig: |
               [DEFAULT]
               glance_api_servers = https://glance-az1-internal.openstack.svc:9292
               [ontap-az1]
               backend_availability_zone = az1
               volume_backend_name=ontap-az1
               volume_driver=cinder.volume.drivers.netapp.common.NetAppDriver
               netapp_server_hostname=10.0.0.5
               netapp_server_port=80
               netapp_storage_protocol=iscsi
               netapp_storage_family=ontap_cluster
               consistencygroup_support=True
             customServiceConfigSecrets:
             - cinder-volume-secrets-az1
             topologyRef:
               name: azone-node-affinity
           ontap-iscsi-az2:
             customServiceConfig: |
               [DEFAULT]
               glance_api_servers = https://glance-az2-internal.openstack.svc:9292
               [ontap-az2]
               backend_availability_zone = az2
               volume_backend_name=ontap-az2
               volume_driver=cinder.volume.drivers.netapp.common.NetAppDriver
               netapp_server_hostname=10.0.0.6
               netapp_server_port=80
               netapp_storage_protocol=iscsi
               netapp_storage_family=ontap_cluster
               consistencygroup_support=True
             customServiceConfigSecrets:
             - cinder-volume-secrets-az2
             topologyRef:
               name: bzone-node-affinity
           ontap-iscsi-az3:
             customServiceConfig: |
               [DEFAULT]
               glance_api_servers = https://glance-az3-internal.openstack.svc:9292
               [ontap-az3]
               backend_availability_zone = az3
               volume_backend_name=ontap-az3
               volume_driver=cinder.volume.drivers.netapp.common.NetAppDriver
               netapp_server_hostname=10.0.0.7
               netapp_server_port=80
               netapp_storage_protocol=iscsi
               netapp_storage_family=ontap_cluster
               consistencygroup_support=True
             customServiceConfigSecrets:
             - cinder-volume-secrets-az3
             topologyRef:
               name: czone-node-affinity
         databaseInstance: openstack
         preserveJobs: false
         secret: osp-secret

   • In this example, the Block Storage backup service under cinderBackup is disabled. If you require the Block Storage backup service:

     • Increase the replica count.
     • Set the topologyRef to run the backup service in the relevant AZ.
     • Set the customServiceConfig for the appropriate back end as described in Storage back ends for backups in Configuring persistent storage.

7. Configure the Compute service (nova):

     nova:
       template:
         apiServiceTemplate:
           replicas: 3
           override:
             service:
               internal:
                 metadata:
                   annotations:
                     metallb.universe.tf/address-pool: internalapi
                     metallb.universe.tf/allow-shared-ip: internalapi
                     metallb.universe.tf/loadBalancerIPs: 172.17.0.80
                 spec:
                   type: LoadBalancer
         metadataServiceTemplate:
           replicas: 3
           override:
             service:
               metadata:
                 annotations:
                   metallb.universe.tf/address-pool: internalapi
                   metallb.universe.tf/allow-shared-ip: internalapi
                   metallb.universe.tf/loadBalancerIPs: 172.17.0.80
               spec:
                 type: LoadBalancer
         schedulerServiceTemplate:
           replicas: 3
         cellTemplates:
           cell0:
             cellDatabaseAccount: nova-cell0
             cellDatabaseInstance: openstack
             cellMessageBusInstance: rabbitmq
             hasAPIAccess: true
           cell1:
             cellDatabaseAccount: nova-cell1
             cellDatabaseInstance: openstack-cell1
             cellMessageBusInstance: rabbitmq-cell1
             conductorServiceTemplate:
               replicas: 1
             noVNCProxyServiceTemplate:
               enabled: true
               networkAttachments:
               - ctlplane
             hasAPIAccess: true
         secret: osp-secret

8. Configure the DNS service for the data plane:

     dns:
       template:
         options:
         - key: server
           values:
           - <IP address for DNS server reachable from dnsmasq pod>
         override:
           service:
             metadata:
               annotations:
                 metallb.universe.tf/address-pool: ctlplane
                 metallb.universe.tf/allow-shared-ip: ctlplane
                 metallb.universe.tf/loadBalancerIPs: 192.168.122.80
             spec:
               type: LoadBalancer
         replicas: 2

   • options: Defines the dnsmasq instances required for each DNS server by using key-value pairs. In this example, there is one key-value pair defined because there is only one DNS server configured to forward requests to.

   • key: Specifies the dnsmasq parameter to customize for the deployed dnsmasq instance. Set to one of the following valid values:

     • server
     • rev-server
     • srv-host
     • txt-record
     • ptr-record
     • rebind-domain-ok
     • naptr-record
     • cname
     • host-record
     • caa-record
     • dns-rr
     • auth-zone
     • synth-domain
     • no-negcache
     • local

   • values: Specifies the value for the DNS server reachable from the dnsmasq pod on the RHOCP cluster network. You can specify a generic DNS server as the value, for example, 1.1.1.1, or a DNS server for a specific domain, for example, /google.com/8.8.8.8.

     Note

     This DNS service, dnsmasq, provides DNS services for nodes on the RHOSO data plane. dnsmasq is different from the RHOSO DNS service (designate) that provides DNS as a service for cloud tenants.

9. Configure the Identity service (keystone):

     keystone:
       template:
         override:
           service:
             internal:
               metadata:
                 annotations:
                   metallb.universe.tf/address-pool: internalapi
                   metallb.universe.tf/allow-shared-ip: internalapi
                   metallb.universe.tf/loadBalancerIPs: 172.17.0.80
               spec:
                 type: LoadBalancer
         databaseInstance: openstack
         secret: osp-secret
         replicas: 3

10. Configure the Image service (glance).

    • If you are using the Image service with Red Hat Ceph Storage, configure the following:

        glance:
          template:
            databaseInstance: openstack
            glanceAPIs:
              az1:
                customServiceConfig: |
                  [DEFAULT]
                  enabled_import_methods = [web-download,copy-image,glance-direct]
                  enabled_backends = az1:rbd
                  [glance_store]
                  default_backend = az1
                  [az1]
                  rbd_store_ceph_conf = /etc/ceph/az1.conf
                  store_description = "az1 RBD backend"
                  rbd_store_pool = images
                  rbd_store_user = openstack
                  rbd_thin_provisioning = True
                networkAttachments:
                - storage
                override:
                  service:
                    internal:
                      metadata:
                        annotations:
                          metallb.universe.tf/address-pool: internalapi
                          metallb.universe.tf/allow-shared-ip: internalapi
                          metallb.universe.tf/loadBalancerIPs: 172.17.0.81
                      spec:
                        type: LoadBalancer
                replicas: 2
                topologyRef:
                  name: zone1-node-affinity
                type: edge
              az2:
                customServiceConfig: |
                  [DEFAULT]
                  enabled_import_methods = [web-download,copy-image,glance-direct]
                  enabled_backends = az1:rbd,az2:rbd
                  [glance_store]
                  default_backend = az2
                  [az2]
                  rbd_store_ceph_conf = /etc/ceph/az2.conf
                  store_description = "az2 RBD backend"
                  rbd_store_pool = images
                  rbd_store_user = openstack
                  rbd_thin_provisioning = True
                  [az1]
                  rbd_store_ceph_conf = /etc/ceph/az1.conf
                  store_description = "az1 RBD backend"
                  rbd_store_pool = images
                  rbd_store_user = openstack
                  rbd_thin_provisioning = True
                networkAttachments:
                - storage
                override:
                  service:
                    internal:
                      metadata:
                        annotations:
                          metallb.universe.tf/address-pool: internalapi
                          metallb.universe.tf/allow-shared-ip: internalapi
                          metallb.universe.tf/loadBalancerIPs: 172.17.0.82
                      spec:
                        type: LoadBalancer
                replicas: 2
                topologyRef:
                  name: zone2-node-affinity
                type: edge
              az3:
                customServiceConfig: |
                  [DEFAULT]
                  enabled_import_methods = [web-download,copy-image,glance-direct]
                  enabled_backends = az1:rbd,az3:rbd
                  [glance_store]
                  default_backend = az3
                  [az3]
                  rbd_store_ceph_conf = /etc/ceph/az3.conf
                  store_description = "az3 RBD backend"
                  rbd_store_pool = images
                  rbd_store_user = openstack
                  rbd_thin_provisioning = True
                  [az1]
                  rbd_store_ceph_conf = /etc/ceph/az1.conf
                  store_description = "az1 RBD backend"
                  rbd_store_pool = images
                  rbd_store_user = openstack
                  rbd_thin_provisioning = True
                networkAttachments:
                - storage
                override:
                  service:
                    internal:
                      metadata:
                        annotations:
                          metallb.universe.tf/address-pool: internalapi
                          metallb.universe.tf/allow-shared-ip: internalapi
                          metallb.universe.tf/loadBalancerIPs: 172.17.0.83
                      spec:
                        type: LoadBalancer
                replicas: 2
                topologyRef:
                  name: zone3-node-affinity
                type: edge
              default:
                customServiceConfig: |
                  [DEFAULT]
                  enabled_import_methods = [web-download,copy-image,glance-direct]
                  enabled_backends = az1:rbd,az2:rbd,az3:rbd
                  [glance_store]
                  default_backend = az1
                  [az1]
                  rbd_store_ceph_conf = /etc/ceph/az1.conf
                  store_description = "az1 RBD backend"
                  rbd_store_pool = images
                  rbd_store_user = openstack
                  rbd_thin_provisioning = True
                  [az2]
                  rbd_store_ceph_conf = /etc/ceph/az2.conf
                  store_description = "az2 RBD backend"
                  rbd_store_pool = images
                  rbd_store_user = openstack
                  rbd_thin_provisioning = True
                  [az3]
                  rbd_store_ceph_conf = /etc/ceph/az3.conf
                  store_description = "az3 RBD backend"
                  rbd_store_pool = images
                  rbd_store_user = openstack
                  rbd_thin_provisioning = True
                networkAttachments:
                - storage
                override:
                  service:
                    internal:
                      metadata:
                        annotations:
                          metallb.universe.tf/address-pool: internalapi
                          metallb.universe.tf/allow-shared-ip: internalapi
                          metallb.universe.tf/loadBalancerIPs: 172.17.0.80
                      spec:
                        type: LoadBalancer
                replicas: 3
                type: split
            keystoneEndpoint: default
            storage:
              storageClass: local-storage
              storageRequest: 10G
          uniquePodNames: true

    • If you are using the Image service with third-party storage, configure the following. In this example, you have three AZs and you are using the Block Storage service as a back end for the Image service. You create a separate glanceAPI server set for each AZ, which uses multistore to write to more than one Block Storage service back end. If you configure multistore to use the Block Storage service as its back end, refer to step 25 when you complete the procedure to create the distributed control plane because you must create volume types for cinder-volume services when the control plane is created.

      This example uses Block Storage service iSCSI to store Image service images, which can support multipath, and multipath options such as cinder_use_multipath and cinder_do_extend_attached are enabled. If the storage protocol you are using or your environment does not support multipath, then do not use these options.

            glanceAPIs:
              az1:
                customServiceConfig: |
                  [DEFAULT]
                  enabled_backends = az1:cinder
                  enabled_import_methods = [web-download,copy-image,glance-direct]
                  [glance_store]
                  default_backend = az1
                  [az1]
                  store_description = AZ1 NetApp iscsi cinder backend
                  cinder_store_auth_address = {{ .KeystoneInternalURL }}
                  cinder_store_user_name = {{ .ServiceUser }}
                  cinder_store_password = {{ .ServicePassword }}
                  cinder_store_project_name = service
                  cinder_catalog_info = volumev3::internalURL
                  cinder_use_multipath = true
                  cinder_do_extend_attached = true
                  cinder_volume_type = glance-ontap-az1
                networkAttachments:
                - storage
                override:
                  service:
                    internal:
                      metadata:
                        annotations:
                          metallb.universe.tf/address-pool: internalapi
                          metallb.universe.tf/allow-shared-ip: internalapi
                          metallb.universe.tf/loadBalancerIPs: 172.17.0.81
                      spec:
                        type: LoadBalancer
                replicas: 2
                topologyRef:
                  name: azone-node-affinity
                type: edge
              az2:
                customServiceConfig: |
                  [DEFAULT]
                  enabled_backends = az1:cinder,az2:cinder
                  [glance_store]
                  default_backend = az2
                  [az2]
                  store_description = AZ2 NetApp iscsi cinder backend
                  cinder_store_auth_address = {{ .KeystoneInternalURL }}
                  cinder_store_user_name = {{ .ServiceUser }}
                  cinder_store_password = {{ .ServicePassword }}
                  cinder_store_project_name = service
                  cinder_catalog_info = volumev3::internalURL
                  cinder_use_multipath = true
                  cinder_do_extend_attached = true
                  cinder_volume_type = glance-ontap-az2
                  [az1]
                  store_description = AZ1 NetApp iscsi cinder backend
                  cinder_store_auth_address = {{ .KeystoneInternalURL }}
                  cinder_store_user_name = {{ .ServiceUser }}
                  cinder_store_password = {{ .ServicePassword }}
                  cinder_store_project_name = service
                  cinder_catalog_info = volumev3::internalURL
                  cinder_use_multipath = true
                  cinder_do_extend_attached = true
                  cinder_volume_type = glance-ontap-az1
                networkAttachments:
                - storage
                override:
                  service:
                    internal:
                      metadata:
                        annotations:
                          metallb.universe.tf/address-pool: internalapi
                          metallb.universe.tf/allow-shared-ip: internalapi
                          metallb.universe.tf/loadBalancerIPs: 172.17.0.82
                      spec:
                        type: LoadBalancer
                replicas: 2
                topologyRef:
                  name: bzone-node-affinity
                type: edge
              az3:
                customServiceConfig: |
                  [DEFAULT]
                  enabled_backends = az1:cinder,az3:cinder
                  [glance_store]
                  default_backend = az3
                  [az3]
                  store_description = AZ3 NetApp iscsi cinder backend
                  cinder_store_auth_address = {{ .KeystoneInternalURL }}
                  cinder_store_user_name = {{ .ServiceUser }}
                  cinder_store_password = {{ .ServicePassword }}
                  cinder_store_project_name = service
                  cinder_catalog_info = volumev3::internalURL
                  cinder_use_multipath = true
                  cinder_do_extend_attached = true
                  cinder_volume_type = glance-ontap-az3
                  [az1]
                  store_description = AZ1 NetApp iscsi cinder backend
                  cinder_store_auth_address = {{ .KeystoneInternalURL }}
                  cinder_store_user_name = {{ .ServiceUser }}
                  cinder_store_password = {{ .ServicePassword }}
                  cinder_store_project_name = service
                  cinder_catalog_info = volumev3::internalURL
                  cinder_use_multipath = true
                  cinder_do_extend_attached = true
                  cinder_volume_type = glance-ontap-az1
                networkAttachments:
                - storage
                override:
                  service:
                    internal:
                      metadata:
                        annotations:
                          metallb.universe.tf/address-pool: internalapi
                          metallb.universe.tf/allow-shared-ip: internalapi
                          metallb.universe.tf/loadBalancerIPs: 172.17.0.83
                      spec:
                        type: LoadBalancer
                replicas: 2
                topologyRef:
                  name: czone-node-affinity
                type: edge
              default:
                customServiceConfig: |
                  [DEFAULT]
                  enabled_backends = az1:cinder,az2:cinder,az3:cinder
                  [glance_store]
                  default_backend = az1
                  [az1]
                  store_description = AZ1 NetApp iscsi cinder backend
                  cinder_store_auth_address = {{ .KeystoneInternalURL }}
                  cinder_store_user_name = {{ .ServiceUser }}
                  cinder_store_password = {{ .ServicePassword }}
                  cinder_store_project_name = service
                  cinder_catalog_info = volumev3::internalURL
                  cinder_use_multipath = true
                  cinder_do_extend_attached = true
                  cinder_volume_type = glance-ontap-az1
                  [az2]
                  store_description = AZ2 NetApp iscsi cinder backend
                  cinder_store_auth_address = {{ .KeystoneInternalURL }}
                  cinder_store_user_name = {{ .ServiceUser }}
                  cinder_store_password = {{ .ServicePassword }}
                  cinder_store_project_name = service
                  cinder_catalog_info = volumev3::internalURL
                  cinder_use_multipath = true
                  cinder_do_extend_attached = true
                  cinder_volume_type = glance-ontap-az2
                  [az3]
                  store_description = AZ3 NetApp iscsi cinder backend
                  cinder_store_auth_address = {{ .KeystoneInternalURL }}
                  cinder_store_user_name = {{ .ServiceUser }}
                  cinder_store_password = {{ .ServicePassword }}
                  cinder_store_project_name = service
                  cinder_catalog_info = volumev3::internalURL
                  cinder_use_multipath = true
                  cinder_do_extend_attached = true
                  cinder_volume_type = glance-ontap-az3

11. Configure the Key Management service (barbican):

      barbican:
        template:
          databaseInstance: openstack
          secret: osp-secret
          barbicanAPI:
            replicas: 3
            override:
              service:
                internal:
                  metadata:
                    annotations:
                      metallb.universe.tf/address-pool: internalapi
                      metallb.universe.tf/allow-shared-ip: internalapi
                      metallb.universe.tf/loadBalancerIPs: 172.17.0.80
                  spec:
                    type: LoadBalancer
          barbicanWorker:
            replicas: 3
          barbicanKeystoneListener:
            replicas: 1

12. Configure the Networking service (neutron):

      neutron:
        template:
          customServiceConfig: |
            [DEFAULT]
            vlan_transparent = true
            debug = true
            [ovs]
            igmp_snooping_enable = true
          databaseInstance: openstack
          networkAttachments:
          - internalapi
          override:
            service:
              internal:
                metadata:
                  annotations:
                    metallb.universe.tf/address-pool: internalapi
                    metallb.universe.tf/allow-shared-ip: internalapi
                    metallb.universe.tf/loadBalancerIPs: 172.17.0.80
                spec:
                  type: LoadBalancer
          replicas: 3
          secret: osp-secret

13. Ensure that the Object Storage service (swift) is disabled:

      swift:
        enabled: false

14. Configure OVN:

      ovn:
        template:
          ovnController:
            external-ids:
              enable-chassis-as-gateway: false
            networkAttachment: tenant
            nicMappings:
              datacentre: ospbr
          ovnDBCluster:
            ovndbcluster-nb:
              dbType: NB
              networkAttachment: internalapi
              replicas: 3
              storageRequest: 10G
            ovndbcluster-sb:
              dbType: SB
              networkAttachment: internalapi
              replicas: 3
              storageRequest: 10G
          ovnNorthd:
            networkAttachment: internalapi

15. Configure the Placement service (placement):

      placement:
        template:
          override:
            service:
              internal:
                metadata:
                  annotations:
                    metallb.universe.tf/address-pool: internalapi
                    metallb.universe.tf/allow-shared-ip: internalapi
                    metallb.universe.tf/loadBalancerIPs: 172.17.0.80
                spec:
                  type: LoadBalancer
          databaseInstance: openstack
          replicas: 3
          secret: osp-secret

16. Configure the Telemetry service (ceilometer, prometheus):

      telemetry:
        enabled: true
        template:
          metricStorage:
            enabled: true
            dashboardsEnabled: true
            dataplaneNetwork: ctlplane
            networkAttachments:
              - ctlplane
            monitoringStack:
              alertingEnabled: true
              scrapeInterval: 30s
              storage:
                strategy: persistent
                retention: 24h
                persistent:
                  pvcStorageRequest: 20G
          autoscaling:
            enabled: false
            aodh:
              databaseAccount: aodh
              databaseInstance: openstack
              passwordSelector:
                aodhService: AodhPassword
              rabbitMqClusterName: rabbitmq
              serviceUser: aodh
              secret: osp-secret
            heatInstance: heat
          ceilometer:
            template:
              passwordSelector:
                service: CeilometerPassword
              secret: osp-secret
              serviceUser: ceilometer
          logging:
            enabled: false

    • metricStorage.dataplaneNetwork: Defines the network that you use to scrape dataplane node_exporter endpoints.
    • metricStorage.networkAttachments: Lists the networks that each service pod is attached to. Specify by using the NetworkAttachmentDefinition resource names. You configure a NIC for the service for each network attachment that you specify. If you do not configure the isolated networks that each service pod is attached to, then the default pod network is used. You must create a networkAttachment that matches the network that you specify as the dataplaneNetwork, so that Prometheus can scrape data from the dataplane nodes.
    • autoscaling: You must have the autoscaling field present, even if autoscaling is disabled. For more information about autoscaling, see Autoscaling for Instances.

17. Configure the Shared File Systems service (manila).

    • If you are using the Shared File Systems service with Red Hat Ceph Storage, add the following configuration:

        manila:
          enabled: true
          template:
            manilaAPI:
              customServiceConfig: |
                [DEFAULT]
                enabled_share_protocols=nfs,cephfs
              networkAttachments:
              - internalapi
              override:
                service:
                  internal:
                    metadata:
                      annotations:
                        metallb.universe.tf/address-pool: internalapi
                        metallb.universe.tf/allow-shared-ip: internalapi
                        metallb.universe.tf/loadBalancerIPs: 172.17.0.80
                    spec:
                      type: LoadBalancer
              replicas: 3
            manilaScheduler:
              replicas: 3
            manilaShares:
              az1:
                customServiceConfig: |
                  [DEFAULT]
                  enabled_share_backends = cephfs_az1
                  enabled_share_protocols = cephfs
                  [cephfs_az1]
                  driver_handles_share_servers = False
                  share_backend_name = cephfs_az1
                  share_driver = manila.share.drivers.cephfs.driver.CephFSDriver
                  cephfs_conf_path = /etc/ceph/az1.conf
                  cephfs_cluster_name = az1
                  cephfs_auth_id=openstack
                  cephfs_volume_mode = 0755
                  cephfs_protocol_helper_type = CEPHFS
                  backend_availability_zone = az1
                networkAttachments:
                - storage
                replicas: 1
                topologyRef:
                  name: zone1-node-affinity
              az2:
                customServiceConfig: |
                  [DEFAULT]
                  enabled_share_backends = cephfs_az2
                  enabled_share_protocols = cephfs
                  [cephfs_az2]
                  driver_handles_share_servers = False
                  share_backend_name = cephfs_az2
                  share_driver = manila.share.drivers.cephfs.driver.CephFSDriver
                  cephfs_conf_path = /etc/ceph/az2.conf
                  cephfs_cluster_name = az2
                  cephfs_auth_id=openstack
                  cephfs_volume_mode = 0755
                  cephfs_protocol_helper_type = CEPHFS
                  backend_availability_zone = az2
                networkAttachments:
                - storage
                replicas: 1
                topologyRef:
                  name: zone2-node-affinity
              az3:
                customServiceConfig: |
                  [DEFAULT]
                  enabled_share_backends = cephfs_az3
                  enabled_share_protocols = cephfs
                  [cephfs_az3]
                  driver_handles_share_servers = False
                  share_backend_name = cephfs_az3
                  share_driver = manila.share.drivers.cephfs.driver.CephFSDriver
                  cephfs_conf_path = /etc/ceph/az3.conf
                  cephfs_cluster_name = az3
                  cephfs_auth_id=openstack
                  cephfs_volume_mode = 0755
                  cephfs_protocol_helper_type = CEPHFS
                  backend_availability_zone = az3
                networkAttachments:
                - storage
                replicas: 1
                topologyRef:
                  name: zone3-node-affinity

    • If you are using the Shared File Systems service with third-party storage, configure the following. In this example, you configure the Shared File Systems service in three AZs with NetApp.

      In the following example, driver_handles_share_servers is set to True. If you do not have this option, then set the driver_handles_share_servers field to False. For more information, see Verifying the distributed control plane and Configuring the Shared File Systems service (manila) in Configuring persistent storage.

        manila:
          apiOverride:
            route:
              haproxy.router.openshift.io/timeout: 60s
          enabled: true
          template:
            manilaAPI:
              customServiceConfig: |
                [DEFAULT]
                storage_availability_zone = az1,az2,az3
                default_share_type = nfs-multiaz
                enabled_share_protocols=nfs
                debug = true
              networkAttachments:
              - internalapi
              override:
                service:
                  internal:
                    metadata:
                      annotations:
                        metallb.universe.tf/address-pool: internalapi
                        metallb.universe.tf/allow-shared-ip: internalapi
                        metallb.universe.tf/loadBalancerIPs: 172.17.0.80
                    spec:
                      type: LoadBalancer
              replicas: 3
            manilaScheduler:
              replicas: 3
            manilaShares:
              az1:
                customServiceConfig: |
                  [DEFAULT]
                  enabled_share_backends = nfs_az1
                  enabled_share_protocols = nfs
                  [nfs_az1]
                  driver_handles_share_servers = True
                  share_backend_name = nfs_az
                  backend_availability_zone = az1
                  share_driver=manila.share.drivers.netapp.common.NetAppDriver
                  netapp_storage_family=ontap_cluster
                  netapp_transport_type=http
                customServiceConfigSecrets:
                - osp-secret-manila-az1
                networkAttachments:
                - storage
                replicas: 1
                topologyRef:
                  name: azone-node-affinity
              az2:
                customServiceConfig: |
                  [DEFAULT]
                  enabled_share_backends = nfs_az2
                  enabled_share_protocols = nfs
                  [nfs_az2]
                  driver_handles_share_servers = True
                  share_backend_name = nfs_az
                  backend_availability_zone = az2
                  share_driver=manila.share.drivers.netapp.common.NetAppDriver
                  netapp_storage_family=ontap_cluster
                  netapp_transport_type=http
                customServiceConfigSecrets:
                - osp-secret-manila-az2
                networkAttachments:
                - storage
                replicas: 1
                topologyRef:
                  name: bzone-node-affinity
              az3:
                customServiceConfig: |
                  [DEFAULT]
                  enabled_share_backends = nfs_az3
                  enabled_share_protocols = nfs
                  [nfs_az3]
                  driver_handles_share_servers = True
                  share_backend_name = nfs_az
                  backend_availability_zone = az3
                  share_driver=manila.share.drivers.netapp.common.NetAppDriver
                  netapp_storage_family=ontap_cluster
                  netapp_transport_type=http
                customServiceConfigSecrets:
                - osp-secret-manila-az3
                networkAttachments:
                - storage
                replicas: 1
                topologyRef:
                  name: czone-node-affinity
            preserveJobs: false

18. Disable the Load-balancing service (octavia) because it is not currently supported when enable-chassis-as-gateway is disabled in the OVN service. For more information, see OSPRH-10766.

      octavia:
        enabled: false

19. Configure the Orchestration service (heat):

      heat:
        cnfAPIOverride:
          route: {}
        enabled: true
        template:
          databaseInstance: openstack
          heatAPI:
            override:
              service:
                internal:
                  metadata:
                    annotations:
                      metallb.universe.tf/address-pool: internalapi
                      metallb.universe.tf/allow-shared-ip: internalapi
                      metallb.universe.tf/loadBalancerIPs: 172.17.0.80
                  spec:
                    type: LoadBalancer
            replicas: 3
          heatEngine:
            override:
              service:
                internal:
                  metadata:
                    annotations:
                      metallb.universe.tf/address-pool: internalapi
                      metallb.universe.tf/allow-shared-ip: internalapi
                      metallb.universe.tf/loadBalancerIPs: 172.17.0.80
                  spec:
                    type: LoadBalancer
            replicas: 3
          secret: osp-secret

20. Configure the Dashboard service (horizon):

      horizon:
        enabled: true
        template:
          replicas: 2
          secret: osp-secret

21. Add the following service configurations to implement high availability (HA):

    • A MariaDB Galera cluster for use by all RHOSO services (openstack), and a MariaDB Galera cluster for use by the Compute service for cell1 (openstack-cell1):

        galera:
          enabled: true
          templates:
            openstack:
              storageRequest: 5000M
              secret: osp-secret
              replicas: 3
            openstack-cell1:
              storageRequest: 5000M
              secret: osp-secret
              replicas: 3

    • A single memcached cluster that contains three memcached servers:

        memcached:
          templates:
            memcached:
              replicas: 3

    • A RabbitMQ cluster for use by all RHOSO services (rabbitmq), and a RabbitMQ cluster for use by the Compute service for cell1 (rabbitmq-cell1):

        rabbitmq:
          templates:
            rabbitmq:
              replicas: 3
              override:
                service:
                  metadata:
                    annotations:
                      metallb.universe.tf/address-pool: internalapi
                      metallb.universe.tf/loadBalancerIPs: 172.17.0.85
                  spec:
                    type: LoadBalancer
            rabbitmq-cell1:
              replicas: 3
              override:
                service:
                  metadata:
                    annotations:
                      metallb.universe.tf/address-pool: internalapi
                      metallb.universe.tf/loadBalancerIPs: 172.17.0.86
                  spec:
                    type: LoadBalancer

      Note

      You cannot configure multiple RabbitMQ instances on the same virtual IP (VIP) address because all RabbitMQ instances use the same port. If you need to expose multiple RabbitMQ instances to the same network, then you must use distinct IP addresses.

22. Create the control plane:

    $ oc create -f distributed_control_plane.yaml -n openstack

23. Wait until RHOCP creates the resources related to the OpenStackControlPlane CR. Run the following command to check the status:

    $ oc get openstackcontrolplane -n openstack
    NAME 						STATUS 	MESSAGE
    distributed-control-plane 	Unknown 	Setup started

    The OpenStackControlPlane resources are created when the status is "Setup complete".

    Tip

    Append the -w option to the end of the get command to track deployment progress.

    Note

    Creating the control plane also creates an OpenStackClient pod that you can access through a remote shell (rsh) to run OpenStack CLI commands.

    $ oc rsh -n openstack openstackclient

24. Optional: Confirm that the control plane is deployed by reviewing the pods in the openstack namespace:

    $ oc get pods -n openstack

    The control plane is deployed when all the pods are either completed or running.

25. If you are using the Image service with third-party storage, and you configure multistore to use the Block Storage service as its back end, each glance store specifies a unique cinder_volume_type. You create the volume types with the --private option to prevent them from being exposed to cloud users. To allow the Image service to access these volume types, you set the --project for each type to the service project ID.

    1. Open a remote shell connection to the OpenStackClient pod:

       $ oc rsh -n openstack openstackclient

    2. Identify the service project ID:

       $ sh-5.1$ openstack project list
       +----------------------------------+---------+
       | ID                               | Name    |
       +----------------------------------+---------+
       | 439f0ee839144b4c8640b9153a596a30 | admin   |
       | 7a8946c6ec7c4d0488c592f0306eaa35 | service |
       +----------------------------------+---------+
       $ sh-5.1$

    3. Store the service user project ID for the Image service in a shell variable:

       $ SERVICE_PROJECT_ID=$(openstack project show service -c id -f value)

    4. Create a type for each of the three cinder-volume services in each AZ. Set its project ID and availability zone. Use the spec key RESKEY:availability_zones to pass the AZ to the Block Storage service even though the Image service only passes the type when it requests a volume:

       $ openstack volume type create --private --project "$SERVICE_PROJECT_ID" --property "RESKEY:availability_zones=az1" glance-ontap-az1
       $ openstack volume type create --private --project "$SERVICE_PROJECT_ID" --property "RESKEY:availability_zones=az2" glance-ontap-az2
       $ openstack volume type create --private --project "$SERVICE_PROJECT_ID" --property "RESKEY:availability_zones=az3" glance-ontap-az3

    5. Exit the openstackclient pod:

       $ exit

Procedure

1. Verify that the service pods are running in the correct AZs based on their worker node:

   $ oc get pods -o wide -l service=<service_name>

   • Replace <service_name> with cinder, glance, or manila to verify the service pods for the specific service.

     For example, if the worker-6 node is located in AZ3, then the output from the following command confirms that the cinder-dc29b-volume-az3-0 pod is running correctly in AZ3:

     $ oc get pods -o wide -l service=cinder
     NAME                         	READY   STATUS  	RESTARTS    	AGE 	IP           	NODE   	NOMINATED NODE   READINESS GATES
     cinder-dc29b-api-0           	2/2 	Running 	0           	4m24s     192.172.41.120   worker-7   <none>       	<none>
     cinder-dc29b-api-1           	2/2 	Running 	0           	4m48s     192.172.28.60    worker-5   <none>       	<none>
     cinder-dc29b-api-2           	2/2 	Running 	0           	5m5s      192.172.24.237   worker-0   <none>       	<none>
     cinder-dc29b-backup-0        	2/2 	Running 	0           	5m9s      192.172.24.234   worker-0   <none>       	<none>
     cinder-dc29b-scheduler-0     	2/2 	Running 	0           	5m5s	192.172.41.118   worker-7   <none>       	<none>
     cinder-dc29b-volume-az1-0    	2/2 	Running 	0           	5m9s	192.172.32.150   worker-2   <none>       	<none>
     cinder-dc29b-volume-az2-0    	2/2 	Running 	0           	5m9s	192.172.16.33    worker-3   <none>       	<none>
     cinder-dc29b-volume-az3-0    	2/2 	Running 	0           	5m8s	192.172.21.170   worker-6   <none>       	<none>

2. Open a remote shell connection to the openstackclient pod:

   $ oc rsh -n openstack openstackclient

3. Confirm that the internal service endpoints are registered with each service:

   $ openstack endpoint list -c 'Service Name' -c Interface -c URL --service glance
   +--------------+-----------+---------------------------------------------------------------+
   | Service Name | Interface | URL                                                           |
   +--------------+-----------+---------------------------------------------------------------+
   | glance       | internal  | https://glance-internal.openstack.svc                     |
   | glance       | public    | https://glance-default-public-openstack.apps.ostest.test.metalkube.org |
   +--------------+-----------+---------------------------------------------------------------+

4. If you are using the Block Storage service (cinder) with Red Hat Ceph Storage, confirm the cinder-volume service is running in each availability zone (AZ):

   $ openstack volume service list
   +------------------+--------------------------------+------+---------+-------+----------------------------+
   | Binary       	| Host                       	| Zone | Status  | State | Updated At             	|
   +------------------+--------------------------------+------+---------+-------+----------------------------+
   | cinder-scheduler | cinder-dc29b-scheduler-0   	| az1  | enabled | up	| 2025-05-21T14:07:11.000000 |
   | cinder-backup	| cinder-dc29b-backup-0      	| az1  | enabled | up	| 2025-05-21T14:07:18.000000 |
   | cinder-volume	| cinder-dc29b-volume-az1-0@ceph | az1  | enabled | up	| 2025-05-21T14:07:13.000000 |
   | cinder-volume	| cinder-dc29b-volume-az2-0@ceph | az2  | enabled | up	| 2025-05-21T14:07:16.000000 |
   | cinder-volume	| cinder-dc29b-volume-az3-0@ceph | az3  | enabled | up	| 2025-05-21T14:07:15.000000 |
   +------------------+--------------------------------+------+---------+-------+----------------------------+

5. If you are using the Block Storage service with third-party storage, verify the following.

   1. Confirm you can create volumes by AZ:

      $ openstack volume create --size 1 --availability-zone az1 vol_az1
      $ openstack volume create --size 1 --availability-zone az2 vol_az2
      $ openstack volume create --size 1 --availability-zone az3 vol_az3

   2. As an administrator, confirm you can create a volume in a specific AZ by passing only the type:

      $ openstack volume create --size 1 --type glance-ontap-az1 vol_az1_by_type
      $ openstack volume create --size 1 --type glance-ontap-az2 vol_az2_by_type
      $ openstack volume create --size 1 --type glance-ontap-az3 vol_az3_by_type

   3. Verify that volumes with each glance type were created by confirming they are available, and then you can delete them.

6. If you are using the Image service (glance) with third-party storage, verify the following to test image creation per AZ.

   1. Observe the available stores:

      $ glance stores-info
      +----------+----------------------------------------------------------------------------------+
      | Property | Value                                                                            |
      +----------+----------------------------------------------------------------------------------+
      | stores   | [{"id": "az1", "description": "AZ1 NetApp iscsi cinder backend", "default":      |
      |          | "true"}, {"id": "az2", "description": "AZ2 NetApp iscsi cinder backend"}, {"id": |
      |          | "az3", "description": "AZ3 NetApp iscsi cinder backend"}]                        |
      +----------+----------------------------------------------------------------------------------+

   2. Create an image without passing any store-related parameters to check the result:

      sh-5.1$ openstack image create --disk-format qcow2 --container-format bare --public --file ./cirros-0.5.2-x86_64-disk.img cirros-default
      +------------------+----------------------------------------------------------------------------------------------------------------------------------------------------+
      | Field            | Value                                                                                                                                              |
      +------------------+----------------------------------------------------------------------------------------------------------------------------------------------------+
      | container_format | bare                                                                                                                                               |
      | created_at       | 2025-06-18T22:55:21Z                                                                                                                               |
      | disk_format      | qcow2                                                                                                                                              |
      | file             | /v2/images/10fcfc90-e178-4776-ac76-853a7082844b/file                                                                                               |
      | id               | 10fcfc90-e178-4776-ac76-853a7082844b                                                                                                               |
      | min_disk         | 0                                                                                                                                                  |
      | min_ram          | 0                                                                                                                                                  |
      | name             | cirros-default                                                                                                                                     |
      | owner            | 439f0ee839144b4c8640b9153a596a30                                                                                                                   |
      | properties       | os_hidden='False', owner_specified.openstack.md5='', owner_specified.openstack.object='images/cirros-default', owner_specified.openstack.sha256='' |
      | protected        | False                                                                                                                                              |
      | schema           | /v2/schemas/image                                                                                                                                  |
      | status           | queued                                                                                                                                             |
      | tags             |                                                                                                                                                    |
      | updated_at       | 2025-06-18T22:55:21Z                                                                                                                               |
      | visibility       | public                                                                                                                                             |
      +------------------+----------------------------------------------------------------------------------------------------------------------------------------------------+
      sh-5.1$

      • The image is created in az1 because it is the default AZ:

        sh-5.1$ openstack image show 10fcfc90-e178-4776-ac76-853a7082844b | grep stores
        | properties       | os_hash_algo='sha512', os_hash_value='6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', os_hidden='False', owner_specified.openstack.md5='', owner_specified.openstack.object='images/cirros-default', owner_specified.openstack.sha256='', stores='az1' |
        sh-5.1$

   3. Create an image and pass a parameter so that it goes directly into store az1:

      $ glance image-create --disk-format raw --container-format bare --name cirros-test --file cirros-0.5.2-x86_64-disk.img --store az1
      +------------------+----------------------------------------------------------------------------------+
      | Property         | Value                                                                            |
      +------------------+----------------------------------------------------------------------------------+
      | checksum         | b874c39491a2377b8490f5f1e89761a4                                                 |
      | container_format | bare                                                                             |
      | created_at       | 2025-06-18T22:47:10Z                                                             |
      | disk_format      | raw                                                                              |
      | id               | 5073f3f0-e4ac-4cca-9883-fecade29a1f3                                             |
      | min_disk         | 0                                                                                |
      | min_ram          | 0                                                                                |
      | name             | cirros-test                                                                      |
      | os_hash_algo     | sha512                                                                           |
      | os_hash_value    | 6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe3622 |
      |                  | 5e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869                                 |
      | os_hidden        | False                                                                            |
      | owner            | 439f0ee839144b4c8640b9153a596a30                                                 |
      | protected        | False                                                                            |
      | size             | 16300544                                                                         |
      | status           | active                                                                           |
      | stores           | az1                                                                              |
      | tags             | []                                                                               |
      | updated_at       | 2025-06-18T22:48:00Z                                                             |
      | virtual_size     | 16300544                                                                         |
      | visibility       | shared                                                                           |
      +------------------+----------------------------------------------------------------------------------+

   4. Import the image to az2:

      $ glance image-import 5073f3f0-e4ac-4cca-9883-fecade29a1f3 --stores az2 --import-method copy-image
      +-----------------------+----------------------------------------------------------------------------------+
      | Property              | Value                                                                            |
      +-----------------------+----------------------------------------------------------------------------------+
      | checksum              | b874c39491a2377b8490f5f1e89761a4                                                 |
      | container_format      | bare                                                                             |
      | created_at            | 2025-06-18T22:47:10Z                                                             |
      | disk_format           | raw                                                                              |
      | id                    | 5073f3f0-e4ac-4cca-9883-fecade29a1f3                                             |
      | min_disk              | 0                                                                                |
      | min_ram               | 0                                                                                |
      | name                  | cirros-test                                                                      |
      | os_glance_import_task | 7a141f7f-409a-4ea9-b66f-0658a83e907b                                             |
      | os_hash_algo          | sha512                                                                           |
      | os_hash_value         | 6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe3622 |
      |                       | 5e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869                                 |
      | os_hidden             | False                                                                            |
      | owner                 | 439f0ee839144b4c8640b9153a596a30                                                 |
      | protected             | False                                                                            |
      | size                  | 16300544                                                                         |
      | status                | active                                                                           |
      | stores                | az1                                                                              |
      | tags                  | []                                                                               |
      | updated_at            | 2025-06-18T22:48:00Z                                                             |
      | virtual_size          | 16300544                                                                         |
      | visibility            | shared                                                                           |
      +-----------------------+----------------------------------------------------------------------------------+

   5. Observe that the image is in the stores for az1 and az2:

      sh-5.1$ openstack image show 5073f3f0-e4ac-4cca-9883-fecade29a1f3 | grep stores
      | properties       | os_glance_failed_import='', os_glance_importing_to_stores='', os_hash_algo='sha512', os_hash_value='6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869', os_hidden='False', stores='az1,az2' |
      sh-5.1$

   6. Observe the Block Storage service volumes in az1 and az2, which store the image ID from the previous step:

      sh-5.1$ openstack volume list --all | grep 5073f3f0-e4ac-4cca-9883-fecade29a1f3
      | 713f25b2-accc-4f4f-b5ba-8c71e2fac3aa | image-5073f3f0-e4ac-4cca-9883-fecade29a1f3 | available |    1 |                                 |
      | b1917ec2-2d8e-42cc-b58c-55db48f1e054 | image-5073f3f0-e4ac-4cca-9883-fecade29a1f3 | available |    1 |                                 |
      sh-5.1$

   7. Confirm the AZ, type, and host of each volume that backs the image:

      sh-5.1$ openstack volume show b1917ec2-2d8e-42cc-b58c-55db48f1e054 -c type -c availability_zone -c os-vol-host-attr:host
      +-----------------------+---------------------------------------------------------------+
      | Field                 | Value                                                         |
      +-----------------------+---------------------------------------------------------------+
      | availability_zone     | az1                                                           |
      | os-vol-host-attr:host | cinder-cbca0-volume-ontap-iscsi-az1-0@ontap#cinder_iscsi_pool |
      | type                  | glance-ontap-az1                                              |
      +-----------------------+---------------------------------------------------------------+
      sh-5.1$ openstack volume show 713f25b2-accc-4f4f-b5ba-8c71e2fac3aa -c type -c availability_zone -c os-vol-host-attr:host
      +-----------------------+---------------------------------------------------------------+
      | Field                 | Value                                                         |
      +-----------------------+---------------------------------------------------------------+
      | availability_zone     | az2                                                           |
      | os-vol-host-attr:host | cinder-cbca0-volume-ontap-iscsi-az2-0@ontap#cinder_iscsi_pool |
      | type                  | glance-ontap-az2                                              |
      +-----------------------+---------------------------------------------------------------+
      sh-5.1$

7. If you are using the Shared File Systems service (manila) with Red Hat Ceph Storage, confirm the manila-share service is running in each AZ:

   $ openstack share service list
   +----+------------------+----------------------+------+---------+-------+----------------------------+
   | ID | Binary       	| Host             	| Zone | Status  | State | Updated At             	|
   +----+------------------+----------------------+------+---------+-------+----------------------------+
   |  1 | manila-scheduler | hostgroup        	| nova | enabled | up	| 2025-05-21T14:10:08.498066 |
   |  4 | manila-share 	| hostgroup@cephfs_az3 | az3  | enabled | up	| 2025-05-21T14:10:04.973221 |
   |  7 | manila-share 	| hostgroup@cephfs_az1 | az1  | enabled | up	| 2025-05-21T14:10:05.228611 |
   | 10 | manila-share 	| hostgroup@cephfs_az2 | az2  | enabled | up	| 2025-05-21T14:10:04.724407 |
   +----+------------------+----------------------+------+---------+-------+----------------------------+

8. If you are using the Shared File Systems service with third-party storage and DHSS is disabled, confirm the following.

   1. As an administrator, create a share type:

      $ openstack share type create nfs-multiaz False --extra-specs share_backend_name=nfs_az

   2. Observe the share type:

      $ openstack share type list
      +--------------------------------------+--------------+------------+------------+--------------------------------------+-----------------------------+-------------+
      | ID                                   | Name         | Visibility | Is Default | Required Extra Specs                 | Optional Extra Specs        | Description |
      +--------------------------------------+--------------+------------+------------+--------------------------------------+-----------------------------+-------------+
      | a63ecc00-33f5-4bd8-9d93-a1fb31f2fe79 | nfs-multiaz  | public     | False      | driver_handles_share_servers : False | share_backend_name : nfs_az | None        |
      +--------------------------------------+--------------+------------+------------+--------------------------------------+-----------------------------+-------------+

   3. You can run the following commands as a non-admin user. Create a share in each AZ:

      openstack share create nfs 1 --name nfsaz1 --availability-zone az1
      openstack share create nfs 1 --name nfsaz2 --availability-zone az2
      openstack share create nfs 1 --name nfsaz3 --availability-zone az3

      • The openstack share create command does not need to pass --share-type nfs-multiaz because of the following configuration in the manila template in the OpenStackControlPlane CR:

              manilaAPI:
                customServiceConfig: |
                  [DEFAULT]
                  storage_availability_zone = az1,az2,az3
                  default_share_type = nfs-multiaz

        If the default_share_type field is not set, then pass the --share-type nfs-multiaz option with the openstack share create command.

   4. Observe the shares and their AZs:

      $ openstack share list
      +--------------------------------------+--------+------+-------------+-----------+-----------+-----------------+-------------------------------+-------------------+
      | ID                                   | Name   | Size | Share Proto | Status    | Is Public | Share Type Name | Host                          | Availability Zone |
      +--------------------------------------+--------+------+-------------+-----------+-----------+-----------------+-------------------------------+-------------------+
      | ef1fa319-4661-4d18-880b-a24b5e708234 | nfsaz1 |    1 | NFS         | available | False     | nfs-multiaz     | hostgroup@nfs_az1#n2_nvme_15T | az1               |
      | f91e1e59-e101-4e1f-8b92-fb49edd706cf | nfsaz2 |    1 | NFS         | available | False     | nfs-multiaz     | hostgroup@nfs_az2#n2_nvme_15T | az2               |
      | 09c9f709-1cf6-4279-ba39-8c80669b118a | nfsaz3 |    1 | NFS         | available | False     | nfs-multiaz     | hostgroup@nfs_az3#n2_nvme_15T | az3               |
      +--------------------------------------+--------+------+-------------+-----------+-----------+-----------------+-------------------------------+-------------------+

   5. Get the NFS paths of a share:

      Example:

      $ openstack share show nfsaz1 | grep path
      |                                       | path = 10.0.0.42:/share_9afcc8c4_2a09_4d05_8cdc_f28eb58a3dff

   6. Grant access to the appropriate IP range based on the Compute service (nova) instance to access the share:

      $ openstack share access create nfsaz1 ip 10.0.0.0/24 --access-level rw

   7. When you deploy the Compute service, you can then mount the share by using a command like the following:

      Example:

      $ mount -t nfs 10.0.0.42:/share_<UUID> /mnt/share

9. If you are using the Shared File Systems service with third-party storage and DHSS is enabled, confirm the following.

   1. As an OpenStack administrator, create a share type:

      $ openstack share type create nfs-multiaz true --extra-specs share_backend_name='nfs_az'

   2. As a project administrator, create a network and subnet for the Shared File Systems service to use:

      $ openstack network create --project rhoso --provider-network-type vlan manila_net
      openstack subnet create --network manila_net --subnet-range <subnet> --dns-nameserver <dns> manila-subnetaz1

   3. As a project user, create the share network by using the values from the previous step and define a share for each AZ:

      $ openstack share network create --name share_net --neutron-net-id <net_id> --neutron-subnet-id <sub_net_id>
      $ openstack share create nfs 1 --name nfsaz1 --share-network share_net --availability-zone az1
      $ openstack share create nfs 1 --name nfsaz2 --share-network share_net --availability-zone az2
      $ openstack share create nfs 1 --name nfsaz3 --share-network share_net --availability-zone az3

      • In this example, the share network does not have an AZ set. It is a default share network that spans all AZs.
      • You can create Networking service (neutron) networks and subnets for each AZ and limit share creation in AZs to their corresponding networks and subnets. To isolate share creation, create multiple share network subnets in the Shared File Systems service, each with its own Networking service network, subnet, and AZ.

      • The openstack share create command does not need to pass --share-type nfs-multiaz because of the following configuration in the manila template in the OpenStackControlPlane CR:

              manilaAPI:
                customServiceConfig: |
                  [DEFAULT]
                  storage_availability_zone = az1,az2,az3
                  default_share_type = nfs-multiaz

        If the default_share_type field is not set, then pass the --share-type nfs-multiaz option with the openstack share create command.

   4. Grant access to the appropriate IP range based on the Compute service instance to access the share:

      $ openstack share access create nfsaz1 ip 10.0.0.0/24 --access-level rw

   5. When you deploy the Compute service, you can then mount the share by using a command like the following:

      Example:

      $ mount -t nfs 10.0.0.42:/share_<UUID> /mnt/share

10. Exit the OpenStackClient pod:

    $ exit

Procedure

1. For unprovisioned nodes, create the SSH key pair for Ansible:

   $ ssh-keygen -f <key_file_name> -N "" -t rsa -b 4096

   • Replace <key_file_name> with the name to use for the key pair.

2. Create the Secret CR for Ansible and apply it to the cluster:

   $ oc create secret generic dataplane-ansible-ssh-private-key-secret \
   --save-config \
   --dry-run=client \
   --from-file=ssh-privatekey=<key_file_name> \
   --from-file=ssh-publickey=<key_file_name>.pub \
   [--from-file=authorized_keys=<key_file_name>.pub] -n openstack \
   -o yaml | oc apply -f -

   • Replace <key_file_name> with the name and location of your SSH key pair file.
   • Optional: Only include the --from-file=authorized_keys option for bare-metal nodes that must be provisioned when creating the data plane.

3. If you are creating Compute nodes, create a secret for migration.

   1. Create the SSH key pair for instance migration:

      $ ssh-keygen -f ./nova-migration-ssh-key -t ecdsa-sha2-nistp521 -N ''

   2. Create the Secret CR for migration and apply it to the cluster:

      $ oc create secret generic nova-migration-ssh-key \
      --save-config \
      --from-file=ssh-privatekey=nova-migration-ssh-key \
      --from-file=ssh-publickey=nova-migration-ssh-key.pub \
      -n openstack \
      -o yaml | oc apply -f -

4. For nodes that have not been registered to the Red Hat Customer Portal, create the Secret CR for subscription-manager credentials to register the nodes:

   $ oc create secret generic subscription-manager \
   --from-literal rhc_auth='{"login": {"username": "<subscription_manager_username>", "password": "<subscription_manager_password>"}}'

   • Replace <subscription_manager_username> with the username you set for subscription-manager.
   • Replace <subscription_manager_password> with the password you set for subscription-manager.

5. Create a Secret CR that contains the Red Hat registry credentials:

   $ oc create secret generic redhat-registry --from-literal edpm_container_registry_logins='{"registry.redhat.io": {"<username>": "<password>"}}'

   • Replace <username> and <password> with your Red Hat registry username and password credentials.

     For information about how to create your registry service account, see the Knowledge Base article Creating Registry Service Accounts.

6. If you are creating Compute nodes, create a secret for libvirt.

   1. Create a file on your workstation named secret_libvirt.yaml to define the libvirt secret:

      apiVersion: v1
      kind: Secret
      metadata:
       name: libvirt-secret
       namespace: openstack
      type: Opaque
      data:
       LibvirtPassword: <base64_password>

      • Replace <base64_password> with a base64-encoded string with maximum length 63 characters. You can use the following command to generate a base64-encoded password:

        $ echo -n <password> | base64

        Tip

        If you do not want to base64-encode the username and password, you can use the stringData field instead of the data field to set the username and password.

   2. Create the Secret CR:

      $ oc apply -f secret_libvirt.yaml -n openstack

7. Verify that the Secret CRs are created:

   $ oc describe secret dataplane-ansible-ssh-private-key-secret
   $ oc describe secret nova-migration-ssh-key
   $ oc describe secret subscription-manager
   $ oc describe secret redhat-registry
   $ oc describe secret libvirt-secret

1. Create the ConfigMap custom resources (CRs) to configure the nodes.
2. Create a custom service for the node set that runs the playbook for the service.
3. Include the ConfigMap CRs in the custom service.

Procedure

1. Create a file named nova-extra-config-az1.yaml on your workstation to define the ConfigMap CR for the Compute node set for the AZ.

2. Define a new configuration file to apply to the Compute nodes in the node set for the AZ that adds the Image service (glance) endpoint for the local AZ and sets the cross_az_attach field to false.

   • If you are using Red Hat Ceph Storage, include RBD options:

     Example:

     apiVersion: v1
     kind: ConfigMap
     metadata:
       name: nova-extra-config-az1
     data:
       03-ceph-nova.conf:
         [libvirt]
         images_type = rbd
         images_rbd_pool = vms
         images_rbd_ceph_conf = /etc/ceph/az1.conf
         images_rbd_glance_store_name = az1
         images_rbd_glance_copy_poll_interval = 15
         images_rbd_glance_copy_timeout = 600
         rbd_user = openstack
         rbd_secret_uuid=$FSID
         hw_disk_discard = unmap
         [glance]
         endpoint_override = https://glance-az1-internal.openstack.svc:9292
         valid_interfaces = internal
         [cinder]
         cross_az_attach = False
         catalog_info = volumev3:cinderv3:internalURL

   • If you are using a third-party storage device with multipath, then set volume_use_multipath to True:

     Example:

     apiVersion: v1
     kind: ConfigMap
     metadata:
       name: nova-extra-config-az1
     data:
       25-nova-extra.conf: |
         [libvirt]
         volume_use_multipath = True
         [glance]
         endpoint_override = https://glance-az1-internal.openstack.svc:9292
         valid_interfaces = internal
         [cinder]
         cross_az_attach = False
         catalog_info = volumev3:cinderv3:internalURL

   • data.<filename>: The file name must follow the naming convention of ##-<name>-nova.conf. Files are evaluated by the Compute service alphabetically. A filename that starts with 01 will be evaluated by the Compute service before a filename that starts with 02. When the same configuration option occurs in multiple files, the configuration option is set to the last value read.
   • rbd_secret_uuid: The $FSID value should contain the actual FSID as described in Obtaining the Red Hat Ceph Storage file system identifier in Configuring persistent storage. The FSID itself does not need to be considered secret.

   When the service is deployed, it adds the configuration to the etc/<service>/<service>.conf.d/ directory in the service container. For example, for a Compute feature, the configuration file is added to etc/nova/nova.conf.d/ in the nova_compute container.

   + For more information on creating ConfigMap objects, see Creating and using config maps in the RHOCP Nodes guide.

3. Create the ConfigMap CR:

   $ oc create -f nova-extra-config-az1.yaml

4. Create a file named nova-custom-az1.yaml on your workstation to define the OpenStackDataPlaneService CR for the Compute node set for the AZ:

   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneService
   metadata:
     name: nova-custom-az1
   spec:
     addCertMounts: false
     caCerts: combined-ca-bundle
     edpmServiceType: nova
     playbook: osp.edpm.nova
     tlsCerts:
       default:
         contents:
         - dnsnames
         - ips
         edpmRoleServiceName: nova
         issuer: osp-rootca-issuer-internal
         networks:
         - ctlplane

5. Add the ConfigMap CR to the custom service:

     dataSources:
     - configMapRef:
         name: ceph-nova-az1
     - secretRef:
         name: nova-migration-ssh-key

6. Specify the Secret CR for the cell that the node set that runs this service connects to:

   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneService
   metadata:
     name: <nodeset>-service
   spec:
     ...
     dataSources:
       - configMapRef:
           name: feature-configmap
       - secretRef:
           name: nova-migration-ssh-key
       - secretRef:
           name: nova-cell1-compute-config

7. Create the custom service:

   $ oc create -f nova-custom-az1.yaml

Procedure

1. Create a file on your workstation to define the OpenStackDataPlaneNodeSet CR for the node set for Zone 1, for example, compute_node_set_zone1.yaml:

   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneNodeSet
   metadata:
     name: compute-node-set-zone1
     namespace: openstack
   spec:
     env:
     - name: ANSIBLE_FORCE_COLOR
       value: "True"
     - name: ANSIBLE_TIMEOUT
       value: "60"
     - name: ANSIBLE_SSH_TIMEOUT
       value: "60"
     - name: ANSIBLE_SSH_RETRIES
       value: "60"

   • metadata.name: The OpenStackDataPlaneNodeSet CR name must be unique, contain only lower case alphanumeric characters and - (hyphens) or . (periods), start and end with an alphanumeric character, and have a maximum length of 53 characters. Update the name in this example to a name that reflects the nodes in the set.
   • spec.env: An optional field that lists the environment variables to pass to the pod.

2. Add the list of services to execute for this node set:

   • Specify the following services for a set of Compute nodes, replacing the nova service with the custom service you created for the node set:

       services:
       - download-cache
       - bootstrap
       - configure-network
       - validate-network
       - install-os
       - configure-os
       - frr
       - ssh-known-hosts
       - run-os
       - reboot-os
       - install-certs
       - ceph-client
       - ovn
       - neutron-metadata
       - libvirt
       - *nova-custom-az1*

   • Specify the following services for a set of Networker nodes:

       services:
       - download-cache
       - bootstrap
       - configure-network
       - validate-network
       - install-os
       - configure-os
       - frr
       - ssh-known-hosts
       - run-os
       - reboot-os
       - install-certs
       - ovn
       - neutron-metadata
       - ovn-bgp-agent

3. Connect the data plane to the control plane network:

   spec:
     ...
     networkAttachments:
     - ctlplane

4. Specify that the nodes in this set are pre-provisioned:

     preProvisioned: true

5. Add the SSH key secret that you created to enable Ansible to connect to the data plane nodes:

     nodeTemplate:
       ansibleSSHPrivateKeySecret: <secret-key>

   • Replace <secret-key> with the name of the SSH key Secret CR you created for this node set in Creating the data plane secrets, for example, dataplane-ansible-ssh-private-key-secret.

6. If you are creating a set of Compute nodes, enable Compute service access to the Red Hat Ceph Storage secret:

     nodeTemplate:
       ...
       extraMounts:
       - extraVolType: Ceph
         mounts:
         - mountPath: /etc/ceph
           name: ceph
           readOnly: true
         volumes:
         - name: ceph
           secret:
             secretName: ceph-conf-files-az1

   Note

   Networker nodes do not require access to the Red Hat Ceph Storage secret.

7. Specify the management network:

     nodeTemplate:
       ...
       managementNetwork: ctlplane

8. Specify the Secret CRs used to source the usernames and passwords to register the operating system of your nodes and to enable repositories. The following example demonstrates how to register your nodes to Red Hat Content Delivery Network (CDN). For information about how to register your nodes with Red Hat Satellite 6.13, see Managing Hosts.

     nodeTemplate:
       ...
       ansible:
         ansibleUser: cloud-admin
         ansiblePort: 22
         ansibleVarsFrom:
           - secretRef:
               name: subscription-manager
           - secretRef:
               name: redhat-registry
         ansibleVars:
           rhc_release: 9.4
           rhc_repositories:
               - {name: "*", state: disabled}
               - {name: "rhel-9-for-x86_64-baseos-eus-rpms", state: enabled}
               - {name: "rhel-9-for-x86_64-appstream-eus-rpms", state: enabled}
               - {name: "rhel-9-for-x86_64-highavailability-eus-rpms", state: enabled}
               - {name: "fast-datapath-for-rhel-9-x86_64-rpms", state: enabled}
               - {name: "rhoso-18.0-for-rhel-9-x86_64-rpms", state: enabled}
               - {name: "rhceph-7-tools-for-rhel-9-x86_64-rpms", state: enabled}
           edpm_bootstrap_release_version_package: []

   • ansibleUser: The user associated with the secret you created in Creating the data plane secrets.
   • ansibleVars: The Ansible variables that customize the set of nodes. For a list of Ansible variables that you can use, see https://openstack-k8s-operators.github.io/edpm-ansible/. For a complete list of the Red Hat Customer Portal registration commands, see https://access.redhat.com/solutions/253273. For information about how to log into registry.redhat.io, see https://access.redhat.com/RegistryAuthentication#creating-registry-service-accounts-6.

9. Configure the networks that the OVN BGP agent uses to communicate with FRRounting (FRR) on the data plane, and the password to authenticate with the BGP peer:

     nodeTemplate:
       ...
       ansible:
         ...
         ansibleVars:
           ...
           edpm_frr_bfd: false
           edpm_frr_bgp_ipv4_src_network: bgpmainnet
           edpm_frr_bgp_ipv6_src_network: bgpmainnetv6
           edpm_frr_bgp_neighbor_password: f00barZ

10. If you are creating a set of Networker nodes, enable the edpm_enable_chassis_gw field:

        ansible:
          ...
          ansibleVars:
            ...
            edpm_enable_chassis_gw: true

11. Add the network configuration template to apply to your nodes:

            edpm_network_config_hide_sensitive_logs: false
            edpm_network_config_os_net_config_mappings:
              edpm-z1-compute-0:
                nic1: 6a:fe:54:3f:8a:02
              edpm-z1-compute-1:
                nic1: 6b:fe:54:3f:8a:02
            neutron_physical_bridge_name: br-ex
            neutron_public_interface_name: eth1
            edpm_network_config_template: |
              ---
              {% set mtu_list = [ctlplane_mtu] %}
              {% for network in nodeset_networks %}
              {{ mtu_list.append(lookup('vars', networks_lower[network] ~ '_mtu')) }}
              {%- endfor %}
              {% set min_viable_mtu = mtu_list | max %}
              network_config:
              - type: ovs_bridge
                name: {{ neutron_physical_bridge_name }}
                use_dhcp: false
              - type: interface
                name: nic1
                use_dhcp: true
                defroute: false
              - type: interface
                name: nic2
                use_dhcp: false
                defroute: false
                dns_servers: {{ ctlplane_dns_nameservers }}
                domain: {{ dns_search_domains }}
                addresses:
                  - ip_netmask: {{ ctlplane_ip }}/{{ ctlplane_cidr }}
              - type: interface
                name: nic3
                use_dhcp: false
                addresses:
                - ip_netmask: {{ lookup('vars', 'bgpnet0_ip') }}/30
              - type: interface
                name: nic4
                use_dhcp: false
                addresses:
                - ip_netmask: {{ lookup('vars', 'bgpnet1_ip') }}/30
              - type: interface
                name: lo
                addresses:
                - ip_netmask: {{ lookup('vars', 'bgpmainnet_ip') }}/32
                - ip_netmask: {{ lookup('vars', 'bgpmainnetv6_ip') }}/128
                - ip_netmask: {{ lookup('vars', 'internalapi_ip') }}/32
                - ip_netmask: {{ lookup('vars', 'storage_ip') }}/32
                - ip_netmask: {{ lookup('vars', 'tenant_ip') }}/32

    • nic1: Update to the MAC address assigned to the NIC to use for network configuration on the Compute node.
    • ip_netmask: {{ lookup('vars', 'storage_ip') }}/32: Not required for sets of Networker nodes.

12. Disable validations that are not required when using BGP:

        	  edpm_nodes_validation_validate_controllers_icmp: false
        	  edpm_nodes_validation_validate_gateway_icmp: false

13. Configure the OVN BGP Agent to not expose tenant networks:

        	  edpm_ovn_bgp_agent_expose_tenant_networks: false

14. Configure OVN to establish tunnels over BGP by using the bgpmainnet instead of the local tenant network:

            edpm_ovn_encap_ip: '{{ lookup(''vars'', ''bgpmainnet_ip'') }}

15. Define each node in this node set:

      nodes:
        edpm-z1-compute-0:
          hostName: edpm-z1-compute-0
          networks:
          - name: ctlplane
            subnetName: subnet1
            defaultRoute: true
            fixedIP: 192.168.122.100
          - name: internalapi
            subnetName: subnet1
          - name: storage
            subnetName: subnet1
          - name: tenant
            subnetName: subnet1
          - name: Bgpnet0
            subnetName: subnet1
            fixedIP: 100.64.0.2
          - name: Bgpnet1
            subnetName: subnet1
            fixedIP: 100.65.0.2
          - name: Bgpmainnet
            subnetName: subnet1
            fixedIP: 99.99.0.2
          - name: BgpmainnetV6
            subnetName: subnet0
            fixedIP: f00d:f00d:f00d:f00d:f00d:f00d:f00d:0012
          ansible:
            ansibleHost: 192.168.122.100
            ansibleVars:
              edpm_frr_bgp_peers:
              - 100.64.0.1
              - 100.65.0.1
              edpm_ovn_bgp_agent_local_ovn_peer_ips:
              - 100.64.0.1
              - 100.65.0.1
        edpm-z1-compute-1:
          hostName: edpm-z1-compute-1
          networks:
          - name: ctlplane
            subnetName: subnet1
            defaultRoute: true
            fixedIP: 192.168.122.101
          - name: internalapi
            subnetName: subnet1
          - name: storage
            subnetName: subnet1
          - name: tenant
            subnetName: subnet1
          - name: Bgpnet0
            subnetName: subnet1
            fixedIP: 100.64.0.6
          - name: Bgpnet1
            subnetName: subnet1
            fixedIP: 100.65.0.6
          - name: Bgpmainnet
            subnetName: subnet1
            fixedIP: 99.99.0.3
          - name: BgpmainnetV6
            subnetName: subnet0
            fixedIP: f00d:f00d:f00d:f00d:f00d:f00d:f00d:0013
        edpm-z1-compute-2:
          hostName: edpm-z1-compute-2
          networks:
          - name: ctlplane
            subnetName: subnet1
            defaultRoute: true
            fixedIP: 192.168.122.102
          - name: internalapi
            subnetName: subnet1
          - name: storage
            subnetName: subnet1
          - name: tenant
            subnetName: subnet1
          - name: Bgpnet0
            subnetName: subnet1
            fixedIP: 100.64.0.10
          - name: Bgpnet1
            subnetName: subnet1
            fixedIP: 100.65.0.10
          - name: Bgpmainnet
            subnetName: subnet1
            fixedIP: 99.99.0.4
          - name: BgpmainnetV6
            subnetName: subnet0
            fixedIP: f00d:f00d:f00d:f00d:f00d:f00d:f00d:0014

    • nodes.<node_ref>: The node definition reference, for example, edpm-compute-0. Each node in the node set must have a node definition.
    • networks: Defines the IPAM and the DNS records for the node.
    • networks.fixedIP: The storage network is not required by sets of Networker nodes.
    • networks.name: Specifies a predictable IP address for the network that must be in the allocation range defined for the network in the NetConfig CR.

    • ansible.ansibleVars: Node-specific Ansible variables that customize the node.

      Note
      • Nodes defined within the nodes section can configure the same Ansible variables that are configured in the nodeTemplate section. Where an Ansible variable is configured for both a specific node and within the nodeTemplate section, the node-specific values override those from the nodeTemplate section.
      • You do not need to replicate all the nodeTemplate Ansible variables for a node to override the default and set some node-specific values. You only need to configure the Ansible variables you want to override for the node.
      • Many ansibleVars include edpm in the name, which stands for "External Data Plane Management".
16. Save the definition file.

17. Create the node set for Zone 1:

    $ oc create --save-config -f compute_node_set_zone1.yaml -n openstack

18. Verify that the resources have been created by confirming that the status is SetupReady:

    $ oc wait openstackdataplanenodeset compute-node-set-zone1 --for condition=SetupReady --timeout=10m

    When the status is SetupReady the command returns a condition met message, otherwise it returns a timeout error.

19. Verify that the Secret resource was created for the node set:

    $ oc get secret | grep openstack-data-plane
    compute-node-set-zone1 Opaque 1 3m50s

20. Verify the services were created:

    $ oc get openstackdataplaneservice -n openstack
    NAME                AGE
    bootstrap           46m
    ceph-client         46m
    ceph-hci-pre        46m
    configure-network   46m
    configure-os        46m
    ...

Procedure

1. Create a file on your workstation named openstack_data_plane_deploy.yaml to define the OpenStackDataPlaneDeployment CR:

   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneDeployment
   metadata:
     name: data-plane-deploy
     namespace: openstack

   • metadata.name: The OpenStackDataPlaneDeployment CR name must be unique, must consist of lower case alphanumeric characters, - (hyphen) or . (period), and must start and end with an alphanumeric character. Update the name in this example to a name that reflects the node sets in the deployment.

2. Add all the OpenStackDataPlaneNodeSet CRs that you want to deploy:

   spec:
     nodeSets:
       - openstack-data-plane
       - <nodeSet_name>
       - ...
       - <nodeSet_name>

   • Replace <nodeSet_name> with the names of the OpenStackDataPlaneNodeSet CRs that you want to include in your data plane deployment.
3. Save the openstack_data_plane_deploy.yaml deployment file.

4. Deploy the data plane:

   $ oc create -f openstack_data_plane_deploy.yaml -n openstack

   You can view the Ansible logs while the deployment executes:

   $ oc get pod -l app=openstackansibleee -w
   $ oc logs -l app=openstackansibleee -f --max-log-requests 10

   If the oc logs command returns an error similar to the following error, increase the --max-log-requests value:

   error: you are attempting to follow 19 log streams, but maximum allowed concurrency is 10, use --max-log-requests to increase the limit

5. Verify that the data plane is deployed:

   $ oc wait openstackdataplanedeployment data-plane-deploy --for=condition=Ready --timeout=<timeout_value>
   $ oc wait openstackdataplanenodeset openstack-data-plane --for=condition=Ready --timeout=<timeout_value>

   • Replace <timeout_value> with a value in minutes you want the command to wait for completion of the task. For example, if you want the command to wait 60 minutes, you would use the value 60m. If the completion status of SetupReady for oc wait openstackdataplanedeployment or NodeSetReady for oc wait openstackdataplanenodeset is not returned in this time frame, the command returns a timeout error. Use a value that is appropriate to the size of your deployment. Give larger deployments more time to complete deployment tasks.

     For information about the data plane conditions and states, see Data plane conditions and states in Deploying Red Hat OpenStack Services on OpenShift.

6. Map the Compute nodes to the Compute cell that they are connected to:

   $ oc rsh nova-cell0-conductor-0 nova-manage cell_v2 discover_hosts --verbose

   If you did not create additional cells, this command maps the Compute nodes to cell1.

7. Access the remote shell for the openstackclient pod and verify that the deployed Compute nodes are visible on the control plane:

   $ oc rsh -n openstack openstackclient
   $ openstack hypervisor list

   If some Compute nodes are missing from the hypervisor list, retry the previous step. If the Compute nodes are still missing from the list, check the status and health of the nova-compute services on the deployed data plane nodes.

8. Verify that the hypervisor hostname is a fully qualified domain name (FQDN):

   $ hostname -f

   If the hypervisor hostname is not an FQDN, for example, if it was registered as a short name or full name instead, contact Red Hat Support.

Procedure

1. Verify that the Compute service is using the third-party storage for the Image service and the Block Storage service in each AZ. In this example, you have Compute nodes in all three AZs, and you have put them into their own aggregates by using a script like the following:

   Example:

   function make_aggregate() {
       OS="oc rsh openstackclient openstack"
       AZ=az$1
       RACK=r$1
       $OS aggregate create $AZ
       $OS aggregate set --zone $AZ $AZ
       for I in $(seq 0 2); do
           $OS aggregate add host $AZ ${RACK}-compute-${I}.ctlplane.example.com
       done
       $OS compute service list -c Host -c Zone
   }
   
   make_aggregate 1
   make_aggregate 2
   make_aggregate 3

2. Open a remote shell connection to the openstackclient pod:

   $ oc rsh -n openstack openstackclient

3. Create an ephemeral instance in AZ1 by using the image created earlier:

   $ openstack server create --flavor c1 --image 3ae0d8d8-18f5-452e-a58e-63fb7aba308a --nic net-id=private --availability-zone az1 vm-az1

4. Observe the instance:

   sh-5.1$ openstack server list
   +--------------------------------------+--------+--------+----------------------+------------------+--------+
   | ID                                   | Name   | Status | Networks             | Image            | Flavor |
   +--------------------------------------+--------+--------+----------------------+------------------+--------+
   | c1f8b608-41d4-422a-ab89-82762353a784 | vm-az1 | ACTIVE | private=192.168.0.21 | cirros-priv-type | c1     |
   +--------------------------------------+--------+--------+----------------------+------------------+--------+

5. Create a volume from an image in AZ2 by using the image which was imported into AZ2 earlier:

   $ openstack volume create --size 8 vm_root_az2 --image 3ae0d8d8-18f5-452e-a58e-63fb7aba308a --availability-zone az2

6. Observe the volume:

   sh-5.1$ openstack volume list
   +--------------------------------------+-------------+-----------+------+-------------+
   | ID                                   | Name        | Status    | Size | Attached to |
   +--------------------------------------+-------------+-----------+------+-------------+
   | 119958b1-0fca-4412-81d1-1d9b856ce85a | vm_root_az2 | available |    8 |             |
   +--------------------------------------+-------------+-----------+------+-------------+

7. Create an instance in AZ2 with the Block Storage service volume from the previous step as the root volume:

   $ openstack server create --flavor c1 --volume 119958b1-0fca-4412-81d1-1d9b856ce85a --nic net-id=private --availability-zone az2 vm-az2

8. Observe the instances:

   sh-5.1$ openstack server list
   +--------------------------------------+--------+--------+-----------------------+--------------------------+--------+
   | ID                                   | Name   | Status | Networks              | Image                    | Flavor |
   +--------------------------------------+--------+--------+-----------------------+--------------------------+--------+
   | 54e1a625-08d1-4df5-838c-c8ea676d2e06 | vm-az2 | ACTIVE | private=192.168.0.138 | N/A (booted from volume) | c1     |
   | c1f8b608-41d4-422a-ab89-82762353a784 | vm-az1 | ACTIVE | private=192.168.0.21  | cirros-priv-type         | c1     |
   +--------------------------------------+--------+--------+-----------------------+--------------------------+--------+
   sh-5.1$

9. Exit the openstackclient pod:

   $ exit