Chapter 10. Configuring a Single Keystone Multiple OpenStacks multi-region deployment to simplify user management and configuration

1. The interdependence between the workload and the central regions requires that every region must provide the following unique identifications for a successful SKMO deployment:

   • A unique namespace to differentiate the RHOSO deployment of each region. For more information, see Create a unique namespace for each workload region.

     Important

     The RHOSO deployment namespace forms part of the DNS name for each OpenStack service. If you do not use different RHOSO namespaces for every region, conflicts occur between services in your different regions.

   • A unique region name defined by the spec of their Identity (keystone) service in the OpenStackControlPlane custom resource (CR). For more information, see Modify the deployment of each workload region.

     Note

     When the central Red Hat OpenStack Services on OpenShift (RHOSO) region is deployed, this region is called regionOne by default. If you use a workload region naming convention, then you can rename the region name of the central region to make it more easily identifiable. For more information, see Rename the central region.

   • The OpenStack services that communicate with the Identity service must use uniquely named Identity service users to simplify the task of managing their individual credentials. For more information, see Modify the deployment of each workload region.

     Warning

     If you do not specify unique Identity service user names for all the OpenStack services that communicate with the Identity service, then changing the password for a service user disrupts this service in all the workload regions that use this same user - unless you schedule a maintenance window for all of these regions first.

2. The interdependence between the workload and the central regions imposes the following restrictions that must be met by the logical networking topology of your DNS configuration:

   • Each workload region must resolve the DNS name of the Identity service in the central region and access it.
   • The data plane (EDPM) nodes in each workload region must resolve the DNS name of the Identity service in the central region and access it.
   • After deploying the workload regions, the Dashboard (horizon) service in the central region must resolve the DNS names in the service catalog of every workload region and access them.

3. The interdependence between the Identity services in each workload region and the Identity service of the central region changes how the service-to-service communication for the workload regions are routed. For more information, see Create the public and private endpoints of each workload region.

   In a normal OpenStack deployment like the central region, the Identity service has both a public and internal endpoint. These endpoints exist in separate networks to keep the internal service-to-service communication separate from public traffic. However, the workload regions are forced to send all of their internal service-to-service communication traffic to the public endpoint and therefore the public network of the central region. Even though the internal service-to-service communication traffic of the workload regions is encrypted, it is more vulnerable to DDOS attacks because it is not isolated on a separate internal network making it easier for external attackers to intercept these messages.

4. The barbican-keystone-listener service requires access to the RabbitMQ message queue so that when a project is deleted by the Identity service (keystone), it can tell the Key Manager service (barbican) to clean up the related secrets and the other artifacts that it manages.

   In an SKMO deployment, the RabbitMQ message queue of the central region and not the workload regions contain the necessary Identity service (keystone) messages. For this reason, the barbican-keystone-listener services in the workload regions cannot know when projects are deleted so that the Key Manager service can clean them up. Therefore you must implement a third-party application like Skupper and configure your SKMO deployment to allow the barbican-keystone-listener services in the workload regions to access the RabbitMQ message queue in the central region to clean up deleted projects. For more information, see Enable RabbitMQ access for barbican-keystone-listener in SKMO.

Procedure

1. Deploy the central region called regionOne by default, unless you rename it. For more information, see Rename the central region.

   The deployment of the central region does not require a data plane.

2. Create the default administrator Identity service user for each workload region in the central region. These Identity service users must be granted the admin role in the admin project of the central region. For more information, see Create the default administrator user for each workload region.

3. Create the catalog entries for the public and private endpoints of the Identity service in each workload region by using the Identity service in the central region. For more information, see Create the public and private endpoints of each workload region.

   Note

   Both the public and private endpoints of the Identity service in each workload region point to the public Identity service endpoint in the central region.

4. Modify and deploy each workload region. An important part of this deployment modification involves creating a unique region name and unique Identity service users for each workload region. For more information, see Modify the deployment of each workload region.
5. After you deploy a workload region of a multi-region Red Hat OpenStack Services on OpenShift (RHOSO) Single Keystone Multiple OpenStacks (SKMO) deployment you must configure the deployed central region to trust this workload region. For more information, see Configure the central region to trust a deployed workload region.

Procedure

1. In the central region, set the default RHOSO namespace:

   $ oc project <central-region-namespace>

   where:

   <central-region-namespace>

   Specifies the name of the unique namespace for the central region, for example openstack.

2. Edit the OpenStackControlPlane CR of your central region on your workstation:

   $ oc edit openstackcontrolplane <name>

   where:

   <name>

   Specifies the name of your YAML OpenStackControlPlane CR. You can use the following command to retrieve this name: oc get openstackcontrolplane.

3. Configure the region parameter of the Identity service (keystone):

   ...
     spec:
       ...
       keystone:
         ...
         template:
           ...
           region: <central-region-name>

   where:

   <central-region-name>

   Specifies the region name for your central region.

4. Save and close the editor to apply this change.

5. Wait for the deployment of the control plane to reach the Ready status:

   $ oc wait openstackcontrolplane <name> --for=condition=Ready --timeout=600s

Procedure

1. In the central region, set the default RHOSO namespace:

   $ oc project <central-region-namespace>

   where:

   <central-region-namespace>

   Specifies the name of the unique namespace for the central region, for example openstack.

2. Access the remote shell for the OpenStackClient pod from your workstation to run OpenStack CLI commands:

   $ oc rsh openstackclient

3. Create the default administrator user for each workload region:

   $ openstack user create --domain Default --project <central-region-admin-project> --project-domain Default --password <workload-region-admin-password> <workload-region-admin-name>

   where:

   <central-region-admin-project>

   Specifies the name of the admin project in the central region that is admin by default.

   <workload-region-admin-password>

   Specifies the password of the default administrator user of each`workload` region.

   Note

   Set this password as the value of the AdminPassword: parameter of the Secret custom resource (CR) file that you must create to provide secure access to the RHOSO service pods when you deploy each workload region.

   <workload-region-admin-name>

   Specifies the name of the default administrator user of each workload region, for example admin-two.

4. Add the following roles to each default workload region administrator user:

   $ openstack role add --project admin --project-domain Default --user <workload-region-admin-name> --user-domain Default admin
   $ openstack role add --system all --user <workload-region-admin-name> --user-domain Default admin

Procedure

1. In the central region, set the default RHOSO namespace:

   $ oc project <central-region-namespace>

   where:

   <central-region-namespace>

   Specifies the name of the unique namespace for the central region, for example openstack.

2. Access the remote shell for the OpenStackClient pod from your workstation to run OpenStack CLI commands:

   $ oc rsh openstackclient

3. Obtain the public Identity service endpoint of the central region:

   $ openstack endpoint list --region <central-region-name> --service keystone --interface public

   where:

   <central-region-name>

   Specifies the name of your central region, which is regionOne by default.

4. Copy this URL that is referenced in this procedure as <central-region-public-keystone-url>.

5. Create the public and private endpoints of each workload region to specify the public Identity service endpoint of the central region:

   $ openstack endpoint create --region <workload-region-name> keystone public <central-region-public-keystone-url>
   $ openstack endpoint create --region <workload-region-name> keystone internal <central-region-public-keystone-url>

   where:

   <workload-region-name>

   Specifies the name of the required workload region, for example regionTwo.

   This creates the catalog entries for the public and private endpoints of the Identity service in each workload region in the Identity service catalog of the central region.

Procedure

1. Create a unique RHOSO deployment namespace for each workload region to differentiate these RHOSO deployments from each other. For more information, see Create a unique namespace for each workload region.
2. Specify the password of the administrator user that you manually created for each workload region as the value of the AdminPassword: parameter when you create the Secret custom resource (CR) to provide secure access to the RHOSO service pods in each workload region. For more information, see Create the default administrator user for each workload region and Provide secure access to RHOSO services in the Deploying Red Hat OpenStack Services on OpenShift guide.
3. Obtain the CA certificate from the central region and add it to a secret in each workload region so it can be trusted. For more information, see Configure each workload region to trust the central region.

4. Create a modified OpenStackControlPlane custom resource (CR) for each workload region. For more information, see Modify the control plane CR of each workload region.

   This involves defining a unique region name defined in the spec of the Identity (keystone) service for each workload region and creating unique Identity service users.

Procedure

1. In the deployed workload region, set the default RHOSO namespace:

   $ oc project <workload-region-namespace>

   where:

   <workload-region-namespace>

   Specifies the name of the unique namespace for this workload region, for example openstack-two.

2. Obtain the CA certificate of this workload region and extract it into a file, for example regionTwo-ca.crt:

   Note

   To decode the certificate before creating the output .crt file add | base64 -d to this command.

   $ oc get secret rootca-public -o yaml | yq '.data."ca.crt"' > regionTwo-ca.crt

3. Copy the regionTwo-ca.crt file to the deployed central region.

4. In the central region, set the default RHOSO namespace:

   $ oc project <central-region-namespace>

   where:

   <central-region-namespace>

   Specifies the name of the unique namespace for the central region, for example openstack.

5. Edit your OpenStackControlPlane CR of the central region:

   $ oc edit openstackcontrolplane <name>

   where:

   <name>

   Specifies the name of your YAML OpenStackControlPlane CR. You can use the following command to retrieve this name: oc get openstackcontrolplane.

6. If your OpenStackControlPlane CR in the central region contains the spec.tls.caBundleSecretName parameter:

   1. Obtain the name of the secret that contains the PEM-formatted bundle containing all the other custom CA certificates in chains of trust if applicable that the central region trusts, for example custom-ca-certs:

      ...
        spec:
          ...
          tls:
            ...
            caBundleSecretName: custom-ca-certs

   2. Edit the specified secret, in this example custom-ca-certs

      $ oc edit secret custom-ca-certs

   3. Append the contents of the CA certificate of the deployed workload region, for example regionTwo-ca.crt to the PEM-formatted bundle containing all the other custom CA certificates that the central region trusts.
   4. Save and exit the editor to automatically apply the changes to the secret, in this example custom-ca-certs.

7. If your OpenStackControlPlane CR in the central region does not contain the spec.tls.caBundleSecretName parameter:

   1. Create a PEM-formatted bundle, for example custom-ca-certs.pem that includes the contents of this regionTwo-ca.crt file.

   2. Create a manifest file for the secret in the central region that specifies the contents of the custom-ca-certs.pem bundle created in the previous step. In this example the manifest file is called custom-ca-certs.yaml and the secret is called custom-ca-certs:

      apiVersion: v1
      data:
        custom-ca-certs.pem: <contents-of-PEM-bundle>
      kind: Secret
      metadata:
        annotations:
        name: custom-ca-certs
        namespace: <namespace>
      type: Opaque

      where:

      <namespace>

      Specifies the namespace of the central region, in this example openstack.

      <contents-of-PEM-bundle>

      Specifies the base64 encoded string of the contents of the PEM-formatted bundle you created called custom-ca-certs.pem that includes the CA certificate from regionTwo. You can get this base64 encoded string by using the following command: cat custom-ca-certs.pem | base64 -w0.

3. Create the secret in the central region from the manifest file. In this example the manifest file is called custom-ca-certs.yaml:

   $ oc apply -f custom-ca-certs.yaml

4. Edit your OpenStackControlPlane CR of the central region:

   $ oc edit openstackcontrolplane <name>

   where:

   <name>

   Specifies with the name of your YAML OpenStackControlPlane CR. You can use the following command to retrieve this name: oc get openstackcontrolplane.

5. Add the secret that you have created, in this example custom-ca-certs:

   ...
     spec:
       ...
       tls:
         ...
         caBundleSecretName: custom-ca-certs

6. Save and close the editor to automatically apply this change.

7. Wait for the deployment of the control plane of the central region to reach the Ready status:

   $ oc wait openstackcontrolplane <name> --for=condition=Ready --timeout=600s