Deploy a multi-region OpenStack with centralized user management

Deploy a Single Keystone Multiple OpenStacks (SKMO) multi-region environment to centralize user management and eliminate the need for separate user credentials in each region.

In standard multi-region RHOSO deployments, each region is isolated with its own Identity (keystone) and Dashboard (horizon) services. This requires separate user accounts for each region, making credential management and rotation difficult.

Single Keystone Multiple OpenStacks multi-region deployment architecture
Copy link

Review the architecture of a Single Keystone Multiple OpenStacks (SKMO) multi-region deployment to understand how centralized Identity and Dashboard services simplify user management across the multiple regions.

The SKMO deployment requires the following architecture:

  • A single central region and multiple workload regions. Important
    When you deploy each workload region, you must define a unique RHOSO namespace, a unique region name, and unique Identity service user names for all the OpenStack services that communicate with the Identity service. For more information about the unique networking and configuration requirements of the SKMO deployment, see SKMO deployment requirements and constraints.
  • The central region provides the Dashboard (horizon) service that is shared by all the regions of the SKMO deployment.
  • The central region provides a centralized Identity (keystone) service:
    • You must use the Identity service of the central region to create the default administrator user for each workload region.
    • You must use the Identity service of the central region to create the catalog entries for the public and private endpoints of the Identity service for each workload region.
  • The centralized Identity and Dashboard services provide a single pane of glass for the simplified configuration and management of the users. Each user has a single set of credentials. You can enable or disable their access to every region in the central region. For more information, see Deploy SKMO.

SKMO deployment requirements and constraints
Copy link

Plan your Single Keystone Multiple OpenStacks (SKMO) multi-region deployment by reviewing the unique namespace, networking, and configuration requirements for the interdependent regional architecture.

The central region provides centralized Dashboard (horizon) and Identity (keystone) services that are relied upon by all the workload regions. You must implement the following requirements for a successful SKMO deployment:

  1. The interdependence between the workload and the central regions requires that every region must provide the following unique identifications for a successful SKMO deployment:
    • A unique namespace to differentiate the RHOSO deployment of each region. For more information, see Create a unique namespace for each workload region. Important
      The RHOSO deployment namespace forms part of the DNS name for each OpenStack service. If you do not use different RHOSO namespaces for every region, conflicts occur between services in your different regions.
    • A unique region name defined by the spec of their Identity (keystone) service in the OpenStackControlPlane custom resource (CR). For more information, see Connect each workload region control plane to the shared central region services. Note
      When the central Red Hat OpenStack Services on OpenShift (RHOSO) region is deployed, this region is called regionOne by default. If you use a workload region naming convention, then you can rename the region name of the central region to make it more easily identifiable. For more information, see Rename the central region.
    • The OpenStack services that communicate with the Identity service must use uniquely named Identity service users to simplify the task of managing their individual credentials. For more information, see Connect each workload region control plane to the shared central region services. Warning
      If you do not specify unique Identity service user names for all the OpenStack services that communicate with the Identity service, then changing the password for a service user disrupts this service in all the workload regions that use this same user - unless you schedule a maintenance window for all of these regions first.
  2. The interdependence between the OpenStack services in each workload region and the central region requires a successful SKMO deployment to implement Red Hat Service Interconnect (Skupper). For more information, see Deploy Red Hat Service Interconnect for SKMO.

Deploy SKMO
Copy link

Deploy a Single Keystone Multiple OpenStacks (SKMO) multi-region environment with centralized Identity (keystone) and Dashboard (horizon) services so that you can manage all users and their regional access from a single location.

Before you begin
Copy link

About this task
Copy link

Each user has a single set of credentials and their access to every workload region can be enabled or disabled in the central region.

Note
You must not manually configure the Dashboard (horizon) service of the central region to connect to the various workload regions because a Managing regions dropdown list is added to the UI automatically for a SKMO deployment to allow users to select the required workload regions. For more information, see Automatic region selection in SKMO Dashboard.

Procedure
Copy link

  1. You must create a unique namespace for every region of your SKMO deployment. For more information, see Create a unique namespace for each workload region.
    Note
    The default namespace of the central region is openstack.
  2. Deploy the central region called regionOne by default, unless you rename it. For more information, see Rename the central region.

    The deployment of the central region does not require a data plane.

  3. Create the default administrator Identity service user for each workload region in the central region. These Identity service users must be granted the admin role in the admin project of the central region. For more information, see Create the default administrator user for each workload region.
  4. Install Red Hat Service Interconnect (Skupper) and create and link Sites between the central and all the workload namespaces. For more information, see Deploy Red Hat Service Interconnect for SKMO.
  5. Expose the private endpoint of the OpenStack Identity service (keystone) of the central region through Red Hat Service Interconnect (Skupper) as a private virtual service inside the namespace of each workload region. For more information, see Route the private workload region endpoints through Skupper.
  6. Create the catalog entries for the public and private endpoints of the Identity service in each workload region by using the Identity service in the central region. For more information, see Create the public and private endpoints of each workload region.
  7. Modify and deploy each workload region. For more information, see Connect each workload region control plane to the shared central region services.
    Important
    An important part of this deployment modification involves creating a unique region name and unique Identity service users for each workload region.
  8. After you deploy a workload region of a multi-region Red Hat OpenStack Services on OpenShift (RHOSO) Single Keystone Multiple OpenStacks (SKMO) deployment you must configure the deployed central region to trust this workload region. For more information, see Configure the central region to trust each deployed workload region.
  9. Red Hat Service Interconnect (Skupper) must connect the central RabbitMQ service to each workload namespace so that the barbican-keystone-listener service in each workload region can know when projects are deleted so that the Key Manager service can clean them up. For more information, see Connect the central RabbitMQ service to each workload namespace with Skupper,
  10. If your workload region data plane contains compute nodes then you must expose the routed Red Hat Service Interconnect (Skupper) endpoint to these nodes that they must be able to resolve and connect to on startup. For more information, see Expose the routed Skupper endpoint to data plane compute nodes.
  11. You can configure the OpenStack service users in your workload regions of your SKMO deployment to use application credentials for their authentication for the multiple security benefits they provide such as near zero downtime password rotation. For more information, see Configure application credentials to authenticate OpenStack services in workload regions.
    Note
    Do this before you deploy the data planes of the workload regions, otherwise you must redeploy the data planes again to adopt the application credentials.
  12. Deploy the data planes of the workload regions.

Create a unique namespace for each workload region
Copy link

You must create a unique namespace for each workload region of your SKMO multi-region deployment to differentiate services and avoid DNS conflicts between regions.

Before you begin
Copy link

  • You are logged on to a workstation that has access to the RHOCP cluster, as a user with cluster-admin privileges.
  • You have the oc command line tool installed on your workstation.

About this task
Copy link

Important
The RHOSO deployment namespace forms part of the DNS name for each OpenStack service. If you do not use different namespaces for every region, conflicts occur between services in your different regions.

Procedure
Copy link

  1. Create a project in your deployed RHOSO environment: 
    $ oc new-project <workload-region-namespace>

    where:

    <workload-region-namespace>
    Specifies the name of the unique namespace for each workload region, for example openstack-two.
  2. Ensure that this namespace is labeled to enable privileged pod creation by the OpenStack Operators: 
    $ oc get namespace <workload-region-namespace> -ojsonpath='{.metadata.labels}' | jq
{
  "kubernetes.io/metadata.name": "<workload-region-namespace>",
  "pod-security.kubernetes.io/enforce": "privileged",
  "security.openshift.io/scc.podSecurityLabelSync": "false"
}
  3. If the security context constraint (SCC) is not "privileged" use the following commands to change it: 
    $ oc label ns <workload-region-namespace> security.openshift.io/scc.podSecurityLabelSync=false --overwrite
$ oc label ns <workload-region-namespace> pod-security.kubernetes.io/enforce=privileged --overwrite

Rename the central region
Copy link

Rename the central region from its default name of regionOne so that it is more easily identifiable alongside your workload regions.

Before you begin
Copy link

  • You are logged on to a workstation that has access to the RHOSO control plane as a user with cluster-admin privileges.
  • You have the oc command line tool installed on your workstation.

Procedure
Copy link

  1. In the central region, set the default RHOSO namespace: 
    $ oc project <central-region-namespace>

    where:

    <central-region-namespace>
    Specifies the name of the unique namespace for the central region, for example openstack.
  2. Edit the OpenStackControlPlane CR of your central region on your workstation: 
    $ oc edit openstackcontrolplane <name>

    where:

    <name>
    Specifies the name of your YAML OpenStackControlPlane CR. You can use the following command to retrieve this name: oc get openstackcontrolplane.
  3. Configure the region parameter of the Identity service (keystone): 
    ...
  spec:
    ...
    keystone:
      ...
      template:
        ...
        region: <central-region-name>

    where:

    <central-region-name>
    Specifies the region name for your central region.
  4. Save and close the editor to apply this change.
  5. Wait for the deployment of the control plane to reach the Ready status: 
    $ oc wait openstackcontrolplane <name> --for=condition=Ready --timeout=600s

Create the default administrator user for each workload region
Copy link

Create the default administrator user for each workload region by using the Identity service (keystone) of the central region in a Single Keystone Multiple OpenStacks (SKMO) deployment so that each region has its own admin credentials for management.

Before you begin
Copy link

  • You are logged on to a workstation that has access to the RHOSO control plane as a user with cluster-admin privileges.
  • You have the oc command line tool installed on your workstation.

About this task
Copy link

These workload administrator users must be granted the admin role in the admin project of the central region. For more information, see Deploy SKMO.

Procedure
Copy link

  1. In the central region, set the default RHOSO namespace: 
    $ oc project <central-region-namespace>

    where:

    <central-region-namespace>
    Specifies the name of the unique namespace for the central region, for example openstack.
  2. Access the remote shell for the OpenStackClient pod from your workstation to run OpenStack CLI commands: 
    $ oc rsh openstackclient
  3. Create the default administrator user for each workload region: 
    $ openstack user create --domain Default --project <central-region-admin-project> --project-domain Default --password <workload-region-admin-password> <workload-region-admin-name>

    where:

    <central-region-admin-project>
    Specifies the name of the admin project in the central region that is admin by default.
    <workload-region-admin-password>

    Specifies the password of the default administrator user of each workload region.

    Note
    Set this password as the value of the AdminPassword: parameter of the Secret custom resource (CR) file that you must create to provide secure access to the RHOSO service pods when you deploy each workload region.
    <workload-region-admin-name>
    Specifies the name of the default administrator user of each workload region, for example admin-two.
  4. Add the following roles to each default workload region administrator user: 
    $ openstack role add --project admin --project-domain Default --user <workload-region-admin-name> --user-domain Default admin
$ openstack role add --system all --user <workload-region-admin-name> --user-domain Default admin

Deploy Red Hat Service Interconnect for SKMO
Copy link

Red Hat Service Interconnect (Skupper) creates a network of linked Skupper Sites to connect the OpenStack services of the central and the workload regions together.

Before you begin
Copy link

  • You are logged on to a workstation that has access to the Red Hat OpenShift Container Platform (RHOCP) cluster, as a user with cluster-admin privileges.
  • You have the oc command line tool installed on your workstation.
  • You have created unique RHOSO namespaces for the central and workload regions. For more information, see Create a unique namespace for each workload region.
  • You require an active Red Hat customer subscription for Red Hat Application Services.

About this task
Copy link

Note
This same procedure is required whether the central and workload regions run in one OpenShift cluster by using different namespaces or in separate OpenShift clusters.

Procedure
Copy link

  1. Open the Red Hat Product Downloads page on the Customer Portal.
  2. Scroll down to the Integration and Automation section.
  3. Select Red Hat Service Interconnect, which is currently the last option in this list.
  4. Scroll down to the Skupper Deployments ( Yamls ) installation option of the latest release of Red Hat Service Interconnect.
  5. Click the Download Now button.
  6. Extract the skupper-cluster-scope-<latest-version>-rh.yaml file from the downloaded tarball.

    where:

    <latest-version>
    Specifies the latest three digit version of Red Hat Service Interconnect. In this example this is 2.2.1.
  7. Apply this YAML file on every OpenShift cluster: 
    $ oc apply -f skupper-cluster-scope-2.2.1-rh.yaml
  8. Wait for the Skupper operator to be ready in the skupper namespace: 
    $ oc -n skupper rollout status deployment --timeout=5m
  9. Verify that this Skupper operator is deployed: 
    $ oc -n skupper get deploy
# NAME                         READY   UP-TO-DATE   AVAILABLE
skupper-controller-manager   1/1     1            1         (downstream)
  10. Create a YAML file on your workstation, for example, skupper-site-central.yaml with the following contents to define the Skupper Site Custom Resource (CR) in the namespace of the central region, which requires linkAccess so that this site can issue tokens: 
    apiVersion: skupper.io/v2alpha1
kind: Site
metadata:
  name: <skupper-site-central-name>
  namespace: <central-region-namespace>
spec:
  linkAccess: route

    where:

    <skupper-site-central-name>
    Specifies the name of the Skupper Site in the central region, which in this example is openstack the same as the namespace.
    <central-region-namespace>
    Specifies the namespace of the central region, which in this example is openstack.
  11. Apply this file in the namespace of the central region to create the central Skupper Site: 
    $ oc apply -n <central-region-namespace> -f skupper-site-central.yaml
  12. Wait for the central Skupper Site to be Ready: 
    $ oc -n <central-region-namespace> wait site/<skupper-site-central-name> --for='condition=Ready' --timeout=5m
  13. Create a YAML file on your workstation, for example, skupper-site-workload.yaml with the following contents to define the Skupper Site CR in the namespace of the workload region, which does not require linkAccess: 
    apiVersion: skupper.io/v2alpha1
kind: Site
metadata:
  name: <skupper-site-workload-name>
  namespace: <workload-region-namespace>
spec:
  linkAccess: none

    where:

    <skupper-site-workload-name>
    Specifies the name of the Skupper Site in the workload region, which in this example is openstack2 the same as the namespace.
    <workload-region-namespace>
    Specifies the namespace of the workload region, which in this example is openstack2.
  14. Apply this file in the namespace of this workload region to create this workload Skupper Site: 
    $ oc apply -n <workload-region-namespace> -f skupper-site-workload.yaml
    Note
    You must define and apply a separate file to create a separate Skupper Site for each of your workload regions.
  15. Wait for this workload Skupper Site to be Ready: 
    $ oc -n <workload-region-namespace> wait site/<skupper-site-workload-name> --for='condition=Ready' --timeout=5m
    Note
    You must repeat this process for every workload region.
  16. Create a YAML file on your workstation, for example, skupper-access-grant-link-to-openstack2.yaml with the following contents to define the Skupper AccessGrant CR in the namespace of the central region: 
    apiVersion: skupper.io/v2alpha1
kind: AccessGrant
metadata:
  name: <skupper-workload-access-grant-name>
  namespace: <central-region-namespace>
spec:
  redemptionsAllowed: 1
  expirationWindow: 15m

    where:

    <skupper-workload-access-grant-name>
    Specifies the name of the Skupper AccessGrant in the central region, which. For example, link-to-openstack2 because the AccessGrant holds a one-time token that a specific workload Skupper Site redeems, which in this example is called openstack2.
  17. Apply this file in the namespace of the central region to create the Skupper AccessGrant: 
    $ oc apply -n <central-region-namespace> -f skupper-access-grant-link-to-openstack2.yaml
  18. Wait for the Skupper AccessGrant to be Ready: 
    $ oc -n <central-region-namespace> wait accessgrant/<skupper-workload-access-grant-name> \ --for='condition=Ready' --timeout=5m
  19. Create a YAML file on your workstation, for example, skupper-access-token-for-openstack2.yaml with the following contents to define the Skupper AccessToken CR in the namespace of the workload region. This access token contains values extracted from the access grant: 
    apiVersion: skupper.io/v2alpha1
kind: AccessToken
metadata:
  name: <skupper-access-token-name>
  namespace: <workload-region-namespace>
spec:
  url: "<url>"
  ca: "<ca>"
  code: "<code>"

    where:

    <skupper-access-token-name>
    Specifies the name of the Skupper AccessToken in the workload region, for example, link-to-openstack.
    <url>
    Specifies the URL extracted from the access grant by using the following command: oc -n <central-region-namespace> get accessgrant/<skupper-workload-access-grant-name> \ -o jsonpath='{.status.url}'.
    <ca>
    Specifies the CA extracted from the access grant by using the following command: oc -n <central-region-namespace> get accessgrant/<skupper-workload-access-grant-name> \ -o jsonpath='{.status.ca}'.
    <code>
    Specifies the code extracted from the access grant by using the following command: oc -n <central-region-namespace> get accessgrant/<skupper-workload-access-grant-name> \ -o jsonpath='{.status.code}'.
  20. Apply this file in the namespace of this workload region to consume this workload Skupper AccessToken to create a Skupper Link between this workload region and the central region: 
    $ oc apply -n <workload-region-namespace> -f skupper-access-token-for-openstack2.yaml
  21. Wait for the Skupper Link between this workload region and the central region to be Ready: 
    $ oc -n <workload-region-namespace> wait link --all --for='condition=Ready' --timeout=5m
  22. Verify that the Skupper Sites in the namespace of the central region and this workload region are ready: 
    $ oc get site -n <central-region-namespace> 
$ oc get site -n <workload-region-namespace>
  23. Verify that a Skupper Link is displayed in the namespace of this workload region: 
    $ oc get link -n <workload-region-namespace>

Route the private workload region endpoints through Skupper
Copy link

Use Red Hat Service Interconnect (Skupper) to expose the private endpoint of the OpenStack Identity service (keystone) of the central region as a private virtual service inside the namespace of each workload region.

Before you begin
Copy link

  • You are logged on to a workstation that has access to the Red Hat OpenShift Container Platform (RHOCP) cluster, as a user with cluster-admin privileges.
  • You have the oc command line tool installed on your workstation.
  • cert-manager is running on the RHOCP cluster
  • Red Hat Service Interconnect (Skupper) is installed and Sites are linked between the central and all the workload namespaces. For more information, see  Deploy Red Hat Service Interconnect for SKMO.
  • You have deployed the control plane of the central region.
  • You have not deployed the control plane of the workload region.

About this task
Copy link

Without using Skupper the private endpoints of each workload region must connect to the public endpoint of the OpenStack Identity service (keystone) of the central region. In this case, even though the internal service-to-service communication traffic of the workload regions is encrypted it is more vulnerable to DDOS attacks because it is not isolated on a separate internal network making it easier for external attackers to intercept these messages.

By using Skupper the service-to-service authentication traffic travels over the Skupper mTLS application-network tunnel and never leaves the cluster network.

Procedure
Copy link

  1. Create a YAML file on your workstation, for example, skupper-connector-keystone-internal.yaml with the following contents to define the Skupper Connector Custom Resource (CR) in the namespace of the central region: 
    apiVersion: skupper.io/v2alpha1
kind: Connector
metadata:
  name: keystone-internal
  namespace: <central-region-namespace>
spec:
  routingKey: keystone-internal
  host: keystone-internal.<central-region-namespace>.svc.cluster.local
  port: 5000
  type: tcp
  tlsCredentials: <keystone-tls-secret>
  verifyHostname: false

    where:

    <central-region-namespace>
    Specifies the namespace of the central region, which in this example is openstack.
    <keystone-tls-secret>
    Specifies the TLS Secret name that Keystone uses for its internal endpoint that you can get by using the following command: oc -n <central-region-namespace> get keystoneapi keystone \ -o jsonpath='{.spec.tls.api.internal.secretName}'.
  2. Apply this file in the namespace of the central region to create the Skupper Connector: 
    $ oc apply -n <central-region-namespace> -f skupper-connector-keystone-internal.yaml
  3. Wait for the Skupper Connector to report the Configured: True status: 
    $ oc -n <central-region-namespace> wait connector keystone-internal \
  --for=jsonpath='{.status.conditions[?(@.type=="Configured")].status}'=True \
  --timeout=5m
  4. Create a YAML file on your workstation, for example, skupper-listener-cert-keystone.yaml with the following contents to define the Skupper Listener TLS certificate CR in the namespace of the workload region: 
    apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: skupper-keystone-regionone
  namespace: <workload-region-namespace>
spec:
  secretName: cert-skupper-keystone-regionone
  issuerRef:
    name: rootca-internal
    kind: Issuer
  dnsNames:
    - <skupper-listener-host>
    - <skupper-listener-host>.<workload-region-namespace>.svc
    - <skupper-listener-host>.<workload-region-namespace>.svc.cluster.local
  usages:
    - digital signature
    - key encipherment
    - server auth

    where:

    <workload-region-namespace>
    Specifies the namespace of the workload region, for example openstack2.
    <skupper-listener-host>
    Specifies the hostname used in the DNS names for this certificate, which must match the hostname used when creating the Skupper Listener CR. In this example this is keystone-regionone.
  5. Apply the Skupper Listener TLS certificate CR file in the namespace of the workload region: 
    $ oc apply -n <workload-region-namespace> -f skupper-listener-cert-keystone.yaml
  6. Wait for cert-manager to issue the certificate: 
    $ oc -n <workload-region-namespace> wait certificate skupper-keystone-regionone \ --for=condition=Ready --timeout=2m
  7. Create a YAML file on your workstation, for example, skupper-listener-keystone-internal.yaml with the following contents to define the Skupper Listener CR in the namespace of the workload region: 
    apiVersion: skupper.io/v2alpha1
kind: Listener
metadata:
  name: keystone-internal
  namespace: <workload-region-namespace>
spec:
  routingKey: keystone-internal
  host: <skupper-listener-host>
  port: 5000
  type: tcp
  tlsCredentials: cert-skupper-keystone-regionone
    • spec.tlsCredentials specifies the spec.secretName of the Skupper Listener TLS certificate CR, in this example, cert-skupper-keystone-regionone
  8. Apply the Skupper Listener CR file in the namespace of the workload region: 
    $ oc apply -n <workload-region-namespace> -f skupper-listener-keystone-internal.yaml

What to do next
Copy link

  1. Modify the control plane CR of each workload region and specify the Skupper virtual internal endpoint of the Identity service (keystone). For more information, see Connect each workload region control plane to the shared central region services.
  2. If the data plane of a workload region contains compute nodes, then before you deploy this data plane you must expose the routed Skupper endpoint to the data plane compute nodes. For more information, see Expose the routed Skupper endpoint to data plane compute nodes.
  3. Once Skupper OpenStack Identity service routing is fully configured and the control plane and data plane of the workload region have been deployed, the OpenStack service users can use application credentials for authentication to enable near zero downtime password rotation. For more information, see Configure application credentials to authenticate OpenStack services in workload regions.

Create the public and private endpoints of each workload region
Copy link

Create the public and private Identity service (keystone) endpoints for each workload region in a Single Keystone Multiple OpenStacks (SKMO) deployment so that all the workloadregions can authenticate through the central region.

Before you begin
Copy link

  • You are logged on to a workstation that has access to the RHOSO control plane as a user with cluster-admin privileges.
  • You have the oc command line tool installed on your workstation.
  • You have routed the private workload region endpoints through Red Hat Service Interconnect (Skupper). For more information, see Route the private workload region endpoints through Skupper.

Procedure
Copy link

  1. In the central region, set the default RHOSO namespace: 
    $ oc project <central-region-namespace>

    where:

    <central-region-namespace>
    Specifies the name of the unique namespace for the central region, for example openstack.
  2. Access the remote shell for the OpenStackClient pod from your workstation to run OpenStack CLI commands: 
    $ oc rsh openstackclient
  3. Obtain the public Identity service endpoint of the central region: 
    $ openstack endpoint list --region <central-region-name> --service keystone --interface public

    where:

    <central-region-name>
    Specifies the name of your central region, which is regionOne by default.
  4. Copy this URL that is referenced in this procedure as <central-region-public-keystone-url>.
  5. Create the public endpoint of each workload region to specify the public Identity service endpoint of the central region: 
    $ openstack endpoint create --region <workload-region-name> keystone public <central-region-public-keystone-url>

    where:

    <workload-region-name>
    Specifies the name of the required workload region, for example regionTwo.
  6. Create the private endpoints of each workload region to specify the private virtual service created by Skupper inside the namespace of each workload region: 
    $ openstack endpoint create --region <workload-region-name> keystone internal https://keystone-regionone.<workload-region-namespace>.svc.cluster.local:5000

    where:

    <workload-region-namespace>
    Specifies the namespace of the required workload region, for example openstack2.

    This creates the catalog entries for the public and private endpoints of the Identity service in each workload region in the Identity service catalog of the central region.

Modify the workload region deployment to share the central region services
Copy link

Modify the deployment of each workload region to share the Dashboard (horizon) and Identity (keystone) services of the central region across all workload regions.

About this task
Copy link

This deployment modification involves creating a unique region name and unique Identity service users for each workload region. For more information, see SKMO deployment requirements and constraints and Deploy SKMO.

Procedure
Copy link

  1. Create a unique RHOSO deployment namespace for each workload region to differentiate these RHOSO deployments from each other. For more information, see Create a unique namespace for each workload region.
  2. Specify the password of the administrator user that you manually created for each workload region as the value of the AdminPassword: parameter when you create the Secret custom resource (CR) to provide secure access to the RHOSO service pods in each workload region. For more information, see Create the default administrator user for each workload region and Provide secure access to services.
  3. Obtain the CA certificate from the central region and add it to a secret in each workload region so it can be trusted. For more information, see Configure each workload region to trust the central region.
  4. Create a modified OpenStackControlPlane custom resource (CR) for each workload region. For more information, see Connect each workload region control plane to the shared central region services.

    This involves defining a unique region name defined in the spec of the Identity (keystone) service for each workload region and creating unique Identity service users.

Configure the central region to trust each deployed workload region
Copy link

Configure the central region of your SKMO multi-region deployment to trust a deployed workload region so that the shared Identity service can communicate securely with the services of the workload region.

Before you begin
Copy link

  • You are logged on to a workstation that has access to your Red Hat OpenShift Container Platform (RHOCP) cluster as a user with cluster-admin privileges.
  • You have the oc command line tool installed on your workstation.

Procedure
Copy link

  1. In the deployed workload region, set the default RHOSO namespace: 
    $ oc project <workload-region-namespace>

    where:

    <workload-region-namespace>
    Specifies the name of the unique namespace for this workload region, for example openstack-two.
  2. Obtain the CA certificate of this workload region and extract it into a file, for example regionTwo-ca.crt:
    Note
    To decode the certificate before creating the output .crt file add | base64 -d to this command.
     
    $ oc get secret rootca-public -o yaml | yq '.data."ca.crt"' > regionTwo-ca.crt
  3. Copy the regionTwo-ca.crt file to the deployed central region.
  4. In the central region, set the default RHOSO namespace: 
    $ oc project <central-region-namespace>

    where:

    <central-region-namespace>
    Specifies the name of the unique namespace for the central region, for example openstack.
  5. Edit your OpenStackControlPlane CR of the central region: 
    $ oc edit openstackcontrolplane <name>

    where:

    <name>
    Specifies the name of your YAML OpenStackControlPlane CR. You can use the following command to retrieve this name: oc get openstackcontrolplane.
  6. If your OpenStackControlPlane CR in the central region contains the spec.tls.caBundleSecretName parameter:
    1. Obtain the name of the secret that contains the PEM-formatted bundle containing all the other custom CA certificates in chains of trust if applicable that the central region trusts, for example custom-ca-certs: 
      ...
  spec:
    ...
    tls:
      ...
      caBundleSecretName: custom-ca-certs
    2. Edit the specified secret, in this example custom-ca-certs 
      $ oc edit secret custom-ca-certs
    3. Append the contents of the CA certificate of the deployed workload region, for example regionTwo-ca.crt to the PEM-formatted bundle containing all the other custom CA certificates that the central region trusts.
    4. Save and exit the editor to automatically apply the changes to the secret, in this example custom-ca-certs.
  7. If your OpenStackControlPlane CR in the central region does not contain the spec.tls.caBundleSecretName parameter:
    1. Create a PEM-formatted bundle, for example custom-ca-certs.pem that includes the contents of this regionTwo-ca.crt file.
    2. Create a manifest file for the secret in the central region that specifies the contents of the custom-ca-certs.pem bundle created in the previous step. In this example the manifest file is called custom-ca-certs.yaml and the secret is called custom-ca-certs: 
      apiVersion: v1
data:
  custom-ca-certs.pem: <contents-of-PEM-bundle>
kind: Secret
metadata:
  annotations:
  name: custom-ca-certs
  namespace: <namespace>
type: Opaque

      where:

      <namespace>
      Specifies the namespace of the central region, in this example openstack.
      <contents-of-PEM-bundle>
      Specifies the base64 encoded string of the contents of the PEM-formatted bundle you created called custom-ca-certs.pem that includes the CA certificate from regionTwo. You can get this base64 encoded string by using the following command: cat custom-ca-certs.pem | base64 -w0.
    3. Create the secret in the central region from the manifest file. In this example the manifest file is called custom-ca-certs.yaml: 
      $ oc apply -f custom-ca-certs.yaml
    4. Edit your OpenStackControlPlane CR of the central region: 
      $ oc edit openstackcontrolplane <name>

      where:

      <name>
      Specifies with the name of your YAML OpenStackControlPlane CR. You can use the following command to retrieve this name: oc get openstackcontrolplane.
    5. Add the secret that you have created, in this example custom-ca-certs: 
      ...
  spec:
    ...
    tls:
      ...
      caBundleSecretName: custom-ca-certs
    6. Save and close the editor to automatically apply this change.
    7. Wait for the deployment of the control plane of the central region to reach the Ready status: 
      $ oc wait openstackcontrolplane <name> --for=condition=Ready --timeout=600s

What to do next
Copy link

  • Extract the catalog entries for this workload region and make sure that the Dashboard (horizon) service in the central region can resolve the DNS names in the service catalog of this workload region and access them.

Connect the central RabbitMQ service to each workload namespace with Skupper
Copy link

Red Hat Service Interconnect (Skupper) must connect the central RabbitMQ service to each workload namespace so that the barbican-keystone-listener service in each workload region can know when projects are deleted so that the Key Manager service can clean them up.

Before you begin
Copy link

  • You are logged on to a workstation that has access to the Red Hat OpenShift Container Platform (RHOCP) cluster, as a user with cluster-admin privileges.
  • You have the oc command line tool installed on your workstation.
  • Red Hat Service Interconnect (Skupper) is installed and Sites are linked between the central and all the workload namespaces. For more information, see  Deploy Red Hat Service Interconnect for SKMO.
  • You have deployed the control plane of the central region.
  • You have deployed the control plane of the workload region.

Procedure
Copy link

  1. Create a YAML file on your workstation, for example, skupper-connector-central.yaml with the following contents to define the Skupper Connector Custom Resource (CR) in the namespace of the central region: 
    apiVersion: skupper.io/v2alpha1
kind: Connector
metadata:
  name: <skupper-connector-name>
  namespace: <central-region-namespace>
spec:
  routingKey: <skupper-routing-key>
  host: <rabbitmq-host>
  port: 5671
  type: tcp
  tlsCredentials: <rabbitmq-cert-secret>
  verifyHostname: true

    where:

    <skupper-connector-name>
    Specifies the Skupper Connector CR name, which should be the same as the Skupper Listener CR name for ease of reference, which in this example is: rabbitmq-keystone.
    <central-region-namespace>
    Specifies the namespace of the central region, which in this example is openstack.
    <skupper-routing-key>
    Specifies an arbitrary Skupper Routing Key for the RabbitMQ Service to identify the Connectors and associated Listeners in the Skupper network. In this example, this is rabbit-keystone.
    <rabbitmq-host>
    Specifies the internal DNS entry for the RabbitMQ Service in the central region, which is: rabbitmq.<central-region-namespace>.svc.cluster.local.
    <rabbitmq-cert-secret>
    Specifies the name of the OpenShift secret containing the TLS certificate used for the RabbitMQ service in the central region, which you can obtain from the following command: oc -n <central-region-namespace> get rabbitmqs rabbitmq -o jsonpath='{.spec.tls.secretName}{"\n"}'.
  2. Apply this file in the namespace of the central region to create the central Skupper Connector: 
    $ oc apply -n <central-region-namespace> -f skupper-connector-central.yaml
  3. Create a YAML file on your workstation, for example, skupper-listener-workload-cert.yaml with the following contents to define the Skupper Listener certificate CR in the namespace of the workload region: 
    apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: <skupper-listener-cert-name>
  namespace: <workload-region-namespace>
spec:
  secretName: <skupper-listener-cert-secret>
  issuerRef:
    name: rootca-internal
    kind: Issuer
  dnsNames:
    - <skupper-listener-host>
    - <skupper-listener-host>.<workload-region-namespace>.svc
    - <skupper-listener-host>.<workload-region-namespace>.svc.cluster.local
  usages:
    - digital signature
    - key encipherment
    - server auth

    where:

    <skupper-listener-cert-name>
    Specifies the Skupper Listener certificate CR name, which even though it is in the namespace of the workload region, in this example this is skupper-rabbitmq-regionone.
    <workload-region-namespace>
    Specifies the namespace of the workload region, which in this example is openstack2.
    <skupper-listener-cert-secret>
    Specifies the name of the secret used to store the new certificate. In this example this is cert-skupper-rabbitmq-regionone to match the CR name.
    <skupper-listener-host>
    Specifies the hostname used in the DNS names for this certificate, which must match the hostname used when creating the Skupper Listener CR. In this example this is rabbitmq-regionone.
  4. Apply this file in the namespace of this workload region to create the Skupper Listener workload certificate: 
    $ oc apply -n <workload-region-namespace> -f skupper-listener-workload-cert.yaml
  5. Create a YAML file on your workstation, for example, skupper-listener-workload.yaml with the following contents to define the Skupper Listener CR in the namespace of the workload region: 
    apiVersion: skupper.io/v2alpha1
kind: Listener
metadata:
  name: <skupper-listener-name>
  namespace: <workload-region-namespace>
spec:
  routingKey: <skupper-routing-key>
  host: <skupper-listener-host>
  port: 5671
  type: tcp
  tlsCredentials: <skupper-listener-cert-secret>

    where:

    <skupper-listener-name>
    Specifies the Skupper Listener CR name, which should be the same as the Skupper Connector CR name for ease of reference, which in this example is: rabbitmq-keystone.
  6. Apply this file in the namespace of this workload region to create the Skupper Listener for this workload region: 
    $ oc apply -n <workload-region-namespace> -f skupper-listener-workload.yaml
  7. Verify that the status of the Skupper Connector in the central region is Ready: 
    $ oc get connectors.skupper.io

    Example output:

    NAME                ROUTING KEY       PORT   HOST                                   SELECTOR   STATUS   HAS MATCHING LISTENER   MESSAGE
rabbitmq-keystone   rabbit-keystone   5671   rabbitmq.openstack.svc.cluster.local              Ready    true                    OK
  8. Verify that the status of the Skupper Listener in the workload region is Ready: 
    $ oc get listeners.skupper.io

    Example output:

    NAME                ROUTING KEY       PORT   HOST                 STATUS   HAS MATCHING CONNECTOR   MESSAGE
rabbitmq-keystone   rabbit-keystone   5671   rabbitmq-regionone   Ready    true                     OK
  9. Set the namespace of the central region: 
    $ oc project <central-region-namespace>
  10. Edit the OpenStackControlPlane CR of the central region on your workstation: 
    $ oc edit openstackcontrolplane <central-oscp-name>

    where:

    <central-oscp-name>
    Specifies the name of the YAML OpenStackControlPlane CR, which you can use the following command to retrieve: oc get openstackcontrolplane.
  11. Add the following configuration to your OpenStackControlPlane CR in the central region to create a unique queue for the barbican-keystone-listener service, which in this example is called barbican-listener-regionone: 
    ...
 spec:
 ...
    barbican:
    ...
      template:
      ...
        barbicanKeystoneListener:
          customServiceConfig: |
            [keystone_notifications]
            pool_name = barbican-listener-regionone
  12. Save and close the editor to apply these changes
  13. Wait for the deployment of the control plane of the central region to reach the Ready status: 
    $ oc wait openstackcontrolplane <central-oscp-name> --for=condition=Ready --timeout=600s
  14. Create a YAML file on your workstation, for example, barbican-regiontwo-transport.yaml with the following contents to define the TransportURL CR in the namespace of the central region, which in this example is openstack2. This CR creates a new RabbitMQ username and password to configure the barbican-keystone-listener service in the workload region, which in this example is called regiontwo: 
    apiVersion: rabbitmq.openstack.org/v1beta1
kind: TransportURL
metadata:
  name: <transport-url-name>
  namespace: <central-region-namespace>
spec:
  rabbitmqClusterName: rabbitmq
  username: <transport-url-username>

    where:

    <transport-url-name>
    Specifies the TransportURL CR name, which in this example is: barbican-regiontwo.
    <transport-url-username>
    Specifies the username of this TransportURL, which in this example is: barbican-keystone-listener-regiontwo.
  15. Apply this file in the namespace of this workload region to create the TransportURL CR: 
    $ oc apply -n <central-region-namespace> -f barbican-regiontwo-transport.yaml

    The TransportURL CR should create the following two Secrets:

    • A Secret that contains the newly created TransportURL connection string, which adheres to this naming convention: rabbitmq-transport-url-<transport-url-name>, which in this example is called rabbitmq-transport-url-barbican-regiontwo.
    • A Secret that contains the credentials for the new RabbitMQ user created for the new TransportURL, which adheres to this naming convention: rabbitmq-user-<transport-url-name>-<transport-url-username>-user, which in this example is called rabbitmq-user-barbican-regiontwo-barbican-keystone-listener-regiontwo-user.
  16. Set the namespace of the workload region: 
    $ oc project <workload-region-namespace>
  17. Edit the OpenStackControlPlane CR of the workload region on your workstation: 
    $ oc edit openstackcontrolplane <workload-oscp-name>

    where:

    <workload-oscp-name>
    Specifies the name of the YAML OpenStackControlPlane CR, which you can use the following command to retrieve: oc get openstackcontrolplane.
  18. Add the following configuration to your OpenStackControlPlane CR in the workload region to use the new RabbitMQ username and password created to configure the barbican-keystone-listener service in the workload region and to create a unique queue for this service: 
    spec:
...
  barbican:
  ...
    template:
    ...
      barbicanKeystoneListener:
        customServiceConfig: |
          [DEFAULT]
          transport_url = rabbit://<rabbitmq-username>:<rabbitmq-password>@<skupper-listener-host>:5671/?ssl=1
          [keystone_notifications]
          pool_name = <barbican-keystone-listener-pool-name>

    where:

    <rabbitmq-username>
    You can retrieve this RabbitMQ username from the Secret that the TransportURL created containing the credentials for the new RabbitMQ user, which in this example is called rabbitmq-user-barbican-regiontwo-barbican-keystone-listener-regiontwo-user, by using the following command: oc get secret/rabbitmq-user-barbican-regiontwo-barbican-keystone-listener-regiontwo-user -o json | jq -r .data.username | base64 -d.
    <rabbitmq-password>
    You can retrieve this RabbitMQ password from the Secret that the TransportURL created containing the credentials for the new RabbitMQ user, which in this example is called rabbitmq-user-barbican-regiontwo-barbican-keystone-listener-regiontwo-user, by using the following command: oc get secret/rabbitmq-user-barbican-regiontwo-barbican-keystone-listener-regiontwo-user -o json | jq -r .data.password | base64 -d.
    <barbican-keystone-listener-pool-name>
    Specifies the unique name of the queue for this barbican-keystone-listener service, which in this example is barbican-listener-regiontwo.
  19. Save and close the editor to apply these changes
  20. Wait for the deployment of the control plane of the workload region to reach the Ready status: 
    $ oc wait openstackcontrolplane <workload-oscp-name> --for=condition=Ready --timeout=600s

Expose the routed Skupper endpoint to data plane compute nodes
Copy link

If your workload region data plane contains compute nodes then you must expose the routed Red Hat Service Interconnect (Skupper) endpoint to these nodes that they must be able to resolve and connect to on startup.

Before you begin
Copy link

  • You are logged on to a workstation that has access to the Red Hat OpenShift Container Platform (RHOCP) cluster, as a user with cluster-admin privileges.
  • You have the oc command line tool installed on your workstation.
  • Red Hat Service Interconnect (Skupper) is installed and Sites are linked between the central and all the workload namespaces. For more information, see  Deploy Red Hat Service Interconnect for SKMO.
  • You have routed the private workload region endpoints through Skupper. For more information, see Route the private workload region endpoints through Skupper.
  • You have deployed the control plane of the central region.
  • You have deployed the control plane of the workload region.
  • You have not deployed the data plane of the workload region.
  • The workload region data plane includes compute nodes.
  • MetalLB must be configured with an address pool for the internalapi network of the workload region.
  • A DNSMasq LoadBalancer Service must be serving DNS to the data plane compute nodes of the workload region

About this task
Copy link

Note
Perform this procedure immediately before you deploy the data plane of every workload region but only if it contains compute nodes.

Procedure
Copy link

  1. Create a YAML file on your workstation, for example, keystone-regionone-lb.yaml with the following contents to define the LoadBalancer service Custom Resource (CR) in the namespace of the workload region: 
    apiVersion: v1
kind: Service
metadata:
  name: keystone-regionone-lb
  namespace: <workload-region-namespace>
  annotations:
    metallb.universe.tf/address-pool: <metallb-address-pool>
spec:
  type: LoadBalancer
  selector:
    application: skupper-router
    skupper.io/component: router
  ports:
    - name: keystone-internal
      port: 5000
      protocol: TCP
      targetPort: 1024        # Skupper router application port

    where:

    <workload-region-namespace>
    Specifies the namespace of the workload region, for example openstack2.
    <metallb-address-pool>
    Specifies the configured MetalLB address pool for the for workload region, for example internalapi2.
  2. Apply this file in the namespace of the workload region to create the LoadBalancer service: 
    $ oc apply -n <workload-region-namespace> -f keystone-regionone-lb.yaml
  3. Verify that the LoadBalancer service is assigned: 
    $ oc -n <workload-region-namespace> get svc keystone-regionone-lb
  4. Wait for MetalLB to assign an IP in the namespace of the workload region: 
    $ oc -n <workload-region-namespace> get svc keystone-regionone-lb \ -o jsonpath='{.status.loadBalancer.ingress[0].ip}'
  5. Make note of the specified IP address, for example 172.17.10.87.
  6. Create a YAML file on your workstation, for example, keystone-skupper-dnsdata.yaml with the following contents to define the DNSData CR in the namespace of the workload region: 
    apiVersion: network.openstack.org/v1beta1
kind: DNSData
metadata:
  name: keystone-skupper
  namespace: <workload-region-namespace>
spec:
  dnsDataLabelSelectorValue: dnsdata
  hosts:
    - hostnames:
        - keystone-regionone.<workload-region-namespace>.svc
        - keystone-regionone.<workload-region-namespace>.svc.cluster.local
      ip: <metallb-ip>

    where:

    <metallb-ip>
    Specifies the IP address that MetalLB assigned, which in this example is 172.17.10.87.
  7. Apply this DNSData CR file in the namespace of the workload region: 
    $ oc apply -n <workload-region-namespace> -f keystone-skupper-dnsdata.yaml

    The dnsDataLabelSelectorValue: dnsdata label in the DNSData CR causes the DNSMasq operator to inject the specified host entries into the running dnsmasq pods without a restart.

  8. Verify that the DNSData CR is reconciled: 
    $ oc -n <workload-region-namespace> get dnsdata keystone-skupper

    where:

    placeholder
    Specifies the placeholder .
  9. placeholder

    where:

    placeholder
    Specifies the placeholder .
  10. placeholder

    where:

    placeholder
    Specifies the placeholder .

What to do next
Copy link

  1. Deploy the data plane of the workload region.
  2. Verify DNS resolution from a data plane compute node in the workload region: 
    $ ssh <data-plane-node> getent hosts keystone-regionone.<workload-region-namespace>.svc.cluster.local

    where:

    <data-plane-node>
    Specifies the address of the compute node in the data plane of the workload region.
  3. Verify that the OpenStack Compute service (nova) in the workload region is running and registered: 
    $ oc -n <workload-region-namespace> exec openstackclient -- openstack compute service list \ --service nova-compute

Automatic region selection in SKMO Dashboard
Copy link

In a Single Keystone Multiple OpenStacks (SKMO) multi-region deployment, do not configure the Dashboard (horizon) login page with a region dropdown list because the central region provides a shared Identity (keystone) service for all regions.

In a standard multi-region OpenStack deployment, each region is an isolated deployment with its own Dashboard and Identity service, so the login page includes a dropdown list for users to select the required region. In the SKMO deployment, the central region provides a centralized Identity service for the entire multi-region deployment.

Note
The SKMO Dashboard automatically provides the Managing regions dropdown list in the UI to allow users to select the required workload region.

Configure application credentials to authenticate OpenStack services in workload regions
Copy link

You can configure the OpenStack service users in your workload regions of your SKMO deployment to use application credentials for their authentication for the multiple security benefits they provide such as near zero downtime password rotation (nZDPR).

Before you begin
Copy link

  • You are logged on to a workstation that has access to the Red Hat OpenShift Container Platform (RHOCP) cluster, as a user with cluster-admin privileges.
  • You have the oc command line tool installed on your workstation.
  • You have deployed the control plane of the central region.
  • You have deployed the control plane of the workload region.
  • You have routed the private workload region endpoints through Skupper. For more information, see Route the private workload region endpoints through Skupper.
  • You have not deployed the data plane of the workload region.

About this task
Copy link

Note
Do this before you deploy the data planes of the workload regions, otherwise you must redeploy the data planes again to adopt the application credentials.

Procedure
Copy link

  1. Set the namespace of the workload region: 
    $ oc project <workload-region-namespace>

    where:

    <workload-region-namespace>
    Specifies the namespace of the workload region, for example openstack2.
  2. Edit the OpenStackControlPlane CR of the workload region on your workstation: 
    $ oc edit openstackcontrolplane <name>

    where:

    <name>
    Specifies the name of the YAML OpenStackControlPlane CR, which you can use the following command to retrieve: oc get openstackcontrolplane.
  3. Add the following configuration to use application credentials for the following OpenStack services in the workload region: 
    apiVersion: core.openstack.org/v1beta1
kind: OpenStackControlPlane
metadata:
  name: controlplane
  namespace: <workload-region-namespace>
spec:
  # ... other control plane configuration ...

  applicationCredential:
    enabled: true
    expirationDays: 730
    gracePeriodDays: 182
    roles:
      - service
      - admin
    unrestricted: false

  barbican:
    applicationCredential:
      enabled: true
    # ... other barbican configuration ...

  cinder:
    applicationCredential:
      enabled: true
    # ... other cinder configuration ...

  glance:
    applicationCredential:
      enabled: true
    # ... other glance configuration ...

  neutron:
    applicationCredential:
      enabled: true
    # ... other neutron configuration ...

  nova:
    applicationCredential:
      enabled: true
      roles:
        - admin
        - service
        - member
    # ... other nova configuration ...

  placement:
    applicationCredential:
      enabled: true
    # ... other placement configuration ...
  4. Save and close the editor to apply these changes.
  5. Wait for the deployment of the control plane of the workload region to reach the Ready status: 
    $ oc wait openstackcontrolplane <name> --for=condition=Ready --timeout=600s
  6. Redeploy the data plane to update the application credentials for the Compute (nova) OpenStack service

    To save time you can create an override for the data plane redeployment YAML file to update only the Compute services. In this example this file is called: dataplane-redeploy-appcreds.yaml

    cat > ~/dataplane-redeploy-appcreds.yaml << EOF
apiVersion: dataplane.openstack.org/v1beta1
kind: OpenStackDataPlaneDeployment
metadata:
 name: appcreds-redeploy-$(date +%Y-%m-%d)
 namespace: <workload-region-namespace>
spec:
 backoffLimit: 6
 deploymentRequeueTime: 15
 nodeSets:
 - openstack-edpm
 preserveJobs: true
 servicesOverride:
   - nova
EOF
  7. Update and deploy the overridden data plane deployment: 
    $ oc apply -f dataplane-redeploy-appcreds.yaml
  8. Wait for the data plane deployment to reach the Ready status: 
    $ oc wait openstackdataplanedeployment <name> --for condition=Ready --timeout=1200s

    where:

    <name>
    Specifies the name of your OpenStackDataPlaneDeployment CR. The name of the CR can be different from the YAML file name. Use the following command to retrieve this CR name: oc get openstackdataplanedeployment.
  9. placeholder

    where:

    placeholder
    Specifies the placeholder .
  10. placeholder

    where:

    placeholder
    Specifies the placeholder .

Results
Copy link

  • Run the oc get appcred command to list the OpenStack service application credentials and their statuses. This command generates a table with the following important columns:
    NAME
    Specifies the name of the application credential, which adheres to the following format: ac-<service>, where <service> specifies the upstream OpenStack service name, for example, ac-barbican.
    ACID
    Specifies the ID of the application credential, for example,d38dc4310fbf4601bbe9f4234eb24114. When an application credential is rotated it obtains a new ID and a new secret.
    SECRETNAME

    Specifies the name of the new immutable secret created for this application credential. This secret name includes the first 5 characters of the ID of the application credential to make it unique. This also makes it easy to identify which secret is associated with which application credential. The secret name adheres to the following format: ac-<service>-<first-5-ID-characters>-secret, for example, ac-barbican-d38dc-secret.

    Note
    RHOSO creates and manages this secret.
    ROTATIONELIGIBLE
    Specifies the start of the configured grace period window when this application credential is rotated. This timestamp is calculated as follows Date of expiry - GracePeriodDays.
    STATUS

    This column verifies the successful creation of the application credential when it specifies a value of True.

    Note
    A missing value or a value of False means the application credential is still being created or has encountered an error. In this case, run the oc describe appcred <name> command to view a more detailed message, where <name> specifies the name of the application credential, for example, ac-barbican.