Chapter 4. Adopting Red Hat OpenStack Platform control plane services

Procedure

1. Patch the OpenStackControlPlane CR to deploy the Identity service:

   $ oc patch openstackcontrolplane openstack --type=merge --patch '
   spec:
     keystone:
       enabled: true
       apiOverride:
         route: {}
       template:
         override:
           service:
             internal:
               metadata:
                 annotations:
                   metallb.universe.tf/address-pool: internalapi
                   metallb.universe.tf/allow-shared-ip: internalapi
                   metallb.universe.tf/loadBalancerIPs: <172.17.0.80>
               spec:
                 type: LoadBalancer
         databaseInstance: openstack
         secret: osp-secret
   '

   where:

   <172.17.0.80>

   Specifies the load balancer IP in your environment. If you use IPv6, change the load balancer IP to the load balancer IP in your environment, for example, metallb.universe.tf/loadBalancerIPs: fd00:bbbb::80.

2. Create an alias to use the openstack command in the Red Hat OpenStack Services on OpenShift (RHOSO) deployment:

   $ alias openstack="oc exec -t openstackclient -- openstack"

3. Remove services and endpoints that still point to the RHOSP control plane, excluding the Identity service and its endpoints:

   $ openstack endpoint list | grep keystone | awk '/admin/{ print $2; }' | xargs ${BASH_ALIASES[openstack]} endpoint delete || true
   > for service in aodh heat heat-cfn barbican cinderv3 glance gnocchi manila manilav2 neutron nova placement swift ironic-inspector ironic octavia; do
   >  openstack service list | awk "/ $service /{ print \$2; }" | xargs -r ${BASH_ALIASES[openstack]} service delete || true
   > done

Verification

1. Verify that you can access the OpenStackClient pod. For more information, see Accessing the OpenStackClient pod in Maintaining the Red Hat OpenStack Services on OpenShift deployment.

2. Confirm that the Identity service endpoints are defined and are pointing to the control plane FQDNs:

   $ openstack endpoint list | grep keystone

3. Wait for the OpenStackControlPlane resource to become Ready:

   $ oc wait --for=condition=Ready --timeout=1m OpenStackControlPlane openstack

1. Create the secret that holds the domain-specific LDAP configuration files that the Identity service uses. Each file within the secret corresponds to an LDAP domain.
2. Patch the OpenStackControlPlane custom resource (CR) to enable domain-specific drivers for the Identity service and mount a secret that contains the LDAP configurations.

Procedure

1. To create the keystone-domains secret that stores the actual LDAP configuration files that Identity service uses, create a local file that includes your LDAP configuration, for example, keystone.myldapdomain.conf:

   The following example file includes the configuration for a single LDAP domain. If you have multiple LDAP domains, create a configuration file for each, for example, keystone.DOMAIN_ONE.conf, keystone.DOMAIN_TWO.conf.

   [identity]
   driver = ldap
   [ldap]
   url = ldap://<ldap_server_host>:<ldap_server_port>
   user = <bind_dn_user>
   password = <bind_dn_password>
   suffix = <user_tree_dn>
   query_scope = sub
   # User configuration
   user_tree_dn = <user_tree_dn>
   user_objectclass = <user_object_class>
   user_id_attribute = <user_id_attribute>
   user_name_attribute = <user_name_attribute>
   user_mail_attribute = <user_mail_attribute>
   user_enabled_attribute = <user_enabled_attribute>
   user_enabled_default = true
   # Group configuration
   group_tree_dn = <group_tree_dn>
   group_objectclass = <group_object_class>
   group_id_attribute = <group_id_attribute>
   group_name_attribute = <group_name_attribute>
   group_member_attribute = <group_member_attribute>
   group_members_are_ids = true

   • Replace the values, such as <ldap_server_host>, <bind_dn_user>, <user_tree_dn>, and so on, with your LDAP server details.

2. Create the secret from this file:

   $ oc create secret generic keystone-domains --from-file=<keystone.DOMAIN_NAME.conf>

   • Replace <keystone.DOMAIN_NAME.conf> with the name of your local configuration file. If applicable, include additional configuration files by using the --from-file option. After creating the secret, you can remove the local configuration file if it is no longer needed, or store it securely.

     Important

     The name of the file that you provide to --from-file, for example keystone.DOMAIN_NAME.conf, is critical. The Identity service uses this filename to map incoming authentication requests for a domain to the correct LDAP configuration. Ensure that DOMAIN_NAME matches the name of the domain you are configuring in the Identity service.

3. Patch the OpenStackControlPlane CR:

   $ oc patch openstackcontrolplane <cr_name> --type=merge -p '
   spec:
     keystone:
       template:
         customServiceConfig: |
             [identity]
             domain_specific_drivers_enabled = true
         extraMounts:
           - name: v1
             region: r1
             extraVol:
               - propagation:
                 - Keystone
                 extraVolType: Conf
                 volumes:
                 - name: keystone-domains
                   secret:
                     secretName: keystone-domains
                 mounts:
                 - name: keystone-domains
                   mountPath: "/etc/keystone/domains"
                   readOnly: true

   • Replace <cr_name> with the name of your OpenStackControlPlane CR (for example, openstack).

   • This patch does the following:

     • Sets spec.keystone.template.customServiceConfig. Ensure that you do not overwrite any previously defined value.

     • Defines spec.keystone.template.extraMounts to mount a secret named keystone-domains into the Identity service pods at /etc/keystone/domains. This secret contains your LDAP configuration files.

       Note

       You might need to wait a few minutes for the changes to propagate and for the Identity service pods to be updated.

Verification

1. Verify that users from the LDAP domain are accessible:

   $ oc exec -t openstackclient -- openstack user list --domain <domain_name>

   • Replace <domain_name> with your LDAP domain name.

     This command returns a list of users from your LDAP server.

2. Verify that groups from the LDAP domain are accessible:

   $ oc exec -t openstackclient -- openstack group list --domain <domain_name>

   This command returns a list of groups from your LDAP server.

3. Test authentication with an LDAP user:

   $ oc exec -t openstackclient -- openstack --os-auth-url <keystone_auth_url> --os-identity-api-version 3 --os-user-domain-name <domain_name> --os-username <ldap_username> --os-password <ldap_password> token issue

   • Replace <keystone_auth_url> with the Identity service authentication URL.

   • Replace <ldap_username> and <ldap_password> with valid LDAP user credentials.

     If successful, this command returns a token, confirming that LDAP authentication is working correctly.

4. Verify group membership for an LDAP user:

   $ oc exec -t openstackclient -- openstack group contains user --group-domain <domain_name> --user-domain <domain_name> <group_name> <username>

   • Replace <domain_name>, <group_name>, and <username> with the appropriate values from your LDAP server.

     This command verifies that the user is properly associated with the group through LDAP.

Procedure

1. Add the kek secret:

   $ oc set data secret/osp-secret "BarbicanSimpleCryptoKEK=$($CONTROLLER1_SSH "python3 -c \"import configparser; c = configparser.ConfigParser(); c.read('/var/lib/config-data/puppet-generated/barbican/etc/barbican/barbican.conf'); print(c['simple_crypto_plugin']['kek'])\"")"

2. Patch the OpenStackControlPlane CR to deploy the Key Manager service:

   $ oc patch openstackcontrolplane openstack --type=merge --patch '
   spec:
     barbican:
       enabled: true
       apiOverride:
         route: {}
       template:
         databaseInstance: openstack
         databaseAccount: barbican
         messagingBus:
           cluster: rabbitmq
         secret: osp-secret
         simpleCryptoBackendSecret: osp-secret
         serviceAccount: barbican
         serviceUser: barbican
         passwordSelectors:
           service: BarbicanPassword
           simplecryptokek: BarbicanSimpleCryptoKEK
         barbicanAPI:
           replicas: 1
           override:
             service:
               internal:
                 metadata:
                   annotations:
                     metallb.universe.tf/address-pool: internalapi
                     metallb.universe.tf/allow-shared-ip: internalapi
                     metallb.universe.tf/loadBalancerIPs: <172.17.0.80>
                 spec:
                   type: LoadBalancer
         barbicanWorker:
           replicas: 1
         barbicanKeystoneListener:
           replicas: 1
   '

   where:

   <172.17.0.80>

   Specifies the load balancer IP in your environment. If you use IPv6, change the load balancer IP to the load balancer IP in your environment, for example, metallb.universe.tf/loadBalancerIPs: fd00:bbbb::80.

   messagingBus.Cluster

   For more information about RHOSO RabbitMQ clusters, see RHOSO RabbitMQ clusters in Monitoring high availability services.

Procedure

1. Update your existing NetConfig CR to add subnets for each remote site. Each service network must include a subnet for the central site and each remote site. Use unique VLAN IDs for each site. For example:

   • Central site: VLANs 20-23
   • Edge site 1: VLANs 30-33

   • Edge site 2: VLANs 40-43

     apiVersion: network.openstack.org/v1beta1
     kind: NetConfig
     metadata:
       name: netconfig
     spec:
       networks:
       - name: ctlplane
         dnsDomain: ctlplane.example.com
         subnets:
         - name: <subnet1>
           allocationRanges:
           - end: 192.168.122.120
             start: 192.168.122.100
           cidr: 192.168.122.0/24
           gateway: 192.168.122.1
         - name: <ctlplanesite1>
           allocationRanges:
           - end: 192.168.133.120
             start: 192.168.133.100
           cidr: 192.168.133.0/24
           gateway: 192.168.133.1
         - name: <ctlplanesite2>
           allocationRanges:
           - end: 192.168.144.120
             start: 192.168.144.100
           cidr: 192.168.144.0/24
           gateway: 192.168.144.1
       - name: internalapi
         dnsDomain: internalapi.example.com
         subnets:
         - name: subnet1
           allocationRanges:
           - end: 172.17.0.250
             start: 172.17.0.100
           cidr: 172.17.0.0/24
           vlan: 20
         - name: internalapisite1
           allocationRanges:
           - end: 172.17.10.250
             start: 172.17.10.100
           cidr: 172.17.10.0/24
           vlan: 30
         - name: internalapisite2
           allocationRanges:
           - end: 172.17.20.250
             start: 172.17.20.100
           cidr: 172.17.20.0/24
           vlan: 40
       - name: storage
         dnsDomain: storage.example.com
         subnets:
         - name: subnet1
           allocationRanges:
           - end: 172.18.0.250
             start: 172.18.0.100
           cidr: 172.18.0.0/24
           vlan: 21
         - name: storagesite1
           allocationRanges:
           - end: 172.18.10.250
             start: 172.18.10.100
           cidr: 172.18.10.0/24
           vlan: 31
         - name: storagesite2
           allocationRanges:
           - end: 172.18.20.250
             start: 172.18.20.100
           cidr: 172.18.20.0/24
           vlan: 41
       - name: tenant
         dnsDomain: tenant.example.com
         subnets:
         - name: subnet1
           allocationRanges:
           - end: 172.19.0.250
             start: 172.19.0.100
           cidr: 172.19.0.0/24
           vlan: 22
         - name: tenantsite1
           allocationRanges:
           - end: 172.19.10.250
             start: 172.19.10.100
           cidr: 172.19.10.0/24
           vlan: 32
         - name: tenantsite2
           allocationRanges:
           - end: 172.19.20.250
             start: 172.19.20.100
           cidr: 172.19.20.0/24
           vlan: 42

     where:

     <subnet1>

     Specifies a user-defined subnet name for the central site subnet.

     <ctlplanesite1>

     Specifies a user-defined subnet for the first DCN edge site.

     <ctlplanesite2>

     Specifies a user-defined subnet for the second DCN edge site.

     Note

     You must have the storagemgmt network on OpenShift nodes when using DCN with Swift storage. It is not necessary when using Red Hat Ceph Storage.

2. Update the NetworkAttachmentDefinition CR for the internalapi network to include routes to remote site subnets. These routes fields enable control plane pods attached to the internalapi network, such as OVN Southbound database, to communicate with Compute nodes at remote sites through the central site gateway, and are required for DCN:

   apiVersion: k8s.cni.cncf.io/v1
   kind: NetworkAttachmentDefinition
   metadata:
     name: internalapi
   spec:
     config: |
       {
         "cniVersion": "0.3.1",
         "name": "internalapi",
         "type": "macvlan",
         "master": "internalapi",
         "ipam": {
           "type": "whereabouts",
           "range": "172.17.0.0/24",
           "range_start": "172.17.0.30",
           "range_end": "172.17.0.70",
           "routes": [
             { "dst": "172.17.10.0/24", "gw": "172.17.0.1" },
             { "dst": "172.17.20.0/24", "gw": "172.17.0.1" }
           ]
         }
       }

3. Update the NetworkAttachmentDefinition CR for the ctlplane network to include routes to remote site subnets:

   apiVersion: k8s.cni.cncf.io/v1
   kind: NetworkAttachmentDefinition
   metadata:
     name: ctlplane
   spec:
     config: |
       {
         "cniVersion": "0.3.1",
         "name": "ctlplane",
         "type": "macvlan",
         "master": "ospbr",
         "ipam": {
           "type": "whereabouts",
           "range": "192.168.122.0/24",
           "range_start": "192.168.122.30",
           "range_end": "192.168.122.70",
           "routes": [
             { "dst": "192.168.133.0/24", "gw": "192.168.122.1" },
             { "dst": "192.168.144.0/24", "gw": "192.168.122.1" }
           ]
         }
       }

4. Update the NetworkAttachmentDefinition CR for the storage network to include routes to remote site subnets:

   apiVersion: k8s.cni.cncf.io/v1
   kind: NetworkAttachmentDefinition
   metadata:
     name: storage
   spec:
     config: |
       {
         "cniVersion": "0.3.1",
         "name": "storage",
         "type": "macvlan",
         "master": "storage",
         "ipam": {
           "type": "whereabouts",
           "range": "172.18.0.0/24",
           "range_start": "172.18.0.30",
           "range_end": "172.18.0.70",
           "routes": [
             { "dst": "172.18.10.0/24", "gw": "172.18.0.1" },
             { "dst": "172.18.20.0/24", "gw": "172.18.0.1" }
           ]
         }
       }

5. Update the NetworkAttachmentDefinition CR for the tenant network to include routes to remote site subnets:

   apiVersion: k8s.cni.cncf.io/v1
   kind: NetworkAttachmentDefinition
   metadata:
     name: tenant
   spec:
     config: |
       {
         "cniVersion": "0.3.1",
         "name": "tenant",
         "type": "macvlan",
         "master": "tenant",
         "ipam": {
           "type": "whereabouts",
           "range": "172.19.0.0/24",
           "range_start": "172.19.0.30",
           "range_end": "172.19.0.70",
           "routes": [
             { "dst": "172.19.10.0/24", "gw": "172.19.0.1" },
             { "dst": "172.19.20.0/24", "gw": "172.19.0.1" }
           ]
         }
       }

   Note

   Adjust the IP ranges, subnets, and gateway addresses in all NAD configurations to match your network topology. The master interface name must match the interface on the OpenShift nodes where the VLAN is configured.

6. If you have already deployed OVN services, restart the OVN Southbound database pods to pick up the new routes:

   $ oc delete pod -l service=ovsdbserver-sb

   The pods are automatically recreated with the updated network configuration.

7. Configure the Networking service (neutron) to recognize all site physnets. In the OpenStackControlPlane CR, ensure the Networking service configuration includes all physnets:

   apiVersion: core.openstack.org/v1beta1
   kind: OpenStackControlPlane
   metadata:
     name: openstack
   spec:
     neutron:
       template:
         customServiceConfig: |
           [ml2_type_vlan]
           network_vlan_ranges = leaf0:1:1000,leaf1:1:1000,leaf2:1:1000
           [ovn]
           ovn_emit_need_to_frag = false

   where:

   leaf0

   Represents the physnet for the central site.

   leaf1

   Represents the physnet for the first remote site.

   leaf2

   Represents the physnet for the second remote site.

   Note

   Adjust the physnet names to match your Red Hat OpenStack Platform deployment. Common conventions include leaf0/leaf1/leaf2 or datacentre/dcn1/dcn2.

Verification

1. Verify that the NetConfig CR is created with all subnets:

   $ oc get netconfig netconfig -o yaml | grep -A2 "name: subnet1\|name: .*site"

2. Verify that each NetworkAttachmentDefinition includes routes to remote site subnets:

   for nad in ctlplane internalapi storage tenant; do
     echo "=== $nad ==="
     oc get net-attach-def $nad -o jsonpath='{.spec.config}' | jq '.ipam.routes
   done

3. After restarting OVN SB pods, verify they have routes to remote site subnets:

   $ oc exec $(oc get pod -l service=ovsdbserver-sb -o name | head -1) -- ip route show | grep 172.17

   Sample output:

   172.17.10.0/24 via 172.17.0.1 dev internalapi
   172.17.20.0/24 via 172.17.0.1 dev internalapi

Procedure

1. Create the swift-conf secret that includes the Object Storage service hash path suffix and prefix:

   $ oc apply -f - <<EOF
   apiVersion: v1
   kind: Secret
   metadata:
     name: swift-conf
   type: Opaque
   data:
     swift.conf: $($CONTROLLER1_SSH sudo cat /var/lib/config-data/puppet-generated/swift/etc/swift/swift.conf | base64 -w0)
   EOF

2. Create the swift-ring-files ConfigMap that includes the Object Storage service ring files:

   $ oc apply -f - <<EOF
   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: swift-ring-files
   binaryData:
     swiftrings.tar.gz: $($CONTROLLER1_SSH "cd /var/lib/config-data/puppet-generated/swift/etc/swift && tar cz *.builder *.ring.gz backups/ | base64 -w0")
     account.ring.gz: $($CONTROLLER1_SSH "base64 -w0 /var/lib/config-data/puppet-generated/swift/etc/swift/account.ring.gz")
     container.ring.gz: $($CONTROLLER1_SSH "base64 -w0 /var/lib/config-data/puppet-generated/swift/etc/swift/container.ring.gz")
     object.ring.gz: $($CONTROLLER1_SSH "base64 -w0 /var/lib/config-data/puppet-generated/swift/etc/swift/object.ring.gz")
   EOF

3. Patch the OpenStackControlPlane custom resource to deploy the Object Storage service:

   $ oc patch openstackcontrolplane openstack --type=merge --patch '
   spec:
     swift:
       enabled: true
       template:
         memcachedInstance: memcached
         swiftRing:
           ringReplicas: 3
         swiftStorage:
           replicas: 0
           networkAttachments:
           - storage
           storageClass: local-storage
           storageRequest: 10Gi
         swiftProxy:
           secret: osp-secret
           replicas: 2
           encryptionEnabled: false
           passwordSelectors:
             service: SwiftPassword
           serviceUser: swift
           override:
             service:
               internal:
                 metadata:
                   annotations:
                     metallb.universe.tf/address-pool: internalapi
                     metallb.universe.tf/allow-shared-ip: internalapi
                     metallb.universe.tf/loadBalancerIPs: <172.17.0.80>
                 spec:
                   type: LoadBalancer
           networkAttachments:
           - storage
   '

   Note

   If SwiftEncryptionEnabled: true was set in Red Hat OpenStack Platform, ensure that spec.swift.swiftProxy.encryptionEnabled is set to true and that the Key Manager service (barbican) adoption is complete before proceeding.

   • spec.swift.swiftStorage.storageClass must match the RHOSO deployment storage class.
   • metallb.universe.tf/loadBalancerIPs: <172.17.0.80> specifies the load balancer IP in your environment. If you use IPv6, change the load balancer IP to the load balancer IP in your environment, for example, metallb.universe.tf/loadBalancerIPs: fd00:bbbb::80.
   • spec.swift.swiftProxy.networkAttachments must match the network attachment for the previous Object Storage service configuration from the RHOSP deployment.

Procedure

1. Patch the OpenStackControlPlane CR to deploy the Compute service:

   Note

   This procedure assumes that Compute service metadata is deployed on the top level and not on each cell level. If the RHOSP deployment has a per-cell metadata deployment, adjust the following patch as needed. You cannot run the metadata service in cell0. To enable the metadata services of a local cell, set the enabled property in the metadataServiceTemplate field of the local cell to true in the OpenStackControlPlane CR.

   $ rm -f celltemplates
   $ for CELL in $(echo $RENAMED_CELLS); do
   > cat >> celltemplates << EOF
   >       ${CELL}:
   >         hasAPIAccess: true
   >         cellDatabaseAccount: nova-$CELL
   >         cellDatabaseInstance: openstack-$CELL
   >         cellMessageBusInstance: rabbitmq-$CELL
   >         metadataServiceTemplate:
   >           enabled: false
   >           override:
   >               service:
   >                 metadata:
   >                   annotations:
   >                     metallb.universe.tf/address-pool: internalapi
   >                     metallb.universe.tf/allow-shared-ip: internalapi
   >                     metallb.universe.tf/loadBalancerIPs: 172.17.0.$(( 79 + ${CELL##*cell} ))
   >                 spec:
   >                   type: LoadBalancer
   >           customServiceConfig: |
   >             [workarounds]
   >             disable_compute_service_check_for_ffu=true
   >         conductorServiceTemplate:
   >           customServiceConfig: |
   >             [workarounds]
   >             disable_compute_service_check_for_ffu=true
   >EOF
   >done
   
   $ cat > oscp-patch.yaml << EOF
   >spec:
   >  nova:
   >   enabled: true
   >   apiOverride:
   >     route: {}
   >   template:
   >     secret: osp-secret
   >     apiDatabaseAccount: nova-api
   >     apiServiceTemplate:
   >       override:
   >         service:
   >           internal:
   >             metadata:
   >               annotations:
   >                 metallb.universe.tf/address-pool: internalapi
   >                 metallb.universe.tf/allow-shared-ip: internalapi
   >                 metallb.universe.tf/loadBalancerIPs: <172.17.0.80>
   >             spec:
   >               type: LoadBalancer
   >       customServiceConfig: |
   >         [workarounds]
   >         disable_compute_service_check_for_ffu=true
   >     metadataServiceTemplate:
   >       enabled: true
   >       override:
   >         service:
   >           metadata:
   >             annotations:
   >               metallb.universe.tf/address-pool: internalapi
   >               metallb.universe.tf/allow-shared-ip: internalapi
   >               metallb.universe.tf/loadBalancerIPs: <172.17.0.80>
   >           spec:
   >             type: LoadBalancer
   >       customServiceConfig: |
   >         [workarounds]
   >         disable_compute_service_check_for_ffu=true
   >     schedulerServiceTemplate:
   >       customServiceConfig: |
   >         [workarounds]
   >         disable_compute_service_check_for_ffu=true
   >     cellTemplates:
   >       cell0:
   >         hasAPIAccess: true
   >         cellDatabaseAccount: nova-cell0
   >         cellDatabaseInstance: openstack
   >         cellMessageBusInstance: rabbitmq
   >         conductorServiceTemplate:
   >           customServiceConfig: |
   >             [workarounds]
   >             disable_compute_service_check_for_ffu=true
   >EOF
   $ cat celltemplates >> oscp-patch.yaml
   $ oc patch openstackcontrolplane openstack  --type=merge --patch-file=oscp-patch.yaml

   • ${CELL}.hasAPIAccess specifies upcall access to the API. In the source cloud, cells are always configured with the main Nova API database upcall access. You can disable upcall access to the API by setting hasAPIAccess to false. However, do not make changes to the API during adoption.
   • ${CELL}.cellDatabaseInstance specifies the database instance that is used by the cell. The database instance names must match the names that are defined in the OpenStackControlPlane CR that you created in when you deployed the back-end services as described in Deploying back-end services.
   • ${CELL}.cellMessageBusInstance specifies the message bus instance that is used by the cell. The message bus instance names must match the names that are defined in the OpenStackControlPlane CR.
   • metallb.universe.tf/loadBalancerIPs: <172.17.0.80> specifies the load balancer IP in your environment. If you use IPv6, change the load balancer IP to the load balancer IP in your environment, for example, metallb.universe.tf/loadBalancerIPs: fd00:bbbb::80.

2. If you are adopting the Compute service with the Bare Metal Provisioning service (ironic), append the novaComputeTemplates field with the following content in each cell in the Compute service CR patch. For example:

           cell1:
             novaComputeTemplates:
               standalone:
                 customServiceConfig: |
                   [DEFAULT]
                   host = <hostname>
                   [workarounds]
                   disable_compute_service_check_for_ffu=true
                 computeDriver: ironic.IronicDriver
           ...

   • Replace <hostname> with the hostname of the node that is running the ironic Compute driver in the source cloud.

3. Wait for the CRs for the Compute control plane services to be ready:

   $ oc wait --for condition=Ready --timeout=300s Nova/nova

   Note

   The local Conductor services are started for each cell, while the superconductor runs in cell0. Note that disable_compute_service_check_for_ffu is mandatory for all imported Compute services until the external data plane is imported, and until the Compute services are fast-forward upgraded. For more information, see Adopting Compute services to the RHOSO data plane and Performing a fast-forward upgrade on Compute services.

Procedure

1. Create a new file, for example, cinder_api.patch, and apply the configuration:

   $ oc patch openstackcontrolplane openstack --type=merge --patch-file=<patch_name>

   • Replace <patch_name> with the name of your patch file.

     The following example shows a cinder_api.patch file:

     spec:
       extraMounts:
       - extraVol:
         - extraVolType: Ceph
           mounts:
           - mountPath: /etc/ceph
             name: ceph
             readOnly: true
           propagation:
           - CinderVolume
           - CinderBackup
           - Glance
           volumes:
           - name: ceph
             projected:
               sources:
               - secret:
                   name: ceph-conf-files
       cinder:
         enabled: true
         apiOverride:
           route: {}
         template:
           databaseInstance: openstack
           databaseAccount: cinder
           secret: osp-secret
           cinderAPI:
             override:
               service:
                 internal:
                   metadata:
                     annotations:
                       metallb.universe.tf/address-pool: internalapi
                       metallb.universe.tf/allow-shared-ip: internalapi
                       metallb.universe.tf/loadBalancerIPs: <172.17.0.80>
                   spec:
                     type: LoadBalancer
             replicas: 1
             customServiceConfig: |
               [DEFAULT]
               default_volume_type=tripleo
           cinderScheduler:
             replicas: 0
           cinderBackup:
             networkAttachments:
             - storage
             replicas: 0
           cinderVolumes:
             ceph:
               networkAttachments:
               - storage
               replicas: 0

     where:

     <172.17.0.80>

     Specifies the load balancer IP in your environment. If you use IPv6, change the load balancer IP to the load balancer IP in your environment, for example, metallb.universe.tf/loadBalancerIPs: fd00:bbbb::80.

2. Retrieve the list of the previous scheduler and backup services:

   $ openstack volume service list
   
   +------------------+------------------------+------+---------+-------+----------------------------+
   | Binary           | Host                   | Zone | Status  | State | Updated At                 |
   +------------------+------------------------+------+---------+-------+----------------------------+
   | cinder-scheduler | standalone.localdomain | nova | enabled | down  | 2024-11-04T17:47:14.000000 |
   | cinder-backup    | standalone.localdomain | nova | enabled | down  | 2024-11-04T17:47:14.000000 |
   | cinder-volume    | hostgroup@tripleo_ceph | nova | enabled | down  | 2024-11-04T17:47:14.000000 |
   +------------------+------------------------+------+---------+-------+----------------------------+

3. Remove services for hosts that are in the down state:

   $ oc exec -t cinder-api-0 -c cinder-api -- cinder-manage service remove <service_binary> <service_host>

   • Replace <service_binary> with the name of the binary, for example, cinder-backup.
   • Replace <service_host> with the host name, for example, cinder-backup-0.

4. Deploy the scheduler, backup, and volume services:

   • Create another file, for example, cinder_services.patch, and apply the configuration:

     $ oc patch openstackcontrolplane openstack --type=merge --patch-file=<patch_name>

   • Replace <patch_name> with the name of your patch file.

   • The following example shows a cinder_services.patch file for a Ceph RBD deployment:

     spec:
       cinder:
         enabled: true
         template:
           cinderScheduler:
             replicas: 1
           cinderBackup:
             networkAttachments:
             - storage
             replicas: 1
             customServiceConfig: |
               [DEFAULT]
               backup_driver=cinder.backup.drivers.ceph.CephBackupDriver
               backup_ceph_conf=/etc/ceph/ceph.conf
               backup_ceph_user=openstack
               backup_ceph_pool=backups
           cinderVolumes:
             ceph:
               networkAttachments:
               - storage
               replicas: 1
               customServiceConfig: |
                 [tripleo_ceph]
                 backend_host=hostgroup
                 volume_backend_name=tripleo_ceph
                 volume_driver=cinder.volume.drivers.rbd.RBDDriver
                 rbd_ceph_conf=/etc/ceph/ceph.conf
                 rbd_user=openstack
                 rbd_pool=volumes
                 rbd_flatten_volume_from_snapshot=False
                 report_discard_supported=True

     Note

     Ensure that you use the same configuration group name for the driver that you used in the source cluster. In this example, the driver configuration group in customServiceConfig is called tripleo_ceph because it reflects the value of the configuration group name in the cinder.conf file of the source OpenStack cluster.

5. Configure the NetApp NFS Block Storage volume service:

   1. Create a secret that includes sensitive information such as hostnames, passwords, and usernames to access the third-party NetApp NFS storage. You can find the credentials in the cinder.conf file that was generated from the director deployment:

      $ oc apply -f - <<EOF
      apiVersion: v1
      kind: Secret
      metadata:
        labels:
          service: cinder
          component: cinder-volume
        name: cinder-volume-ontap-secrets
      type: Opaque
      stringData:
        ontap-cinder-secrets: |
          [tripleo_netapp]
          netapp_login= netapp_username
          netapp_password= netapp_password
          netapp_vserver= netapp_vserver
          nas_host= netapp_nfsip
          nas_share_path=/netapp_nfspath
          netapp_pool_name_search_pattern=(netapp_poolpattern)
      EOF

   2. Patch the OpenStackControlPlane CR to deploy NetApp NFS Block Storage volume back end:

      $ oc patch openstackcontrolplane openstack --type=merge --patch-file=<cinder_netappNFS.patch>

      • Replace <cinder_netappNFS.patch> with the name of the patch file for your NetApp NFS Block Storage volume back end.

        The following example shows a cinder_netappNFS.patch file that configures a NetApp NFS Block Storage volume service:

        spec:
          cinder:
            enabled: true
            template:
              cinderVolumes:
                ontap-nfs:
                  networkAttachments:
                    - storage
                  customServiceConfig: |
                    [tripleo_netapp]
                    volume_backend_name=ontap-nfs
                    volume_driver=cinder.volume.drivers.netapp.common.NetAppDriver
                    nfs_snapshot_support=true
                    nas_secure_file_operations=false
                    nas_secure_file_permissions=false
                    netapp_server_hostname= netapp_backendip
                    netapp_server_port=80
                    netapp_storage_protocol=nfs
                    netapp_storage_family=ontap_cluster
                  customServiceConfigSecrets:
                  - cinder-volume-ontap-secrets

6. Configure the NetApp iSCSI Block Storage volume service:

   1. Create a secret that includes sensitive information such as hostnames, passwords, and usernames to access the third-party NetApp iSCSI storage. You can find the credentials in the cinder.conf file that was generated from the director deployment:

      $ oc apply -f - <<EOF
      apiVersion: v1
      kind: Secret
      metadata:
        labels:
          service: cinder
          component: cinder-volume
        name: cinder-volume-ontap-secrets
      type: Opaque
      stringData:
        ontap-cinder-secrets: |
          [tripleo_netapp]
          netapp_server_hostname = netapp_host
          netapp_login = netapp_username
          netapp_password = netapp_password
          netapp_vserver = netapp_vserver
          netapp_pool_name_search_pattern=(netapp_poolpattern)
      EOF

7. Patch the OpenStackControlPlane custom resource (CR) to deploy the NetApp iSCSI Block Storage volume back end:

   $ oc patch openstackcontrolplane openstack --type=merge --patch-file=<cinder_netappISCSI.patch>

   • Replace <cinder_netappISCSI.patch> with the name of the patch file for your NetApp iSCSI Block Storage volume back end.

     The following example shows a cinder_netappISCSI.patch file that configures a NetApp iSCSI Block Storage volume service:

     spec:
       cinder:
         enabled: true
         template:
           cinderVolumes:
             ontap-iscsi:
               networkAttachments:
                 - storage
               customServiceConfig: |
                 [tripleo_netapp]
                 volume_backend_name=ontap-iscsi
                 volume_driver=cinder.volume.drivers.netapp.common.NetAppDriver
                 netapp_storage_protocol=iscsi
                 netapp_storage_family=ontap_cluster
                 consistencygroup_support=True
               customServiceConfigSecrets:
                 - cinder-volume-ontap-secrets

8. Check if all the services are up and running:

   $ openstack volume service list
   
   +------------------+--------------------------+------+---------+-------+----------------------------+
   | Binary           | Host                     | Zone | Status  | State | Updated At                 |
   +------------------+--------------------------+------+---------+-------+----------------------------+
   | cinder-volume    | hostgroup@tripleo_netapp | nova | enabled | up    | 2023-06-28T17:00:03.000000 |
   | cinder-scheduler | cinder-scheduler-0       | nova | enabled | up    | 2023-06-28T17:00:02.000000 |
   | cinder-backup    | cinder-backup-0          | nova | enabled | up    | 2023-06-28T17:00:01.000000 |
   +------------------+--------------------------+------+---------+-------+----------------------------+

9. Apply the DB data migrations:

   Note

   You are not required to run the data migrations at this step, but you must run them before the next upgrade. However, for adoption, you can run the migrations now to ensure that there are no issues before you run production workloads on the deployment.

   $ oc exec -it cinder-scheduler-0 -- cinder-manage db online_data_migrations

Verification

1. Ensure that the openstack alias is defined:

   $ alias openstack="oc exec -t openstackclient -- openstack"

2. Confirm that Block Storage service endpoints are defined and pointing to the control plane FQDNs:

   $ openstack endpoint list --service <endpoint>

   • Replace <endpoint> with the name of the endpoint that you want to confirm.

3. Confirm that the Block Storage services are running:

   $ openstack volume service list

   Note

   Cinder API services do not appear in the list. However, if you get a response from the openstack volume service list command, that means at least one of the cinder API services is running.

4. Confirm that you have your previous volume types, volumes, snapshots, and backups:

   $ openstack volume type list
   $ openstack volume list
   $ openstack volume snapshot list
   $ openstack volume backup list

5. To confirm that the configuration is working, perform the following steps:

   1. Create a volume from an image to check that the connection to Image Service (glance) is working:

      $ openstack volume create --image cirros --bootable --size 1 disk_new

   2. Back up the previous attached volume:

      $ openstack --os-volume-api-version 3.47 volume create --backup <backup_name>

      • Replace <backup_name> with the name of your new backup location.

        Note

        You do not boot a Compute service (nova) instance by using the new volume from image or try to detach the previous volume because the Compute service and the Block Storage service are still not connected.

Procedure

1. Retrieve the fsid for each Red Hat Ceph Storage cluster in your DCN deployment. The fsid is used as the rbd_secret_uuid for libvirt integration:

   $ oc get secret ceph-conf-central -o json | jq -r '.data | to_entries[] | select(.key | endswith(".conf")) | "\(.key): \(.value | @base64d)"' | grep fsid

2. Create a patch file for the Block Storage service with multiple Red Hat Ceph Storage back ends. The following example shows a DCN deployment with a central site and two edge sites:

   $ cat << EOF > cinder_dcn_patch.yaml
   spec:
     cinder:
       enabled: true
       template:
         cinderAPI:
           customServiceConfig: |
             [DEFAULT]
             default_availability_zone = az-central
         cinderScheduler:
           replicas: 1
         cinderVolumes:
           central:
             networkAttachments:
             - storage
             replicas: 1
             customServiceConfig: |
               [DEFAULT]
               enabled_backends = central
               glance_api_servers = http://glance-central-internal.openstack.svc:9292
               [central]
               backend_host = hostgroup
               volume_backend_name = central
               volume_driver = cinder.volume.drivers.rbd.RBDDriver
               rbd_ceph_conf = /etc/ceph/central.conf
               rbd_user = openstack
               rbd_pool = volumes
               rbd_flatten_volume_from_snapshot = False
               report_discard_supported = True
               rbd_secret_uuid = <central_fsid>
               rbd_cluster_name = central
               backend_availability_zone = az-central
           dcn1:
             networkAttachments:
             - storage
             replicas: 1
             customServiceConfig: |
               [DEFAULT]
               enabled_backends = dcn1
               glance_api_servers = http://glance-dcn1-internal.openstack.svc:9292
               [dcn1]
               backend_host = hostgroup
               volume_backend_name = dcn1
               volume_driver = cinder.volume.drivers.rbd.RBDDriver
               rbd_ceph_conf = /etc/ceph/dcn1.conf
               rbd_user = openstack
               rbd_pool = volumes
               rbd_flatten_volume_from_snapshot = False
               report_discard_supported = True
               rbd_secret_uuid = <dcn1_fsid>
               rbd_cluster_name = dcn1
               backend_availability_zone = az-dcn1
           dcn2:
             networkAttachments:
             - storage
             replicas: 1
             customServiceConfig: |
               [DEFAULT]
               enabled_backends = dcn2
               glance_api_servers = http://glance-dcn2-internal.openstack.svc:9292
               [dcn2]
               backend_host = hostgroup
               volume_backend_name = dcn2
               volume_driver = cinder.volume.drivers.rbd.RBDDriver
               rbd_ceph_conf = /etc/ceph/dcn2.conf
               rbd_user = openstack
               rbd_pool = volumes
               rbd_flatten_volume_from_snapshot = False
               report_discard_supported = True
               rbd_secret_uuid = <dcn2_fsid>
               rbd_cluster_name = dcn2
               backend_availability_zone = az-dcn2
   EOF

   where:

   <central_fsid>

   Specifies the fsid of the central Red Hat Ceph Storage cluster, used as the libvirt secret UUID.

   <dcn1_fsid>

   Specifies the fsid of the DCN1 edge Red Hat Ceph Storage cluster.

   <dcn2_fsid>

   Specifies the fsid of the DCN2 edge Red Hat Ceph Storage cluster.

   Note
   • You must configure each CinderVolume with the backend_availability_zone value that matches your Compute service availability zone for that site, because cross_az_attach = False is set in the Compute service configuration. If the names do not match, instances cannot attach volumes. Replace the examples (az-central, az-dcn1, az-dcn2) with the names used in your Red Hat OpenStack Platform deployment.
   • Each CinderVolume points to its local Image service API endpoint through glance_api_servers. This ensures that volume creation from images uses the local Image service and Red Hat Ceph Storage cluster. The examples use http:// for the Image service endpoints. If your Red Hat OpenStack Platform deployment uses TLS for internal endpoints, use https:// instead, and ensure that you have completed the TLS migration. For more information, see Migrating TLS-e to the RHOSO deployment.
   • The rbd_cluster_name setting identifies which Red Hat Ceph Storage cluster configuration to use from the mounted secrets.
   • Adjust the number of edge sites and their names to match your DCN deployment.

3. Patch the OpenStackControlPlane CR to deploy the Block Storage service with multiple Red Hat Ceph Storage back ends:

   $ oc patch openstackcontrolplane openstack --type=merge --patch-file cinder_dcn_patch.yaml

4. Configure the Block Storage service backup service. In this example, the backup service runs at the central site and uses the central Red Hat Ceph Storage cluster. Add the cinderBackups section to your patch file and re-apply it:

   $ cat << EOF >> cinder_dcn_patch.yaml
         cinderBackups:
           central:
             networkAttachments:
             - storage
             replicas: 1
             customServiceConfig: |
               [DEFAULT]
               backup_driver=cinder.backup.drivers.ceph.CephBackupDriver
               backup_ceph_conf=/etc/ceph/central.conf
               backup_ceph_user=openstack
               backup_ceph_pool=backups
               storage_availability_zone=az-central
   EOF
   $ oc patch openstackcontrolplane openstack --type=merge --patch-file cinder_dcn_patch.yaml

   Note

   Unlike a single-site Red Hat Ceph Storage deployment where the backup config references /etc/ceph/ceph.conf, in a DCN deployment the Red Hat Ceph Storage configuration files in the ceph-conf-files secret are named by cluster. Set backup_ceph_conf to the path of the Red Hat Ceph Storage configuration file for whichever cluster hosts your backups pool. In this example the file is named central.conf, so the path is /etc/ceph/central.conf. Using a path that does not match a file in the secret will cause the backup service to fail with a conf_read_file error.

   Set storage_availability_zone to match the availability zone of the volumes you want to back up. The backup scheduler uses this to route backup requests to a service in the correct zone. If the backup service zone does not match the volume zone, backup creation fails with Service not found for creating backup.

5. Verify that the Block Storage service volume services are running for each availability zone:

   $ openstack volume service list --service cinder-volume
   
   +------------------+---------------------+------------+---------+-------+----------------------------+
   | Binary           | Host                | Zone       | Status  | State | Updated At                 |
   +------------------+---------------------+------------+---------+-------+----------------------------+
   | cinder-volume    | hostgroup@central   | az-central | enabled | up    | 2024-01-01T00:00:00.000000 |
   | cinder-volume    | hostgroup@dcn1      | az-dcn1    | enabled | up    | 2024-01-01T00:00:00.000000 |
   | cinder-volume    | hostgroup@dcn2      | az-dcn2    | enabled | up    | 2024-01-01T00:00:00.000000 |
   +------------------+---------------------+------------+---------+-------+----------------------------+

6. Verify that the Block Storage service backup service is running and in the correct availability zone:

   $ openstack volume service list --service cinder-backup
   
   +---------------+-------------------------+------------+---------+-------+----------------------------+
   | Binary        | Host                    | Zone       | Status  | State | Updated At                 |
   +---------------+-------------------------+------------+---------+-------+----------------------------+
   | cinder-backup | cinder-backup-central-0 | az-central | enabled | up    | 2024-01-01T00:00:00.000000 |
   +---------------+-------------------------+------------+---------+-------+----------------------------+

7. Test the backup service by creating a volume, backing it up, and restoring the backup:

   $ openstack volume create --size 1 backup-test-vol
   
   $ openstack volume backup create --name backup-test-backup backup-test-vol
   
   $ openstack volume backup show backup-test-backup
   +-----------------------+--------------------------------------+
   | Field                 | Value                                |
   +-----------------------+--------------------------------------+
   | container             | backups                              |
   | fail_reason           | None                                 |
   | name                  | backup-test-backup                   |
   | size                  | 1                                    |
   | status                | available                            |
   +-----------------------+--------------------------------------+
   
   $ openstack volume backup restore backup-test-backup backup-test-restore

   Note

   Some versions of the Red Hat OpenShift Container Platform (RHOCP) client display a cannot unpack non-iterable VolumeBackupsRestore object error after the restore command. This is a known issue in the client, the restore operation might not have failed. Verify by checking the restored volume status directly.

   $ openstack volume show backup-test-restore -c status -c availability_zone -c os-vol-host-attr:host -f value
   available
   az-central
   hostgroup@central#central

Verification

1. Verify that the Dashboard service instance is successfully deployed and ready:

   $ oc get horizon

2. Confirm that the Dashboard service is reachable and returns a 200 status code:

   PUBLIC_URL=$(oc get horizon horizon -o jsonpath='{.status.endpoint}')
   curl --silent --output /dev/stderr --head --write-out "%{http_code}" "$PUBLIC_URL/dashboard/auth/login/?next=/dashboard/" -k | grep 200

Procedure

1. Retrieve the existing auth_encryption_key and service passwords. You use these passwords to patch the osp-secret. In the following example, the auth_encryption_key is used as HeatAuthEncryptionKey and the service password is used as HeatPassword:

   [stack@rhosp17 ~]$ grep -E 'HeatPassword|HeatAuth|HeatStackDomainAdmin' ~/overcloud-deploy/overcloud/overcloud-passwords.yaml
     HeatAuthEncryptionKey: Q60Hj8PqbrDNu2dDCbyIQE2dibpQUPg2
     HeatPassword: dU2N0Vr2bdelYH7eQonAwPfI3
     HeatStackDomainAdminPassword: dU2N0Vr2bdelYH7eQonAwPfI3

2. Log in to a Controller node and verify the auth_encryption_key value in use:

   [stack@rhosp17 ~]$ ansible -i overcloud-deploy/overcloud/config-download/overcloud/tripleo-ansible-inventory.yaml overcloud-controller-0 -m shell -a "grep auth_encryption_key /var/lib/config-data/puppet-generated/heat/etc/heat/heat.conf | grep -Ev '^#|^$'" -b
   overcloud-controller-0 | CHANGED | rc=0 >>
   auth_encryption_key=Q60Hj8PqbrDNu2dDCbyIQE2dibpQUPg2

3. Encode the password to Base64 format:

   $ echo Q60Hj8PqbrDNu2dDCbyIQE2dibpQUPg2 | base64
   UTYwSGo4UHFickROdTJkRENieUlRRTJkaWJwUVVQZzIK

4. Patch the osp-secret to update the HeatAuthEncryptionKey and HeatPassword parameters. These values must match the values in the director Orchestration service configuration:

   $ oc patch secret osp-secret --type='json' -p='[{"op" : "replace" ,"path" : "/data/HeatAuthEncryptionKey" ,"value" : "UTYwSGo4UHFickROdTJkRENieUlRRTJkaWJwUVVQZzIK"}]'
   secret/osp-secret patched

5. Patch the OpenStackControlPlane CR to deploy the Orchestration service:

   $ oc patch openstackcontrolplane openstack --type=merge --patch '
   spec:
     heat:
       enabled: true
       apiOverride:
         route: {}
       template:
         databaseInstance: openstack
         databaseAccount: heat
         secret: osp-secret
         memcachedInstance: memcached
         passwordSelectors:
           authEncryptionKey: HeatAuthEncryptionKey
           service: HeatPassword
           stackDomainAdminPassword: HeatStackDomainAdminPassword
   '

Verification

1. Ensure that the statuses of all the CRs are Setup complete:

   $ oc get Heat,HeatAPI,HeatEngine,HeatCFNAPI
   NAME                           STATUS   MESSAGE
   heat.heat.openstack.org/heat   True     Setup complete
   
   NAME                                  STATUS   MESSAGE
   heatapi.heat.openstack.org/heat-api   True     Setup complete
   
   NAME                                        STATUS   MESSAGE
   heatengine.heat.openstack.org/heat-engine   True     Setup complete
   
   NAME                                        STATUS   MESSAGE
   heatcfnapi.heat.openstack.org/heat-cfnapi   True     Setup complete

2. Check that the Orchestration service is registered in the Identity service:

   $ oc exec -it openstackclient -- openstack service list -c Name -c Type
   +------------+----------------+
   | Name       | Type           |
   +------------+----------------+
   | heat       | orchestration  |
   | glance     | image          |
   | heat-cfn   | cloudformation |
   | ceilometer | Ceilometer     |
   | keystone   | identity       |
   | placement  | placement      |
   | cinderv3   | volumev3       |
   | nova       | compute        |
   | neutron    | network        |
   +------------+----------------+

   $ oc exec -it openstackclient -- openstack endpoint list --service=heat -f yaml
   - Enabled: true
     ID: 1da7df5b25b94d1cae85e3ad736b25a5
     Interface: public
     Region: regionOne
     Service Name: heat
     Service Type: orchestration
     URL: http://heat-api-public-openstack-operators.apps.okd.bne-shift.net/v1/%(tenant_id)s
   - Enabled: true
     ID: 414dd03d8e9d462988113ea0e3a330b0
     Interface: internal
     Region: regionOne
     Service Name: heat
     Service Type: orchestration
     URL: http://heat-api-internal.openstack-operators.svc:8004/v1/%(tenant_id)s

3. Check that the Orchestration service engine services are running:

   $ oc exec -it openstackclient -- openstack orchestration service list -f yaml
   - Binary: heat-engine
     Engine ID: b16ad899-815a-4b0c-9f2e-e6d9c74aa200
     Host: heat-engine-6d47856868-p7pzz
     Hostname: heat-engine-6d47856868-p7pzz
     Status: up
     Topic: engine
     Updated At: '2023-10-11T21:48:01.000000'
   - Binary: heat-engine
     Engine ID: 887ed392-0799-4310-b95c-ac2d3e6f965f
     Host: heat-engine-6d47856868-p7pzz
     Hostname: heat-engine-6d47856868-p7pzz
     Status: up
     Topic: engine
     Updated At: '2023-10-11T21:48:00.000000'
   - Binary: heat-engine
     Engine ID: 26ed9668-b3f2-48aa-92e8-2862252485ea
     Host: heat-engine-6d47856868-p7pzz
     Hostname: heat-engine-6d47856868-p7pzz
     Status: up
     Topic: engine
     Updated At: '2023-10-11T21:48:00.000000'
   - Binary: heat-engine
     Engine ID: 1011943b-9fea-4f53-b543-d841297245fd
     Host: heat-engine-6d47856868-p7pzz
     Hostname: heat-engine-6d47856868-p7pzz
     Status: up
     Topic: engine
     Updated At: '2023-10-11T21:48:01.000000'

4. Verify that you can see your Orchestration service stacks:

   $ openstack stack list -f yaml
   - Creation Time: '2023-10-11T22:03:20Z'
     ID: 20f95925-7443-49cb-9561-a1ab736749ba
     Project: 4eacd0d1cab04427bc315805c28e66c9
     Stack Name: test-networks
     Stack Status: CREATE_COMPLETE
     Updated Time: null

Procedure

1. Migrate the server certificate authority (CA) passphrase from the previous deployment:

   SERVER_CA_PASSPHRASE=$($CONTROLLER1_SSH grep ^ca_private_key_passphrase /var/lib/config-data/puppet-generated/octavia/etc/octavia/octavia.conf)
   
   oc apply -f - <<EOF
   apiVersion: v1
   kind: Secret
   metadata:
     name: octavia-ca-passphrase
   type: Opaque
   data:
     server-ca-passphrase: $(echo -n $SERVER_CA_PASSPHRASE | base64 -w0)
   EOF

2. To isolate the management network, add the network interface for the VLAN base interface:

   $ oc get --no-headers nncp | cut -f 1 -d ' ' | grep -v nncp-dns | while read; do
   
   interfaces=$(oc get nncp $REPLY -o jsonpath="{.spec.desiredState.interfaces[*].name}")
   
   (echo $interfaces | grep -w -q "octbr\|enp6s0.24") || \
           oc patch nncp $REPLY --type json --patch '
   [{
       "op": "add",
       "path": "/spec/desiredState/interfaces/-",
       "value": {
         "description": "Octavia VLAN host interface",
         "name": "enp6s0.24",
         "state": "up",
         "type": "vlan",
         "vlan": {
           "base-iface": "<enp6s0>",
           "id": 24
           }
       }
   },
   {
       "op": "add",
       "path": "/spec/desiredState/interfaces/-",
       "value": {
         "description": "Octavia Bridge",
         "mtu": <mtu>,
         "state": "up",
         "type": "linux-bridge",
         "name": "octbr",
         "bridge": {
           "options": { "stp": { "enabled": "false" } },
           "port": [ { "name": "enp6s0.24" } ]
           }
       }
   }]'
   
   done

   where:

   <enp6s0>

   Specifies the name of the network interface in your RHOCP setup.

   <mtu>

   Specifies the mtu value in your environment.

3. To connect pods that manage load balancer virtual machines (amphorae) and the OpenvSwitch pods the OVN operator manages, configure the Load-balancing service network attachment definition:

   $ cat  octavia-nad.yaml << EOF_CAT
   apiVersion: k8s.cni.cncf.io/v1
   kind: NetworkAttachmentDefinition
   metadata:
     labels:
       osp/net: octavia
     name: octavia
   spec:
     config: |
       {
         "cniVersion": "0.3.1",
         "name": "octavia",
         "type": "bridge",
         "bridge": "octbr",
         "ipam": {
           "type": "whereabouts",
           "range": "172.23.0.0/24",
           "range_start": "172.23.0.30",
           "range_end": "172.23.0.70",
           "routes": [
              {
                "dst": "172.24.0.0/16",
                "gw" : "172.23.0.150"
              }
            ]
         }
       }
   EOF_CAT

4. Create the NetworkAttachmentDefinition CR:

   $ oc apply -f octavia-nad.yaml

5. Enable the Load-balancing service in RHOCP:

   $ oc patch openstackcontrolplane openstack --type=merge --patch '
   spec:
     ovn:
       template:
         ovnController:
           networkAttachment: tenant
           nicMappings:
             octavia: octbr
     octavia:
       enabled: true
       template:
         amphoraImageContainerImage: quay.io/gthiemonge/octavia-amphora-image
         octaviaHousekeeping:
           networkAttachments:
             - octavia
         octaviaHealthManager:
           networkAttachments:
             - octavia
         octaviaWorker:
           networkAttachments:
             - octavia
   '

6. Wait for the Load-balancing service control plane services CRs to be ready:

   $ oc wait --for condition=Ready --timeout=600s octavia.octavia.openstack.org/octavia

7. Ensure that the Load-balancing service is registered in the Identity service:

   $ alias openstack="oc exec -t openstackclient -- openstack"
   $ openstack service list | grep load-balancer
   | bd078ca6f90c4b86a48801f45eb6f0d7 | octavia   | load-balancer |
   $ openstack endpoint list --service load-balancer
   +----------------------------------+-----------+--------------+---------------+---------+-----------+---------------------------------------------------+
   | ID                               | Region    | Service Name | Service Type  | Enabled | Interface | URL                                               |
   +----------------------------------+-----------+--------------+---------------+---------+-----------+---------------------------------------------------+
   | f1ae7756b6164baf9cb82a1a670067a2 | regionOne | octavia      | load-balancer | True    | public    | https://octavia-public-openstack.apps-crc.testing |
   | ff3222b4621843669e89843395213049 | regionOne | octavia      | load-balancer | True    | internal  | http://octavia-internal.openstack.svc:9876        |
   +----------------------------------+-----------+--------------+---------------+---------+-----------+---------------------------------------------------+

Procedure

1. Patch the OpenStackControlPlane CR to deploy cluster-observability-operator:

   $ oc create -f - <<EOF
   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: cluster-observability-operator
     namespace: openshift-operators
   spec:
     channel: stable
     installPlanApproval: Automatic
     name: cluster-observability-operator
     source: redhat-operators
     sourceNamespace: openshift-marketplace
   EOF

2. Wait for the installation to succeed:

   $ oc wait --for jsonpath="{.status.phase}"=Succeeded csv --namespace=openshift-operators -l operators.coreos.com/cluster-observability-operator.openshift-operators

3. Patch the OpenStackControlPlane CR to deploy Ceilometer services:

   $ oc patch openstackcontrolplane openstack --type=merge --patch '
   spec:
     telemetry:
       enabled: true
       template:
         ceilometer:
           passwordSelector:
             ceilometerService: CeilometerPassword
           enabled: true
           secret: osp-secret
           serviceUser: ceilometer
   '

4. Enable the metrics storage back end:

   $ oc patch openstackcontrolplane openstack --type=merge --patch '
   spec:
     telemetry:
       template:
         metricStorage:
           enabled: true
           monitoringStack:
             alertingEnabled: true
             scrapeInterval: 30s
             storage:
               strategy: persistent
               retention: 24h
               persistent:
                 pvcStorageRequest: 20G
   '

Verification

1. Verify that the alertmanager and prometheus pods are available:

   $ oc get pods -l alertmanager=metric-storage
   NAME                            READY   STATUS    RESTARTS   AGE
   alertmanager-metric-storage-0   2/2     Running   0          46s
   alertmanager-metric-storage-1   2/2     Running   0          46s
   
   $ oc get pods -l prometheus=metric-storage
   NAME                          READY   STATUS    RESTARTS   AGE
   prometheus-metric-storage-0   3/3     Running   0          46s

2. Inspect the resulting Ceilometer pods:

   CEILOMETETR_POD=`oc get pods -l service=ceilometer | tail -n 1 | cut -f 1 -d' '`
   oc exec -t $CEILOMETETR_POD -c ceilometer-central-agent -- cat /etc/ceilometer/ceilometer.conf

3. Inspect enabled pollsters:

   $ oc get secret ceilometer-config-data -o jsonpath="{.data['polling\.yaml\.j2']}"  | base64 -d

4. Optional: Override default pollsters according to the requirements of your environment:

   $ oc patch openstackcontrolplane controlplane --type=merge --patch '
   spec:
     telemetry:
       template:
         ceilometer:
             defaultConfigOverwrite:
               polling.yaml.j2: |
                 ---
                 sources:
                   - name: pollsters
                     interval: 100
                     meters:
                       - volume.*
                       - image.size
             enabled: true
             secret: osp-secret
   '

Next steps

1. Optional: Patch the OpenStackControlPlane CR to include logging:

   $ oc patch openstackcontrolplane openstack --type=merge --patch '
   spec:
     telemetry:
       template:
         logging:
           enabled: false
           ipaddr: 172.17.0.80
           port: 10514
           cloNamespace: openshift-logging
   '

Procedure

1. Patch the OpenStackControlPlane CR to deploy the autoscaling services:

   $ oc patch openstackcontrolplane openstack --type=merge --patch '
   spec:
     telemetry:
       enabled: true
       template:
         autoscaling:
           enabled: true
           aodh:
             passwordSelector:
               aodhService: AodhPassword
             databaseAccount: aodh
             databaseInstance: openstack
             secret: osp-secret
             serviceUser: aodh
           heatInstance: heat
   '

2. Inspect the aodh pods:

   $ AODH_POD=`oc get pods -l service=aodh | tail -n 1 | cut -f 1 -d' '`
   $ oc exec -t $AODH_POD -c aodh-api -- cat /etc/aodh/aodh.conf

3. Check whether the aodh API service is registered in the Identity service:

   $ openstack endpoint list | grep aodh
   | d05d120153cd4f9b8310ac396b572926 | regionOne | aodh  | alarming  | True    | internal  | http://aodh-internal.openstack.svc:8042  |
   | d6daee0183494d7a9a5faee681c79046 | regionOne | aodh  | alarming  | True    | public    | http://aodh-public.openstack.svc:8042    |

4. Optional: Create aodh alarms with the PrometheusAlarm alarm type:

   Note

   You must use the PrometheusAlarm alarm type instead of GnocchiAggregationByResourcesAlarm.

   $ openstack alarm create --name high_cpu_alarm \
   --type prometheus \
   --query "(rate(ceilometer_cpu{resource_name=~'cirros'})) * 100" \
   --alarm-action 'log://' \
   --granularity 15 \
   --evaluation-periods 3 \
   --comparison-operator gt \
   --threshold 7000000000

   1. Verify that the alarm is enabled:

      $ openstack alarm list
      +--------------------------------------+------------+------------------+-------------------+----------+
      | alarm_id                             | type       | name             | state  | severity | enabled  |
      +--------------------------------------+------------+------------------+-------------------+----------+
      | 209dc2e9-f9d6-40e5-aecc-e767ce50e9c0 | prometheus | prometheus_alarm |   ok   |    low   |   True   |
      +--------------------------------------+------------+------------------+-------------------+----------+

Procedure

1. Update your ssh parameters according to your environment in the os-diff.cfg. Os-diff uses the ssh parameters to connect to your director node, and then query and download the configuration files:

   ssh_cmd=ssh -F ssh.config standalone
   container_engine=podman
   connection=ssh
   remote_config_path=/tmp/tripleo

   Ensure that the ssh command you provide in ssh_cmd parameter is correct and includes key authentication.

2. Enable the services that you want to include in the /etc/os-diff/config.yaml file, and disable the services that you want to exclude from the file. Ensure that you have the correct permissions to edit the file:

   $ chown ospng:ospng /etc/os-diff/config.yaml

   The following example enables the default Identity service (keystone) to be included in the /etc/os-diff/config.yaml file:

   # service name and file location
   services:
     # Service name
     keystone:
       # Bool to enable/disable a service (not implemented yet)
       enable: true
       # Pod name, in both OCP and podman context.
       # It could be strict match or will only just grep the podman_name
       # and work with all the pods which matched with pod_name.
       # To enable/disable use strict_pod_name_match: true/false
       podman_name: keystone
       pod_name: keystone
       container_name: keystone-api
       # pod options
       # strict match for getting pod id in TripleO and podman context
       strict_pod_name_match: false
       # Path of the config files you want to analyze.
       # It could be whatever path you want:
       # /etc/<service_name> or /etc or /usr/share/<something> or even /
       # @TODO: need to implement loop over path to support multiple paths such as:
       # - /etc
       # - /usr/share
       path:
         - /etc/
         - /etc/keystone
         - /etc/keystone/keystone.conf
         - /etc/keystone/logging.conf

   Repeat this step for each RHOSP service that you want to disable or enable.

3. If you use non-containerized services, such as the ovs-external-ids, pull the configuration or the command output. For example:

   services:
     ovs_external_ids:
       hosts:
         - standalone
       service_command: "ovs-vsctl list Open_vSwitch . | grep external_ids | awk -F ': ' '{ print $2; }'"
       cat_output: true
       path:
         - ovs_external_ids.json
       config_mapping:
         ovn-bridge-mappings: edpm_ovn_bridge_mappings
         ovn-bridge: edpm_ovn_bridge
         ovn-encap-type: edpm_ovn_encap_type
         ovn-monitor-all: ovn_monitor_all
         ovn-remote-probe-interval: edpm_ovn_remote_probe_interval
         ovn-ofctrl-wait-before-clear: edpm_ovn_ofctrl_wait_before_clear

   Note

   You must correctly configure an SSH configuration file or equivalent for non-standard services, such as OVS. The ovs_external_ids service does not run in a container, and the OVS data is stored on each host of your cloud, for example, controller_1/controller_2/, and so on.

   • hosts specifies the list of hosts, for example, compute-1, compute-2.
   • service_command: "ovs-vsctl list Open_vSwitch . | grep external_ids | awk -F ': ' '{ print $2; }'" runs against the hosts.
   • cat_output: true provides os-diff with the output of the command and stores the output in a file that is specified by the key path.
   • config_mapping provides a mapping between, in this example, the data plane custom resource definition and the ovs-vsctl output.

   • ovn-bridge-mappings: edpm_ovn_bridge_mappings must be a list of strings, for example, ["datacentre:br-ex"].

     1. Compare the values:

        $ os-diff diff ovs_external_ids.json edpm.crd --crd --service ovs_external_ids

        For example, to check the /etc/yum.conf on every host, you must put the following statement in the config.yaml file. The following example uses a file called yum_config:

        services:
          yum_config:
            hosts:
              - undercloud
              - controller_1
              - compute_1
              - compute_2
            service_command: "cat /etc/yum.conf"
            cat_output: true
            path:
              - yum.conf

4. Pull the configuration:

   Note

   The following command pulls all the configuration files that are included in the /etc/os-diff/config.yaml file. You can configure os-diff to update this file automatically according to your running environment by using the --update or --update-only option. These options set the podman information into the config.yaml for all running containers. The podman information can be useful later, when all the Red Hat OpenStack Platform services are turned off.

   Note that when the config.yaml file is populated automatically you must provide the configuration paths manually for each service.

   # will only update the /etc/os-diff/config.yaml
   os-diff pull --update-only

   # will update the /etc/os-diff/config.yaml and pull configuration
   os-diff pull --update

   # will update the /etc/os-diff/config.yaml and pull configuration
   os-diff pull

   The configuration is pulled and stored by default in the following directory:

   /tmp/tripleo/

Procedure

1. To restore the source cloud to a working state, start the RHOSP control plane services that you previously stopped during the adoption procedure:

   ServicesToStart=("tripleo_horizon.service"
                    "tripleo_keystone.service"
                    "tripleo_barbican_api.service"
                    "tripleo_barbican_worker.service"
                    "tripleo_barbican_keystone_listener.service"
                    "tripleo_cinder_api.service"
                    "tripleo_cinder_api_cron.service"
                    "tripleo_cinder_scheduler.service"
                    "tripleo_cinder_volume.service"
                    "tripleo_cinder_backup.service"
                    "tripleo_glance_api.service"
                    "tripleo_manila_api.service"
                    "tripleo_manila_api_cron.service"
                    "tripleo_manila_scheduler.service"
                    "tripleo_neutron_api.service"
                    "tripleo_placement_api.service"
                    "tripleo_nova_api_cron.service"
                    "tripleo_nova_api.service"
                    "tripleo_nova_conductor.service"
                    "tripleo_nova_metadata.service"
                    "tripleo_nova_scheduler.service"
                    "tripleo_nova_vnc_proxy.service"
                    "tripleo_aodh_api.service"
                    "tripleo_aodh_api_cron.service"
                    "tripleo_aodh_evaluator.service"
                    "tripleo_aodh_listener.service"
                    "tripleo_aodh_notifier.service"
                    "tripleo_ceilometer_agent_central.service"
                    "tripleo_ceilometer_agent_compute.service"
                    "tripleo_ceilometer_agent_ipmi.service"
                    "tripleo_ceilometer_agent_notification.service"
                    "tripleo_ovn_cluster_north_db_server.service"
                    "tripleo_ovn_cluster_south_db_server.service"
                    "tripleo_ovn_cluster_northd.service"
                    "tripleo_octavia_api.service"
                    "tripleo_octavia_health_manager.service"
                    "tripleo_octavia_rsyslog.service"
                    "tripleo_octavia_driver_agent.service"
                    "tripleo_octavia_housekeeping.service"
                    "tripleo_octavia_worker.service")
   
   PacemakerResourcesToStart=("galera-bundle"
                              "haproxy-bundle"
                              "rabbitmq-bundle"
                              "openstack-cinder-volume"
                              "openstack-cinder-backup"
                              "openstack-manila-share")
   
   echo "Starting systemd OpenStack services"
   for service in ${ServicesToStart[*]}; do
       for i in {1..3}; do
           SSH_CMD=CONTROLLER${i}_SSH
           if [ ! -z "${!SSH_CMD}" ]; then
               if ${!SSH_CMD} sudo systemctl is-enabled $service &> /dev/null; then
                   echo "Starting the $service in controller $i"
                   ${!SSH_CMD} sudo systemctl start $service
               fi
           fi
       done
   done
   
   echo "Checking systemd OpenStack services"
   for service in ${ServicesToStart[*]}; do
       for i in {1..3}; do
           SSH_CMD=CONTROLLER${i}_SSH
           if [ ! -z "${!SSH_CMD}" ]; then
               if ${!SSH_CMD} sudo systemctl is-enabled $service &> /dev/null; then
                   if ! ${!SSH_CMD} systemctl show $service | grep ActiveState=active >/dev/null; then
                       echo "ERROR: Service $service is not running on controller $i"
                   else
                       echo "OK: Service $service is running in controller $i"
                   fi
               fi
           fi
       done
   done
   
   echo "Starting pacemaker OpenStack services"
   for i in {1..3}; do
       SSH_CMD=CONTROLLER${i}_SSH
       if [ ! -z "${!SSH_CMD}" ]; then
           echo "Using controller $i to run pacemaker commands"
           for resource in ${PacemakerResourcesToStart[*]}; do
               if ${!SSH_CMD} sudo pcs resource config $resource &>/dev/null; then
                   echo "Starting $resource"
                   ${!SSH_CMD} sudo pcs resource enable $resource
               else
                   echo "Service $resource not present"
               fi
           done
           break
       fi
   done
   
   echo "Checking pacemaker OpenStack services"
   for i in {1..3}; do
       SSH_CMD=CONTROLLER${i}_SSH
       if [ ! -z "${!SSH_CMD}" ]; then
           echo "Using controller $i to run pacemaker commands"
           for resource in ${PacemakerResourcesToStop[*]}; do
               if ${!SSH_CMD} sudo pcs resource config $resource &>/dev/null; then
                   if ${!SSH_CMD} sudo pcs resource status $resource | grep Started >/dev/null; then
                       echo "OK: Service $resource is started"
                   else
                       echo "ERROR: Service $resource is stopped"
                   fi
               fi
           done
           break
       fi
   done

2. If the Ceph NFS service is running on the deployment as a Shared File Systems service (manila) back end, you must restore the Pacemaker order and colocation constraints for the openstack-manila-share service:

   $ sudo pcs constraint order start ceph-nfs then openstack-manila-share kind=Optional id=order-ceph-nfs-openstack-manila-share-Optional
   $ sudo pcs constraint colocation add openstack-manila-share with ceph-nfs score=INFINITY id=colocation-openstack-manila-share-ceph-nfs-INFINITY

3. Verify that the source cloud is operational again, for example, you can run openstack CLI commands such as openstack server list, or check that you can access the Dashboard service (horizon).

4. Remove the partially or fully deployed control plane so that you can attempt the adoption again later:

   $ oc delete --ignore-not-found=true --wait=false openstackcontrolplane/openstack
   $ oc patch openstackcontrolplane openstack --type=merge --patch '
   > metadata:
   >   finalizers: []
   > ' || true
   >
   >while oc get pod | grep rabbitmq-server-0; do
   >    sleep 2
   >done
   >while oc get pod | grep openstack-galera-0; do
   >    sleep 2
   >done
   
   $ oc delete --ignore-not-found=true --wait=false pod mariadb-copy-data
   $ oc delete --ignore-not-found=true --wait=false pvc mariadb-data
   $ oc delete --ignore-not-found=true --wait=false pod ovn-copy-data
   $ oc delete --ignore-not-found=true secret osp-secret