Adopting a Red Hat OpenStack Platform 17.1 deployment

Procedure

1. Log in to the Compute node host as a user with the appropriate permissions.

2. List the instances that are running on the host:

   $ sudo machinectl list

   Sample output

   MACHINE                  CLASS SERVICE      OS VERSION ADDRESSES
   qemu-1-instance-000000b9 vm    libvirt-qemu -  -       -
   qemu-2-instance-000000c2 vm    libvirt-qemu -  -       -
   
   2 machines listed.

3. Verify that the systemd-machined service is running:

   $ sudo systemctl status systemd-machined.service

   Sample output

   systemd-machined.service - Virtual Machine and Container Registration Service
        Loaded: loaded (/usr/lib/systemd/system/systemd-machined.service; static)
        Active: active (running) since Mon 2025-06-16 11:42:07 EDT; 2min 48s ago
          Docs: man:systemd-machined.service(8)
                man:org.freedesktop.machine1(5)
      Main PID: 136614 (systemd-machine)
        Status: "Processing requests..."
         Tasks: 1 (limit: 838860)
        Memory: 1.4M
           CPU: 33ms
        CGroup: /system.slice/systemd-machined.service
                └─136614 /usr/lib/systemd/systemd-machined
   
   Jun 16 11:42:07 computehost001 systemd[1]: Starting Virtual Machine and Container Registration Service...
   Jun 16 11:42:07 computehost001 systemd[1]: Started Virtual Machine and Container Registration Service.
   Jun 16 11:43:44 computehost001 systemd-machined[136614]: New machine qemu-1-instance-000000b9.
   Jun 16 11:43:51 computehost001 systemd-machined[136614]: New machine qemu-2-instance-000000c2.

   Important

   If the systemd-machined service is running, skip the rest of this procedure. Ensure that you verify that the systemd-machined service is running each Compute node host in the cluster.

4. If the systemd-machined service is not running, before you can install the systemd-container package, live migrate all virtual machines from the host. For more information about live migration, see Rebooting Compute nodes in Performing a minor update of Red Hat OpenStack Platform.

5. Install the systemd-container on the host:

   • If you upgraded your environment from an earlier version of Red Hat OpenStack Platform, reboot the Compute host to automatically install the systemd-container.

   • If you deployed a new RHOSO environment, install the systemd-container manually by using the following command. Rebooting the Compute host is not required:

     $ sudo dnf -y install systemd-container

     Note

     If your Compute host is not running a virtual machine, you can install the systemd-container automatically or manually.

6. Repeat this procedure on each Compute host in the cluster where the systemd-machined service is not running.

1. Remove custom policies.
2. Run the adoption.
3. Re-add custom policies by using the new SRBAC syntax.

Procedure

1. Configure the /etc/os-diff/os-diff.cfg file and the /etc/os-diff/ssh.config file according to your environment. To allow os-diff to connect to your clouds and pull files from the services that you describe in the config.yaml file, you must set the following options in the os-diff.cfg file:

   [Default]
   
   local_config_dir=/tmp/
   service_config_file=config.yaml
   
   [Tripleo]
   
   ssh_cmd=ssh -F ssh.config
   director_host=standalone
   container_engine=podman
   connection=ssh
   remote_config_path=/tmp/tripleo
   local_config_path=/tmp/
   
   [Openshift]
   
   ocp_local_config_path=/tmp/ocp
   connection=local
   ssh_cmd=""

   • ssh_cmd=ssh -F ssh.config instructs os-diff to access your director host through SSH. The default value is ssh -F ssh.config. However, you can set the value without an ssh.config file, for example, ssh -i /home/user/.ssh/id_rsa stack@my.undercloud.local.
   • director_host=standalone specifies the host to use to access your cloud, and the podman/docker binary is installed and allowed to interact with the running containers. You can leave this key blank.

2. If you use a host file to connect to your cloud, configure the ssh.config file to allow os-diff to access your RHOSP environment, for example:

   Host *
       IdentitiesOnly yes
   
   Host virthost
       Hostname virthost
       IdentityFile ~/.ssh/id_rsa
       User root
       StrictHostKeyChecking no
       UserKnownHostsFile=/dev/null
   
   
   Host standalone
       Hostname standalone
       IdentityFile <path to SSH key>
       User root
       StrictHostKeyChecking no
       UserKnownHostsFile=/dev/null
   
   Host crc
       Hostname crc
       IdentityFile ~/.ssh/id_rsa
       User stack
       StrictHostKeyChecking no
       UserKnownHostsFile=/dev/null

   • Replace <path to SSH key> with the path to your SSH key. You must provide a value for IdentityFile to get full working access to your RHOSP environment.

3. If you use an inventory file to connect to your cloud, generate the ssh.config file from your Ansible inventory, for example, tripleo-ansible-inventory.yaml file:

   $ os-diff configure -i tripleo-ansible-inventory.yaml -o ssh.config --yaml

Prerequisites

1. Define the following shell variables. Replace the example values with values that are correct for your environment:

   Note

   If you use IPv6, define the SOURCE_MARIADB_IP value without brackets. For example, SOURCE_MARIADB_IP=fd00:bbbb::2.

   $ PASSWORD_FILE="$HOME/overcloud-passwords.yaml"
   $ MARIADB_IMAGE=registry.redhat.io/rhoso/openstack-mariadb-rhel9:18.0
   $ declare -A TRIPLEO_PASSWORDS
   $ CELLS="default cell1 cell2"
   $ for CELL in $(echo $CELLS); do
   >    TRIPLEO_PASSWORDS[$CELL]="$PASSWORD_FILE"
   > done
   $ declare -A SOURCE_DB_ROOT_PASSWORD
   $ for CELL in $(echo $CELLS); do
   >     SOURCE_DB_ROOT_PASSWORD[$CELL]=$(cat ${TRIPLEO_PASSWORDS[$CELL]} | grep ' MysqlRootPassword:' | awk -F ': ' '{ print $2; }')
   > done

2. Define the following shell variables. Replace the example values with values that are correct for your environment:

   $ MARIADB_CLIENT_ANNOTATIONS='--annotations=k8s.v1.cni.cncf.io/networks=internalapi'
   $ MARIADB_RUN_OVERRIDES="$MARIADB_CLIENT_ANNOTATIONS"

   Note

   For environments that are enabled with border gateway protocol (BGP), the network annotation must include a default route to enable proper routing. Use the following instead:

   $ MARIADB_CLIENT_ANNOTATIONS='--annotations=k8s.v1.cni.cncf.io/networks=[{"name":"internalapi","namespace":"openstack","default-route":["<172.17.0.1>"]}]'
   $ MARIADB_RUN_OVERRIDES="$MARIADB_CLIENT_ANNOTATIONS"

   where:

   <172.17.0.1>

   Replace with the gateway IP address of your internalapi network.

   $ CONTROLLER1_SSH="ssh -i *<path to SSH key>* root@*<node IP>*"
   
   
   $ declare -A SOURCE_MARIADB_IP
   $ SOURCE_MARIADB_IP[default]=*<galera cluster VIP>*
   $ SOURCE_MARIADB_IP[cell1]=*<galera cell1 cluster VIP>*
   $ SOURCE_MARIADB_IP[cell2]=*<galera cell2 cluster VIP>*
   # ...

   • Provide CONTROLLER1_SSH settings with SSH connection details for any non-cell Controller of the source director cloud.
   • For each cell that is defined in CELLS, replace SOURCE_MARIADB_IP[*]= ..., with the records lists for the cell names and VIP addresses of MariaDB Galera clusters, including the cells, of the source director cloud.

   • To get the values for SOURCE_MARIADB_IP, query the puppet-generated configurations in a Controller and CellController node:

     $ sudo grep -rI 'listen mysql' -A10 /var/lib/config-data/puppet-generated/ | grep bind

     Note

     The source cloud always uses the same password for cells databases. For that reason, the same passwords file is used for all cells stacks. However, split-stack topology allows using different passwords files for each stack.

Procedure

1. If your source RHOSP environment uses border gateway protocol (BGP) for Layer 3 networking, create a BGPConfiguration custom resource to enable BGP routing:

   $ cat << EOF > bgp.yaml
   apiVersion: network.openstack.org/v1beta1
   kind: BGPConfiguration
   metadata:
     name: bgpconfiguration
     namespace: openstack
   spec: {}
   EOF
   
   $ oc apply -f bgp.yaml

   The BGPConfiguration resource enables BGP route advertisement between the Red Hat OpenShift Container Platform (RHOCP) cluster and the source cloud, which is necessary for the mariadb-client pod to reach the source MariaDB cluster.

2. Create a persistent mariadb-client pod for database operations:

   $ oc delete pod mariadb-client || true
   $ oc run mariadb-client ${MARIADB_RUN_OVERRIDES} -q --image ${MARIADB_IMAGE} --restart=Never -- /usr/bin/sleep infinity

   This creates a long-running pod that is used for all subsequent database operations, avoiding the need to create temporary pods for each command.

3. Wait for the mariadb-client pod to be able to reach the source MariaDB:

   $ oc rsh mariadb-client mysql -rsh "${SOURCE_MARIADB_IP[default]}" -uroot -p"${SOURCE_DB_ROOT_PASSWORD[default]}" -e 'select 1;'

   Note

   For BGP-enabled environments, this command might take a few moments to succeed while BGP routes are advertised and propagated through the network. The mariadb-client pod needs to receive the route to the source MariaDB IP address through BGP before it can establish a connection. If the command fails, wait a few seconds and retry. The connection should succeed once the BGP route advertisement is complete.

   For IPv6 environments, this command might take a few moments to succeed while the network IPv6 stack completes its setup. If the command fails, wait a few seconds and retry.

   For standard deployments, such as non-BGP deployments or IPv4 deployments, this command should succeed immediately.

4. Export the shell variables for the following outputs and test the connection to the RHOSP database:

   $ unset PULL_OPENSTACK_CONFIGURATION_DATABASES
   $ declare -xA PULL_OPENSTACK_CONFIGURATION_DATABASES
   $ for CELL in $(echo $CELLS); do
   >     PULL_OPENSTACK_CONFIGURATION_DATABASES[$CELL]=$(oc rsh mariadb-client \
   >         mysql -rsh "${SOURCE_MARIADB_IP[$CELL]}" -uroot -p"${SOURCE_DB_ROOT_PASSWORD[$CELL]}" -e 'SHOW databases;')
   > done

   If the connection is successful, the expected output is nothing.

   Note

   The nova, nova_api, and nova_cell0 databases are included in the same database host for the main overcloud Orchestration service (heat) stack. Additional cells use the nova database of their local Galera clusters.

5. Run mysqlcheck on the RHOSP database to check for inaccuracies:

   $ unset PULL_OPENSTACK_CONFIGURATION_MYSQLCHECK_NOK
   $ declare -xA PULL_OPENSTACK_CONFIGURATION_MYSQLCHECK_NOK
   $ run_mysqlcheck() {
   >     PULL_OPENSTACK_CONFIGURATION_MYSQLCHECK_NOK=$(oc rsh mariadb-client \
   >         mysqlcheck --all-databases -h ${SOURCE_MARIADB_IP[$CELL]} -u root -p"${SOURCE_DB_ROOT_PASSWORD[$CELL]}" | grep -v OK)
   > }
   $ for CELL in $(echo $CELLS); do
   >     run_mysqlcheck $CELL
   > done
   $ if [ "$PULL_OPENSTACK_CONFIGURATION_MYSQLCHECK_NOK" != "" ]; then
   >     for CELL in $(echo $CELLS); do
   >         MYSQL_UPGRADE=$(oc rsh mariadb-client \
   >             mysql_upgrade --skip-version-check -v -h ${SOURCE_MARIADB_IP[$CELL]} -u root -p"${SOURCE_DB_ROOT_PASSWORD[$CELL]}")
   >         # rerun mysqlcheck to check if problem is resolved
   >         run_mysqlcheck
   >     done
   > fi

6. Get the Compute service (nova) cell mappings:

   export PULL_OPENSTACK_CONFIGURATION_NOVADB_MAPPED_CELLS=$(oc rsh mariadb-client \
       mysql -rsh "${SOURCE_MARIADB_IP[default]}" -uroot -p"${SOURCE_DB_ROOT_PASSWORD[default]}" nova_api -e \
       'select uuid,name,transport_url,database_connection,disabled from cell_mappings;')

7. Get the hostnames of the registered Compute services:

   $ unset PULL_OPENSTACK_CONFIGURATION_NOVA_COMPUTE_HOSTNAMES
   $ declare -xA PULL_OPENSTACK_CONFIGURATION_NOVA_COMPUTE_HOSTNAMES
   $ for CELL in $(echo $CELLS); do
   >     PULL_OPENSTACK_CONFIGURATION_NOVA_COMPUTE_HOSTNAMES[$CELL]=$(oc rsh mariadb-client \
   >         mysql -rsh "${SOURCE_MARIADB_IP[$CELL]}" -uroot -p"${SOURCE_DB_ROOT_PASSWORD[$CELL]}" -e \
   >             "select host from nova.services where services.binary='nova-compute' and deleted=0;")
   > done

8. Get the list of the mapped Compute service cells:

   export PULL_OPENSTACK_CONFIGURATION_NOVAMANAGE_CELL_MAPPINGS=$($CONTROLLER1_SSH sudo podman exec -it nova_conductor nova-manage cell_v2 list_cells)

9. Store the exported variables for future use:

   $ unset SRIOV_AGENTS
   $ declare -xA SRIOV_AGENTS
   $ for CELL in $(echo $CELLS); do
   >     RCELL=$CELL
   >     [ "$CELL" = "$DEFAULT_CELL_NAME" ] && RCELL=default
   >     cat > ~/.source_cloud_exported_variables_$CELL << EOF
   > unset PULL_OPENSTACK_CONFIGURATION_DATABASES
   > unset PULL_OPENSTACK_CONFIGURATION_MYSQLCHECK_NOK
   > unset PULL_OPENSTACK_CONFIGURATION_NOVA_COMPUTE_HOSTNAMES
   > declare -xA PULL_OPENSTACK_CONFIGURATION_DATABASES
   > declare -xA PULL_OPENSTACK_CONFIGURATION_MYSQLCHECK_NOK
   > declare -xA PULL_OPENSTACK_CONFIGURATION_NOVA_COMPUTE_HOSTNAMES
   > PULL_OPENSTACK_CONFIGURATION_DATABASES[$CELL]="$(oc rsh mariadb-client \
   >     mysql -rsh ${SOURCE_MARIADB_IP[$RCELL]} -uroot -p${SOURCE_DB_ROOT_PASSWORD[$RCELL]} -e 'SHOW databases;')"
   > PULL_OPENSTACK_CONFIGURATION_MYSQLCHECK_NOK[$CELL]="$(oc rsh mariadb-client \
   >     mysqlcheck --all-databases -h ${SOURCE_MARIADB_IP[$RCELL]} -u root -p${SOURCE_DB_ROOT_PASSWORD[$RCELL]} | grep -v OK)"
   > PULL_OPENSTACK_CONFIGURATION_NOVA_COMPUTE_HOSTNAMES[$CELL]="$(oc rsh mariadb-client \
   >     mysql -rsh ${SOURCE_MARIADB_IP[$RCELL]} -uroot -p${SOURCE_DB_ROOT_PASSWORD[$RCELL]} -e \
   >     "select host from nova.services where services.binary='nova-compute' and deleted=0;")"
   > if [ "$RCELL" = "default" ]; then
   >     PULL_OPENSTACK_CONFIGURATION_NOVADB_MAPPED_CELLS="$(oc rsh mariadb-client \
   >         mysql -rsh ${SOURCE_MARIADB_IP[$RCELL]} -uroot -p${SOURCE_DB_ROOT_PASSWORD[$RCELL]} nova_api -e \
   >             'select uuid,name,transport_url,database_connection,disabled from cell_mappings;')"
   >     PULL_OPENSTACK_CONFIGURATION_NOVAMANAGE_CELL_MAPPINGS="$($CONTROLLER1_SSH sudo podman exec -it nova_conductor nova-manage cell_v2 list_cells)"
   > fi
   > EOF
   > done
   $ chmod 0600 ~/.source_cloud_exported_variables*

   • declare -xA SRIOV_AGENTS gets the neutron-sriov-nic-agent configuration to use for the data plane adoption if neutron-sriov-nic-agent agents are running in your RHOSP deployment.

10. Clean up the mariadb-client pod:

    $ oc delete pod mariadb-client

    The mariadb-client pod is no longer needed after all the data is exported and stored.

Procedure

1. Ensure that you are using the Red Hat OpenShift Container Platform (RHOCP) namespace where you want the control plane to be deployed:

   $ oc project openstack

2. Create the RHOSP secret. For more information, see Providing secure access to the Red Hat OpenStack Services on OpenShift services in Deploying Red Hat OpenStack Services on OpenShift.

3. If the $ADMIN_PASSWORD is different than the password you set in osp-secret, amend the AdminPassword key in the osp-secret:

   $ oc set data secret/osp-secret "AdminPassword=$ADMIN_PASSWORD"

4. Set service account passwords in osp-secret to match the service account passwords from the original deployment:

   $ oc set data secret/osp-secret "AodhPassword=$AODH_PASSWORD"
   $ oc set data secret/osp-secret "BarbicanPassword=$BARBICAN_PASSWORD"
   $ oc set data secret/osp-secret "CeilometerPassword=$CEILOMETER_PASSWORD"
   $ oc set data secret/osp-secret "CinderPassword=$CINDER_PASSWORD"
   $ oc set data secret/osp-secret "GlancePassword=$GLANCE_PASSWORD"
   $ oc set data secret/osp-secret "HeatAuthEncryptionKey=$HEAT_AUTH_ENCRYPTION_KEY"
   $ oc set data secret/osp-secret "HeatPassword=$HEAT_PASSWORD"
   $ oc set data secret/osp-secret "HeatStackDomainAdminPassword=$HEAT_STACK_DOMAIN_ADMIN_PASSWORD"
   $ oc set data secret/osp-secret "IronicPassword=$IRONIC_PASSWORD"
   $ oc set data secret/osp-secret "IronicInspectorPassword=$IRONIC_PASSWORD"
   $ oc set data secret/osp-secret "ManilaPassword=$MANILA_PASSWORD"
   $ oc set data secret/osp-secret "MetadataSecret=$METADATA_SECRET"
   $ oc set data secret/osp-secret "NeutronPassword=$NEUTRON_PASSWORD"
   $ oc set data secret/osp-secret "NovaPassword=$NOVA_PASSWORD"
   $ oc set data secret/osp-secret "OctaviaPassword=$OCTAVIA_PASSWORD"
   $ oc set data secret/osp-secret "PlacementPassword=$PLACEMENT_PASSWORD"
   $ oc set data secret/osp-secret "SwiftPassword=$SWIFT_PASSWORD"

5. Deploy the OpenStackControlPlane CR. Ensure that you only enable the DNS, Galera, Memcached, and RabbitMQ services. All other services must be disabled:

   $ oc apply -f - <<EOF
   apiVersion: core.openstack.org/v1beta1
   kind: OpenStackControlPlane
   metadata:
     name: openstack
   spec:
     secret: osp-secret
     storageClass: <storage_class>
   
     barbican:
       enabled: false
       template:
         barbicanAPI: {}
         barbicanWorker: {}
         barbicanKeystoneListener: {}
   
     cinder:
       enabled: false
       template:
         cinderAPI: {}
         cinderScheduler: {}
         cinderBackup: {}
         cinderVolumes: {}
   
     dns:
       template:
         override:
           service:
             metadata:
               annotations:
                 metallb.universe.tf/address-pool: ctlplane
                 metallb.universe.tf/allow-shared-ip: ctlplane
                 metallb.universe.tf/loadBalancerIPs: <loadBalancer_IP>
   
             spec:
               type: LoadBalancer
         options:
         - key: server
           values:
           - <DNS_server_IP>
         replicas: 1
   
     glance:
       enabled: false
       template:
         glanceAPIs: {}
   
     heat:
       enabled: false
       template: {}
   
     horizon:
       enabled: false
       template: {}
   
     ironic:
       enabled: false
       template:
         ironicConductors: []
   
     keystone:
       enabled: false
       template: {}
   
     manila:
       enabled: false
       template:
         manilaAPI: {}
         manilaScheduler: {}
         manilaShares: {}
   
     galera:
       enabled: true
       templates:
         openstack:
           secret: osp-secret
           replicas: 3
           storageRequest: 5G
         openstack-cell1:
           secret: osp-secret
           replicas: 3
           storageRequest: 5G
         openstack-cell2:
           secret: osp-secret
           replicas: 1
           storageRequest: 5G
         openstack-cell3:
           secret: osp-secret
           replicas: 1
           storageRequest: 5G
     memcached:
       enabled: true
       templates:
         memcached:
           replicas: 3
   
     neutron:
       enabled: false
       template: {}
   
     nova:
       enabled: false
       template: {}
   
     ovn:
       enabled: false
       template:
         ovnController:
           networkAttachment: tenant
         ovnNorthd:
           replicas: 0
         ovnDBCluster:
           ovndbcluster-nb:
             replicas: 3
             dbType: NB
             networkAttachment: internalapi
           ovndbcluster-sb:
             replicas: 3
             dbType: SB
             networkAttachment: internalapi
   
     placement:
       enabled: false
       template: {}
   
     rabbitmq:
       templates:
         rabbitmq:
           override:
             service:
               metadata:
                 annotations:
                   metallb.universe.tf/address-pool: internalapi
                   metallb.universe.tf/loadBalancerIPs: <loadBalancer_IP>
               spec:
                 type: LoadBalancer
         rabbitmq-cell1:
           persistence:
             storage: 10Gi
           override:
             service:
               metadata:
                 annotations:
                   metallb.universe.tf/address-pool: internalapi
                   metallb.universe.tf/loadBalancerIPs: <loadBalancer_IP>
   
               spec:
                 type: LoadBalancer
         rabbitmq-cell2:
           persistence:
             storage: 10Gi
           override:
             service:
               metadata:
                 annotations:
                   metallb.universe.tf/address-pool: internalapi
                   metallb.universe.tf/loadBalancerIPs: <loadBalancer_IP>
               spec:
                 type: LoadBalancer
         rabbitmq-cell3:
           persistence:
             storage: 10Gi
           override:
             service:
               metadata:
                 annotations:
                   metallb.universe.tf/address-pool: internalapi
                   metallb.universe.tf/loadBalancerIPs: <loadBalancer_IP>
               spec:
                 type: LoadBalancer
     telemetry:
       enabled: false
     tls:
       podLevel:
         enabled: false
       ingress:
         enabled: false
     swift:
       enabled: false
       template:
         swiftRing:
           ringReplicas: 3
         swiftStorage:
           replicas: 0
         swiftProxy:
           replicas: 2
   EOF

   where:

   <DNS_server_IP>

   Specifies the value for the DNS server reachable from the dnsmasq pod on the RHOCP cluster network. You can specify a generic DNS server as the value, for example, 1.1.1.1, or a DNS server for a specific domain, for example, /google.com/8.8.8.8.

   Note

   This DNS service, dnsmasq, provides DNS services for nodes on the RHOSO data plane. dnsmasq is different from the RHOSO DNS service (designate) that provides DNS as a service for cloud tenants.

   <storage_class>

   Specifies an existing storage class in your RHOCP cluster.

   <loadBalancer_IP>

   Specifies the LoadBalancer IP address. If you use IPv6, change the load balancer IPs to the IPs in your environment, for example:

   ...
   metallb.universe.tf/allow-shared-ip: ctlplane
   metallb.universe.tf/loadBalancerIPs: fd00:aaaa::80
   ...
   metallb.universe.tf/address-pool: internalapi
   metallb.universe.tf/loadBalancerIPs: fd00:bbbb::85
   ...
   metallb.universe.tf/address-pool: internalapi
   metallb.universe.tf/loadBalancerIPs: fd00:bbbb::86

   • galera.openstack-cell1 provides the required infrastructure database and messaging services for the Compute cells, for example, cell1, cell2, and cell3. Adjust the values for fields such as replicas, storage, or storageRequest, for each Compute cell as needed.

   • spec.tls specifies whether TLS-e is enabled. If you enabled TLS-e in your RHOSP environment, set tls to the following:

     spec:
       ...
       tls:
         podLevel:
           enabled: true
           internal:
             ca:
               customIssuer: rootca-internal
           libvirt:
             ca:
               customIssuer: rootca-internal
           ovn:
             ca:
               customIssuer: rootca-internal
         ingress:
           ca:
             customIssuer: rootca-internal
           enabled: true

Procedure

1. Create the ceph-conf-files secret that includes the Red Hat Ceph Storage configuration:

   $ oc apply -f - <<EOF
   apiVersion: v1
   data:
     ceph.client.openstack.keyring: $CEPH_KEY
     ceph.conf: $CEPH_CONF
   kind: Secret
   metadata:
     name: ceph-conf-files
   type: Opaque
   EOF

   The content of the file should be similar to the following example:

   apiVersion: v1
   kind: Secret
   metadata:
     name: ceph-conf-files
   stringData:
     ceph.client.openstack.keyring: |
       [client.openstack]
           key = <secret key>
           caps mgr = "allow *"
           caps mon = "allow r, profile rbd"
           caps osd = "pool=vms, profile rbd pool=volumes, profile rbd pool=images, allow rw pool manila_data'
     ceph.conf: |
       [global]
       fsid = 7a1719e8-9c59-49e2-ae2b-d7eb08c695d4
       mon_host = 10.1.1.2,10.1.1.3,10.1.1.4

   where:

   mon_host

   specifies the addresses of the cluster’s monitors. If you use IPv6, use brackets for the mon_host. For example: mon_host = [v2:[fd00:cccc::100]:3300/0,v1:[fd00:cccc::100]:6789/0]

   Note

   For Distributed Compute Node (DCN) deployments with multiple Red Hat Ceph Storage clusters, create one secret per site. Each secret contains only the keys that the respective site requires. For more information on the rationale and key distribution pattern, see Red Hat Ceph Storage migration for Distributed Compute Node deployments.

   The Red Hat Ceph Storage configuration files for all clusters are available on the RHOSP controller at either /var/lib/tripleo-config/ceph/, or /etc/ceph. Copy them locally and create the per-site secrets:

   $ CEPH_SSH="ssh root@<controller>"
   $ CEPH_DIR="/var/lib/tripleo-config/ceph"
   $ TMPDIR=$(mktemp -d)
   
   $ $CEPH_SSH "cat ${CEPH_DIR}/central.conf" > ${TMPDIR}/central.conf
   $ $CEPH_SSH "sudo cat ${CEPH_DIR}/central.client.openstack.keyring" > ${TMPDIR}/central.client.openstack.keyring
   $ $CEPH_SSH "cat ${CEPH_DIR}/dcn1.conf" > ${TMPDIR}/dcn1.conf
   $ $CEPH_SSH "sudo cat ${CEPH_DIR}/dcn1.client.openstack.keyring" > ${TMPDIR}/dcn1.client.openstack.keyring
   $ $CEPH_SSH "cat ${CEPH_DIR}/dcn2.conf" > ${TMPDIR}/dcn2.conf
   $ $CEPH_SSH "sudo cat ${CEPH_DIR}/dcn2.client.openstack.keyring" > ${TMPDIR}/dcn2.client.openstack.keyring
   
   # Central site secret: contains all clusters
   $ oc create secret generic ceph-conf-central \
       --from-file=${TMPDIR}/central.conf \
       --from-file=${TMPDIR}/central.client.openstack.keyring \
       --from-file=${TMPDIR}/dcn1.conf \
       --from-file=${TMPDIR}/dcn1.client.openstack.keyring \
       --from-file=${TMPDIR}/dcn2.conf \
       --from-file=${TMPDIR}/dcn2.client.openstack.keyring \
       -n openstack
   
   # DCN1 edge site secret: central + local only
   $ oc create secret generic ceph-conf-dcn1 \
       --from-file=${TMPDIR}/central.conf \
       --from-file=${TMPDIR}/central.client.openstack.keyring \
       --from-file=${TMPDIR}/dcn1.conf \
       --from-file=${TMPDIR}/dcn1.client.openstack.keyring \
       -n openstack
   
   # DCN2 edge site secret: central + local only
   $ oc create secret generic ceph-conf-dcn2 \
       --from-file=${TMPDIR}/central.conf \
       --from-file=${TMPDIR}/central.client.openstack.keyring \
       --from-file=${TMPDIR}/dcn2.conf \
       --from-file=${TMPDIR}/dcn2.client.openstack.keyring \
       -n openstack
   
   $ rm -rf ${TMPDIR}

   Repeat for each additional edge site. Each edge site secret must include the central cluster files and only the files for that edge site’s local cluster.

   When configuring extraMounts on the OpenStackControlPlane, use propagation labels matching the service instance names (for example, central, dcn1, dcn2) so that each pod mounts only its site-specific secret.

2. In your OpenStackControlPlane CR, inject the Red Hat Ceph Storage configuration into the RHOSP service pods using extraMounts. For a single-cluster deployment, propagate one secret to all services:

   $ oc patch openstackcontrolplane openstack --type=merge --patch '
   spec:
     extraMounts:
       - name: v1
         region: r1
         extraVol:
           - propagation:
             - CinderVolume
             - CinderBackup
             - GlanceAPI
             - ManilaShare
             extraVolType: Ceph
             volumes:
             - name: ceph
               projected:
                 sources:
                 - secret:
                     name: ceph-conf-files
             mounts:
             - name: ceph
               mountPath: "/etc/ceph"
               readOnly: true
   '

   For a DCN deployment with per-site secrets, use propagation labels matching each service instance name so that each pod receives only the keys for its site:

   $ oc patch openstackcontrolplane openstack --type=merge --patch '
   spec:
     extraMounts:
     - name: v1
       region: r1
       extraVol:
       - extraVolType: Ceph
         propagation:
         - central
         - CinderBackup
         - ManilaShare
         volumes:
         - name: ceph-central
           projected:
             sources:
             - secret:
                 name: ceph-conf-central
         mounts:
         - name: ceph-central
           mountPath: "/etc/ceph"
           readOnly: true
       - extraVolType: Ceph
         propagation:
         - dcn1
         volumes:
         - name: ceph-dcn1
           projected:
             sources:
             - secret:
                 name: ceph-conf-dcn1
         mounts:
         - name: ceph-dcn1
           mountPath: "/etc/ceph"
           readOnly: true
       - extraVolType: Ceph
         propagation:
         - dcn2
         volumes:
         - name: ceph-dcn2
           projected:
             sources:
             - secret:
                 name: ceph-conf-dcn2
         mounts:
         - name: ceph-dcn2
           mountPath: "/etc/ceph"
           readOnly: true
   '

   The propagation label central matches the Image service and Block Storage service pod instances named central. The CinderBackup and ManilaShare labels are service-type propagation and apply to all Block Storage service backup and Shared File Systems service pods, which run only at the central site. Replace central, dcn1, and dcn2 with the instance names used in your deployment.

Procedure

1. If your deployment enables CephFS through NFS as a back end for Shared File Systems service (manila), remove the following Pacemaker ordering and co-location constraints that govern the Virtual IP address of the ceph-nfs service and the manila-share service:

   # check the co-location and ordering constraints concerning "manila-share"
   sudo pcs constraint list --full
   
   # remove these constraints
   sudo pcs constraint remove colocation-openstack-manila-share-ceph-nfs-INFINITY
   sudo pcs constraint remove order-ceph-nfs-openstack-manila-share-Optional

2. Disable RHOSP control plane services:

   # Update the services list to be stopped
   ServicesToStop=("tripleo_aodh_api.service"
                   "tripleo_aodh_api_cron.service"
                   "tripleo_aodh_evaluator.service"
                   "tripleo_aodh_listener.service"
                   "tripleo_aodh_notifier.service"
                   "tripleo_ceilometer_agent_central.service"
                   "tripleo_ceilometer_agent_notification.service"
                   "tripleo_designate_api.service"
                   "tripleo_designate_backend_bind9.service"
                   "tripleo_designate_central.service"
                   "tripleo_designate_mdns.service"
                   "tripleo_designate_producer.service"
                   "tripleo_designate_worker.service"
                   "tripleo_octavia_api.service"
                   "tripleo_octavia_health_manager.service"
                   "tripleo_octavia_rsyslog.service"
                   "tripleo_octavia_driver_agent.service"
                   "tripleo_octavia_housekeeping.service"
                   "tripleo_octavia_worker.service"
                   "tripleo_horizon.service"
                   "tripleo_keystone.service"
                   "tripleo_barbican_api.service"
                   "tripleo_barbican_worker.service"
                   "tripleo_barbican_keystone_listener.service"
                   "tripleo_cinder_api.service"
                   "tripleo_cinder_api_cron.service"
                   "tripleo_cinder_scheduler.service"
                   "tripleo_cinder_volume.service"
                   "tripleo_cinder_backup.service"
                   "tripleo_collectd.service"
                   "tripleo_glance_api.service"
                   "tripleo_gnocchi_api.service"
                   "tripleo_gnocchi_metricd.service"
                   "tripleo_gnocchi_statsd.service"
                   "tripleo_manila_api.service"
                   "tripleo_manila_api_cron.service"
                   "tripleo_manila_scheduler.service"
                   "tripleo_neutron_api.service"
                   "tripleo_placement_api.service"
                   "tripleo_nova_api_cron.service"
                   "tripleo_nova_api.service"
                   "tripleo_nova_conductor.service"
                   "tripleo_nova_metadata.service"
                   "tripleo_nova_scheduler.service"
                   "tripleo_nova_vnc_proxy.service"
                   "tripleo_aodh_api.service"
                   "tripleo_aodh_api_cron.service"
                   "tripleo_aodh_evaluator.service"
                   "tripleo_aodh_listener.service"
                   "tripleo_aodh_notifier.service"
                   "tripleo_ceilometer_agent_central.service"
                   "tripleo_ceilometer_agent_compute.service"
                   "tripleo_ceilometer_agent_ipmi.service"
                   "tripleo_ceilometer_agent_notification.service"
                   "tripleo_ovn_cluster_northd.service"
                   "tripleo_ironic_neutron_agent.service"
                   "tripleo_ironic_api.service"
                   "tripleo_ironic_inspector.service"
                   "tripleo_ironic_conductor.service"
                   "tripleo_ironic_inspector_dnsmasq.service"
                   "tripleo_ironic_pxe_http.service"
                   "tripleo_ironic_pxe_tftp.service"
                   "tripleo_unbound.service")
   
   PacemakerResourcesToStop=("openstack-cinder-volume"
                             "openstack-cinder-backup"
                             "openstack-manila-share")
   
   echo "Stopping systemd OpenStack services"
   for service in ${ServicesToStop[*]}; do
       for i in {1..3}; do
           SSH_CMD=CONTROLLER${i}_SSH
           if [ ! -z "${!SSH_CMD}" ]; then
               echo "Stopping the $service in controller $i"
               if ${!SSH_CMD} sudo systemctl is-active $service; then
                   ${!SSH_CMD} sudo systemctl stop $service
               fi
           fi
       done
   done
   
   echo "Checking systemd OpenStack services"
   for service in ${ServicesToStop[*]}; do
       for i in {1..3}; do
           SSH_CMD=CONTROLLER${i}_SSH
           if [ ! -z "${!SSH_CMD}" ]; then
               if ! ${!SSH_CMD} systemctl show $service | grep ActiveState=inactive >/dev/null; then
                   echo "ERROR: Service $service still running on controller $i"
               else
                   echo "OK: Service $service is not running on controller $i"
               fi
           fi
       done
   done
   
   echo "Stopping pacemaker OpenStack services"
   for i in {1..3}; do
       SSH_CMD=CONTROLLER${i}_SSH
       if [ ! -z "${!SSH_CMD}" ]; then
           echo "Using controller $i to run pacemaker commands"
           for resource in ${PacemakerResourcesToStop[*]}; do
               if ${!SSH_CMD} sudo pcs resource config $resource &>/dev/null; then
                   echo "Stopping $resource"
                   ${!SSH_CMD} sudo pcs resource disable $resource
               else
                   echo "Service $resource not present"
               fi
       done
           break
       fi
   done
   
   echo "Checking pacemaker OpenStack services"
   for i in {1..3}; do
       SSH_CMD=CONTROLLER${i}_SSH
       if [ ! -z "${!SSH_CMD}" ]; then
           echo "Using controller $i to run pacemaker commands"
           for resource in ${PacemakerResourcesToStop[*]}; do
               if ${!SSH_CMD} sudo pcs resource config $resource &>/dev/null; then
                   if ! ${!SSH_CMD} sudo pcs resource status $resource | grep Started; then
                       echo "OK: Service $resource is stopped"
                   else
                       echo "ERROR: Service $resource is started"
                   fi
               fi
           done
           break
       fi
       done

   If the status of each service is OK, then the services stopped successfully.

3. For Distributed Compute Node (DCN) deployments where Image service, Block Storage service, and Red Hat Ceph Storage services run on edge Compute nodes, stop the Image service, Block Storage service, and etcd services on all edge Compute nodes with the DistributedComputeHCI role:

   Note

   The DistributedComputeHCI role runs GlanceApiEdge, CinderVolumeEdge, and Etcd services. A minimum of three nodes per site use this role. Skip this step if your DCN deployment does not run these services on edge Compute nodes. The examples in this procedure use hyper-converged (HCI) roles. If your deployment does not use HCI, the same services apply to the DistributedCompute role, which runs the same GlanceApiEdge, CinderVolumeEdge, and Etcd services but without Ceph OSD, Ceph Monitor, or Ceph Manager.

   1. Define shell variables for your DistributedComputeHCI edge Compute nodes:

      # DCN1 edge site DistributedComputeHCI nodes
      DCN1_HCI0_SSH="ssh -i <path to SSH key> root@<dcn1-hci-0 IP>"
      DCN1_HCI1_SSH="ssh -i <path to SSH key> root@<dcn1-hci-1 IP>"
      DCN1_HCI2_SSH="ssh -i <path to SSH key> root@<dcn1-hci-2 IP>"
      
      # DCN2 edge site DistributedComputeHCI nodes
      DCN2_HCI0_SSH="ssh -i <path to SSH key> root@<dcn2-hci-0 IP>"
      DCN2_HCI1_SSH="ssh -i <path to SSH key> root@<dcn2-hci-1 IP>"
      DCN2_HCI2_SSH="ssh -i <path to SSH key> root@<dcn2-hci-2 IP>"

      where:

      <path to SSH key>

      Specifies the path to your SSH key for each DistributedComputeHCI edge Compute node on each DCN edge site.

      <dcn1-hci-0 IP>, <dcn1-hci-1 IP>, <dcn1-hci-2 IP>

      Specifies the IP address for each DistributedComputeHCI edge Compute node within the DCN1 edge site.

      <dcn2-hci-0 IP>, <dcn2-hci-1 IP>, <dcn2-hci-2 IP>

      Specifies the IP address for each DistributedComputeHCI edge Compute node within the DCN2 edge site.

   2. Stop the storage services on all DistributedComputeHCI nodes:

      # Services to stop on DistributedComputeHCI edge compute nodes
      DCN_HCI_SERVICES=("tripleo_glance_api_internal.service"
                         "tripleo_cinder_volume.service"
                         "tripleo_etcd.service")
      
      # List of all DistributedComputeHCI node SSH commands
      DCN_HCI_NODES=("$DCN1_HCI0_SSH"
                     "$DCN1_HCI1_SSH"
                     "$DCN1_HCI2_SSH"
                     "$DCN2_HCI0_SSH"
                     "$DCN2_HCI1_SSH"
                     "$DCN2_HCI2_SSH")
      
      echo "Stopping storage services on DistributedComputeHCI nodes"
      for node_ssh in "${DCN_HCI_NODES[@]}"; do
          [ -z "$node_ssh" ] && continue
          echo "Processing node: $node_ssh"
          for service in "${DCN_HCI_SERVICES[@]}"; do
              if $node_ssh sudo systemctl is-active $service 2>/dev/null; then
                  echo "Stopping $service"
                  $node_ssh sudo systemctl stop $service
              fi
          done
      done
      
      echo "Checking storage services on DistributedComputeHCI nodes"
      for node_ssh in "${DCN_HCI_NODES[@]}"; do
          [ -z "$node_ssh" ] && continue
          for service in "${DCN_HCI_SERVICES[@]}"; do
              if ! $node_ssh systemctl show $service 2>/dev/null | grep ActiveState=inactive >/dev/null; then
                  echo "ERROR: Service $service still running on $node_ssh"
              else
                  echo "OK: Service $service is not running on $node_ssh"
              fi
          done
      done

      Note
      • On edge sites, the Image service runs with the service name tripleo_glance_api_internal.service, which is different from the tripleo_glance_api.service on the central controller.
      • The Block Storage service volume service (tripleo_cinder_volume.service) uses the same service name on both edge sites and the central controller.
      • The etcd service (tripleo_etcd.service) is used as a distributed lock manager (DLM) for the Block Storage service volume service running in active/active mode on edge sites.

4. If your DCN deployment includes DistributedComputeHCIScaleOut nodes, stop the HAProxy service on those nodes:

   Note

   The DistributedComputeHCIScaleOut role is used to scale compute and storage capacity beyond the initial three DistributedComputeHCI nodes at each site. These nodes run HAProxyEdge, which proxies Image service requests to the GlanceApiEdge instances on DistributedComputeHCI nodes. Skip this step if your DCN deployment does not include DistributedComputeHCIScaleOut nodes. For non-HCI deployments, the equivalent role is DistributedComputeScaleOut, which runs the same HAProxyEdge service.

   1. Define shell variables for your DistributedComputeHCIScaleOut edge Compute nodes:

      # DCN1 edge site DistributedComputeHCIScaleOut nodes
      DCN1_SCALEOUT0_SSH="ssh -i <path to SSH key> root@<dcn1-scaleout-0 IP>"
      DCN1_SCALEOUT1_SSH="ssh -i <path to SSH key> root@<dcn1-scaleout-1 IP>"
      
      # DCN2 edge site DistributedComputeHCIScaleOut nodes
      DCN2_SCALEOUT0_SSH="ssh -i <path to SSH key> root@<dcn2-scaleout-0 IP>"
      DCN2_SCALEOUT1_SSH="ssh -i <path to SSH key> root@<dcn2-scaleout-1 IP>"

      where:

      <dcn1-scaleout-0 IP>, <dcn1-scaleout-1 IP>

      Specifies the IP address for each DistributedComputeHCIScaleOut node within the DCN1 edge site.

      <dcn2-scaleout-0 IP>, <dcn2-scaleout-1 IP>

      Specifies the IP address for each DistributedComputeHCIScaleOut node within the DCN2 edge site.

   2. Stop the services on all DistributedComputeHCIScaleOut nodes:

      # Services to stop on DistributedComputeHCIScaleOut edge compute nodes
      DCN_SCALEOUT_SERVICES=("tripleo_haproxy_edge.service")
      
      # List of all DistributedComputeHCIScaleOut node SSH commands
      DCN_SCALEOUT_NODES=("$DCN1_SCALEOUT0_SSH"
                           "$DCN1_SCALEOUT1_SSH"
                           "$DCN2_SCALEOUT0_SSH"
                           "$DCN2_SCALEOUT1_SSH")
      
      echo "Stopping services on DistributedComputeHCIScaleOut nodes"
      for node_ssh in "${DCN_SCALEOUT_NODES[@]}"; do
          [ -z "$node_ssh" ] && continue
          echo "Processing node: $node_ssh"
          for service in "${DCN_SCALEOUT_SERVICES[@]}"; do
              if $node_ssh sudo systemctl is-active $service 2>/dev/null; then
                  echo "Stopping $service"
                  $node_ssh sudo systemctl stop $service
              fi
          done
      done
      
      echo "Checking services on DistributedComputeHCIScaleOut nodes"
      for node_ssh in "${DCN_SCALEOUT_NODES[@]}"; do
          [ -z "$node_ssh" ] && continue
          for service in "${DCN_SCALEOUT_SERVICES[@]}"; do
              if ! $node_ssh systemctl show $service 2>/dev/null | grep ActiveState=inactive >/dev/null; then
                  echo "ERROR: Service $service still running on $node_ssh"
              else
                  echo "OK: Service $service is not running on $node_ssh"
              fi
          done
      done

      Note
      • The HAProxy edge service (tripleo_haproxy_edge.service) provided a local Image service endpoint on DistributedComputeHCIScaleOut nodes, proxying requests to the GlanceApiEdge instances on DistributedComputeHCI nodes. During adoption, Red Hat OpenShift Container Platform (RHOCP) Kubernetes service endpoints backed by MetalLB replace HAProxy.

Procedure

1. Check that the source Galera database clusters in each cell have its members online and synced:

   for CELL in $(echo $CELLS); do
     MEMBERS=SOURCE_GALERA_MEMBERS_$(echo ${CELL}|tr '[:lower:]' '[:upper:]')[@]
     for i in "${!MEMBERS}"; do
       echo "Checking for the database node $i WSREP status Synced"
       oc rsh mariadb-copy-data mysql \
         -h "$i" -uroot -p"${SOURCE_DB_ROOT_PASSWORD[$CELL]}" \
         -e "show global status like 'wsrep_local_state_comment'" | \
         grep -qE "\bSynced\b"
     done
   done

   Note

   Each additional Compute service (nova) v2 cell runs a dedicated Galera database cluster, so the command checks each cell.

2. Get the count of source databases with the NOK (not-OK) status:

   for CELL in $(echo $CELLS); do
     oc rsh mariadb-copy-data mysql -h "${SOURCE_MARIADB_IP[$CELL]}" -uroot -p"${SOURCE_DB_ROOT_PASSWORD[$CELL]}" -e "SHOW databases;"
   done

3. Check that mysqlcheck had no errors:

   for CELL in $(echo $CELLS); do
     set +u
     . ~/.source_cloud_exported_variables_$CELL
     set -u
     test -z "${PULL_OPENSTACK_CONFIGURATION_MYSQLCHECK_NOK[$CELL]}" || [ "${PULL_OPENSTACK_CONFIGURATION_MYSQLCHECK_NOK[$CELL]}" = " " ] && echo "OK" || echo "CHECK FAILED"
   done

4. Test the connection to the control plane upcall and cells databases:

   for CELL in $(echo "super $RENAMED_CELLS"); do
     oc rsh mariadb-copy-data mysql -rsh "${PODIFIED_MARIADB_IP[$CELL]}" -uroot -p"${PODIFIED_DB_ROOT_PASSWORD[$CELL]}" -e 'SHOW databases;'
   done

   Note

   You must transition Compute services that you import later into a superconductor architecture by deleting the old service records in the cell databases, starting with cell1. New records are registered with different hostnames that are provided by the Compute service operator. All Compute services, except the Compute agent, have no internal state, and you can safely delete their service records. You also need to rename the former default cell to DEFAULT_CELL_NAME.

5. Create a dump of the original databases:

   for CELL in $(echo $CELLS); do
     oc rsh mariadb-copy-data << EOF
       mysql -h"${SOURCE_MARIADB_IP[$CELL]}" -uroot -p"${SOURCE_DB_ROOT_PASSWORD[$CELL]}" \
       -N -e "show databases" | grep -E -v "schema|mysql|gnocchi|aodh" | \
       while read dbname; do
         echo "Dumping $CELL cell \${dbname}";
         mysqldump -h"${SOURCE_MARIADB_IP[$CELL]}" -uroot -p"${SOURCE_DB_ROOT_PASSWORD[$CELL]}" \
           --single-transaction --complete-insert --skip-lock-tables --lock-tables=0 \
           "\${dbname}" > /backup/"${CELL}.\${dbname}".sql;
       done
   EOF
   done

6. Restore the databases from .sql files into the control plane MariaDB:

   for CELL in $(echo $CELLS); do
     RCELL=$CELL
     [ "$CELL" = "default" ] && RCELL=$DEFAULT_CELL_NAME
     oc rsh mariadb-copy-data << EOF
       declare -A db_name_map
       db_name_map['nova']="nova_$RCELL"
       db_name_map['ovs_neutron']='neutron'
       db_name_map['ironic-inspector']='ironic_inspector'
       declare -A db_cell_map
       db_cell_map['nova']="nova_$DEFAULT_CELL_NAME"
       db_cell_map["nova_$RCELL"]="nova_$RCELL"
       declare -A db_server_map
       db_server_map['default']=${PODIFIED_MARIADB_IP['super']}
       db_server_map["nova"]=${PODIFIED_MARIADB_IP[$DEFAULT_CELL_NAME]}
       db_server_map["nova_$RCELL"]=${PODIFIED_MARIADB_IP[$RCELL]}
       declare -A db_server_password_map
       db_server_password_map['default']=${PODIFIED_DB_ROOT_PASSWORD['super']}
       db_server_password_map["nova"]=${PODIFIED_DB_ROOT_PASSWORD[$DEFAULT_CELL_NAME]}
       db_server_password_map["nova_$RCELL"]=${PODIFIED_DB_ROOT_PASSWORD[$RCELL]}
       cd /backup
       for db_file in \$(ls ${CELL}.*.sql); do
         db_name=\$(echo \${db_file} | awk -F'.' '{ print \$2; }')
         [[ "$CELL" != "default" && ! -v "db_cell_map[\${db_name}]" ]] && continue
         if [[ "$CELL" == "default" && -v "db_cell_map[\${db_name}]" ]] ; then
           target=$DEFAULT_CELL_NAME
         elif [[ "$CELL" == "default" && ! -v "db_cell_map[\${db_name}]" ]] ; then
           target=super
         else
           target=$RCELL
         fi
         renamed_db_file="\${target}_new.\${db_name}.sql"
         mv -f \${db_file} \${renamed_db_file}
         if [[ -v "db_name_map[\${db_name}]" ]]; then
           echo "renaming $CELL cell \${db_name} to \$target \${db_name_map[\${db_name}]}"
           db_name=\${db_name_map[\${db_name}]}
         fi
         db_server=\${db_server_map["default"]}
         if [[ -v "db_server_map[\${db_name}]" ]]; then
           db_server=\${db_server_map[\${db_name}]}
         fi
         db_password=\${db_server_password_map['default']}
         if [[ -v "db_server_password_map[\${db_name}]" ]]; then
           db_password=\${db_server_password_map[\${db_name}]}
         fi
         echo "creating $CELL cell \${db_name} in \$target \${db_server}"
         mysql -h"\${db_server}" -uroot "-p\${db_password}" -e \
           "CREATE DATABASE IF NOT EXISTS \${db_name} DEFAULT \
           CHARACTER SET ${CHARACTER_SET} DEFAULT COLLATE ${COLLATION};"
         echo "importing $CELL cell \${db_name} into \$target \${db_server} from \${renamed_db_file}"
         mysql -h "\${db_server}" -uroot "-p\${db_password}" "\${db_name}" < "\${renamed_db_file}"
       done
       if [ "$CELL" = "default" ] ; then
         mysql -h "\${db_server_map['default']}" -uroot -p"\${db_server_password_map['default']}" -e \
           "update nova_api.cell_mappings set name='$DEFAULT_CELL_NAME' where name='default';"
       fi
       mysql -h "\${db_server_map["nova_$RCELL"]}" -uroot -p"\${db_server_password_map["nova_$RCELL"]}" -e \
         "delete from nova_${RCELL}.services where host not like '%nova_${RCELL}-%' and services.binary != 'nova-compute';"
   EOF
   done

   • db_name_map defines which common databases to rename when importing them.
   • db_cell_map defines which cells databases to import, and how to rename them, if needed.
   • db_cell_map["nova_$RCELL"]="nova_$RCELL" omits importing special cell0 databases of the cells, as its contents cannot be consolidated during adoption.
   • db_server_map defines which databases to import into which servers, usually dedicated for cells.
   • db_server_password_map defines the root passwords map for database servers. You can only use the same password for now.
   • renamed_db_file="\${target}_new.\${db_name}.sql" assigns which databases to import into which hosts when extracting databases from the default cell.

1. Check that the databases are imported correctly:

   $ set +u
   $ . ~/.source_cloud_exported_variables_default
   $ set -u
   $ dbs=$(oc exec openstack-galera-0 -c galera -- mysql -rs -uroot -p"${PODIFIED_DB_ROOT_PASSWORD['super']}" -e 'SHOW databases;')
   $ echo $dbs | grep -Eq '\bkeystone\b' && echo "OK" || echo "CHECK FAILED"
   $ echo $dbs | grep -Eq '\bneutron\b' && echo "OK" || echo "CHECK FAILED"
   $ echo "${PULL_OPENSTACK_CONFIGURATION_DATABASES[@]}" | grep -Eq '\bovs_neutron\b' && echo "OK" || echo "CHECK FAILED"
   $ novadb_mapped_cells=$(oc exec openstack-galera-0 -c galera -- mysql -rs -uroot -p"${PODIFIED_DB_ROOT_PASSWORD['super']}" \
   >   nova_api -e 'select uuid,name,transport_url,database_connection,disabled from cell_mappings;')
   > uuidf='\S{8,}-\S{4,}-\S{4,}-\S{4,}-\S{12,}'
   > default=$(printf "%s\n" "$PULL_OPENSTACK_CONFIGURATION_NOVADB_MAPPED_CELLS" | sed -rn "s/^($uuidf)\s+default\b.*$/\1/p")
   > difference=$(diff -ZNua \
   >   <(printf "%s\n" "$PULL_OPENSTACK_CONFIGURATION_NOVADB_MAPPED_CELLS") \
   >   <(printf "%s\n" "$novadb_mapped_cells")) || true
   > if [ "$DEFAULT_CELL_NAME" != "default" ]; then
   >   printf "%s\n" "$difference" | grep -qE "^\-$default\s+default\b" && echo "OK" || echo "CHECK FAILED"
   >   printf "%s\n" "$difference" | grep -qE "^\+$default\s+$DEFAULT_CELL_NAME\b" && echo "OK" || echo "CHECK FAILED"
   >   [ $(grep -E "^[-\+]$uuidf" <<<"$difference" | wc -l) -eq 2 ] && echo "OK" || echo "CHECK FAILED"
   > else
   >   [ "x$difference" = "x" ] && echo "OK" || echo "CHECK FAILED"
   > fi
   > for CELL in $(echo $RENAMED_CELLS); do
   >   RCELL=$CELL
   >   [ "$CELL" = "$DEFAULT_CELL_NAME" ] && RCELL=default
   >   set +u
   >   . ~/.source_cloud_exported_variables_$RCELL
   >   set -u
   >   c1dbs=$(oc exec openstack-$CELL-galera-0 -c galera -- mysql -rs -uroot -p${PODIFIED_DB_ROOT_PASSWORD[$CELL]} -e 'SHOW databases;')
   >   echo $c1dbs | grep -Eq "\bnova_${CELL}\b" && echo "OK" || echo "CHECK FAILED"
   >   novadb_svc_records=$(oc exec openstack-$CELL-galera-0 -c galera -- mysql -rs -uroot -p${PODIFIED_DB_ROOT_PASSWORD[$CELL]} \
   >     nova_$CELL -e "select host from services where services.binary='nova-compute' and deleted=0 order by host asc;")
   >   diff -Z <(echo "x$novadb_svc_records") <(echo "x${PULL_OPENSTACK_CONFIGURATION_NOVA_COMPUTE_HOSTNAMES[@]}") && echo "OK" || echo "CHECK FAILED"
   > done

   • echo "${PULL_OPENSTACK_CONFIGURATION_DATABASES[@]}" | grep -Eq '\bovs_neutron\b' && echo "OK" || echo "CHECK FAILED" ensures that the Networking service (neutron) database is renamed from ovs_neutron.
   • nova_api -e 'select uuid,name,transport_url,database_connection,disabled from cell_mappings;') ensures that the default cell is renamed to $DEFAULT_CELL_NAME, and the cell UUIDs are retained.
   • for CELL in $(echo $RENAMED_CELLS); do ensures that the registered Compute services names have not changed.
   • c1dbs=$(oc exec openstack-$CELL-galera-0 -c galera -- mysql -rs -uroot -p${PODIFIED_DB_ROOT_PASSWORD[$CELL]} -e 'SHOW databases;') ensures Compute service cells databases are extracted to separate database servers, and renamed from nova to nova_cell<X>.
   • diff -Z <(echo "x$novadb_svc_records") <(echo "x${PULL_OPENSTACK_CONFIGURATION_NOVA_COMPUTE_HOSTNAMES[@]}") && echo "OK" || echo "CHECK FAILED" ensures that the registered Compute service name has not changed.

2. Delete the mariadb-data pod and the mariadb-copy-data persistent volume claim that contains the database backup:

   Note

   Consider taking a snapshot of them before deleting.

   $ oc delete pod mariadb-copy-data
   $ oc delete pvc mariadb-data

Procedure

1. Prepare a temporary PersistentVolume claim and the helper pod for the OVN backup. Adjust the storage requests for a large database, if needed:

   $ oc apply -f - <<EOF
   ---
   apiVersion: cert-manager.io/v1
   kind: Certificate
   metadata:
     name: ovn-data-cert
   spec:
     commonName: ovn-data-cert
     secretName: ovn-data-cert
     issuerRef:
       name: rootca-internal
   ---
   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: ovn-data
   spec:
     storageClassName: $STORAGE_CLASS
     accessModes:
       - ReadWriteOnce
     resources:
       requests:
         storage: 10Gi
   ---
   apiVersion: v1
   kind: Pod
   metadata:
     name: ovn-copy-data
     annotations:
       openshift.io/scc: anyuid
       k8s.v1.cni.cncf.io/networks: internalapi
     labels:
       app: adoption
   spec:
     containers:
     - image: $OVSDB_IMAGE
       command: [ "sh", "-c", "sleep infinity"]
       name: adoption
       volumeMounts:
       - mountPath: /backup
         name: ovn-data
       - mountPath: /etc/pki/tls/misc
         name: ovn-data-cert
         readOnly: true
     securityContext:
       allowPrivilegeEscalation: false
       capabilities:
         drop: ALL
       runAsNonRoot: true
       seccompProfile:
         type: RuntimeDefault
     volumes:
     - name: ovn-data
       persistentVolumeClaim:
         claimName: ovn-data
     - name: ovn-data-cert
       secret:
         secretName: ovn-data-cert
   EOF

2. Wait for the pod to be ready:

   $ oc wait --for=condition=Ready pod/ovn-copy-data --timeout=30s

3. If the podified internalapi cidr is different than the source internalapi cidr, add an iptables accept rule on the Controller nodes:

   $ $CONTROLLER1_SSH sudo iptables -I INPUT -s {PODIFIED_INTERNALAPI_NETWORK} -p tcp -m tcp --dport 6641 -m conntrack --ctstate NEW -j ACCEPT
   $ $CONTROLLER1_SSH sudo iptables -I INPUT -s {PODIFIED_INTERNALAPI_NETWORK} -p tcp -m tcp --dport 6642 -m conntrack --ctstate NEW -j ACCEPT

4. Back up your OVN databases:

   • If you did not enable TLS everywhere, run the following command:

     $ oc exec ovn-copy-data -- bash -c "ovsdb-client backup tcp:$SOURCE_OVSDB_IP:6641 > /backup/ovs-nb.db"
     $ oc exec ovn-copy-data -- bash -c "ovsdb-client backup tcp:$SOURCE_OVSDB_IP:6642 > /backup/ovs-sb.db"

   • If you enabled TLS everywhere, run the following command:

     $ oc exec ovn-copy-data -- bash -c "ovsdb-client backup --ca-cert=/etc/pki/tls/misc/ca.crt --private-key=/etc/pki/tls/misc/tls.key --certificate=/etc/pki/tls/misc/tls.crt ssl:$SOURCE_OVSDB_IP:6641 > /backup/ovs-nb.db"
     $ oc exec ovn-copy-data -- bash -c "ovsdb-client backup --ca-cert=/etc/pki/tls/misc/ca.crt --private-key=/etc/pki/tls/misc/tls.key --certificate=/etc/pki/tls/misc/tls.crt ssl:$SOURCE_OVSDB_IP:6642 > /backup/ovs-sb.db"

5. Start the control plane OVN database services prior to import, with northd disabled:

   $ oc patch openstackcontrolplane openstack --type=merge --patch '
   spec:
     ovn:
       enabled: true
       template:
         ovnDBCluster:
           ovndbcluster-nb:
             replicas: 3
             dbType: NB
             storageRequest: 10G
             networkAttachment: internalapi
           ovndbcluster-sb:
             replicas: 3
             dbType: SB
             storageRequest: 10G
             networkAttachment: internalapi
         ovnNorthd:
           replicas: 0
   '

6. Wait for the OVN database services to reach the Running phase:

   $ oc wait --for=jsonpath='{.status.phase}'=Running pod --selector=service=ovsdbserver-nb
   $ oc wait --for=jsonpath='{.status.phase}'=Running pod --selector=service=ovsdbserver-sb

7. Fetch the OVN database IP addresses on the clusterIP service network:

   PODIFIED_OVSDB_NB_IP=$(oc get svc --selector "statefulset.kubernetes.io/pod-name=ovsdbserver-nb-0" -ojsonpath='{.items[0].spec.clusterIP}')
   PODIFIED_OVSDB_SB_IP=$(oc get svc --selector "statefulset.kubernetes.io/pod-name=ovsdbserver-sb-0" -ojsonpath='{.items[0].spec.clusterIP}')

8. If you are using IPv6, adjust the address to the format expected by ovsdb-* tools:

   PODIFIED_OVSDB_NB_IP=[$PODIFIED_OVSDB_NB_IP]
   PODIFIED_OVSDB_SB_IP=[$PODIFIED_OVSDB_SB_IP]

9. Upgrade the database schema for the backup files:

   1. If you did not enable TLS everywhere, use the following command:

      $ oc exec ovn-copy-data -- bash -c "ovsdb-client get-schema tcp:$PODIFIED_OVSDB_NB_IP:6641 > /backup/ovs-nb.ovsschema && ovsdb-tool convert /backup/ovs-nb.db /backup/ovs-nb.ovsschema"
      $ oc exec ovn-copy-data -- bash -c "ovsdb-client get-schema tcp:$PODIFIED_OVSDB_SB_IP:6642 > /backup/ovs-sb.ovsschema && ovsdb-tool convert /backup/ovs-sb.db /backup/ovs-sb.ovsschema"

   2. If you enabled TLS everywhere, use the following command:

      $ oc exec ovn-copy-data -- bash -c "ovsdb-client get-schema --ca-cert=/etc/pki/tls/misc/ca.crt --private-key=/etc/pki/tls/misc/tls.key --certificate=/etc/pki/tls/misc/tls.crt ssl:$PODIFIED_OVSDB_NB_IP:6641 > /backup/ovs-nb.ovsschema && ovsdb-tool convert /backup/ovs-nb.db /backup/ovs-nb.ovsschema"
      $ oc exec ovn-copy-data -- bash -c "ovsdb-client get-schema --ca-cert=/etc/pki/tls/misc/ca.crt --private-key=/etc/pki/tls/misc/tls.key --certificate=/etc/pki/tls/misc/tls.crt ssl:$PODIFIED_OVSDB_SB_IP:6642 > /backup/ovs-sb.ovsschema && ovsdb-tool convert /backup/ovs-sb.db /backup/ovs-sb.ovsschema"

10. Restore the database backup to the new OVN database servers:

    1. If you did not enable TLS everywhere, use the following command:

       $ oc exec ovn-copy-data -- bash -c "ovsdb-client restore tcp:$PODIFIED_OVSDB_NB_IP:6641 < /backup/ovs-nb.db"
       $ oc exec ovn-copy-data -- bash -c "ovsdb-client restore tcp:$PODIFIED_OVSDB_SB_IP:6642 < /backup/ovs-sb.db"

    2. If you enabled TLS everywhere, use the following command:

       $ oc exec ovn-copy-data -- bash -c "ovsdb-client restore --ca-cert=/etc/pki/tls/misc/ca.crt --private-key=/etc/pki/tls/misc/tls.key --certificate=/etc/pki/tls/misc/tls.crt ssl:$PODIFIED_OVSDB_NB_IP:6641 < /backup/ovs-nb.db"
       $ oc exec ovn-copy-data -- bash -c "ovsdb-client restore --ca-cert=/etc/pki/tls/misc/ca.crt --private-key=/etc/pki/tls/misc/tls.key --certificate=/etc/pki/tls/misc/tls.crt ssl:$PODIFIED_OVSDB_SB_IP:6642 < /backup/ovs-sb.db"

11. Check that the data was successfully migrated by running the following commands against the new database servers, for example:

    $ oc exec -it ovsdbserver-nb-0 -- ovn-nbctl show
    $ oc exec -it ovsdbserver-sb-0 -- ovn-sbctl list Chassis

12. Start the control plane ovn-northd service to keep both OVN databases in sync:

    $ oc patch openstackcontrolplane openstack --type=merge --patch '
    spec:
      ovn:
        enabled: true
        template:
          ovnNorthd:
            replicas: 1
    '

13. If you are running OVN gateway services on RHOCP nodes, enable the control plane ovn-controller service:

    $ oc patch openstackcontrolplane openstack --type=merge --patch '
    spec:
      ovn:
        enabled: true
        template:
          ovnController:
            nicMappings:
              physNet: NIC
    '

    • physNet defines the name of your physical network. NIC is the name of the physical interface that is connected to your physical network.

      Note

      Running OVN gateways on RHOCP nodes might be prone to data plane downtime during Open vSwitch upgrades. Consider running OVN gateways on dedicated Networker data plane nodes for production deployments instead.

14. Delete the ovn-data helper pod and the temporary PersistentVolumeClaim that is used to store OVN database backup files:

    $ oc delete --ignore-not-found=true pod ovn-copy-data
    $ oc delete --ignore-not-found=true pvc ovn-data

    Note

    Consider taking a snapshot of the ovn-data helper pod and the temporary PersistentVolumeClaim before deleting them. For more information, see About volume snapshots in OpenShift Container Platform storage overview.

15. Stop the adopted OVN database servers:

    ServicesToStop=("tripleo_ovn_cluster_north_db_server.service"
                    "tripleo_ovn_cluster_south_db_server.service")
    
    echo "Stopping systemd OpenStack services"
    for service in ${ServicesToStop[*]}; do
        for i in {1..3}; do
            SSH_CMD=CONTROLLER${i}_SSH
            if [ ! -z "${!SSH_CMD}" ]; then
                echo "Stopping the $service in controller $i"
                if ${!SSH_CMD} sudo systemctl is-active $service; then
                    ${!SSH_CMD} sudo systemctl stop $service
                fi
            fi
        done
    done
    
    echo "Checking systemd OpenStack services"
    for service in ${ServicesToStop[*]}; do
        for i in {1..3}; do
            SSH_CMD=CONTROLLER${i}_SSH
            if [ ! -z "${!SSH_CMD}" ]; then
                if ! ${!SSH_CMD} systemctl show $service | grep ActiveState=inactive >/dev/null; then
                    echo "ERROR: Service $service still running on controller $i"
                else
                    echo "OK: Service $service is not running on controller $i"
                fi
            fi
        done
    done

Procedure

1. Patch the OpenStackControlPlane CR to deploy the Identity service:

   $ oc patch openstackcontrolplane openstack --type=merge --patch '
   spec:
     keystone:
       enabled: true
       apiOverride:
         route: {}
       template:
         override:
           service:
             internal:
               metadata:
                 annotations:
                   metallb.universe.tf/address-pool: internalapi
                   metallb.universe.tf/allow-shared-ip: internalapi
                   metallb.universe.tf/loadBalancerIPs: <172.17.0.80>
               spec:
                 type: LoadBalancer
         databaseInstance: openstack
         secret: osp-secret
   '

   where:

   <172.17.0.80>

   Specifies the load balancer IP in your environment. If you use IPv6, change the load balancer IP to the load balancer IP in your environment, for example, metallb.universe.tf/loadBalancerIPs: fd00:bbbb::80.

2. Create an alias to use the openstack command in the Red Hat OpenStack Services on OpenShift (RHOSO) deployment:

   $ alias openstack="oc exec -t openstackclient -- openstack"

3. Remove services and endpoints that still point to the RHOSP control plane, excluding the Identity service and its endpoints:

   $ openstack endpoint list | grep keystone | awk '/admin/{ print $2; }' | xargs ${BASH_ALIASES[openstack]} endpoint delete || true
   > for service in aodh heat heat-cfn barbican cinderv3 glance gnocchi manila manilav2 neutron nova placement swift ironic-inspector ironic octavia; do
   >  openstack service list | awk "/ $service /{ print \$2; }" | xargs -r ${BASH_ALIASES[openstack]} service delete || true
   > done

Verification

1. Verify that you can access the OpenStackClient pod. For more information, see Accessing the OpenStackClient pod in Maintaining the Red Hat OpenStack Services on OpenShift deployment.

2. Confirm that the Identity service endpoints are defined and are pointing to the control plane FQDNs:

   $ openstack endpoint list | grep keystone

3. Wait for the OpenStackControlPlane resource to become Ready:

   $ oc wait --for=condition=Ready --timeout=1m OpenStackControlPlane openstack

1. Create the secret that holds the domain-specific LDAP configuration files that the Identity service uses. Each file within the secret corresponds to an LDAP domain.
2. Patch the OpenStackControlPlane custom resource (CR) to enable domain-specific drivers for the Identity service and mount a secret that contains the LDAP configurations.

Procedure

1. To create the keystone-domains secret that stores the actual LDAP configuration files that Identity service uses, create a local file that includes your LDAP configuration, for example, keystone.myldapdomain.conf:

   The following example file includes the configuration for a single LDAP domain. If you have multiple LDAP domains, create a configuration file for each, for example, keystone.DOMAIN_ONE.conf, keystone.DOMAIN_TWO.conf.

   [identity]
   driver = ldap
   [ldap]
   url = ldap://<ldap_server_host>:<ldap_server_port>
   user = <bind_dn_user>
   password = <bind_dn_password>
   suffix = <user_tree_dn>
   query_scope = sub
   # User configuration
   user_tree_dn = <user_tree_dn>
   user_objectclass = <user_object_class>
   user_id_attribute = <user_id_attribute>
   user_name_attribute = <user_name_attribute>
   user_mail_attribute = <user_mail_attribute>
   user_enabled_attribute = <user_enabled_attribute>
   user_enabled_default = true
   # Group configuration
   group_tree_dn = <group_tree_dn>
   group_objectclass = <group_object_class>
   group_id_attribute = <group_id_attribute>
   group_name_attribute = <group_name_attribute>
   group_member_attribute = <group_member_attribute>
   group_members_are_ids = true

   • Replace the values, such as <ldap_server_host>, <bind_dn_user>, <user_tree_dn>, and so on, with your LDAP server details.

2. Create the secret from this file:

   $ oc create secret generic keystone-domains --from-file=<keystone.DOMAIN_NAME.conf>

   • Replace <keystone.DOMAIN_NAME.conf> with the name of your local configuration file. If applicable, include additional configuration files by using the --from-file option. After creating the secret, you can remove the local configuration file if it is no longer needed, or store it securely.

     Important

     The name of the file that you provide to --from-file, for example keystone.DOMAIN_NAME.conf, is critical. The Identity service uses this filename to map incoming authentication requests for a domain to the correct LDAP configuration. Ensure that DOMAIN_NAME matches the name of the domain you are configuring in the Identity service.

3. Patch the OpenStackControlPlane CR:

   $ oc patch openstackcontrolplane <cr_name> --type=merge -p '
   spec:
     keystone:
       template:
         customServiceConfig: |
             [identity]
             domain_specific_drivers_enabled = true
         extraMounts:
           - name: v1
             region: r1
             extraVol:
               - propagation:
                 - Keystone
                 extraVolType: Conf
                 volumes:
                 - name: keystone-domains
                   secret:
                     secretName: keystone-domains
                 mounts:
                 - name: keystone-domains
                   mountPath: "/etc/keystone/domains"
                   readOnly: true

   • Replace <cr_name> with the name of your OpenStackControlPlane CR (for example, openstack).

   • This patch does the following:

     • Sets spec.keystone.template.customServiceConfig. Ensure that you do not overwrite any previously defined value.

     • Defines spec.keystone.template.extraMounts to mount a secret named keystone-domains into the Identity service pods at /etc/keystone/domains. This secret contains your LDAP configuration files.

       Note

       You might need to wait a few minutes for the changes to propagate and for the Identity service pods to be updated.

Verification

1. Verify that users from the LDAP domain are accessible:

   $ oc exec -t openstackclient -- openstack user list --domain <domain_name>

   • Replace <domain_name> with your LDAP domain name.

     This command returns a list of users from your LDAP server.

2. Verify that groups from the LDAP domain are accessible:

   $ oc exec -t openstackclient -- openstack group list --domain <domain_name>

   This command returns a list of groups from your LDAP server.

3. Test authentication with an LDAP user:

   $ oc exec -t openstackclient -- openstack --os-auth-url <keystone_auth_url> --os-identity-api-version 3 --os-user-domain-name <domain_name> --os-username <ldap_username> --os-password <ldap_password> token issue

   • Replace <keystone_auth_url> with the Identity service authentication URL.

   • Replace <ldap_username> and <ldap_password> with valid LDAP user credentials.

     If successful, this command returns a token, confirming that LDAP authentication is working correctly.

4. Verify group membership for an LDAP user:

   $ oc exec -t openstackclient -- openstack group contains user --group-domain <domain_name> --user-domain <domain_name> <group_name> <username>

   • Replace <domain_name>, <group_name>, and <username> with the appropriate values from your LDAP server.

     This command verifies that the user is properly associated with the group through LDAP.

Procedure

1. Add the kek secret:

   $ oc set data secret/osp-secret "BarbicanSimpleCryptoKEK=$($CONTROLLER1_SSH "python3 -c \"import configparser; c = configparser.ConfigParser(); c.read('/var/lib/config-data/puppet-generated/barbican/etc/barbican/barbican.conf'); print(c['simple_crypto_plugin']['kek'])\"")"

2. Patch the OpenStackControlPlane CR to deploy the Key Manager service:

   $ oc patch openstackcontrolplane openstack --type=merge --patch '
   spec:
     barbican:
       enabled: true
       apiOverride:
         route: {}
       template:
         databaseInstance: openstack
         databaseAccount: barbican
         messagingBus:
           cluster: rabbitmq
         secret: osp-secret
         simpleCryptoBackendSecret: osp-secret
         serviceAccount: barbican
         serviceUser: barbican
         passwordSelectors:
           service: BarbicanPassword
           simplecryptokek: BarbicanSimpleCryptoKEK
         barbicanAPI:
           replicas: 1
           override:
             service:
               internal:
                 metadata:
                   annotations:
                     metallb.universe.tf/address-pool: internalapi
                     metallb.universe.tf/allow-shared-ip: internalapi
                     metallb.universe.tf/loadBalancerIPs: <172.17.0.80>
                 spec:
                   type: LoadBalancer
         barbicanWorker:
           replicas: 1
         barbicanKeystoneListener:
           replicas: 1
   '

   where:

   <172.17.0.80>

   Specifies the load balancer IP in your environment. If you use IPv6, change the load balancer IP to the load balancer IP in your environment, for example, metallb.universe.tf/loadBalancerIPs: fd00:bbbb::80.

   messagingBus.Cluster

   For more information about RHOSO RabbitMQ clusters, see RHOSO RabbitMQ clusters in Monitoring high availability services.

Procedure

1. Update your existing NetConfig CR to add subnets for each remote site. Each service network must include a subnet for the central site and each remote site. Use unique VLAN IDs for each site. For example:

   • Central site: VLANs 20-23
   • Edge site 1: VLANs 30-33

   • Edge site 2: VLANs 40-43

     apiVersion: network.openstack.org/v1beta1
     kind: NetConfig
     metadata:
       name: netconfig
     spec:
       networks:
       - name: ctlplane
         dnsDomain: ctlplane.example.com
         subnets:
         - name: <subnet1>
           allocationRanges:
           - end: 192.168.122.120
             start: 192.168.122.100
           cidr: 192.168.122.0/24
           gateway: 192.168.122.1
         - name: <ctlplanesite1>
           allocationRanges:
           - end: 192.168.133.120
             start: 192.168.133.100
           cidr: 192.168.133.0/24
           gateway: 192.168.133.1
         - name: <ctlplanesite2>
           allocationRanges:
           - end: 192.168.144.120
             start: 192.168.144.100
           cidr: 192.168.144.0/24
           gateway: 192.168.144.1
       - name: internalapi
         dnsDomain: internalapi.example.com
         subnets:
         - name: subnet1
           allocationRanges:
           - end: 172.17.0.250
             start: 172.17.0.100
           cidr: 172.17.0.0/24
           vlan: 20
         - name: internalapisite1
           allocationRanges:
           - end: 172.17.10.250
             start: 172.17.10.100
           cidr: 172.17.10.0/24
           vlan: 30
         - name: internalapisite2
           allocationRanges:
           - end: 172.17.20.250
             start: 172.17.20.100
           cidr: 172.17.20.0/24
           vlan: 40
       - name: storage
         dnsDomain: storage.example.com
         subnets:
         - name: subnet1
           allocationRanges:
           - end: 172.18.0.250
             start: 172.18.0.100
           cidr: 172.18.0.0/24
           vlan: 21
         - name: storagesite1
           allocationRanges:
           - end: 172.18.10.250
             start: 172.18.10.100
           cidr: 172.18.10.0/24
           vlan: 31
         - name: storagesite2
           allocationRanges:
           - end: 172.18.20.250
             start: 172.18.20.100
           cidr: 172.18.20.0/24
           vlan: 41
       - name: tenant
         dnsDomain: tenant.example.com
         subnets:
         - name: subnet1
           allocationRanges:
           - end: 172.19.0.250
             start: 172.19.0.100
           cidr: 172.19.0.0/24
           vlan: 22
         - name: tenantsite1
           allocationRanges:
           - end: 172.19.10.250
             start: 172.19.10.100
           cidr: 172.19.10.0/24
           vlan: 32
         - name: tenantsite2
           allocationRanges:
           - end: 172.19.20.250
             start: 172.19.20.100
           cidr: 172.19.20.0/24
           vlan: 42

     where:

     <subnet1>

     Specifies a user-defined subnet name for the central site subnet.

     <ctlplanesite1>

     Specifies a user-defined subnet for the first DCN edge site.

     <ctlplanesite2>

     Specifies a user-defined subnet for the second DCN edge site.

     Note

     You must have the storagemgmt network on OpenShift nodes when using DCN with Swift storage. It is not necessary when using Red Hat Ceph Storage.

2. Update the NetworkAttachmentDefinition CR for the internalapi network to include routes to remote site subnets. These routes fields enable control plane pods attached to the internalapi network, such as OVN Southbound database, to communicate with Compute nodes at remote sites through the central site gateway, and are required for DCN:

   apiVersion: k8s.cni.cncf.io/v1
   kind: NetworkAttachmentDefinition
   metadata:
     name: internalapi
   spec:
     config: |
       {
         "cniVersion": "0.3.1",
         "name": "internalapi",
         "type": "macvlan",
         "master": "internalapi",
         "ipam": {
           "type": "whereabouts",
           "range": "172.17.0.0/24",
           "range_start": "172.17.0.30",
           "range_end": "172.17.0.70",
           "routes": [
             { "dst": "172.17.10.0/24", "gw": "172.17.0.1" },
             { "dst": "172.17.20.0/24", "gw": "172.17.0.1" }
           ]
         }
       }

3. Update the NetworkAttachmentDefinition CR for the ctlplane network to include routes to remote site subnets:

   apiVersion: k8s.cni.cncf.io/v1
   kind: NetworkAttachmentDefinition
   metadata:
     name: ctlplane
   spec:
     config: |
       {
         "cniVersion": "0.3.1",
         "name": "ctlplane",
         "type": "macvlan",
         "master": "ospbr",
         "ipam": {
           "type": "whereabouts",
           "range": "192.168.122.0/24",
           "range_start": "192.168.122.30",
           "range_end": "192.168.122.70",
           "routes": [
             { "dst": "192.168.133.0/24", "gw": "192.168.122.1" },
             { "dst": "192.168.144.0/24", "gw": "192.168.122.1" }
           ]
         }
       }

4. Update the NetworkAttachmentDefinition CR for the storage network to include routes to remote site subnets:

   apiVersion: k8s.cni.cncf.io/v1
   kind: NetworkAttachmentDefinition
   metadata:
     name: storage
   spec:
     config: |
       {
         "cniVersion": "0.3.1",
         "name": "storage",
         "type": "macvlan",
         "master": "storage",
         "ipam": {
           "type": "whereabouts",
           "range": "172.18.0.0/24",
           "range_start": "172.18.0.30",
           "range_end": "172.18.0.70",
           "routes": [
             { "dst": "172.18.10.0/24", "gw": "172.18.0.1" },
             { "dst": "172.18.20.0/24", "gw": "172.18.0.1" }
           ]
         }
       }

5. Update the NetworkAttachmentDefinition CR for the tenant network to include routes to remote site subnets:

   apiVersion: k8s.cni.cncf.io/v1
   kind: NetworkAttachmentDefinition
   metadata:
     name: tenant
   spec:
     config: |
       {
         "cniVersion": "0.3.1",
         "name": "tenant",
         "type": "macvlan",
         "master": "tenant",
         "ipam": {
           "type": "whereabouts",
           "range": "172.19.0.0/24",
           "range_start": "172.19.0.30",
           "range_end": "172.19.0.70",
           "routes": [
             { "dst": "172.19.10.0/24", "gw": "172.19.0.1" },
             { "dst": "172.19.20.0/24", "gw": "172.19.0.1" }
           ]
         }
       }

   Note

   Adjust the IP ranges, subnets, and gateway addresses in all NAD configurations to match your network topology. The master interface name must match the interface on the OpenShift nodes where the VLAN is configured.

6. If you have already deployed OVN services, restart the OVN Southbound database pods to pick up the new routes:

   $ oc delete pod -l service=ovsdbserver-sb

   The pods are automatically recreated with the updated network configuration.

7. Configure the Networking service (neutron) to recognize all site physnets. In the OpenStackControlPlane CR, ensure the Networking service configuration includes all physnets:

   apiVersion: core.openstack.org/v1beta1
   kind: OpenStackControlPlane
   metadata:
     name: openstack
   spec:
     neutron:
       template:
         customServiceConfig: |
           [ml2_type_vlan]
           network_vlan_ranges = leaf0:1:1000,leaf1:1:1000,leaf2:1:1000
           [ovn]
           ovn_emit_need_to_frag = false

   where:

   leaf0

   Represents the physnet for the central site.

   leaf1

   Represents the physnet for the first remote site.

   leaf2

   Represents the physnet for the second remote site.

   Note

   Adjust the physnet names to match your Red Hat OpenStack Platform deployment. Common conventions include leaf0/leaf1/leaf2 or datacentre/dcn1/dcn2.

Verification

1. Verify that the NetConfig CR is created with all subnets:

   $ oc get netconfig netconfig -o yaml | grep -A2 "name: subnet1\|name: .*site"

2. Verify that each NetworkAttachmentDefinition includes routes to remote site subnets:

   for nad in ctlplane internalapi storage tenant; do
     echo "=== $nad ==="
     oc get net-attach-def $nad -o jsonpath='{.spec.config}' | jq '.ipam.routes
   done

3. After restarting OVN SB pods, verify they have routes to remote site subnets:

   $ oc exec $(oc get pod -l service=ovsdbserver-sb -o name | head -1) -- ip route show | grep 172.17

   Sample output:

   172.17.10.0/24 via 172.17.0.1 dev internalapi
   172.17.20.0/24 via 172.17.0.1 dev internalapi

Procedure

1. Create the swift-conf secret that includes the Object Storage service hash path suffix and prefix:

   $ oc apply -f - <<EOF
   apiVersion: v1
   kind: Secret
   metadata:
     name: swift-conf
   type: Opaque
   data:
     swift.conf: $($CONTROLLER1_SSH sudo cat /var/lib/config-data/puppet-generated/swift/etc/swift/swift.conf | base64 -w0)
   EOF

2. Create the swift-ring-files ConfigMap that includes the Object Storage service ring files:

   $ oc apply -f - <<EOF
   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: swift-ring-files
   binaryData:
     swiftrings.tar.gz: $($CONTROLLER1_SSH "cd /var/lib/config-data/puppet-generated/swift/etc/swift && tar cz *.builder *.ring.gz backups/ | base64 -w0")
     account.ring.gz: $($CONTROLLER1_SSH "base64 -w0 /var/lib/config-data/puppet-generated/swift/etc/swift/account.ring.gz")
     container.ring.gz: $($CONTROLLER1_SSH "base64 -w0 /var/lib/config-data/puppet-generated/swift/etc/swift/container.ring.gz")
     object.ring.gz: $($CONTROLLER1_SSH "base64 -w0 /var/lib/config-data/puppet-generated/swift/etc/swift/object.ring.gz")
   EOF

3. Patch the OpenStackControlPlane custom resource to deploy the Object Storage service:

   $ oc patch openstackcontrolplane openstack --type=merge --patch '
   spec:
     swift:
       enabled: true
       template:
         memcachedInstance: memcached
         swiftRing:
           ringReplicas: 3
         swiftStorage:
           replicas: 0
           networkAttachments:
           - storage
           storageClass: local-storage
           storageRequest: 10Gi
         swiftProxy:
           secret: osp-secret
           replicas: 2
           encryptionEnabled: false
           passwordSelectors:
             service: SwiftPassword
           serviceUser: swift
           override:
             service:
               internal:
                 metadata:
                   annotations:
                     metallb.universe.tf/address-pool: internalapi
                     metallb.universe.tf/allow-shared-ip: internalapi
                     metallb.universe.tf/loadBalancerIPs: <172.17.0.80>
                 spec:
                   type: LoadBalancer
           networkAttachments:
           - storage
   '

   • spec.swift.swiftStorage.storageClass must match the RHOSO deployment storage class.
   • metallb.universe.tf/loadBalancerIPs: <172.17.0.80> specifies the load balancer IP in your environment. If you use IPv6, change the load balancer IP to the load balancer IP in your environment, for example, metallb.universe.tf/loadBalancerIPs: fd00:bbbb::80.

   • spec.swift.swiftProxy.networkAttachments must match the network attachment for the previous Object Storage service configuration from the RHOSP deployment.

     Note

     If SwiftEncryptionEnabled: true was set in Red Hat OpenStack Platform, ensure that spec.swift.swiftProxy.encryptionEnabled is set to true and that the Key Manager service (barbican) adoption is complete before proceeding.

Procedure

1. Patch the OpenStackControlPlane CR to deploy the Compute service:

   Note

   This procedure assumes that Compute service metadata is deployed on the top level and not on each cell level. If the RHOSP deployment has a per-cell metadata deployment, adjust the following patch as needed. You cannot run the metadata service in cell0. To enable the metadata services of a local cell, set the enabled property in the metadataServiceTemplate field of the local cell to true in the OpenStackControlPlane CR.

   $ rm -f celltemplates
   $ for CELL in $(echo $RENAMED_CELLS); do
   > cat >> celltemplates << EOF
   >       ${CELL}:
   >         hasAPIAccess: true
   >         cellDatabaseAccount: nova-$CELL
   >         cellDatabaseInstance: openstack-$CELL
   >         cellMessageBusInstance: rabbitmq-$CELL
   >         metadataServiceTemplate:
   >           enabled: false
   >           override:
   >               service:
   >                 metadata:
   >                   annotations:
   >                     metallb.universe.tf/address-pool: internalapi
   >                     metallb.universe.tf/allow-shared-ip: internalapi
   >                     metallb.universe.tf/loadBalancerIPs: 172.17.0.$(( 79 + ${CELL##*cell} ))
   >                 spec:
   >                   type: LoadBalancer
   >           customServiceConfig: |
   >             [workarounds]
   >             disable_compute_service_check_for_ffu=true
   >         conductorServiceTemplate:
   >           customServiceConfig: |
   >             [workarounds]
   >             disable_compute_service_check_for_ffu=true
   >EOF
   >done
   
   $ cat > oscp-patch.yaml << EOF
   >spec:
   >  nova:
   >   enabled: true
   >   apiOverride:
   >     route: {}
   >   template:
   >     secret: osp-secret
   >     apiDatabaseAccount: nova-api
   >     apiServiceTemplate:
   >       override:
   >         service:
   >           internal:
   >             metadata:
   >               annotations:
   >                 metallb.universe.tf/address-pool: internalapi
   >                 metallb.universe.tf/allow-shared-ip: internalapi
   >                 metallb.universe.tf/loadBalancerIPs: <172.17.0.80>
   >             spec:
   >               type: LoadBalancer
   >       customServiceConfig: |
   >         [workarounds]
   >         disable_compute_service_check_for_ffu=true
   >     metadataServiceTemplate:
   >       enabled: true
   >       override:
   >         service:
   >           metadata:
   >             annotations:
   >               metallb.universe.tf/address-pool: internalapi
   >               metallb.universe.tf/allow-shared-ip: internalapi
   >               metallb.universe.tf/loadBalancerIPs: <172.17.0.80>
   >           spec:
   >             type: LoadBalancer
   >       customServiceConfig: |
   >         [workarounds]
   >         disable_compute_service_check_for_ffu=true
   >     schedulerServiceTemplate:
   >       customServiceConfig: |
   >         [workarounds]
   >         disable_compute_service_check_for_ffu=true
   >     cellTemplates:
   >       cell0:
   >         hasAPIAccess: true
   >         cellDatabaseAccount: nova-cell0
   >         cellDatabaseInstance: openstack
   >         cellMessageBusInstance: rabbitmq
   >         conductorServiceTemplate:
   >           customServiceConfig: |
   >             [workarounds]
   >             disable_compute_service_check_for_ffu=true
   >EOF
   $ cat celltemplates >> oscp-patch.yaml
   $ oc patch openstackcontrolplane openstack  --type=merge --patch-file=oscp-patch.yaml

   • ${CELL}.hasAPIAccess specifies upcall access to the API. In the source cloud, cells are always configured with the main Nova API database upcall access. You can disable upcall access to the API by setting hasAPIAccess to false. However, do not make changes to the API during adoption.
   • ${CELL}.cellDatabaseInstance specifies the database instance that is used by the cell. The database instance names must match the names that are defined in the OpenStackControlPlane CR that you created in when you deployed the back-end services as described in Deploying back-end services.
   • ${CELL}.cellMessageBusInstance specifies the message bus instance that is used by the cell. The message bus instance names must match the names that are defined in the OpenStackControlPlane CR.
   • metallb.universe.tf/loadBalancerIPs: <172.17.0.80> specifies the load balancer IP in your environment. If you use IPv6, change the load balancer IP to the load balancer IP in your environment, for example, metallb.universe.tf/loadBalancerIPs: fd00:bbbb::80.

2. If you are adopting the Compute service with the Bare Metal Provisioning service (ironic), append the novaComputeTemplates field with the following content in each cell in the Compute service CR patch. For example:

           cell1:
             novaComputeTemplates:
               standalone:
                 customServiceConfig: |
                   [DEFAULT]
                   host = <hostname>
                   [workarounds]
                   disable_compute_service_check_for_ffu=true
                 computeDriver: ironic.IronicDriver
           ...

   • Replace <hostname> with the hostname of the node that is running the ironic Compute driver in the source cloud.

3. Wait for the CRs for the Compute control plane services to be ready:

   $ oc wait --for condition=Ready --timeout=300s Nova/nova

   Note

   The local Conductor services are started for each cell, while the superconductor runs in cell0. Note that disable_compute_service_check_for_ffu is mandatory for all imported Compute services until the external data plane is imported, and until the Compute services are fast-forward upgraded. For more information, see Adopting Compute services to the RHOSO data plane and Performing a fast-forward upgrade on Compute services.