Chapter 4. Adopting Red Hat OpenStack Platform control plane services

Procedure

1. Write the patch manifest into a file, for example glance_swift.patch. For example:

   spec:
     glance:
       enabled: true
       apiOverride:
         route: {}
       template:
         databaseInstance: openstack
         storageClass: "local-storage"
         storageRequest: 10G
         customServiceConfig: |
             [DEFAULT]
             enabled_backends = default_backend:swift
             [glance_store]
             default_backend = default_backend
             [default_backend]
             swift_store_create_container_on_put = True
             swift_store_auth_version = 3
             swift_store_auth_address = {{ .KeystoneInternalURL }}
             swift_store_endpoint_type = internalURL
             swift_store_user = service:glance
             swift_store_key = {{ .ServicePassword }}
         glanceAPIs:
           default:
             replicas: 1
             override:
               service:
                 internal:
                   metadata:
                     annotations:
                       metallb.universe.tf/address-pool: internalapi
                       metallb.universe.tf/allow-shared-ip: internalapi
                       metallb.universe.tf/loadBalancerIPs: 172.17.0.80
                 spec:
                   type: LoadBalancer
             networkAttachments:
               - storage

   Having Object Storage service as a backend establishes a dependency between the two services, and any deployed GlanceAPI instance would not work if Image service is configured with Object Storage service that is still not available in the OpenStackControlPlane. Once Object Storage service, and in particular SwiftProxy, has been adopted, it is possible to proceed with the GlanceAPI adoption. For more information, see Adopting the Object Storage service.

2. Verify that SwiftProxy is available:

   $ oc get pod -l component=swift-proxy | grep Running
   swift-proxy-75cb47f65-92rxq   3/3     Running   0

3. Patch the GlanceAPI service deployed in the control plane context:

   $ oc patch openstackcontrolplane openstack --type=merge --patch-file=glance_swift.patch

Procedure

1. Write the patch manifest into a file, for example glance_cinder.patch. For example:

   spec:
     glance:
       enabled: true
       apiOverride:
         route: {}
       template:
         databaseInstance: openstack
         storageClass: "local-storage"
         storageRequest: 10G
         customServiceConfig: |
             [DEFAULT]
             enabled_backends = default_backend:cinder
             [glance_store]
             default_backend = default_backend
             [default_backend]
             rootwrap_config = /etc/glance/rootwrap.conf
             description = Default cinder backend
             cinder_store_auth_address = {{ .KeystoneInternalURL }}
             cinder_store_user_name = {{ .ServiceUser }}
             cinder_store_password = {{ .ServicePassword }}
             cinder_store_project_name = service
             cinder_catalog_info = volumev3::internalURL
             cinder_use_multipath = true
         glanceAPIs:
           default:
             replicas: 1
             override:
               service:
                 internal:
                   metadata:
                     annotations:
                       metallb.universe.tf/address-pool: internalapi
                       metallb.universe.tf/allow-shared-ip: internalapi
                       metallb.universe.tf/loadBalancerIPs: 172.17.0.80
                 spec:
                   type: LoadBalancer
             networkAttachments:
               - storage

   Having Block Storage service as a backend establishes a dependency between the two services, and any deployed GlanceAPI instance would not work if the Image service is configured with Block Storage service that is still not available in the OpenStackControlPlane. Once Block Storage service, and in particular CinderVolume, has been adopted, it is possible to proceed with the GlanceAPI adoption.

2. Verify that CinderVolume is available:

   $ oc get pod -l component=cinder-volume | grep Running
   cinder-volume-75cb47f65-92rxq   3/3     Running   0

3. Patch the GlanceAPI service deployed in the control plane context:

   oc patch openstackcontrolplane openstack --type=merge --patch-file=glance_cinder.patch

1. The Storage network has been propagated to the RHOSP control plane.
2. The Image service is able to reach the Storage network and connect to the nfs-server through the port 2049.

Procedure

1. Adopt the Image service and create a new default GlanceAPI instance connected with the existing NFS Ganesha share.

   cat << EOF > glance_nfs_patch.yaml
   
   spec:
     extraMounts:
     - extraVol:
       - extraVolType: Nfs
         mounts:
         - mountPath: /var/lib/glance/images
           name: nfs
         propagation:
         - Glance
         volumes:
         - name: nfs
           nfs:
             path: /var/nfs
             server: 172.17.3.20
       name: r1
       region: r1
     glance:
       enabled: true
       template:
         databaseInstance: openstack
         customServiceConfig: |
            [DEFAULT]
            enabled_backends = default_backend:file
            [glance_store]
            default_backend = default_backend
            [default_backend]
            filesystem_store_datadir = /var/lib/glance/images/
         storageClass: "local-storage"
         storageRequest: 10G
         glanceAPIs:
           default:
             replicas: 1
             type: single
             override:
               service:
                 internal:
                   metadata:
                     annotations:
                       metallb.universe.tf/address-pool: internalapi
                       metallb.universe.tf/allow-shared-ip: internalapi
                       metallb.universe.tf/loadBalancerIPs: 172.17.0.80
                 spec:
                   type: LoadBalancer
             networkAttachments:
             - storage
   EOF

   Note

   Replace in glance_nfs_patch.yaml the nfs/server IP address with the IP used to reach the nfs-server and make sure the nfs/path points to the exported path in the nfs-server.

2. Patch OpenStackControlPlane to deploy Image service with a NFS Ganesha backend:

   $ oc patch openstackcontrolplane openstack --type=merge --patch-file glance_nfs_patch.yaml

Procedure

1. Test the Image Service (glance) from the Red Hat OpenStack Platform CLI. You can compare and make sure the configuration has been correctly applied to the Image service pods:

   os-diff diff /etc/glance/glance.conf.d/02-config.conf glance_patch.yaml --frompod -p glance-api

   If no line appears, then the configuration is correctly done.

2. Inspect the resulting glance pods:

   GLANCE_POD=`oc get pod |grep glance-default-external-0 | cut -f 1 -d' '`
   oc exec -t $GLANCE_POD -c glance-api -- cat /etc/glance/glance.conf.d/02-config.conf
   
   [DEFAULT]
   enabled_backends=default_backend:rbd
   [glance_store]
   default_backend=default_backend
   [default_backend]
   rbd_store_ceph_conf=/etc/ceph/ceph.conf
   rbd_store_user=openstack
   rbd_store_pool=images
   store_description=Ceph glance store backend.

3. If you use a Ceph backend, ensure that the Ceph secrets are properly mounted:

   oc exec -t $GLANCE_POD -c glance-api -- ls /etc/ceph
   ceph.client.openstack.keyring
   ceph.conf

4. Check that the service is active and the endpoints are properly updated in the RHOSP CLI:

   (openstack)$ service list | grep image
   
   | fc52dbffef36434d906eeb99adfc6186 | glance    | image        |
   
   (openstack)$ endpoint list | grep image
   
   | 569ed81064f84d4a91e0d2d807e4c1f1 | regionOne | glance       | image        | True    | internal  | http://glance-internal-openstack.apps-crc.testing   |
   | 5843fae70cba4e73b29d4aff3e8b616c | regionOne | glance       | image        | True    | public    | http://glance-public-openstack.apps-crc.testing     |
   | 709859219bc24ab9ac548eab74ad4dd5 | regionOne | glance       | image        | True    | admin     | http://glance-admin-openstack.apps-crc.testing      |

5. Check that the images that you previously listed in the source Cloud are available in the adopted service:

   (openstack)$ image list
   +--------------------------------------+--------+--------+
   | ID                                   | Name   | Status |
   +--------------------------------------+--------+--------+
   | c3158cad-d50b-452f-bec1-f250562f5c1f | cirros | active |
   +--------------------------------------+--------+--------+

Procedure

1. It is recommended to write the patch manifest into a file, for example cinder.patch and then apply it with something like:

   oc patch openstackcontrolplane openstack --type=merge --patch-file=cinder.patch

   For example, for the RBD deployment from the Development Guide the cinder.patch would look like this:

   spec:
     extraMounts:
     - extraVol:
       - extraVolType: Ceph
         mounts:
         - mountPath: /etc/ceph
           name: ceph
           readOnly: true
         propagation:
         - CinderVolume
         - CinderBackup
         - Glance
         volumes:
         - name: ceph
           projected:
             sources:
             - secret:
                 name: ceph-conf-files
     cinder:
       enabled: true
       apiOverride:
         route: {}
       template:
         databaseInstance: openstack
         databaseAccount: cinder
         secret: osp-secret
         cinderAPI:
           override:
             service:
               internal:
                 metadata:
                   annotations:
                     metallb.universe.tf/address-pool: internalapi
                     metallb.universe.tf/allow-shared-ip: internalapi
                     metallb.universe.tf/loadBalancerIPs: 172.17.0.80
                 spec:
                   type: LoadBalancer
           replicas: 1
           customServiceConfig: |
             [DEFAULT]
             default_volume_type=tripleo
         cinderScheduler:
           replicas: 1
         cinderBackup:
           networkAttachments:
           - storage
           replicas: 1
           customServiceConfig: |
             [DEFAULT]
             backup_driver=cinder.backup.drivers.ceph.CephBackupDriver
             backup_ceph_conf=/etc/ceph/ceph.conf
             backup_ceph_user=openstack
             backup_ceph_pool=backups
         cinderVolumes:
           ceph:
             networkAttachments:
             - storage
             replicas: 1
             customServiceConfig: |
               [tripleo_ceph]
               backend_host=hostgroup
               volume_backend_name=tripleo_ceph
               volume_driver=cinder.volume.drivers.rbd.RBDDriver
               rbd_ceph_conf=/etc/ceph/ceph.conf
               rbd_user=openstack
               rbd_pool=volumes
               rbd_flatten_volume_from_snapshot=False
               report_discard_supported=True

2. Once the services have been deployed you need to clean up the old scheduler and backup services which will appear as being down while you have others that appear as being up:

   openstack volume service list
   
   +------------------+------------------------+------+---------+-------+----------------------------+
   | Binary           | Host                   | Zone | Status  | State | Updated At                 |
   +------------------+------------------------+------+---------+-------+----------------------------+
   | cinder-backup    | standalone.localdomain | nova | enabled | down  | 2023-06-28T11:00:59.000000 |
   | cinder-scheduler | standalone.localdomain | nova | enabled | down  | 2023-06-28T11:00:29.000000 |
   | cinder-volume    | hostgroup@tripleo_ceph | nova | enabled | up    | 2023-06-28T17:00:03.000000 |
   | cinder-scheduler | cinder-scheduler-0     | nova | enabled | up    | 2023-06-28T17:00:02.000000 |
   | cinder-backup    | cinder-backup-0        | nova | enabled | up    | 2023-06-28T17:00:01.000000 |
   +------------------+------------------------+------+---------+-------+----------------------------+

3. In this case you need to remove services for hosts standalone.localdomain

   oc exec -it cinder-scheduler-0 -- cinder-manage service remove cinder-backup standalone.localdomain
   oc exec -it cinder-scheduler-0 -- cinder-manage service remove cinder-scheduler standalone.localdomain

   The reason why we haven’t preserved the name of the backup service is because we have taken the opportunity to change its configuration to support Active-Active, even though we are not doing so right now because we have 1 replica.

4. Now that the Block Storage services are running, the DB schema migration has been completed and you can proceed to apply the DB data migrations. While it is not necessary to run these data migrations at this precise moment, because you can run them right before the next upgrade, for adoption it is best to run them now to make sure there are no issues before running production workloads on the deployment.

   The command to run the DB data migrations is:

   oc exec -it cinder-scheduler-0 -- cinder-manage db online_data_migrations

1. Ensure that the openstack alias is defined:

   alias openstack="oc exec -t openstackclient -- openstack"

1. See that Block Storage service endpoints are defined and pointing to the control plane FQDNs:

   openstack endpoint list --service cinderv3

2. Check that the Block Storage services are running and up. The API won’t show but if you get a response you know it’s up as well:

   openstack volume service list

3. Check that your old volume types, volumes, snapshots, and backups are there:

   openstack volume type list
   openstack volume list
   openstack volume snapshot list
   openstack volume backup list

1. Create a volume from an image to check that the connection to Image Service (glance) is working.

   openstack volume create --image cirros --bootable --size 1 disk_new

2. Backup the old attached volume to a new backup. Example:

   openstack --os-volume-api-version 3.47 volume create --backup backup restored

Procedure

1. Define the CONTROLLER1_SSH environment variable, if it hasn’t been defined already. Then copy the configuration file from RHOSP 17.1 for reference.

   $CONTROLLER1_SSH cat /var/lib/config-data/puppet-generated/manila/etc/manila/manila.conf | awk '!/^ *#/ && NF' > ~/manila.conf

2. Review this configuration alongside any configuration changes that were noted since RHOSP 17.1. Not all of it makes sense to bring into the new cloud environment:

   • The Shared File Systems service operator is capable of setting up database related configuration ([database]), service authentication (auth_strategy, [keystone_authtoken]), message bus configuration (transport_url, control_exchange), the default paste config (api_paste_config) and inter-service communication configuration ( [neutron], [nova], [cinder], [glance] [oslo_messaging_*]). So all of these can be ignored.
   • Ignore the osapi_share_listen configuration. In Red Hat OpenStack Services on OpenShift (RHOSO) 18.0, you rely on Red Hat OpenShift Container Platform routes and ingress.

   • Pay attention to policy overrides. In RHOSO 18.0, the Shared File Systems service ships with a secure default RBAC, and overrides may not be necessary. If a custom policy is necessary, you must provide it as a ConfigMap. The following sample spec illustrates how a ConfigMap called manila-policy can be set up with the contents of a file called policy.yaml.

       spec:
         manila:
           enabled: true
           template:
             manilaAPI:
               customServiceConfig: |
                  [oslo_policy]
                  policy_file=/etc/manila/policy.yaml
             extraMounts:
             - extraVol:
               - extraVolType: Undefined
                 mounts:
                 - mountPath: /etc/manila/
                   name: policy
                   readOnly: true
                 propagation:
                 - ManilaAPI
                 volumes:
                 - name: policy
                   projected:
                     sources:
                     - configMap:
                         name: manila-policy
                         items:
                           - key: policy
                             path: policy.yaml

   • You must preserve the value of the host option under the [DEFAULT] section as hostgroup.
   • The Shared File Systems service API service needs the enabled_share_protocols option to be added in the customServiceConfig section in manila: template: manilaAPI.
   • If you had scheduler overrides, add them to the customServiceConfig section in manila: template: manilaScheduler.
   • If you had multiple storage backend drivers configured with RHOSP 17.1, you will need to split them up when deploying RHOSO 18.0. Each storage backend driver needs to use its own instance of the manila-share service.

   • If a storage backend driver needs a custom container image, find it on the RHOSP Ecosystem Catalog and set manila: template: manilaShares: <custom name> : containerImage value. The following example illustrates multiple storage backend drivers, using custom container images.

       spec:
         manila:
           enabled: true
           template:
             manilaAPI:
               customServiceConfig: |
                 [DEFAULT]
                 enabled_share_protocols = nfs
               replicas: 3
             manilaScheduler:
               replicas: 3
             manilaShares:
              netapp:
                customServiceConfig: |
                  [DEFAULT]
                  debug = true
                  enabled_share_backends = netapp
                  host = hostgroup
                  [netapp]
                  driver_handles_share_servers = False
                  share_backend_name = netapp
                  share_driver = manila.share.drivers.netapp.common.NetAppDriver
                  netapp_storage_family = ontap_cluster
                  netapp_transport_type = http
                replicas: 1
              pure:
                 customServiceConfig: |
                  [DEFAULT]
                  debug = true
                  enabled_share_backends=pure-1
                  host = hostgroup
                  [pure-1]
                  driver_handles_share_servers = False
                  share_backend_name = pure-1
                  share_driver = manila.share.drivers.purestorage.flashblade.FlashBladeShareDriver
                  flashblade_mgmt_vip = 203.0.113.15
                  flashblade_data_vip = 203.0.10.14
                 containerImage: registry.connect.redhat.com/purestorage/openstack-manila-share-pure-rhosp-18-0
                 replicas: 1

3. If providing sensitive information, such as passwords, hostnames and usernames, it is recommended to use Red Hat OpenShift Container Platform secrets, and the customServiceConfigSecrets key. An example:

   cat << __EOF__ > ~/netapp_secrets.conf
   
   [netapp]
   netapp_server_hostname = 203.0.113.10
   netapp_login = fancy_netapp_user
   netapp_password = secret_netapp_password
   netapp_vserver = mydatavserver
   __EOF__

   oc create secret generic osp-secret-manila-netapp --from-file=~/netapp_secrets.conf -n openstack

   • customConfigSecrets can be used in any service, the following is a config example using the secret you created above.

       spec:
         manila:
           enabled: true
           template:
             < . . . >
             manilaShares:
              netapp:
                customServiceConfig: |
                  [DEFAULT]
                  debug = true
                  enabled_share_backends = netapp
                  host = hostgroup
                  [netapp]
                  driver_handles_share_servers = False
                  share_backend_name = netapp
                  share_driver = manila.share.drivers.netapp.common.NetAppDriver
                  netapp_storage_family = ontap_cluster
                  netapp_transport_type = http
                customServiceConfigSecrets:
                  - osp-secret-manila-netapp
                replicas: 1
         < . . . >

   • If you need to present extra files to any of the services, you can use extraMounts. For example, when using ceph, you’d need the Shared File Systems service ceph user’s keyring file as well as the ceph.conf configuration file available. These are mounted via extraMounts as done with the example below.
   • Ensure that the names of the backends (share_backend_name) remain as they did on RHOSP 17.1.
   • It is recommended to set the replica count of the manilaAPI service and the manilaScheduler service to 3. You should ensure to set the replica count of the manilaShares service/s to 1.
   • Ensure that the appropriate storage management network is specified in the manilaShares section. The example below connects the manilaShares instance with the CephFS backend driver to the storage network.
   • Prior to adopting the manilaShares service for CephFS through NFS, ensure that you have a clustered Ceph NFS service created. You will need to provide the name of the service as ``cephfs_nfs_cluster_id``.

4. Patch OpenStackControlPlane to deploy the Shared File Systems service; here’s an example that uses Native CephFS:

   cat << __EOF__ > ~/manila.patch
   spec:
     manila:
       enabled: true
       apiOverride:
         route: {}
       template:
         databaseInstance: openstack
         databaseAccount: manila
         secret: osp-secret
         manilaAPI:
           replicas: 3
           customServiceConfig: |
             [DEFAULT]
             enabled_share_protocols = cephfs
           override:
             service:
               internal:
                 metadata:
                   annotations:
                     metallb.universe.tf/address-pool: internalapi
                     metallb.universe.tf/allow-shared-ip: internalapi
                     metallb.universe.tf/loadBalancerIPs: 172.17.0.80
                 spec:
                   type: LoadBalancer
         manilaScheduler:
           replicas: 3
         manilaShares:
           cephfs:
             replicas: 1
             customServiceConfig: |
               [DEFAULT]
               enabled_share_backends = tripleo_ceph
               host = hostgroup
               [cephfs]
               driver_handles_share_servers=False
               share_backend_name=cephfs
               share_driver=manila.share.drivers.cephfs.driver.CephFSDriver
               cephfs_conf_path=/etc/ceph/ceph.conf
               cephfs_auth_id=openstack
               cephfs_cluster_name=ceph
               cephfs_volume_mode=0755
               cephfs_protocol_helper_type=CEPHFS
             networkAttachments:
                 - storage
   __EOF__

   Below is an example that uses CephFS through NFS. In this example:

   • The cephfs_ganesha_server_ip option is preserved from the configuration on the old RHOSP 17.1 environment.

   • The cephfs_nfs_cluster_id option is set with the name of the NFS cluster created on Ceph.

     cat << __EOF__ > ~/manila.patch
     spec:
       manila:
         enabled: true
         apiOverride:
           route: {}
         template:
           databaseInstance: openstack
           secret: osp-secret
           manilaAPI:
             replicas: 3
             customServiceConfig: |
               [DEFAULT]
               enabled_share_protocols = cephfs
             override:
               service:
                 internal:
                   metadata:
                     annotations:
                       metallb.universe.tf/address-pool: internalapi
                       metallb.universe.tf/allow-shared-ip: internalapi
                       metallb.universe.tf/loadBalancerIPs: 172.17.0.80
                   spec:
                     type: LoadBalancer
           manilaScheduler:
             replicas: 3
           manilaShares:
             cephfs:
               replicas: 1
               customServiceConfig: |
                 [DEFAULT]
                 enabled_share_backends = cephfs
                 host = hostgroup
                 [cephfs]
                 driver_handles_share_servers=False
                 share_backend_name=tripleo_ceph
                 share_driver=manila.share.drivers.cephfs.driver.CephFSDriver
                 cephfs_conf_path=/etc/ceph/ceph.conf
                 cephfs_auth_id=openstack
                 cephfs_cluster_name=ceph
                 cephfs_protocol_helper_type=NFS
                 cephfs_nfs_cluster_id=cephfs
                 cephfs_ganesha_server_ip=172.17.5.47
               networkAttachments:
                   - storage
     __EOF__

     oc patch openstackcontrolplane openstack --type=merge --patch-file=~/manila.patch

Verification

1. Inspect the resulting Shared File Systems service pods:

   oc get pods -l service=manila

2. Check that the Shared File Systems service API service is registered in Identity service (keystone):

   openstack service list | grep manila

   openstack endpoint list | grep manila
   
   | 1164c70045d34b959e889846f9959c0e | regionOne | manila       | share        | True    | internal  | http://manila-internal.openstack.svc:8786/v1/%(project_id)s        |
   | 63e89296522d4b28a9af56586641590c | regionOne | manilav2     | sharev2      | True    | public    | https://manila-public-openstack.apps-crc.testing/v2                |
   | af36c57adcdf4d50b10f484b616764cc | regionOne | manila       | share        | True    | public    | https://manila-public-openstack.apps-crc.testing/v1/%(project_id)s |
   | d655b4390d7544a29ce4ea356cc2b547 | regionOne | manilav2     | sharev2      | True    | internal  | http://manila-internal.openstack.svc:8786/v2                       |

3. Test the health of the service:

   openstack share service list
   openstack share pool list --detail

4. Check on existing workloads:

   openstack share list
   openstack share snapshot list

5. You can create further resources:

   openstack share create cephfs 10 --snapshot mysharesnap --name myshareclone
   openstack share create nfs 10 --name mynfsshare
   openstack share export location list mynfsshare

Procedure

1. Once the old NFS service is no longer used, you can adjust the configuration for the manila-share service to remove the cephfs_ganesha_server_ip option. Doing this will restart the manila-share process and remove the export locations that pertained to the old NFS service from all the shares.

   cat << __EOF__ > ~/manila.patch
   spec:
     manila:
       enabled: true
       apiOverride:
         route: {}
       template:
         manilaShares:
           cephfs:
             replicas: 1
             customServiceConfig: |
               [DEFAULT]
               enabled_share_backends = cephfs
               host = hostgroup
               [cephfs]
               driver_handles_share_servers=False
               share_backend_name=cephfs
               share_driver=manila.share.drivers.cephfs.driver.CephFSDriver
               cephfs_conf_path=/etc/ceph/ceph.conf
               cephfs_auth_id=openstack
               cephfs_cluster_name=ceph
               cephfs_protocol_helper_type=NFS
               cephfs_nfs_cluster_id=cephfs
             networkAttachments:
                 - storage
   __EOF__

   oc patch openstackcontrolplane openstack --type=merge --patch-file=~/manila.patch

2. To cleanup the standalone ceph nfs service from the RHOSP control plane nodes, you can disable and delete the pacemaker resources associated with the service. Replace <VIP> in the following commands with the IP address assigned to the ceph-nfs service in your environment.

   sudo pcs resource disable ceph-nfs
   sudo pcs resource disable ip-<VIP>
   sudo pcs resource unmanage ceph-nfs
   sudo pcs resource unmanage ip-<VIP>

Procedure

1. Patch the OpenStackControlPlane to deploy the Bare Metal Provisioning service:

   oc patch openstackcontrolplane openstack -n openstack --type=merge --patch '
   spec:
     ironic:
       enabled: true
       template:
         rpcTransport: oslo
         databaseInstance: openstack
         ironicAPI:
           replicas: 1
           override:
             service:
               internal:
                 metadata:
                   annotations:
                     metallb.universe.tf/address-pool: internalapi
                     metallb.universe.tf/allow-shared-ip: internalapi
                     metallb.universe.tf/loadBalancerIPs: 172.17.0.80
                 spec:
                   type: LoadBalancer
         ironicConductors:
         - replicas: 1
           networkAttachments:
             - baremetal
           provisionNetwork: baremetal
           storageRequest: 10G
           customServiceConfig: |
             [neutron]
             cleaning_network=<cleaning network uuid>
             provisioning_network=<provisioning network uuid>
             rescuing_network=<rescuing network uuid>
             inspection_network=<introspection network uuid>
             [conductor]
             automated_clean=true
         ironicInspector:
           replicas: 1
           inspectionNetwork: baremetal
           networkAttachments:
             - baremetal
           dhcpRanges:
             - name: inspector-0
               cidr: 172.20.1.0/24
               start: 172.20.1.190
               end: 172.20.1.199
               gateway: 172.20.1.1
           serviceUser: ironic-inspector
           databaseAccount: ironic-inspector
           passwordSelectors:
             database: IronicInspectorDatabasePassword
             service: IronicInspectorPassword
         ironicNeutronAgent:
           replicas: 1
           rabbitMqClusterName: rabbitmq
         secret: osp-secret
   '

   The operator begins to apply the configuration and start the necessary Bare Metal Provisioning services. Once the services have reached a running state, the Bare Metal Provisioning service automatically begins polling the power state of bare metal nodes for which it is configured to manage.

2. Wait for Bare Metal Provisioning control plane services' custom resources to become ready:

   oc wait --for condition=Ready --timeout=300s ironics.ironic.openstack.org ironic
   //kgilliga: Is "optionally verify the individual service" part of the code block, or is it a separate step?
   # Optionally verify the individual services
   oc wait --for condition=Ready --timeout=300s ironicapis.ironic.openstack.org ironic-api
   oc wait --for condition=Ready --timeout=300s ironicconductors.ironic.openstack.org ironic-conductor
   oc wait --for condition=Ready --timeout=300s ironicinspectors.ironic.openstack.org ironic-inspector
   oc wait --for condition=Ready --timeout=300s ironicneutronagents.ironic.openstack.org ironic-ironic-neutron-agent

3. Update the DNS Nameservers on the provisoning/cleaning/rescue networks. For name resolution to work for Bare Metal Provisioning service operations the DNS nameserver must be set to use the internal DNS servers in the RHOSO control plane:

   openstack subnet set --dns-nameserver 192.168.122.80 provisioning-subnet

4. Your Bare Metal Provisioning service nodes might be missing from a openstack baremetal node list command due to increased access restrictions in the role-based access control model. To see the nodes again, temporarily disable the new role- based access control policy, which you can then re-enable after setting the owner field on the nodes.

   oc patch openstackcontrolplane openstack -n openstack --type=merge --patch '
   spec:
     ironic:
       enabled: true
       template:
         databaseInstance: openstack
         ironicAPI:
           replicas: 1
           customServiceConfig: |
             [oslo_policy]
             enforce_scope=false
             enforce_new_defaults=false
   '

5. Once this configuration has applied, the operator restarts the Ironic API service disabling the new RBAC policy which is enabled by default. After which, you should be able to view bare metal nodes without an owner field:

   openstack baremetal node list -f uuid,provision_state,owner

6. Run the following command to assign all bare metal nodes with no owner to a new project, for example, the "admin" project:

   ADMIN_PROJECT_ID=$(openstack project show -c id -f value --domain default admin)
   for node in $(openstack baremetal node list -f json -c UUID -c Owner | jq -r '.[] | select(.Owner == null) | .UUID'); do openstack baremetal node set --owner $ADMIN_PROJECT_ID $node; done

7. Re-apply the default access control policy:

   oc patch openstackcontrolplane openstack -n openstack --type=merge --patch '
   spec:
     ironic:
       enabled: true
       template:
         databaseInstance: openstack
         ironicAPI:
           replicas: 1
           customServiceConfig: |
             [oslo_policy]
             enforce_scope=true
             enforce_new_defaults=true
   '

Procedure

1. Update your ssh parameters according to your environment in the os-diff.cfg. Os-diff uses those parameters to connect to your director node, query and download the configuration files:

   ssh_cmd=ssh -F ssh.config standalone
   container_engine=podman
   connection=ssh
   remote_config_path=/tmp/tripleo

   Make sure the ssh command you provide in ssh_cmd parameter is correct and with key authentication.

2. Enable or disable the services that you want in the /etc/os-diff/config.yaml file. Make sure that you have the correct rights to edit the file, for example:

   chown ospng:ospng /etc/os-diff/config.yaml

1. If you are using non-containerized services, such as the ovs-external-ids, os-diff can pull configuration or command output:

   services:
     ovs_external_ids:
       hosts:
         - standalone
       service_command: "ovs-vsctl list Open_vSwitch . | grep external_ids | awk -F ': ' '{ print $2; }'"
       cat_output: true
       path:
         - ovs_external_ids.json
       config_mapping:
         ovn-bridge-mappings: edpm_ovn_bridge_mappings
         ovn-bridge: edpm_ovn_bridge
         ovn-encap-type: edpm_ovn_encap_type
         ovn-match-northd-version: ovn_match_northd_version
         ovn-monitor-all: ovn_monitor_all
         ovn-remote-probe-interval: edpm_ovn_remote_probe_interval
         ovn-ofctrl-wait-before-clear: edpm_ovn_ofctrl_wait_before_clear

   This service is not an Red Hat OpenStack Platform service executed in a container, so the description and the behavior is different. It is important to correctly configure an SSH config file or equivalent for non-standard services such as OVS. The ovs_external_ids does not run in a container, and the ovs data is stored on each host of our cloud: controller_1/controller_2/…​

   With the hosts key, os-diff loops on each host and runs the command in the service_command key:

       ovs_external_ids:
           path:
               - ovs_external_ids.json
           hosts:
               - standalone

   The service_command provides the required information. It could be a simple cat from a config file. If you want os-diff to get the output of the command and store the output in a file specified by the key path, set cat_output to true. Then you can provide a mapping between in this case the EDPM CRD, and the ovs-vsctl output with config_mapping:

           service_command: 'ovs-vsctl list Open_vSwitch . | grep external_ids | awk -F '': '' ''{ print $2; }'''
           cat_output: true
           config_mapping:
               ovn-bridge: edpm_ovn_bridge
               ovn-bridge-mappings: edpm_ovn_bridge_mappings
               ovn-encap-type: edpm_ovn_encap_type
               ovn-match-northd-version: ovn_match_northd_version
               ovn-monitor-all: ovn_monitor_all
               ovn-ofctrl-wait-before-clear: edpm_ovn_ofctrl_wait_before_clear
               ovn-remote-probe-interval: edpm_ovn_remote_probe_interval

   Then you can use the following command to compare the values:

   os-diff diff ovs_external_ids.json edpm.crd --crd --service ovs_external_ids

   For example, to check the /etc/yum.conf on every host, you must put the following statement in the config.yaml file. The following example uses a file called yum_config:

   services:
     yum_config:
       hosts:
         - undercloud
         - controller_1
         - compute_1
         - compute_2
       service_command: "cat /etc/yum.conf"
       cat_output: true
       path:
         - yum.conf

2. Pull the configuration:

   This command will pull all the configuration files that are described in the /etc/os-diff/config.yaml file. Os-diff can update this file automatically according to your running environment with the command --update or --update-only. This option sets the podman information into the config.yaml for all running containers. It can be useful later, when all the Red Hat OpenStack Platform services are turned off.