Configuring persistent storage

Procedure

1. Open your OpenStackControlPlane CR file, openstack_control_plane.yaml, on your workstation.

2. Add the extraMounts attribute to the OpenStackControlPlane CR service definition.

   The following example demonstrates adding the extraMounts attribute:

   apiVersion: core.openstack.org/v1beta1
   kind: OpenStackControlPlane
   metadata:
     name: openstack
   spec:
     extraMounts:
       - name: v1
         region: r1
         extraVol:
           extraVolType: Ceph

3. Add the propagation field to specify where in the service definition the extraMount attribute applies.

   The following example adds the propagation field to the previous example:

   apiVersion: core.openstack.org/v1beta1
   kind: OpenStackControlPlane
   metadata:
     name: openstack
   spec:
     glance:
     ...
     extraMounts:
       - name: v1
         region: r1
         extraVol:
           - propagation:
             - Glance
             extraVolType: Ceph

   The propagation field can have one of the following values:

   • Service level propagations:

     • Glance
     • Cinder
     • Manila
     • Horizon
     • Neutron

   • Component level propagations:

     • CinderAPI
     • CinderScheduler
     • CinderVolume
     • CinderBackup
     • GlanceAPI
     • ManilaAPI
     • ManilaScheduler
     • ManilaShare
     • NeutronAPI

   • Back-end propagation:

     • Any back-end in the CinderVolume, ManilaShare , or GlanceAPI maps.

4. Define the volume sources:

   The following example demonstrates adding the volumes field to the previous example to provide a Red Hat Ceph Storage secret to the Image service (glance):

   apiVersion: core.openstack.org/v1beta1
   kind: OpenStackControlPlane
   metadata:
     name: openstack
   spec:
     extraMounts:
       - name: v1
         region: r1
         extraVol:
           extraVolType: Ceph
           volumes:
             - name: ceph
               secret:
                 secretName: ceph-conf-files

   where:

   ceph

   Is the Red Hat Ceph Storage secret name.

5. Define where the different volumes are mounted within the pod.

   The following example demonstrates adding the mounts field to the previous example to provide the location and name of the file that contains the Red Hat Ceph Storage secret:

   apiVersion: core.openstack.org/v1beta1
   kind: OpenStackControlPlane
   metadata:
     name: openstack
   spec:
     extraMounts:
       - name: v1
         region: r1
         extraVol:
           extraVolType: Ceph
           volumes:
             - name: ceph
               secret:
                 secretName: ceph-conf-files
             mounts:
             - name: ceph
               mountPath: "/etc/ceph"
               readOnly: true

   where:

   "/etc/ceph"

   Is the location of the secrets file.

6. Update the control plane:

   $ oc apply -f openstack_control_plane.yaml -n openstack

7. Wait until RHOCP creates the resources related to the OpenStackControlPlane CR. Run the following command to check the status:

   $ oc get openstackcontrolplane -n openstack

   The OpenStackControlPlane resources are created when the status is "Setup complete".

   Tip

   Append the -w option to the end of the get command to track deployment progress.

8. Confirm that the control plane is deployed by reviewing the pods in the openstack namespace:

   $ oc get pods -n openstack

   The control plane is deployed when all the pods are either completed or running.

Procedure

1. Enter the cephadm container client:

   $ sudo cephadm shell

2. Create pools for the Compute service (vms), the Block Storage service (volumes), and the Image service (images):

   $ for P in vms volumes images; do
      ceph osd pool create $P;
      ceph osd pool application enable $P rbd;
   done

3. If you are using the Shared File Systems service, create the cephfs volume. This automatically enables the CephFS Metadata service (MDS) and creates the necessary data and metadata pools on the Ceph cluster:

   $ ceph fs volume create cephfs

4. If you are using the Shared File Systems service with CephFS-NFS, deploy an NFS service on the Red Hat Ceph Storage cluster:

   1. If you are deploying Red Hat Ceph Storage 7 or 8, run the following command:

      $ ceph nfs cluster create cephfs \
      --ingress --virtual-ip=<vip> \
      --ingress-mode=haproxy-protocol

   2. If you are deploying Red Hat Ceph Storage 9, run the following command:

      $ ceph nfs cluster create cephfs \
      --ingress --virtual-ip=<vip> \
      --ingress-mode=haproxy-protocol \
      --enable-nfsv3

      • Replace <vip> with the IP address assigned to the NFS service. The NFS service should be on a dedicated network that isolates NFS traffic while allowing RHOSO users to attach their Compute instances to access shares.

5. Create a CephX key for RHOSO to use to access pools:

   $ ceph auth add client.openstack \
        mgr 'allow *' \
           mon 'profile rbd' \
           osd 'profile rbd pool=vms, profile rbd pool=volumes, profile rbd pool=backups, profile rbd pool=images'

   • If you are using the Shared File Systems service, add osd caps for the CephFS data pool by using the following command instead:

     $ ceph auth add client.openstack \
          mgr 'allow *' \
             mon 'profile rbd' \
             osd 'profile rbd pool=vms, profile rbd pool=volumes, profile rbd pool=backups, profile rbd pool=images, profile rbd pool=cephfs.cephfs.data'

6. Export the CephX key:

   $ ceph auth get client.openstack > /etc/ceph/ceph.client.openstack.keyring

7. Export the configuration file:

   $ ceph config generate-minimal-conf > /etc/ceph/ceph.conf

Procedure

1. Transfer the cephx key and configuration file created in the Creating Red Hat Ceph Storage pools procedure to a host that can create resources in the openstack namespace.

2. Base64 encode these files and store them in KEY and CONF environment variables:

   $ KEY=$(cat /etc/ceph/ceph.client.openstack.keyring | base64 -w 0)
   $ CONF=$(cat /etc/ceph/ceph.conf | base64 -w 0)

3. Create a YAML file to create the Secret resource.

4. Using the environment variables, add the Secret configuration to the YAML file:

   apiVersion: v1
   data:
     ceph.client.openstack.keyring: $KEY
     ceph.conf: $CONF
   kind: Secret
   metadata:
     name: ceph-conf-files
     namespace: openstack
   type: Opaque

5. Save the YAML file.

6. Create the Secret resource:

   $ oc create -f <secret_configuration_file>

   • Replace <secret_configuration_file> with the name of the YAML file you created.

Procedure

1. Check the storage interface defined in your NodeNetworkConfigurationPolicy (nncp) custom resource to confirm that it has the same network configuration as the public_network of the Red Hat Ceph Storage cluster. This is required to enable access to the Red Hat Ceph Storage cluster through the Storage network. The Storage network should have the same network configuration as the public_network of the Red Hat Ceph Storage cluster.

   It is not necessary for RHOSO to access the cluster_network of the Red Hat Ceph Storage cluster.

   Note

   If it does not impact workload performance, the Storage network can be different from the external Red Hat Ceph Storage cluster public_network using routed (L3) connectivity as long as the appropriate routes are added to the Storage network to reach the external Red Hat Ceph Storage cluster public_network.

2. Check the networkAttachments for the default Image service instance in the OpenStackControlPlane CR to confirm that the default Image service is configured to access the Storage network:

   glance:
       enabled: true
       template:
         databaseInstance: openstack
         storage:
           storageRequest: 10G
         glanceAPIs:
           default
             replicas: 3
             override:
               service:
                 internal:
                   metadata:
                     annotations:
                       metallb.universe.tf/address-pool: internalapi
                       metallb.universe.tf/allow-shared-ip: internalapi
                       metallb.universe.tf/loadBalancerIPs: 172.17.0.80
                   spec:
                     type: LoadBalancer
             networkAttachments:
             - storage

3. Confirm the Block Storage service is configured to access the Storage network through MetalLB.
4. Optional: Confirm the Shared File Systems service is configured to access the Storage network through ManilaShare.
5. Confirm the Compute service (nova) is configured to access the Storage network.
6. Confirm the Red Hat Ceph Storage configuration file, /etc/ceph/ceph.conf, contains the IP addresses of the Red Hat Ceph Storage cluster monitors. These IP addresses must be within the Storage network IP address range.
7. Open your openstack_control_plane.yaml file to edit the OpenStackControlPlane CR.

8. Add the extraMounts parameter to define the services that require access to the Red Hat Ceph Storage secret.

   The following is an example of using the extraMounts parameter for this purpose. Only include ManilaShare in the propagation list if you are using the Shared File Systems service (manila):

   apiVersion: core.openstack.org/v1beta1
   kind: OpenStackControlPlane
   spec:
     extraMounts:
       - name: v1
         region: r1
         extraVol:
           - propagation:
             - CinderVolume
             - GlanceAPI
             - ManilaShare
             extraVolType: Ceph
             volumes:
             - name: ceph
               projected:
                 sources:
                 - secret:
                     name: <ceph-conf-files>
             mounts:
             - name: ceph
               mountPath: "/etc/ceph"
               readOnly: true

   • Replace <ceph-conf-files> with the name of your Secret CR created in Creating a Red Hat Ceph Storage secret.

9. Add the customServiceConfig parameter to the glance template to configure the Image service to use the Red Hat Ceph Storage cluster:

   apiVersion: core.openstack.org/v1beta1
   kind: OpenStackControlPlane
   metadata:
     name: openstack
   spec:
     glance:
       template:
         customServiceConfig: |
           [DEFAULT]
           enabled_backends = <backend_name>:rbd
           [glance_store]
           default_backend = <backend_name>
           [<backend_name>]
           rbd_store_ceph_conf = /etc/ceph/ceph.conf
           store_description = "RBD backend"
           rbd_store_pool = images
           rbd_store_user = openstack
         databaseInstance: openstack
         databaseAccount: glance
         secret: osp-secret
         storage:
           storageRequest: 10G
     extraMounts:
       - name: v1
         region: r1
         extraVol:
           - propagation:
             - GlanceAPI
             extraVolType: Ceph
             volumes:
             - name: ceph
               secret:
                 secretName: ceph-conf-files
             mounts:
             - name: ceph
               mountPath: "/etc/ceph"
               readOnly: true

   • Replace <backend_name> with the name of the default back end.

     When you use Red Hat Ceph Storage as a back end for the Image service, image-conversion is enabled by default. For more information, see Planning storage and shared file systems in Planning your deployment.

10. Add the customServiceConfig parameter to the cinder template to configure the Block Storage service to use the Red Hat Ceph Storage cluster. For information about using Block Storage backups, see Configuring the Block Storage backup service.

    apiVersion: core.openstack.org/v1beta1
    kind: OpenStackControlPlane
    spec:
      extraMounts:
        ...
      cinder:
        template:
          cinderVolumes:
            ceph:
              customServiceConfig: |
                [DEFAULT]
                enabled_backends=ceph
                [ceph]
                volume_backend_name=ceph
                volume_driver=cinder.volume.drivers.rbd.RBDDriver
                rbd_ceph_conf=/etc/ceph/ceph.conf
                rbd_user=openstack
                rbd_pool=volumes
                rbd_flatten_volume_from_snapshot=False
                rbd_secret_uuid=<$FSID>

    • Replace <$FSID> with the actual FSID. The FSID itself does not need to be considered secret. For more information, see Obtaining the Red Hat Ceph Storage FSID.

11. Optional: Add the customServiceConfig parameter to the manila template to configure the Shared File Systems service to use native CephFS or CephFS-NFS with the Red Hat Ceph Storage cluster. For more information, see Configuring the Shared File Systems service (manila).

    The following example exposes native CephFS:

    apiVersion: core.openstack.org/v1beta1
    kind: OpenStackControlPlane
    spec:
      extraMounts:
      ...
      manila:
        template:
          manilaAPI:
            customServiceConfig: |
              [DEFAULT]
              enabled_share_protocols=cephfs
          manilaShares:
            share1:
              customServiceConfig: |
                [DEFAULT]
                enabled_share_backends=cephfs
                [cephfs]
                driver_handles_share_servers=False
                share_backend_name=cephfs
                share_driver=manila.share.drivers.cephfs.driver.CephFSDriver
                cephfs_conf_path=/etc/ceph/ceph.conf
                cephfs_auth_id=openstack
                cephfs_cluster_name=ceph
                cephfs_volume_mode=0755
                cephfs_protocol_helper_type=CEPHFS

    The following example exposes CephFS with NFS:

    apiVersion: core.openstack.org/v1beta1
    kind: OpenStackControlPlane
    spec:
      extraMounts:
      ...
      manila:
        template:
          manilaAPI:
            customServiceConfig: |
              [DEFAULT]
              enabled_share_protocols=nfs
          manilaShares:
            share1:
              customServiceConfig: |
                [DEFAULT]
                enabled_share_backends=cephfsnfs
                [cephfsnfs]
                driver_handles_share_servers=False
                share_backend_name=cephfsnfs
                share_driver=manila.share.drivers.cephfs.driver.CephFSDriver
                cephfs_conf_path=/etc/ceph/ceph.conf
                cephfs_auth_id=openstack
                cephfs_cluster_name=ceph
                cephfs_volume_mode=0755
                cephfs_protocol_helper_type=NFS
                cephfs_nfs_cluster_id=cephfs

12. Apply the updates to the OpenStackControlPlane CR:

    $ oc apply -f openstack_control_plane.yaml

Procedure

1. Create a ConfigMap with additional content for the Compute service (nova) configuration file /etc/nova/nova.conf.d/ inside the nova_compute container. This additional content directs the Compute service to use Red Hat Ceph Storage RBD.

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: ceph-nova
   data:
    <03-ceph-nova.conf>: |
     [libvirt]
     images_type=rbd
     images_rbd_pool=vms
     images_rbd_ceph_conf=/etc/ceph/ceph.conf
     images_rbd_glance_store_name=<backend_name>
     images_rbd_glance_copy_poll_interval=15
     images_rbd_glance_copy_timeout=600
     rbd_user=openstack
     rbd_secret_uuid=<$FSID>

   • Replace <03-ceph-nova.conf> with your file name. This file name must follow the naming convention of ##-<name>-nova.conf. Files are evaluated by the Compute service alphabetically. A filename that starts with 01 will be evaluated by the Compute service before a filename that starts with 02. When the same configuration option occurs in multiple files, the last one read wins.
   • Replace <backend_name> with the name of the back end specified in the glance template of the OpenStackControlPlane CR.
   • Replace <$FSID> with the actual FSID, as described in the Obtaining the Ceph FSID section. The FSID itself does not need to be considered secret.

2. Create a custom version of the default nova service to use the new ConfigMap, which in this case is called ceph-nova.

   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneService
   metadata:
     name: nova-custom-ceph
   spec:
     caCerts: combined-ca-bundle
     edpmServiceType: nova
     dataSources:
      - configMapRef:
          name: ceph-nova
      - secretRef:
          name: nova-cell1-compute-config
      - secretRef:
          name: nova-migration-ssh-key
     playbook: osp.edpm.nova

   • The custom service is named nova-custom-ceph. It cannot be named nova because nova is an unchangeable default service. Any custom service that has the same name as a default service name will be overwritten during reconciliation.

3. Apply the ConfigMap and custom service changes:

   $ oc create -f ceph-nova.yaml

4. In your OpenStackDataPlaneNodeSet CR, update the list of services by adding the ceph-client service and replacing the default nova service with the new custom service, for example nova-custom-ceph. Add the extraMounts parameter to define access to the Ceph Storage secret.

   Example:

   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneNodeSet
   spec:
     ...
     services:
     - redhat
     - bootstrap
     - download-cache
     - configure-network
     - validate-network
     - install-os
     - configure-os
     - ssh-known-hosts
     - run-os
     - reboot-os
     - install-certs
     - ceph-client
     - ovn
     - neutron-metadata
     - libvirt
     - nova-custom-ceph
     - telemetry
   
     nodeTemplate:
       extraMounts:
       - extraVolType: Ceph
         volumes:
         - name: ceph
           secret:
             secretName: ceph-conf-files
         mounts:
         - name: ceph
           mountPath: "/etc/ceph"
           readOnly: true

   • You must add the ceph-client service before the ovn, libvirt, and nova-custom-ceph services in the list of services. The ceph-client service configures data plane nodes as clients of a Red Hat Ceph Storage server by distributing the Red Hat Ceph Storage client files.

   • This example might not list all of the services in your environment. You can run the following command to verify the list of services in your environment:

     $ oc get -n openstack crd/openstackdataplanenodesets.dataplane.openstack.org -o yaml |yq -r '.spec.versions.[].schema.openAPIV3Schema.properties.spec.properties.services.default

     For more information, see Data plane services.

5. Save the changes to the services list.

6. Create an OpenStackDataPlaneDeployment CR:

   $ oc create -f <dataplanedeployment_cr_file>

   • Replace <dataplanedeployment_cr_file> with the name of your file.

     The Ansible job for the nova-custom-ceph service copies overrides from the ConfigMap to the Compute service hosts. The Ansible job also uses virsh secret-* commands so the libvirt service retrieves the cephx secret by FSID.

Procedure

1. Configure the RGW to verify users and their roles in the Identity service (keystone) to authenticate with the external RGW service.
2. Deploy and configure a RGW service to handle object storage requests.

Procedure

1. Create a DNSData custom resource (CR) for the external Ceph cluster.

   Example DNSData CR:

   apiVersion: network.openstack.org/v1beta1
   kind: DNSData
   metadata:
     labels:
       component: ceph-storage
       service: ceph
     name: ceph-storage
     namespace: openstack
   spec:
     dnsDataLabelSelectorValue: dnsdata
     hosts:
       - hostnames:
         - ceph-rgw-internal-vip.ceph.local
         ip: <172.18.0.2>
       - hostnames:
         - ceph-rgw-external-vip.ceph.local
         ip: <10.10.10.2>

   • Replace <172.18.0.2> with the correct host for your environment. In this example, the host at the IP address 172.18.0.2 hosts the Ceph RGW endpoint for access on the private storage network. This host passes the CR so that a DNS A and PTR record is created. This enables the host to be accessed by using the host name ceph-rgw-internal-vip.ceph.local.
   • Replace <10.10.10.2> with the correct host for your environment. In this example, the host at the IP address 10.10.10.2 hosts the Ceph RGW endpoint for access on the external network. This host passes the CR so that a DNS A and PTR record is created. This enables the host to be accessed by using the host name ceph-rgw-external-vip.ceph.local.

2. Apply the CR to your environment:

   $ oc apply -f <ceph_dns_yaml>

   • Replace <ceph_dns_yaml> with the name of the DNSData CR file.
3. Update the CoreDNS CR to configure DNS forwarding to the dnsmasq service for requests to the ceph.local domain. For more information about DNS forwarding, see Using DNS forwarding in the RHOCP Networking guide.

4. List the openstack domain DNS cluster IP address:

   $ oc get svc dnsmasq-dns

   Example output:

   $ oc get svc dnsmasq-dns
   dnsmasq-dns     LoadBalancer   10.217.5.130   192.168.122.80    53:30185/UDP     160m

5. Record the DNS cluster IP address from the command output for DNS forwarding.

6. List the CoreDNS CR:

   $ oc -n openshift-dns describe dns.operator/default

7. Edit the CoreDNS CR and add the servers configuration to the spec section with the DNS cluster IP address.

   Example CoreDNS CR updated with the DNS cluster IP address:

   apiVersion: operator.openshift.io/v1
   kind: DNS
   metadata:
     creationTimestamp: "2024-03-25T02:49:24Z"
     finalizers:
     - dns.operator.openshift.io/dns-controller
     generation: 3
     name: default
     resourceVersion: "164142"
     uid: 860b0e61-a48a-470e-8684-3b23118e6083
   spec:
     cache:
       negativeTTL: 0s
       positiveTTL: 0s
     logLevel: Normal
     nodePlacement: {}
     operatorLogLevel: Normal
     servers:
     - forwardPlugin:
         policy: Random
         upstreams:
         - 10.217.5.130:53
       name: ceph
       zones:
       - ceph.local
     upstreamResolvers:
       policy: Sequential
       upstreams:
       - port: 53
         type: SystemResolvConf

   where:

   servers

   Defines DNS forwarding configurations for specific domains.

   upstreams

   Specifies the DNS cluster IP address to which DNS queries are forwarded.

   10.217.5.130:53

   Is the DNS cluster IP address recorded from the oc get svc dnsmasq-dns command.

   zones

   Defines the domain for which DNS queries are forwarded to the upstream server.

8. Create a Certificate CR with the host names from the DNSData CR.

   Example Certificate CR:

   apiVersion: cert-manager.io/v1
   kind: Certificate
   metadata:
     name: cert-ceph-rgw
     namespace: openstack
   spec:
     duration: 43800h0m0s
     issuerRef: {'group': 'cert-manager.io', 'kind': 'Issuer', 'name': 'rootca-public'}
     secretName: cert-ceph-rgw
     dnsNames:
       - ceph-rgw-internal-vip.ceph.local
       - ceph-rgw-external-vip.ceph.local

9. Apply the CR to your environment:

   $ oc apply -f <ceph_cert_yaml>

   • Replace <ceph_cert_yaml> with the name of the Certificate CR file.

10. Extract the certificate and key data from the secret created when the Certificate CR was applied:

    $ oc get secret <ceph_cert_secret_name> -o yaml

    • Replace <ceph_cert_secret_name> with the name used in the secretName field of your Certificate CR.

      Example output:

      [stack@osp-storage-04 ~]$ oc get secret cert-ceph-rgw -o yaml
      apiVersion: v1
      data:
        ca.crt: <CA>
        tls.crt: <b64cert>
        tls.key: <b64key>
      kind: Secret

      • The <b64cert> and <b64key> values are the base64-encoded certificate and key strings that you must use in the next step.

11. Extract and base64 decode the certificate and key information obtained in the previous step.

    Extract and decode the certificate:

    $ oc get secret <ceph_cert_secret_name> -o yaml | awk '/tls.crt/ {print $2}' | base64 -d

    Extract and decode the key:

    $ oc get secret <ceph_cert_secret_name> -o yaml | awk '/tls.key/ {print $2}' | base64 -d

12. If you are using Red Hat Ceph Storage 7 or 8, concatenate the decoded certificate and key values with no spaces in between, and save them in the Ceph Object Gateway service specification.

    The rgw section of the specification file looks like the following:

      service_type: rgw
      service_id: rgw
      service_name: rgw.rgw
      placement:
        hosts:
        - host1
        - host2
      networks:
        - 172.18.0.0/24
      spec:
        rgw_frontend_port: 8082
        rgw_realm: default
        rgw_zone: default
        ssl: true
        rgw_frontend_ssl_certificate: |
        -----BEGIN CERTIFICATE-----
        MIIDkzCCAfugAwIBAgIRAKNgGd++xV9cBOrwDAeEdQUwDQYJKoZIhvcNAQELBQAw
        <redacted>
        -----END CERTIFICATE-----
        -----BEGIN RSA PRIVATE KEY-----
        MIIEpQIBAAKCAQEAyTL1XRJDcSuaBLpqasAuLsGU2LQdMxuEdw3tE5voKUNnWgjB
        <redacted>
        -----END RSA PRIVATE KEY-----

    The ingress section of the specification file looks like the following:

      service_type: ingress
      service_id: rgw.default
      service_name: ingress.rgw.default
      placement:
        count: 1
      spec:
        backend_service: rgw.rgw
        frontend_port: 8080
        monitor_port: 8999
        virtual_interface_networks:
        - 172.18.0.0/24
        virtual_ip: 172.18.0.2/24
        ssl_cert: |
        -----BEGIN CERTIFICATE-----
        MIIDkzCCAfugAwIBAgIRAKNgGd++xV9cBOrwDAeEdQUwDQYJKoZIhvcNAQELBQAw
        <redacted>
        -----END CERTIFICATE-----
        -----BEGIN RSA PRIVATE KEY-----
        MIIEpQIBAAKCAQEAyTL1XRJDcSuaBLpqasAuLsGU2LQdMxuEdw3tE5voKUNnWgjB
        <redacted>
        -----END RSA PRIVATE KEY-----

    where:

    rgw_frontend_ssl_certificate

    Contains the base64 decoded values from both <b64cert> and <b64key> in the previous step with no spaces in between.

    ssl_cert

    Contains the base64 decoded values from both <b64cert> and <b64key> in the previous step with no spaces in between.

13. If you are using Red Hat Ceph Storage 9, save the decoded certificate and key values separately in the Ceph Object Gateway service specification.

    The rgw section of the specification file looks like the following:

      service_type: rgw
      service_id: rgw
      service_name: rgw.rgw
      placement:
        hosts:
        - host1
        - host2
      networks:
        - 172.18.0.0/24
      spec:
        rgw_frontend_port: 8082
        rgw_realm: default
        rgw_zone: default
        ssl: true
        certificate_source: inline
        ssl_cert: |
        -----BEGIN CERTIFICATE-----
        MIIDkzCCAfugAwIBAgIRAKNgGd++xV9cBOrwDAeEdQUwDQYJKoZIhvcNAQELBQAw
        <redacted>
        -----END CERTIFICATE-----
        ssl_key: |
        -----BEGIN PRIVATE KEY-----
        MIIEpQIBAAKCAQEAyTL1XRJDcSuaBLpqasAuLsGU2LQdMxuEdw3tE5voKUNnWgjB
        <redacted>
        -----END PRIVATE KEY-----

    The ingress section of the specification file looks like the following:

      service_type: ingress
      service_id: rgw.default
      service_name: ingress.rgw.default
      placement:
        count: 1
      spec:
        backend_service: rgw.rgw
        frontend_port: 8080
        monitor_port: 8999
        virtual_interface_networks:
        - 172.18.0.0/24
        virtual_ip: 172.18.0.2/24
        ssl: true
        certificate_source: inline
        ssl_cert: |
        -----BEGIN CERTIFICATE-----
        MIIDkzCCAfugAwIBAgIRAKNgGd++xV9cBOrwDAeEdQUwDQYJKoZIhvcNAQELBQAw
        <redacted>
        -----END CERTIFICATE-----
        ssl_key: |
        -----BEGIN PRIVATE KEY-----
        MIIEpQIBAAKCAQEAyTL1XRJDcSuaBLpqasAuLsGU2LQdMxuEdw3tE5voKUNnWgjB
        <redacted>
        -----END PRIVATE KEY-----

    where:

    certificate_source: inline

    Specifies that the certificate and key are embedded directly in the specification.

    ssl_cert

    Contains the base64 decoded certificate value from <b64cert> in the previous step.

    ssl_key

    Contains the base64 decoded key value from <b64key> in the previous step.

    Note

    In Red Hat Ceph Storage 9, the rgw_frontend_ssl_certificate field, which required concatenated certificate and key values, is deprecated. New deployments must use the separate ssl_cert and ssl_key fields.

14. Use the procedure "Deploying the Ceph Object Gateway using the service specification" to deploy Ceph RGW with SSL. For more information, see the Red Hat Ceph Storage Operations Guide:

    • Red Hat Ceph Storage 7 Operations Guide
    • Red Hat Ceph Storage 8 Operations Guide
    • Red Hat Ceph Storage 9 Operations Guide
15. Connect to the openstackclient pod.

16. Verify that DNS forwarding has been successfully configured.

    $ curl --trace - <host_name>

    • Replace <host_name> with the name of the external host previously added to the DNSData CR.

      Example output:

      sh-5.1$ curl https://rgw-external-vip.ceph.local:8080
      <?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult>
      .1$
      sh-5.1$

    • In this example, the openstackclient pod successfully resolved the host name, and no SSL verification errors were encountered.

Procedure

1. Verify which Ceph version the clients in your Ceph Storage cluster are running:

   $ cephadm shell -- ceph osd get-require-min-compat-client

   Example output:

   luminous

2. To set the cluster to use the Clone v2 API and the deferred deletion feature by default, set min-compat-client to mimic. Only clients in the cluster that are running Ceph version 13.2.x (Mimic) can access images with dependencies:

   $ cephadm shell -- ceph osd set-require-min-compat-client mimic

3. Schedule an interval for trash purge in minutes by using the m suffix:

   $ rbd trash purge schedule add --pool <pool> <30m>

   • Replace <pool> with the name of the associated storage pool, for example, volumes in the Block Storage service.
   • Replace <30m> with the interval in minutes that you want to specify for trash purge.

4. Verify a trash purge schedule has been set for the pool:

   $ rbd trash purge schedule list --pool <pool>

Procedure

1. Determine if any parts of the control plane are not properly deployed by assessing whether the Ready condition is not True:

   $ oc get -n openstack OpenStackControlPlane \
     -o jsonpath="{range .items[0].status.conditions[?(@.status!='True')]}{.type} is {.status} due to {.message}{'\n'}{end}"

   1. If you identify a service that is not properly deployed, check the status of the service.

      The following example checks the status of the Compute service:

      $ oc get -n openstack Nova/nova \
        -o jsonpath="{range .status.conditions[?(@.status!='True')]}{.type} is {.status} due to {.message}{'\n'}{end}"

      • You can check the status of all deployed services:

        $ oc get pods -n openstack

      • You can check the logs of a specific service:

        $ oc logs -n openstack <service_pod_name>

        • Replace <service_pod_name> with the name of the service pod you want to check.

   2. If you identify an operator that is not properly deployed, check the status of the operator:

      $ oc get pods -n openstack-operators -lopenstack.org/operator-name

      • You can check the operator logs:

        $ oc logs -n openstack-operators -lopenstack.org/operator-name=<operator_name>

2. Check the Status of the data plane deployment:

   $ oc get -n openstack OpenStackDataPlaneDeployment

   • If the Status of the data plane deployment is False, check the logs of the associated Ansible job:

     $ oc logs -n openstack job/<ansible_job_name>

     • Replace <ansible_job_name> with the name of the associated job. The job name is listed in the Message field of the oc get -n openstack OpenStackDataPlaneDeployment command output.

3. Check the Status of the data plane node set deployment:

   $ oc get -n openstack OpenStackDataPlaneNodeSet

   • If the Status of the data plane node set deployment is False, check the logs of the associated Ansible job:

     $ oc logs -n openstack job/<ansible_job_name>

     • Replace <ansible_job_name> with the name of the associated job. It is listed in the Message field of the oc get -n openstack OpenStackDataPlaneNodeSet command output.

4. If any pods are in the CrashLookBackOff state, you can duplicate them for troubleshooting purposes:

   $ oc debug <pod_name>

   • Replace <pod_name> with the name of the pod to duplicate.

5. Optional: You can route traffic to the duplicate pod during the debug process:

   $ oc debug <pod_name> --keep-labels=true

6. Optional: You can use the oc debug command in the following object debugging activities:

   • To run /bin/sh on a container other than the first one, the command’s default behavior, using the command form oc debug -container <pod_name> <container_name>. This is useful for pods like the API where the first container is tailing a file and the second container is the one you want to debug. If you use this command form, you must first use the command oc get pods | grep <search_string> to find the container name.
   • To create any resource that creates pods such as Deployments, StatefulSets, and Nodes, use the command form oc debug <resource_type>/<resource_name>. An example of creating a StatefulSet would be oc debug StatefulSet/cinder-scheduler.

7. Connect to the pod and confirm that the ceph.client.openstack.keyring and ceph.conf files are present in the /etc/ceph directory.

   $ oc rsh <pod_name>

   • Replace <pod_name> with the name of the applicable pod.
   • If the Ceph configuration files are missing, check the extraMounts parameter in your OpenStackControlPlane CR.

8. Confirm the pod has a network connection to the Red Hat Ceph Storage cluster by connecting to the IP and port of a Ceph Monitor from the pod. The IP and port information is located in /etc/ceph.conf.

   The following is an example of this process:

   $ oc get pods | grep glance | grep external-api-0
   glance-06f7a-default-external-api-0                               3/3     Running     0              2d3h
   $ oc debug --container glance-api glance-06f7a-default-external-api-0
   Starting pod/glance-06f7a-default-external-api-0-debug-p24v9, command was: /usr/bin/dumb-init --single-child -- /bin/bash -c /usr/local/bin/kolla_set_configs && /usr/local/bin/kolla_start
   Pod IP: 192.168.25.50
   If you don't see a command prompt, try pressing enter.
   sh-5.1# cat /etc/ceph/ceph.conf
   # Ansible managed
   
   [global]
   
   fsid = 63bdd226-fbe6-5f31-956e-7028e99f1ee1
   mon host = [v2:192.168.122.100:3300/0,v1:192.168.122.100:6789/0],[v2:192.168.122.102:3300/0,v1:192.168.122.102:6789/0],[v2:192.168.122.101:3300/0,v1:192.168.122.101:6789/0]
   
   
   [client.libvirt]
   admin socket = /var/run/ceph/$cluster-$type.$id.$pid.$cctid.asok
   log file = /var/log/ceph/qemu-guest-$pid.log
   
   sh-5.1# python3
   Python 3.9.19 (main, Jul 18 2024, 00:00:00)
   [GCC 11.4.1 20231218 (Red Hat 11.4.1-3)] on linux
   Type "help", "copyright", "credits" or "license" for more information.
   >>> import socket
   >>> s = socket.socket()
   >>> ip="192.168.122.100"
   >>> port=3300
   >>> s.connect((ip,port))
   >>>

9. Optional: If you cannot connect to a Ceph Monitor, troubleshoot the network connection between the cluster and pod. The previous example uses a Python socket to connect to the IP and port of the Red Hat Ceph Storage cluster from the ceph.conf file.

   • There are two potential outcomes from the execution of the s.connect((ip,port)) function:

     • If the command executes successfully and there is no error similar to the following example, the network connection between the pod and cluster is functioning correctly. If the connection is functioning correctly, the command execution will provide no return value at all.

     • If the command takes a long time to execute and returns an error similar to the following example, the network connection between the pod and cluster is not functioning correctly. It should be investigated further to troubleshoot the connection.

       Traceback (most recent call last):
         File "<stdin>", line 1, in <module>
       TimeoutError: [Errno 110] Connection timed out

10. Examine the cephx key as shown in the following example:

    bash-5.1$ cat /etc/ceph/ceph.client.openstack.keyring
    [client.openstack]
       key = "<redacted>"
       caps mgr = allow *
       caps mon = profile rbd
       caps osd = profile rbd pool=vms, profile rbd pool=volumes, profile rbd pool=backups, profile rbd pool=images
    bash-5.1$

11. List the contents of a pool from the caps osd parameter as shown in the following example:

    $ /usr/bin/rbd --conf /etc/ceph/ceph.conf \
    --keyring /etc/ceph/ceph.client.openstack.keyring \
    --cluster ceph --id openstack \
    ls -l -p <pool_name> | wc -l

    • Replace <pool_name> with the name of the required Red Hat Ceph Storage pool.
    • If this command returns the number 0 or greater, the cephx key provides adequate permissions to connect to, and read information from, the Red Hat Ceph Storage cluster.
    • If this command does not complete but network connectivity to the cluster was confirmed, work with the Ceph administrator to obtain the correct cephx keyring.
    • Check if there is an MTU mismatch on the Storage network. If the network is using jumbo frames (an MTU value of 9000), all switch ports between servers using the interface must be updated to support jumbo frames. If this change is not made on the switch, problems can occur at the Ceph application layer. Verify all hosts using the network can communicate at the desired MTU with a command such as ping -M do -s 8972 <ip_address>.

12. Send test data to the images pool on the Ceph cluster.

    The following is an example of performing this task:

    # DATA=$(date | md5sum | cut -c-12)
    # POOL=images
    # RBD="/usr/bin/rbd --conf /etc/ceph/ceph.conf --keyring /etc/ceph/ceph.client.openstack.keyring --cluster ceph --id openstack"
    # $RBD create --size 1024 $POOL/$DATA

    Tip

    It is possible to be able to read data from the cluster but not have permissions to write data to it, even if write permission was granted in the cephx keyring. If you have write permissions, but you cannot write data to the cluster, the cluster might be overloaded and not able to write new data.

    In the example, the rbd command did not complete successfully and was canceled. It was subsequently confirmed the cluster itself did not have the resources to write new data. The issue was resolved on the cluster itself. There was nothing incorrect with the client configuration.

Procedure

1. Locate the Red Hat Ceph Storage configuration file mapped in the Red Hat OpenShift secret created in Creating a Red Hat Ceph Storage secret.

2. Modify the contents of the configuration file to include troubleshooting-related configuration.

   The following is an example of troubleshooting-related configuration:

   [client.openstack]
   admin socket = /var/run/ceph/$cluster-$type.$id.$pid.$cctid.asok
   log file = /var/log/guest-$pid.log
   debug ms = 1
   debug rbd = 20
   log to file = true

   Note

   For more information about troubleshooting, see the Red Hat Ceph Storage Troubleshooting Guide:

   • Red Hat Ceph Storage 7 Troubleshooting Guide
   • Red Hat Ceph Storage 8 Troubleshooting Guide
   • Red Hat Ceph Storage 9 Troubleshooting Guide
3. Update the secret with the new content.

Procedure

1. Open your OpenStackControlPlane CR file, openstack_control_plane.yaml.

2. Edit the CR file and add the Block Storage service global configuration.

   The following example demonstrates a Block Storage service initial configuration:

   apiVersion: core.openstack.org/v1beta1
   kind: OpenStackControlPlane
   metadata:
     name: openstack
   spec:
     cinder:
       enabled: true
       template:
         customServiceConfig: |
           [DEFAULT]
           quota_volumes = 20
           quota_snapshots = 15

   For a complete list of all initial default parameters, see Initial default parameters.

3. Update the control plane:

   $ oc apply -f openstack_control_plane.yaml -n openstack

4. Wait until RHOCP creates the resources related to the OpenStackControlPlane CR. Run the following command to check the status:

   $ oc get openstackcontrolplane -n openstack

   The OpenStackControlPlane resources are created when the status is "Setup complete".

   Tip

   Append the -w option to the end of the get command to track deployment progress.

Procedure

1. Open your OpenStackControlPlane CR file, openstack_control_plane.yaml.

2. Edit the CR file and add the configuration for the internal Red Hat OpenShift Container Platform (RHOCP) load balancer.

   The following example demonstrates a load balancer configuration:

   apiVersion: core.openstack.org/v1beta1
   kind: OpenStackControlPlane
   metadata:
     name: openstack
   spec:
     cinder:
       template:
         cinderAPI:
           override:
             service:
               internal:
                 metadata:
                   annotations:
                     metallb.universe.tf/address-pool: internalapi
                     metallb.universe.tf/allow-shared-ip: internalapi
                     metallb.universe.tf/loadBalancerIPs: 172.17.0.80
                 spec:
                   type: LoadBalancer

3. Edit the CR file and add the configuration for the number of API service replicas. Run the cinderAPI service in an Active-Active configuration with three replicas.

   The following example demonstrates configuring the cinderAPI service to use three replicas:

   apiVersion: core.openstack.org/v1beta1
   kind: OpenStackControlPlane
   metadata:
     name: openstack
   spec:
     cinder:
       template:
         cinderAPI:
           replicas: 3

4. Edit the CR file and configure cinderAPI options. These options are configured in the customServiceConfig section under the cinderAPI section.

   The following example demonstrates configuring cinderAPI service options and enabling debugging on all services:

   apiVersion: core.openstack.org/v1beta1
   kind: OpenStackControlPlane
   metadata:
     name: openstack
   spec:
     cinder:
       template:
         customServiceConfig: |
           [DEFAULT]
           debug = true
         cinderAPI:
           customServiceConfig: |
             [DEFAULT]
             osapi_volume_workers = 3

   For a listing of commonly used cinderAPI service option parameters, see API service option parameters.

5. Save the file.

6. Update the control plane:

   $ oc apply -f openstack_control_plane.yaml -n openstack

7. Wait until RHOCP creates the resources related to the OpenStackControlPlane CR. Run the following command to check the status:

   $ oc get openstackcontrolplane -n openstack

   The OpenStackControlPlane resources are created when the status is "Setup complete".

   Tip

   Append the -w option to the end of the get command to track deployment progress.

Procedure

1. Open your OpenStackControlPlane CR file, openstack_control_plane.yaml.

2. Edit the CR file and add the configuration for the service down detection timeouts.

   The following example demonstrates this configuration:

   apiVersion: core.openstack.org/v1beta1
   kind: OpenStackControlPlane
   metadata:
     name: openstack
   spec:
     cinder:
       template:
         customServiceConfig: |
           [DEFAULT]
           report_interval = 20
           service_down_time = 120

   • report_interval: The number of seconds between Block Storage service components reporting an operational state in the form of a heartbeat through the database. The default is 10.

   • service_down_time: The maximum number of seconds since the last heartbeat from the component for it to be considered non-operational. The default is 60.

     Note

     Configure these values at the cinder level of the CR instead of the cinderScheduler so that these values are applied to all components consistently.

3. Edit the CR file and add the configuration for the statistics reporting interval.

   The following example demonstrates configuring these values at the cinder level to apply them globally to all services:

   apiVersion: core.openstack.org/v1beta1
   kind: OpenStackControlPlane
   metadata:
     name: openstack
   spec:
     cinder:
       template:
         customServiceConfig: |
           [DEFAULT]
           backend_stats_polling_interval = 120
           backup_driver_stats_polling_interval = 120

   • backend_stats_polling_interval: The number of seconds between requests from the volume for usage statistics from the back end. The default is 60.

   • backup_driver_stats_polling_interval: The number of seconds between requests from the volume for usage statistics from backup service. The default is 60.

     The following example demonstrates configuring these values at the cinderVolume and cinderBackup level to customize settings at the service level.

     apiVersion: core.openstack.org/v1beta1
     kind: OpenStackControlPlane
     metadata:
       name: openstack
     spec:
       cinder:
         template:
           cinderBackup:
             customServiceConfig: |
               [DEFAULT]
               backup_driver_stats_polling_interval = 120
               < rest of the config >
           cinderVolumes:
             nfs:
               customServiceConfig: |
                 [DEFAULT]
                 backend_stats_polling_interval = 120

     Note

     The generation of usage statistics can be resource intensive for some back ends. Setting these values too low can affect back end performance. You may need to tune the configuration of these settings to better suit individual back ends.

4. Perform any additional configuration necessary to customize the cinderScheduler service.

   For more configuration options for the customization of the cinderScheduler service, see Scheduler service parameters.

5. Save the file.

6. Update the control plane:

   $ oc apply -f openstack_control_plane.yaml -n openstack

7. Wait until RHOCP creates the resources related to the OpenStackControlPlane CR. Run the following command to check the status:

   $ oc get openstackcontrolplane -n openstack

   The OpenStackControlPlane resources are created when the status is "Setup complete".

   Tip

   Append the -w option to the end of the get command to track deployment progress.

Procedure

1. Open your OpenStackControlPlane CR file, openstack_control_plane.yaml.

2. Edit the CR file and add the configuration for your back end.

   The following example demonstrates the service configuration for a Red Hat Ceph Storage back end:

   apiVersion: core.openstack.org/v1beta1
   kind: OpenStackControlPlane
   metadata:
     name: openstack
   spec:
     cinder:
       template:
         customServiceConfig: |
           [DEFAULT]
           debug = true
         cinderVolumes:
           ceph:
             networkAttachments:
             - storage
             customServiceConfig: |
               [ceph]
               volume_backend_name = ceph
               volume_driver = cinder.volume.drivers.rbd.RBDDriver

   • ceph: The configuration area for the individual back end. Each unique back end requires an individual configuration area. No back end is deployed by default. The Block Storage service volume service will not run unless at least one back end is configured during deployment. For more information about configuring back ends, see Block Storage service (cinder) back ends and Multiple Block Storage service (cinder) back ends.
   • networkAttachments: The configuration area for the back end network connections.
   • volume_backend_name: The name assigned to this back end.

   • volume_driver: The driver used to connect to this back end.

     For a list of commonly used volume service parameters, see Volume service parameters.

3. Save the file.

4. Update the control plane:

   $ oc apply -f openstack_control_plane.yaml -n openstack

5. Wait until RHOCP creates the resources related to the OpenStackControlPlane CR. Run the following command to check the status:

   $ oc get openstackcontrolplane -n openstack

   The OpenStackControlPlane resources are created when the status is "Setup complete".

   Tip

   Append the -w option to the end of the get command to track deployment progress.

Procedure

1. Open your OpenStackControlPlane CR file, openstack_control_plane.yaml.

2. Edit the CR file and add the AZ configuration.

   The following example demonstrates an AZ configuration:

   apiVersion: core.openstack.org/v1beta1
   kind: OpenStackControlPlane
   metadata:
   name: openstack
   spec:
     cinder:
       template:
         cinderVolumes:
           nfs:
             networkAttachments:
             - storage
             - storageMgmt
             customServiceConfigSecrets:
             - cinder-volume-nfs-secrets
             customServiceConfig: |
                   [nfs]
                   volume_backend_name=nfs
                   backend_availability_zone=zone1
           iSCSI:
             networkAttachments:
             - storage
             - storageMgmt
             customServiceConfig: |
                   [iscsi]
                   volume_backend_name=iscsi
                   backend_availability_zone=zone2

   • backend_availability_zone: The availability zone associated with the back end.
3. Save the file.

4. Update the control plane:

   $ oc apply -f openstack_control_plane.yaml -n openstack

5. Wait until RHOCP creates the resources related to the OpenStackControlPlane CR. Run the following command to check the status:

   $ oc get openstackcontrolplane -n openstack

   The OpenStackControlPlane resources are created when the status is "Setup complete".

   Tip

   Append the -w option to the end of the get command to track deployment progress.

Procedure

1. Create a Secret CR to store the volume connection information.

   The following is an example of a Secret CR:

   apiVersion: v1
   kind: Secret
   metadata:
     name: cinder-volume-nfs-secrets
   type: Opaque
   stringData:
     cinder-volume-nfs-secrets: |
   	[nfs]
   	nas_host=192.168.130.1
   	nas_share_path=/var/nfs/cinder

   where:

   name

   Is the name used when including it in the cinderVolumes back end configuration.

2. Save the file.

3. Update the control plane:

   $ oc apply -f <secret_file_name> -n openstack

   • Replace <secret_file_name> with the name of the file that contains your Secret CR.
4. Open your OpenStackControlPlane CR file, openstack_control_plane.yaml.

5. Edit the CR file and add the configuration for the generic NFS back end.

   The following example demonstrates this configuration:

   apiVersion: core.openstack.org/v1beta1
   kind: OpenStackControlPlane
   metadata:
     name: openstack
   spec:
     cinder:
       template:
         cinderVolumes:
           nfs:
             networkAttachments:
             - storage
             customServiceConfig: |
               [nfs]
               volume_backend_name=nfs
               volume_driver=cinder.volume.drivers.nfs.NfsDriver
               nfs_snapshot_support=true
               nas_secure_file_operations=false
               nas_secure_file_permissions=false
             customServiceConfigSecrets:
             - cinder-volume-nfs-secrets

   • The storageMgmt network is not listed because generic NFS does not have a management interface.
   • cinder-volume-nfs-secret: The name from the Secret CR.
   • If you are configuring multiple generic NFS back ends, ensure that each back end is in an individual configuration section so that one pod is dedicated to each back end.
6. Save the file.

7. Update the control plane:

   $ oc apply -f openstack_control_plane.yaml -n openstack

8. Wait until RHOCP creates the resources related to the OpenStackControlPlane CR. Run the following command to check the status:

   $ oc get openstackcontrolplane -n openstack

   The OpenStackControlPlane resources are created when the status is "Setup complete".

   Tip

   Append the -w option to the end of the get command to track deployment progress.

Procedure

1. Open your OpenStackControlPlane CR file, openstack_control_plane.yaml.

2. Edit the CR file and add the configuration for the directory for converting large Image service images.

   The following example demonstrates how to configure this conversion directory:

   apiVersion: core.openstack.org/v1beta1
   kind: OpenStackControlPlane
   spec:
     extraMounts:
         extraVol:
           - propagation:
             - CinderVolume
             volumes:
             - name: cinder-conversion
               nfs:
                   path: <nfs_share_path>
                   server: <nfs_server>
             mounts:
             - name: cinder-conversion
               mountPath: /var/lib/cinder/conversion
   ...

   • Replace <nfs_share_path> with the path to the conversion directory.

     Note

     The Block Storage volume service (cinder-volume) runs as the cinder user. The cinder user requires write permission for <nfs_share_path>. You can configure this by running the following command on the NFS server: $ chown 42407:42407 <nfs_share_path>.

   • Replace <nfs_server> with the IP address of the NFS server that hosts the conversion directory.
   Note

   This example demonstrates how to create a common conversion directory that all the volume service pods use.

   You can also define a conversion directory for each volume service pod:

   • Define each conversion directory by using an extraMounts section, as demonstrated above, in the cinder section of the OpenStackControlPlane CR file.
   • Set the propagation value to the name of the specific Volume section instead of CinderVolume.
3. Save the file.

4. Update the control plane:

   $ oc apply -f openstack_control_plane.yaml -n openstack

5. Wait until Red Hat OpenShift Container Platform (RHOCP) creates the resources related to the OpenStackControlPlane CR. Run the following command to check the status:

   $ oc get openstackcontrolplane -n openstack

   The OpenStackControlPlane resources are created when the status is "Setup complete".

   Tip

   Append the -w option to the end of the get command to track deployment progress.

Procedure

1. Open your openstack_control_plane.yaml file to edit the OpenStackControlPlane CR.

2. Add the dbPurge parameter to the cinder template to configure database cleanup depending on the service you want to configure.

   The following is an example of using the dbPurge parameter to configure the Block Storage service:

   apiVersion: core.openstack.org/v1beta1
   kind: OpenStackControlPlane
   metadata:
     name: openstack
   spec:
     cinder:
       template:
         dbPurge:
           age: 20
           schedule: 1 0 * * 0

   • age: The number of days a record has been marked for deletion before it is purged. The default value is 30. The minimum value is 1.
   • schedule: When to run the job in a crontab format. The default value is 1 0 * * *. This default value is equivalent to 00:01 daily.

3. Update the control plane:

   $ oc apply -f openstack_control_plane.yaml

Procedure

1. Open your OpenStackControlPlane CR file, openstack_control_plane.yaml, and add the following parameters to the cinder template to set the number of replicas for the cinderBackup parameter:

   apiVersion: core.openstack.org/v1beta1
   kind: OpenStackControlPlane
   metadata:
     name: openstack
   spec:
      …
      cinder:
         template:
           cinderBackup: |
             replicas: <number_of_replicas>
   ...

   • Replace <number_of_replicas> with a value greater than 1.

2. Update the control plane:

   $ oc apply -f openstack_control_plane.yaml -n openstack

3. Wait until RHOCP creates the resources related to the OpenStackControlPlane CR. Run the following command to check the status:

   $ oc get openstackcontrolplane -n openstack

   The OpenStackControlPlane resources are created when the status is "Setup complete".

   Tip

   Append the -w option to the end of the get command to track deployment progress.

Procedure

1. Open your OpenStackControlPlane CR file, openstack_control_plane.yaml, and add the following parameters to the cinder template to set configuration options. In this example, you enable debug logs, double the number of processes, and increase the maximum number of operations per pod to 20.

   Example:

   apiVersion: core.openstack.org/v1beta1
   kind: OpenStackControlPlane
   metadata:
     name: openstack
   spec:
      …
      cinder:
         template:
           customServiceConfig: |
             [DEFAULT]
             debug = true
           cinderBackup:
             customServiceConfig: |
              [DEFAULT]
              backup_workers = 2
              backup_max_operations = 20
   
   ...

2. Update the control plane:

   $ oc apply -f openstack_control_plane.yaml -n openstack

3. Wait until RHOCP creates the resources related to the OpenStackControlPlane CR. Run the following command to check the status:

   $ oc get openstackcontrolplane -n openstack

   The OpenStackControlPlane resources are created when the status is "Setup complete".

   Tip

   Append the -w option to the end of the get command to track deployment progress.

Procedure

1. Open your OpenStackControlPlane CR file, openstack_control_plane.yaml, and add the following parameter to the cinder template to enable data compression. In this example, you enable data compression with an Object Storage service (swift) back end:

   apiVersion: core.openstack.org/v1beta1
   kind: OpenStackControlPlane
   spec:
     cinder:
       template:
         cinderBackup
           customServiceConfig: |
             [DEFAULT]
             backup_driver = cinder.backup.drivers.nfs.SwiftBackupDriver
             backup_compression_algorithm = zstd
           networkAttachments:
           - storage

2. Update the control plane:

   $ oc apply -f openstack_control_plane.yaml -n openstack

3. Wait until RHOCP creates the resources related to the OpenStackControlPlane CR. Run the following command to check the status:

   $ oc get openstackcontrolplane -n openstack

   The OpenStackControlPlane resources are created when the status is "Setup complete".

   Tip

   Append the -w option to the end of the get command to track deployment progress.

Procedure

1. Open your OpenStackControlPlane CR file, openstack_control_plane.yaml, and add the following parameters to the cinder template to configure Ceph RBD as the back end for backups:

   apiVersion: core.openstack.org/v1beta1
   kind: OpenStackControlPlane
   spec:
     cinder:
       template:
         cinderBackup
           customServiceConfig: |
             [DEFAULT]
             backup_driver = cinder.backup.drivers.nfs.CephBackupDriver
             backup_ceph_pool = backups
             backup_ceph_user = openstack
           networkAttachments:
           - storage
           replicas: 1

2. Update the control plane:

   $ oc apply -f openstack_control_plane.yaml -n openstack

3. Wait until RHOCP creates the resources related to the OpenStackControlPlane CR. Run the following command to check the status:

   $ oc get openstackcontrolplane -n openstack

   The OpenStackControlPlane resources are created when the status is "Setup complete".

   Tip

   Append the -w option to the end of the get command to track deployment progress.

Procedure

1. Open your OpenStackControlPlane CR file, openstack_control_plane.yaml, and add the following parameters to the cinder template to configure the Object Storage service as the back end for backups:

   apiVersion: core.openstack.org/v1beta1
   kind: OpenStackControlPlane
   spec:
     cinder:
       template:
         cinderBackup
           customServiceConfig: |
             [DEFAULT]
             backup_driver = cinder.backup.drivers.nfs.SwiftBackupDriver
           networkAttachments:
           - storage
           replicas: 1

2. Update the control plane:

   $ oc apply -f openstack_control_plane.yaml -n openstack

3. Wait until RHOCP creates the resources related to the OpenStackControlPlane CR. Run the following command to check the status:

   $ oc get openstackcontrolplane -n openstack

   The OpenStackControlPlane resources are created when the status is "Setup complete".

   Tip

   Append the -w option to the end of the get command to track deployment progress.

Procedure

1. Create a secret CR file, for example, cinder-backup-nfs-secrets.yaml, and add the following configuration for your NFS share:

   apiVersion: v1
   kind: Secret
   metadata:
     labels:
       service: cinder
       component: cinder-backup
     name: cinder-backup-nfs-secrets
   type: Opaque
   stringData:
     nfs-secrets.conf: |
       [DEFAULT]
       backup_share = <192.168.1.2:/Backups>
       backup_mount_options = <optional>

   • Replace <192.168.1.2:/Backups> with the IP address of your NFS share.
   • Replace <optional> with the mount options for your NFS share.

2. Open your OpenStackControlPlane CR file, openstack_control_plane.yaml, and add the following parameters to the cinder template to add the secret for the NFS share and configure NFS as the back end for backups:

   apiVersion: core.openstack.org/v1beta1
   kind: OpenStackControlPlane
   spec:
     cinder:
       template:
         cinderBackup
           customServiceConfig: |
             [DEFAULT]
             backup_driver = cinder.backup.drivers.nfs.NFSBackupDriver
           customServiceConfigSecrets:
           - cinder-backup-nfs-secrets
           networkAttachments:
           - storage
           replicas: 1

3. Update the control plane:

   $ oc apply -f openstack_control_plane.yaml -n openstack

4. Wait until RHOCP creates the resources related to the OpenStackControlPlane CR. Run the following command to check the status:

   $ oc get openstackcontrolplane -n openstack

   The OpenStackControlPlane resources are created when the status is "Setup complete".

   Tip

   Append the -w option to the end of the get command to track deployment progress.

Procedure

1. Open your OpenStackControlPlane CR file, openstack_control_plane.yaml, and add the following parameters to the cinder template to configure S3 as the back end for backups:

   apiVersion: core.openstack.org/v1beta1
   kind: OpenStackControlPlane
   spec:
     cinder:
       template:
         cinderBackup
           customServiceConfig: |
             [DEFAULT]
             backup_driver = cinder.backup.drivers.s3.S3BackupDriver
             backup_s3_endpoint_url = <user supplied>
             backup_s3_store_access_key = <user supplied>
             backup_s3_store_secret_key = <user supplied>
             backup_s3_store_bucket = volumebackups
             backup_s3_ca_cert_file = /etc/pki/tls/certs/ca-bundle.crt
           networkAttachments:
           - storage

2. Update the control plane:

   $ oc apply -f openstack_control_plane.yaml -n openstack

3. Wait until RHOCP creates the resources related to the OpenStackControlPlane CR. Run the following command to check the status:

   $ oc get openstackcontrolplane -n openstack

   The OpenStackControlPlane resources are created when the status is "Setup complete".

   Tip

   Append the -w option to the end of the get command to track deployment progress.

Procedure

1. Open your OpenStackControlPlane CR file, openstack_control_plane.yaml, and add the following parameters to the glance template to configure the Block Storage service as the back end for the Image service:

   apiVersion: core.openstack.org/v1beta1
   kind: OpenStackControlPlane
   spec:
     ...
     glance:
       template:
         glanceAPIs:
           default:
             replicas: 3 # Configure back end; set to 3 when deploying service
         ...
         customServiceConfig: |
           [DEFAULT]
           enabled_backends = <backend_name>:cinder
           [glance_store]
           default_backend = <backend_name>
           [<backend_name>]
           description = Default cinder backend
           cinder_store_auth_address = {{ .KeystoneInternalURL }}
           cinder_store_user_name = {{ .ServiceUser }}
           cinder_store_password = {{ .ServicePassword }}
           cinder_store_project_name = service
           cinder_catalog_info = volumev3::internalURL
           cinder_use_multipath = true
           [oslo_concurrency]
           lock_path = /var/lib/glance/tmp
   ...

   • Set replicas to 3 for high availability across APIs.
   • Replace <backend_name> with the name of the default cinder back end, for example nfs_store.
   • The /var/lib/glance/tmp directory is where lock files used by oslo.concurrency are stored to coordinate concurrent access to shared resources.

2. Update the control plane:

   $ oc apply -f openstack_control_plane.yaml -n openstack

3. Wait until RHOCP creates the resources related to the OpenStackControlPlane CR. Run the following command to check the status:

   $ oc get openstackcontrolplane -n openstack

   The OpenStackControlPlane resources are created when the status is "Setup complete".

   Tip

   Append the -w option to the end of the get command to track deployment progress.

Procedure

1. Open your OpenStackControlPlane CR file, openstack_control_plane.yaml, and add the following parameters to the glance template to configure the Object Storage service as the back end:

   apiVersion: core.openstack.org/v1beta1
   kind: OpenStackControlPlane
   spec:
     ...
     glance:
       template:
         glanceAPIs:
           default:
             replicas: 3 # Configure back end; set to 3 when deploying service
         ...
         customServiceConfig: |
           [DEFAULT]
           enabled_backends = <backend_name>:swift
           [glance_store]
           default_backend = <backend_name>
           [<backend_name>]
           swift_store_create_container_on_put = True
           swift_store_auth_version = 3
           swift_store_auth_address = {{ .KeystoneInternalURL }}
           swift_store_key = {{ .ServicePassword }}
           swift_store_user = service:glance
           swift_store_endpoint_type = internalURL
   ...

   • Set replicas to 3 for high availability across APIs.
   • Replace <backend_name> with the name of the default back end.

2. Update the control plane:

   $ oc apply -f openstack_control_plane.yaml -n openstack

3. Wait until RHOCP creates the resources related to the OpenStackControlPlane CR. Run the following command to check the status:

   $ oc get openstackcontrolplane -n openstack

   The OpenStackControlPlane resources are created when the status is "Setup complete".

   Tip

   Append the -w option to the end of the get command to track deployment progress.

Procedure

1. Create a configuration file, for example, glance-s3.conf, where you can store the S3 configuration details.

2. Generate the secret and access keys for your S3 storage.

   • If your S3 storage is provisioned by the Ceph Object Gateway (RGW), run the following command to generate the secret and access keys:

     $ radosgw-admin user create --uid="<user_1>" \
     --display-name="<Jane Doe>"

     • Replace <user_1> with the user ID.
     • Replace <Jane Doe> with a display name for the user.

   • If your S3 storage is provisioned by the Object Storage service (swift), run the following command to generate the secret and access keys:

     $ openstackclient openstack credential create --type ec2 \
     --project admin admin \
     '{"access": "<access_key>", "secret": "<secret_key>"}'

     • Replace <access_key> with the user ID.
     • Replace <secret_key> with a display name for the user.

3. Add the S3 configuration details to your glance-s3.conf configuration file:

   [default_backend]
   s3_store_host = <_s3_endpoint_>
   s3_store_access_key = <_s3_access_key_>
   s3_store_secret_key = <_s3_secret_key_>
   s3_store_bucket = <_s3_bucket_>

   • Replace <_s3_endpoint_> with the host where the S3 server is listening. This option can contain a DNS name, for example, _s3.amazonaws.com_, or an IP address.
   • Replace <_s3_access_key_> and <_s3_secret_key_> with the data generated by the S3 back end.
   • Replace <_s3_bucket_> with the bucket name where you want to store images in the S3 back end. If you set s3_store_create_bucket_on_put to True in your OpenStackControlPlane CR file, the bucket name is created automatically, even if the bucket does not already exist.

4. Create a secret from the glance-s3.conf file:

   $ oc create secret generic glances3  \
   --from-file s3glance.conf

5. Open your OpenStackControlPlane CR file, openstack_control_plane.yaml, and add the following parameters to the glance template to configure S3 as the back end:

   apiVersion: core.openstack.org/v1beta1
   kind: OpenStackControlPlane
   spec:
     ...
     glance:
       template:
         ...
         customServiceConfig: |
           [DEFAULT]
           enabled_backends = <backend_name>:s3
           [glance_store]
           default_backend = <backend_name>
           [<backend_name>]
           s3_store_create_bucket_on_put = True
           s3_store_bucket_url_format = "path"
           s3_store_cacert = "/etc/pki/tls/certs/ca-bundle.crt
           s3_store_large_object_size = 0
         glanceAPIs:
           default:
             customServiceConfigSecrets:
             - glances3
   ...

   • Replace <backend_name> with the name of the default back end.
   • Optional: If your S3 storage is accessed by HTTPS, you must set the s3_store_cacert field and point it to the ca-bundle.crt path. The OpenStack control plane is deployed by default with TLS enabled, and a CA certificate is mounted to the pod in /etc/pki/tls/certs/ca-bundle.crt.
   • Optional: Set s3_store_large_object_size to 0 to force multipart upload when you create an image in the S3 back end from a Block Storage service (cinder) volume.

6. Update the control plane:

   $ oc apply -f openstack_control_plane.yaml -n openstack

7. Wait until RHOCP creates the resources related to the OpenStackControlPlane CR. Run the following command to check the status:

   $ oc get openstackcontrolplane -n openstack

   The OpenStackControlPlane resources are created when the status is "Setup complete".

   Tip

   Append the -w option to the end of the get command to track deployment progress.

Procedure

1. Open your OpenStackControlPlane CR file, openstack_control_plane.yaml, and add the extraMounts parameter in the spec section to add the export path and IP address of the NFS share. The path is mapped to /var/lib/glance/images, where the Image service API (glanceAPI) stores and retrieves images:

   apiVersion: core.openstack.org/v1beta1
   kind: OpenStackControlPlane
   metadata:
     name: openstack
   ...
   spec:
     extraMounts:
     - extraVol:
       - extraVolType: Nfs
         mounts:
         - mountPath: /var/lib/glance/images
           name: nfs
         propagation:
         - Glance
         volumes:
         - name: nfs
           nfs:
             path: <nfs_export_path>
             server: <nfs_ip_address>
       name: r1
       region: r1
   ...

   • Replace <nfs_export_path> with the export path of your NFS share.
   • Replace <nfs_ip_address> with the IP address of your NFS share. This IP address must be part of the overlay network that is reachable by the Image service.

2. Add the following parameters to the glance template to configure NFS as the back end:

   ...
   spec:
     extraMounts:
     ...
     glance:
       template:
         glanceAPIs:
           default:
             type: single
             replicas: 3 # Configure back end; set to 3 when deploying service
         ...
         customServiceConfig: |
           [DEFAULT]
           enabled_backends = <backend_name>:file
           [glance_store]
           default_backend = <backend_name>
           [<backend_name>]
           filesystem_store_datadir = /var/lib/glance/images
         databaseInstance: openstack
   ...

   • Set replicas to 3 for high availability across APIs.

   • Replace <backend_name> with the name of the default back end.

     Note

     When you configure an NFS back end, you must set the type to single. By default, the Image service has a split deployment type for an external API service, which is accessible through the public and administrator endpoints for the Identity service (keystone), and an internal API service, which is accessible only through the internal endpoint for the Identity service. The split deployment type is invalid for a file back end because different pods access the same file share.

3. Update the control plane:

   $ oc apply -f openstack_control_plane.yaml -n openstack

4. Wait until RHOCP creates the resources related to the OpenStackControlPlane CR. Run the following command to check the status:

   $ oc get openstackcontrolplane -n openstack

   The OpenStackControlPlane resources are created when the status is "Setup complete".

   Tip

   Append the -w option to the end of the get command to track deployment progress.

Procedure

1. Open your OpenStackControlPlane CR file, openstack_control_plane.yaml, and add the parameters to the glance template to configure the back ends. In the following example, there are two Ceph RBD stores and one Object Storage service (swift) store:

   ...
   spec:
     glance:
       template:
         customServiceConfig: |
           [DEFAULT]
           debug=True
           enabled_backends = ceph-0:rbd,ceph-1:rbd,swift-0:swift
   ...

2. Specify the back end to use as the default back end. In the following example, the default back end is ceph-1:

   ...
         customServiceConfig: |
           [DEFAULT]
           debug=True
           enabled_backends = ceph-0:rbd,ceph-1:rbd,swift-0:swift
           [glance_store]
           default_backend = ceph-1
   ...

3. Add the configuration for each back end type you want to use:

   • Add the configuration for the first Ceph RBD store, ceph-0:

     ...
           customServiceConfig: |
             [DEFAULT]
             ...
             [ceph-0]
             rbd_store_ceph_conf = /etc/ceph/ceph-0.conf
             store_description = "RBD backend"
             rbd_store_pool = images
             rbd_store_user = openstack
     ...

   • Add the configuration for the second Ceph RBD store, ceph-1:

     ...
           customServiceConfig: |
             [DEFAULT]
             ...
             [ceph-0]
             ...
             [ceph-1]
             rbd_store_ceph_conf = /etc/ceph/ceph-1.conf
             store_description = "RBD backend 1"
             rbd_store_pool = images
             rbd_store_user = openstack
     ...

   • Add the configuration for the Object Storage service store, swift-0:

     ...
           customServiceConfig: |
             [DEFAULT]
             ...
             [ceph-0]
             ...
             [ceph-1]
             ...
             [swift-0]
             swift_store_create_container_on_put = True
             swift_store_auth_version = 3
             swift_store_auth_address = {{ .KeystoneInternalURL }}
             swift_store_key = {{ .ServicePassword }}
             swift_store_user = service:glance
             swift_store_endpoint_type = internalURL
     ...

4. Update the control plane:

   $ oc apply -f openstack_control_plane.yaml -n openstack

5. Wait until RHOCP creates the resources related to the OpenStackControlPlane CR. Run the following command to check the status:

   $ oc get openstackcontrolplane -n openstack

   The OpenStackControlPlane resources are created when the status is "Setup complete".

   Tip

   Append the -w option to the end of the get command to track deployment progress.

Procedure

1. Open your OpenStackControlPlane CR file, openstack_control_plane.yaml, and add the glanceAPIs parameter to the glance template to configure multiple glanceAPI instances. In the following example, you create three glanceAPI instances that are named api0, api1, and api2:

   ...
   spec:
     glance:
       template:
         customServiceConfig: |
           [DEFAULT]
           enabled_backends = <backend_name>:rbd
           [glance_store]
           default_backend = <backend_name>
           [<backend_name>]
           rbd_store_ceph_conf = /etc/ceph/ceph.conf
           store_description = "RBD backend"
           rbd_store_pool = images
           rbd_store_user = openstack
         databaseInstance: openstack
         databaseUser: glance
         keystoneEndpoint: api0
         glanceAPIs:
           api0:
             replicas: 1
           api1:
             replicas: 1
           api2:
             replicas: 1
       ...

   • Replace <backend_name> with the name of the default back end.
   • api0 is registered in the Keystone catalog and is the default endpoint for OpenStack CLI operations.
   • api1 and api2 are not default endpoints, but they are active APIs that users can use for image uploads by specifying the --os-image-url parameter when they upload an image.
   • You can update the keystoneEndpoint parameter to change the default endpoint in the Keystone catalog.

2. Add the extraMounts parameter to connect the three glanceAPI instances to a different back end. In the following example, you connect api0, api1, and api2 to three different Ceph Storage clusters that are named ceph0, ceph1, and ceph2:

   spec:
     glance:
       template:
         customServiceConfig: |
           [DEFAULT]
           ...
         extraMounts:
           - name: api0
             region: r1
             extraVol:
               - propagation:
                 - api0
                 volumes:
                 - name: ceph0
                   secret:
                     secretName: <secret_name>
                 mounts:
                 - name: ceph0
                   mountPath: "/etc/ceph"
                   readOnly: true
           - name: api1
             region: r1
             extraVol:
               - propagation:
                 - api1
                 volumes:
                 - name: ceph1
                   secret:
                     secretName: <secret_name>
                 mounts:
                 - name: ceph1
                   mountPath: "/etc/ceph"
                   readOnly: true
           - name: api2
             region: r1
             extraVol:
               - propagation:
                 - api2
                 volumes:
                 - name: ceph2
                   secret:
                     secretName: <secret_name>
                 mounts:
                 - name: ceph2
                   mountPath: "/etc/ceph"
                   readOnly: true
   ...

   • Replace <secret_name> with the name of the secret associated to the Ceph Storage cluster that you are using as the back end for the specific glanceAPI, for example, ceph-conf-files-0 for the ceph0 cluster.

3. Update the control plane:

   $ oc apply -f openstack_control_plane.yaml -n openstack

4. Wait until RHOCP creates the resources related to the OpenStackControlPlane CR. Run the following command to check the status:

   $ oc get openstackcontrolplane -n openstack

   The OpenStackControlPlane resources are created when the status is "Setup complete".

   Tip

   Append the -w option to the end of the get command to track deployment progress.

Procedure

1. Open your OpenStackControlPlane CR file, openstack_control_plane.yaml, and add the following parameters to the swift template:

   apiVersion: core.openstack.org/v1beta1
   kind: OpenStackControlPlane
   metadata:
     name: openstack-control-plane
     namespace: openstack
   spec:
     ...
     swift:
       enabled: true
       template:
         swiftProxy:
           replicas: 2
         swiftRing:
           ringReplicas: 3
         swiftStorage:
           replicas: 3
           storageClass: <swift-storage>
           storageRequest: 100Gi
   ...

   • Increase the swiftProxy/replicas: value to distribute proxy instances across more nodes.
   • Replace the ringReplicas: value to define the number of object copies you want in your cluster.
   • Increase the swiftStorage/replicas: value to define the number of PVs in your cluster.
   • Replace <swift-storage> with the name of the storage class you want the Object Storage service to use.

2. Update the control plane:

   $ oc apply -f openstack_control_plane.yaml -n openstack

3. Wait until RHOCP creates the resources related to the OpenStackControlPlane CR. Run the following command to check the status:

   $ oc get openstackcontrolplane -n openstack

   The OpenStackControlPlane resources are created when the status is "Setup complete".

   Tip

   Append the -w option to the end of the get command to track deployment progress.

Procedure

1. Enable DNS forwarding to resolve data plane hostnames in the control plane pods.

   1. Obtain the clusterIP of the resolver:

      $ oc get svc dnsmasq-dns -o jsonpath=`{.spec.clusterIP}`

   2. Update the default DNS entry to add the clusterIP of the resolver:

      apiVersion: operator.openshift.io/v1
      kind: DNS
      metadata:
        name: default
      spec:
        servers:
        - name: swift
          zones:
          - storage.example.com
          forwardPlugin:
            policy: Random
            upstreams:
            - <clusterIP>

      • Replace <clusterIP> with the clusterIP of the resolver.

2. Enable the swift storage service on the data plane nodes by adding the swift service to the end of the list of services for the NodeSet in your OpenStackDataPlaneNodeSet CR. The service runs the playbooks that are required to configure the Object Storage services:

   Example:

       services:
       - repo-setup
       - bootstrap
       - download-cache
       - configure-network
       - validate-network
       - install-os
       - configure-os
       - ssh-known-hosts
       - run-os
       - reboot-os
       - install-certs
       - swift

3. Define disks to be used by the Object Storage service on data plane nodes.

   • When you define disks, you can do the following:

     • Define the disks in the global nodeTemplate section in your OpenStackDataPlaneNodeSet CR to use the same type of disks for all nodes.
     • Define disks on a per-node basis in the nodes section of your OpenStackDataPlaneNodeSet CR.
     • Assign disks to a specific region or zone.
     • Enable ring management to distribute replicas.

   • You must specify a weight for each disk. If you do not have custom weights in your existing rings, you can set the weight to the GiB capacity of the disk.

     The following example shows the OpenStackDataPlaneNodeSet CR for a data plane with three storage nodes. Each node is configured to use two disks in the nodeTemplate section. The first node edpm-swift-0 is configured to use a third disk in the nodes section:

     Example:

     - apiVersion: dataplane.openstack.org/v1beta1
       kind: OpenStackDataPlaneNodeSet
       metadata:
         name: openstack-edpm-ipam
         namespace: openstack
       spec:
         ...
         networkAttachments:
         - ctlplane
         - storage
         nodeTemplate:
           ansible:
             ansibleVars:
               edpm_swift_disks:
               - device: /dev/vdb
                 path: /srv/node/vdb
                 region: 0
                 weight: 4000
                 zone: 0
               - device: /dev/vdc
                 path: /srv/node/vdc
                 region: 0
                 weight: 4000
                 zone: 0
         nodes:
           edpm-swift-0:
             ansible:
               ansibleVars:
                 edpm_swift_disks:
                 - device: /dev/vdd
                   path: /srv/node/vdd
                   weight: 1000
             hostName: edpm-swift-0
             networks:
             - defaultRoute: true
               fixedIP: 192.168.122.100
               name: ctlplane
               subnetName: subnet1
             - name: internalapi
               subnetName: subnet1
             - name: storage
               subnetName: subnet1
             - name: tenant
               subnetName: subnet1
           edpm-swift-1:
             hostName: edpm-swift-1
             networks:
             - defaultRoute: true
               fixedIP: 192.168.122.101
               name: ctlplane
               subnetName: subnet1
             - name: internalapi
               subnetName: subnet1
             - name: storage
               subnetName: subnet1
             - name: tenant
               subnetName: subnet1
           edpm-swift-2:
             hostName: edpm-swift-2
             networks:
             - defaultRoute: true
               fixedIP: 192.168.122.102
               name: ctlplane
               subnetName: subnet1
             - name: internalapi
               subnetName: subnet1
             - name: storage
               subnetName: subnet1
             - name: tenant
               subnetName: subnet1
         ...
         services:
         - repo-setup
         - bootstrap
         - download-cache
         - configure-network
         - validate-network
         - install-os
         - configure-os
         - ssh-known-hosts
         - run-os
         - reboot-os
         - install-certs
         - swift

Procedure

1. Open your OpenStackControlPlane CR file, openstack_control_plane.yaml, and change the value for partPower under the swiftRing parameter in the swift template:

   apiVersion: core.openstack.org/v1beta1
   kind: OpenStackControlPlane
   metadata:
     name: openstack-control-plane
     namespace: openstack
   spec:
     ...
     swift:
       enabled: true
       template:
         swiftProxy:
           replicas: 2
         swiftRing:
           partPower: 12
           ringReplicas: 3
   ...

   • Replace <12> with the value you want to set for partition power.

     Tip

     You can also configure an additional object server ring for new containers. This is useful if you want to add more disks to an Object Storage service deployment that initially uses a low partition power.

Procedure

1. Open your OpenStackControlPlane CR file, openstack_control_plane.yaml, and add the following parameters to the spec section to enable the Shared File Systems service:

   spec:
     ...
     manila:
       enabled: true
       apiOverride:
         route: {}
       template:
         databaseInstance: openstack
         secret: osp-secret
         manilaAPI:
           replicas: 3
           override:
             service:
               internal:
                 metadata:
                   annotations:
                     metallb.universe.tf/address-pool: internalapi
                     metallb.universe.tf/allow-shared-ip: internalapi
                     metallb.universe.tf/loadBalancerIPs: 172.17.0.80
                 spec:
                   type: LoadBalancer
         manilaScheduler:
           replicas: 3
         manilaShares:
           share1:
             networkAttachments:
             - storage
             replicas: 0 # backend needs to be configured

   Note

   You must configure a back end for the Shared File Systems service. If you do not configure a back end for the Shared File Systems service, then the service is deployed but not activated (replicas: 0).

2. Update the control plane:

   $ oc apply -f openstack_control_plane.yaml -n openstack

3. Wait until RHOCP creates the resources related to the OpenStackControlPlane CR. Run the following command to check the status:

   $ oc get openstackcontrolplane -n openstack

   The OpenStackControlPlane resources are created when the status is "Setup complete".

   Tip

   Append the -w option to the end of the get command to track deployment progress.

Procedure

1. Open your OpenStackControlPlane CR file, openstack_control_plane.yaml, and add the extraMounts parameter in the spec section to present the Ceph configuration files:

   apiVersion: core.openstack.org/v1beta1
   kind: OpenStackControlPlane
   spec:
     extraMounts:
       - name: v1
         region: r1
         extraVol:
           - propagation:
             - ManilaShare
             extraVolType: Ceph
             volumes:
             - name: ceph
               projected:
                 sources:
                 - secret:
                     name: <ceph-conf-files>
             mounts:
             - name: ceph
               mountPath: "/etc/ceph"
               readOnly: true

2. Add the following parameters to the manila template to configure the native CephFS back end:

   apiVersion: core.openstack.org/v1beta1
   kind: OpenStackControlPlane
   spec:
       ...
     manila:
       enabled: true
       template:
         manilaAPI:
           replicas: 3
           customServiceConfig: |
             [DEFAULT]
             debug = true
             enabled_share_protocols=cephfs
         manilaScheduler:
           replicas: 3
         manilaShares:
           cephfsnative:
             replicas: 1
             networkAttachments:
             - storage
             customServiceConfig: |
               [DEFAULT]
               enabled_share_backends=cephfs
               [cephfs]
               driver_handles_share_servers=False
               share_backend_name=cephfs
               share_driver=manila.share.drivers.cephfs.driver.CephFSDriver
               cephfs_conf_path=/etc/ceph/ceph.conf
               cephfs_auth_id=openstack
               cephfs_cluster_name=ceph
               cephfs_volume_mode=0755
               cephfs_protocol_helper_type=CEPHFS
   ...

3. Update the control plane:

   $ oc apply -f openstack_control_plane.yaml -n openstack

4. Wait until RHOCP creates the resources related to the OpenStackControlPlane CR. Run the following command to check the status:

   $ oc get openstackcontrolplane -n openstack

   The OpenStackControlPlane resources are created when the status is "Setup complete".

   Tip

   Append the -w option to the end of the get command to track deployment progress.

Procedure

1. Open your OpenStackControlPlane CR file, openstack_control_plane.yaml, and add the following parameters to the manila template:

   apiVersion: core.openstack.org/v1beta1
   kind: OpenStackControlPlane
   spec:
       ...
     manila:
       enabled: true
       template:
         manilaAPI:
           replicas: 3
           customServiceConfig: |
             [DEFAULT]
             debug = true
             enabled_share_protocols=nfs
         manilaScheduler:
           replicas: 3
         manilaShares:
           share1:
             customServiceConfig: |
               [DEFAULT]
               enabled_share_backends=cephfsnfs
               [cephfsnfs]
               driver_handles_share_servers=False
               share_backend_name=cephfs
               share_driver=manila.share.drivers.cephfs.driver.CephFSDriver
               cephfs_auth_id=openstack
               cephfs_cluster_name=ceph
               cephfs_nfs_cluster_id=cephfs
               cephfs_protocol_helper_type=NFS
             networkAttachments:
             - storage
   ...

2. Update the control plane:

   $ oc apply -f openstack_control_plane.yaml -n openstack

3. Wait until RHOCP creates the resources related to the OpenStackControlPlane CR. Run the following command to check the status:

   $ oc get openstackcontrolplane -n openstack

   The OpenStackControlPlane resources are created when the status is "Setup complete".

   Tip

   Append the -w option to the end of the get command to track deployment progress.

1. Create the server connection secret.
2. Configure the OpenStackControlPlane CR to use the alternative storage system as the back end for the Shared File Systems service.

Procedure

1. Open your OpenStackControlPlane CR file, openstack_control_plane.yaml, and add the following parameters to the manila template to configure the back ends. In this example, there is a CephFS-NFS back end, a native CephFS back end, and a Pure Storage back end:

   apiVersion: core.openstack.org/v1beta1
   kind: OpenStackControlPlane
   spec:
       ...
     manila:
       enabled: true
       template:
         manilaAPI:
           replicas: 3
           customServiceConfig: |
             [DEFAULT]
             debug = true
             enabled_share_protocols=nfs,cephfs,cifs
         manilaScheduler:
           replicas: 3
       ...

2. Add the configuration for each back end you want to use:

   • Add the configuration for the CephFS-NFS back end:

         ...
             customServiceConfig: |
             ...
             manilaShares:
               cephfsnfs:
                 networkAttachments:
                 - storage
                 customServiceConfig: |
                     [DEFAULT]
                     enabled_share_backends=cephfsnfs
                     [cephfsnfs]
                     driver_handles_share_servers=False
                     share_backend_name=cephfs
                     share_driver=manila.share.drivers.cephfs.driver.CephFSDriver
                     cephfs_auth_id=openstack
                     cephfs_cluster_name=ceph
                     cephfs_nfs_cluster_id=cephfs
                     cephfs_protocol_helper_type=NFS
                 replicas: 1
         ...

   • Add the configuration for the native CephFS back end:

         ...
             customServiceConfig: |
             ...
             manilaShares:
               cephfsnfs:
               ...
               cephfs:
                 networkAttachments:
                 - storage
                 customServiceConfig: |
                   [DEFAULT]
                   enabled_share_backends=cephfs
                   [cephfs]
                   driver_handles_share_servers=False
                   share_backend_name=cephfs
                   share_driver=manila.share.drivers.cephfs.driver.CephFSDriver
                   cephfs_conf_path=/etc/ceph/ceph.conf
                   cephfs_auth_id=openstack
                   cephfs_protocol_helper_type=CEPHFS
                 replicas: 1
         ...

   • Add the configuration for the Pure Storage back end:

         ...
             customContainerImages:
               manilaShareImages:
                 pure: registry.connect.redhat.com/purestorage/openstack-manila-share-pure:rhoso18
             manilaShares:
               cephfsnfs:
               ...
               cephfs:
               ...
               pure:
                 networkAttachments:
                 - storage
                 customServiceConfigSecret: |
                 - manila-pure-secret
                 customServiceConfig: |
                   [DEFAULT]
                   debug = true
                   enabled_share_backends=pure
                   [pure]
                   driver_handles_share_servers=False
                   share_backend_name=pure
                   share_driver=manila.share.drivers.purestorage.flashblade.FlashBladeShareDriver
         ...

3. Update the control plane:

   $ oc apply -f openstack_control_plane.yaml -n openstack

4. Wait until RHOCP creates the resources related to the OpenStackControlPlane CR. Run the following command to check the status:

   $ oc get openstackcontrolplane -n openstack

   The OpenStackControlPlane resources are created when the status is "Setup complete".

   Tip

   Append the -w option to the end of the get command to track deployment progress.

Procedure

1. List the manila pods to verify that they are running:

   $ oc -n openstack get pod -l service=manila

   Example output:

   NAME                             READY   STATUS      RESTARTS          AGE
   manila-api-0                     2/2     Running     0                 43h
   manila-api-1                     2/2     Running     0                 43h
   manila-api-2                     2/2     Running     0                 43h
   manila-db-purge-28696321-tkl9g   0/1     Completed   0                 41h
   manila-db-purge-28697761-zxxzc   0/1     Completed   0                 17h
   manila-scheduler-0               2/2     Running     0                 43h
   manila-scheduler-1               2/2     Running     0                 43h
   manila-scheduler-2               2/2     Running     0                 43h
   manila-share-share1-0            2/2     Running     0                 43h

2. Access the remote shell for the openstackclient pod from your workstation:

   $ oc rsh -n openstack openstackclient

3. Run the openstack share service list command:

   $ openstack share service list

   Example output:

   ----------------------------------------------------------------------------------------------- | ID | Binary | Host | Zone | Status | State | Updated At | ----------------------------------------------------------------------------------------------- | 1 | manila-scheduler | hostgroup | nova | enabled | up | 2024-07-25T17:40:27.323342 | | 4 | manila-share | hostgroup@cephfsnfs | nova | enabled | up | 2024-07-25T17:40:49.115386 | -----------------------------------------------------------------------------------------------

4. Verify that the Status entry of every service is up. If not, examine the relevant log files.

5. Exit the openstackclient pod:

   $ exit

Procedure

1. Access the remote shell for the openstackclient pod from your workstation:

   $ oc rsh -n openstack openstackclient

2. Confirm the list of Shared File Systems service back ends:

   $ openstack share service list

   The status of each successfully deployed back end shows as enabled and the state shows as up.

3. Exit the openstackclient pod:

   $ exit

Procedure

1. Open your OpenStackControlPlane CR file, openstack_control_plane.yaml, and add the following parameters to the manila template:

   apiVersion: core.openstack.org/v1beta1
   kind: OpenStackControlPlane
   spec:
       ...
     manila:
       enabled: true
       template:
         manilaShares:
           cephfs:
             customServiceConfig: |
               [cephfs]
               backend_availability_zone = zone_1
             ...
           netapp:
             customServiceConfig: |
               [netapp]
               backend_availability_zone = zone_2
             ...

2. Update the control plane:

   $ oc apply -f openstack_control_plane.yaml -n openstack

3. Wait until RHOCP creates the resources related to the OpenStackControlPlane CR. Run the following command to check the status:

   $ oc get openstackcontrolplane -n openstack

   The OpenStackControlPlane resources are created when the status is "Setup complete".

   Tip

   Append the -w option to the end of the get command to track deployment progress.

Procedure

1. Open your OpenStackControlPlane CR file, openstack_control_plane.yaml, and add the following parameters to the manila template. In this example, you enable the NFS protocol:

   apiVersion: core.openstack.org/v1beta1
   kind: OpenStackControlPlane
   spec:
       ...
         manila:
           enabled: true
           template:
             manilaAPI:
               customServiceConfig: |
                 [DEFAULT]
                 enabled_share_protocols = NFS
             ...

2. Update the control plane:

   $ oc apply -f openstack_control_plane.yaml -n openstack

3. Wait until RHOCP creates the resources related to the OpenStackControlPlane CR. Run the following command to check the status:

   $ oc get openstackcontrolplane -n openstack

   The OpenStackControlPlane resources are created when the status is "Setup complete".

   Tip

   Append the -w option to the end of the get command to track deployment progress.

Procedure

1. Access the remote shell for the openstackclient pod from your workstation:

   $ oc rsh -n openstack openstackclient

2. Run the following command to view the available back-end storage capacity:

   $ openstack share pool list --detail

3. Exit the openstackclient pod:

   $ exit

Procedure

1. Open your openstack_control_plane.yaml file to edit the OpenStackControlPlane CR.

2. Add the dbPurge parameter to the manila template to configure database cleanup.

   The following is an example of using the dbPurge parameter to configure the Shared File Systems service:

   apiVersion: core.openstack.org/v1beta1
   kind: OpenStackControlPlane
   metadata:
     name: openstack
   spec:
     manila:
       template:
         dbPurge:
           age: 20
           schedule: 1 0 * * 0

   • age: The number of days a record has been marked for deletion before it is purged. The default value is 30 days. The minimum value is 1 day.
   • schedule: When to run the job in a crontab format. The default value is 1 0 * * *. This default value is equivalent to 00:01 daily.

3. Update the control plane:

   $ oc apply -f openstack_control_plane.yaml