Deploying and managing OpenShift Container Storage using Google Cloud

Procedure

1. Enable the Key/Value (KV) backend path in Vault.

   For Vault KV secret engine API, version 1:

   $ vault secrets enable -path=ocs kv

   For Vault KV secret engine API, version 2:

   $ vault secrets enable -path=ocs kv-v2

2. Create a policy to restrict users to perform a write or delete operation on the secret using the following commands:

   echo '
   path "ocs/*" {
     capabilities = ["create", "read", "update", "delete", "list"]
   }
   path "sys/mounts" {
   capabilities = ["read"]
   }'| vault policy write ocs -

3. Create a token matching the above policy:

   $ vault token create -policy=ocs -format json

Procedure

1. Navigate in the web console to the click Operators → OperatorHub.
2. Scroll or type a keyword into the Filter by keyword box to search for OpenShift Container Storage Operator.
3. Click Install on the OpenShift Container Storage operator page.

4. On the Install Operator page, the following required options are selected by default:

   1. Update Channel as stable-4.7.
   2. Installation Mode as A specific namespace on the cluster.
   3. Installed Namespace as Operator recommended namespace openshift-storage. If Namespace openshift-storage does not exist, it will be created during the operator installation.
   4. Select Approval Strategy as Automatic or Manual.

   5. Click Install.

      If you selected Automatic updates, then the Operator Lifecycle Manager (OLM) automatically upgrades the running instance of your Operator without any intervention.

      If you selected Manual updates, then the OLM creates an update request. As a cluster administrator, you must then manually approve that update request to have the Operator updated to the new version.

Procedure

1. Log into the OpenShift Web Console.

2. Click Operators → Installed Operators to view all the installed operators.

   Ensure that the Project selected is openshift-storage.

3. Click OpenShift Container Storage > Create Instance link of Storage Cluster.
4. Select Mode is set to Internal by default.

5. In Select capacity and nodes,

   1. Select Storage Class. By default, it is set to standard. However, if you created a storage class to use SSD based disks for better performance, you need to select that storage class.

   2. Select Requested Capacity from the drop down list. It is set to 2 TiB by default. You can use the drop down to modify the capacity value.

      Note

      Once you select the initial storage capacity, cluster expansion is performed only using the selected usable capacity (3 times of raw storage).

   3. In the Select Nodes section, select at least three available nodes.

      For cloud platforms with multiple availability zones, ensure that the Nodes are spread across different Locations/availability zones.

      If the nodes selected do not match the OpenShift Container Storage cluster requirement of an aggregated 30 CPUs and 72 GiB of RAM, a minimal cluster will be deployed. For minimum starting node requirements, see Resource requirements section in Planning guide.

   4. Click Next.

6. (Optional) Security configuration

   1. Select the Enable encryption checkbox to encrypt block and file storage.

   2. Choose any one or both Encryption level:

      • Cluster-wide encryption to encrypt the entire cluster (block and file).

      • Storage class encryption to create encrypted persistent volume (block only) using encryption enabled storage class.

        Important

        Storage class encryption is a Technology Preview feature available only for RBD PVs. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

        For more information, see Technology Preview Features Support Scope.

   3. Select the Connect to an external key management service checkbox. This is optional for cluster-wide encryption.

      1. Key Management Service Provider is set to Vault by default.
      2. Enter Vault Service Name, host Address of Vault server ('https://<hostname or ip>'), Port number and Token.

      3. Expand Advanced Settings to enter additional settings and certificate details based on your Vault configuration:

         1. Enter the Key Value secret path in Backend Path that is dedicated and unique to OpenShift Container Storage.
         2. Enter TLS Server Name and Vault Enterprise Namespace.
         3. Provide CA Certificate, Client Certificate and Client Private Key by uploading the respective PEM encoded certificate file.
         4. Click Save.
   4. Click Next.
7. Review the configuration details. To modify any configuration settings, click Back to go back to the previous configuration page.
8. Click Create.

9. Edit the configmap if Vault Key/Value (KV) secret engine API, version 2 is used for cluster-wide encryption with Key Management System (KMS).

   1. On the OpenShift Web Console, navigate to Workloads → ConfigMaps.
   2. To view the KMS connection details, click ocs-kms-connection-details.

   3. Edit the configmap.

      1. Click Action menu (⋮) → Edit ConfigMap.

      2. Set the VAULT_BACKEND parameter to v2.

         kind: ConfigMap
         apiVersion: v1
         metadata:
           name: ocs-kms-connection-details
         [...]
         data:
           KMS_PROVIDER: vault
           KMS_SERVICE_NAME: vault
         [...]
           VAULT_BACKEND: v2
         [...]

      3. Click Save.

Verification steps

1. On the storage cluster details page, the storage cluster name displays a green tick next to it to indicate that the cluster was created successfully.

2. Verify that the final Status of the installed storage cluster shows as Phase: Ready with a green tick mark.

   • Click Operators → Installed Operators → Storage Cluster link to view the storage cluster installation status.
   • Alternatively, when you are on the Operator Details tab, you can click on the Storage Cluster tab to view the status.
3. To verify that all components for OpenShift Container Storage are successfully installed, see Verifying your OpenShift Container Storage installation.

Procedure

1. Click Workloads → Pods from the left pane of the OpenShift Web Console.

2. Select openshift-storage from the Project drop down list.

   For more information on the expected number of pods for each component and how it varies depending on the number of nodes, see Table 3.1, “Pods corresponding to OpenShift Container storage cluster”.

3. Verify that the following pods are in running and completed state by clicking on the Running and the Completed tabs:

   Table 3.1. Pods corresponding to OpenShift Container storage cluster

   Component	Corresponding pods
   OpenShift Container Storage Operator	ocs-operator-* (1 pod on any worker node) ocs-metrics-exporter-*
   Rook-ceph Operator	rook-ceph-operator-* (1 pod on any worker node)
   Multicloud Object Gateway	noobaa-operator-* (1 pod on any worker node) noobaa-core-* (1 pod on any storage node) noobaa-db-pg-* (1 pod on any storage node) noobaa-endpoint-* (1 pod on any storage node)
   MON	rook-ceph-mon-* (3 pods distributed across storage nodes)
   MGR	rook-ceph-mgr-* (1 pod on any storage node)
   MDS	rook-ceph-mds-ocs-storagecluster-cephfilesystem-* (2 pods distributed across storage nodes)
   CSI	cephfs csi-cephfsplugin-* (1 pod on each worker node) csi-cephfsplugin-provisioner-* (2 pods distributed across worker nodes) rbd csi-rbdplugin-* (1 pod on each worker node) csi-rbdplugin-provisioner-* (2 pods distributed across worker nodes)
   rook-ceph-crashcollector	rook-ceph-crashcollector-* (1 pod on each storage node)
   OSD	rook-ceph-osd-* (1 pod for each device) rook-ceph-osd-prepare-ocs-deviceset-* (1 pod for each device)

1. Delete the volume snapshots that are using OpenShift Container Storage.

   1. List the volume snapshots from all the namespaces.

      $ oc get volumesnapshot --all-namespaces

   2. From the output of the previous command, identify and delete the volume snapshots that are using OpenShift Container Storage.

      $ oc delete volumesnapshot <VOLUME-SNAPSHOT-NAME> -n <NAMESPACE>

2. Delete PVCs and OBCs that are using OpenShift Container Storage.

   In the default uninstall mode (graceful), the uninstaller waits till all the PVCs and OBCs that use OpenShift Container Storage are deleted.

   If you wish to delete the Storage Cluster without deleting the PVCs beforehand, you may set the uninstall mode annotation to "forced" and skip this step. Doing so will result in orphan PVCs and OBCs in the system.

   1. Delete OpenShift Container Platform monitoring stack PVCs using OpenShift Container Storage.

      See Section 4.2, “Removing monitoring stack from OpenShift Container Storage”

   2. Delete OpenShift Container Platform Registry PVCs using OpenShift Container Storage.

      See Section 4.3, “Removing OpenShift Container Platform registry from OpenShift Container Storage”

   3. Delete OpenShift Container Platform logging PVCs using OpenShift Container Storage.

      See Section 4.4, “Removing the cluster logging operator from OpenShift Container Storage”

   4. Delete other PVCs and OBCs provisioned using OpenShift Container Storage.

      • Given below is a sample script to identify the PVCs and OBCs provisioned using OpenShift Container Storage. The script ignores the PVCs that are used internally by Openshift Container Storage.

        #!/bin/bash
        
        RBD_PROVISIONER="openshift-storage.rbd.csi.ceph.com"
        CEPHFS_PROVISIONER="openshift-storage.cephfs.csi.ceph.com"
        NOOBAA_PROVISIONER="openshift-storage.noobaa.io/obc"
        RGW_PROVISIONER="openshift-storage.ceph.rook.io/bucket"
        
        NOOBAA_DB_PVC="noobaa-db"
        NOOBAA_BACKINGSTORE_PVC="noobaa-default-backing-store-noobaa-pvc"
        
        # Find all the OCS StorageClasses
        OCS_STORAGECLASSES=$(oc get storageclasses | grep -e "$RBD_PROVISIONER" -e "$CEPHFS_PROVISIONER" -e "$NOOBAA_PROVISIONER" -e "$RGW_PROVISIONER" | awk '{print $1}')
        
        # List PVCs in each of the StorageClasses
        for SC in $OCS_STORAGECLASSES
        do
                echo "======================================================================"
                echo "$SC StorageClass PVCs and OBCs"
                echo "======================================================================"
                oc get pvc  --all-namespaces --no-headers 2>/dev/null | grep $SC | grep -v -e "$NOOBAA_DB_PVC" -e "$NOOBAA_BACKINGSTORE_PVC"
                oc get obc  --all-namespaces --no-headers 2>/dev/null | grep $SC
                echo
        done

        Note

        Omit RGW_PROVISIONER for cloud platforms.

      • Delete the OBCs.

        $ oc delete obc <obc name> -n <project name>

      • Delete the PVCs.

        $ oc delete pvc <pvc name> -n <project-name>

        Note

        Ensure that you have removed any custom backing stores, bucket classes, etc., created in the cluster.

3. Delete the Storage Cluster object and wait for the removal of the associated resources.

   $ oc delete -n openshift-storage storagecluster --all --wait=true

4. Check for cleanup pods if the uninstall.ocs.openshift.io/cleanup-policy was set to delete(default) and ensure that their status is Completed.

   $ oc get pods -n openshift-storage | grep -i cleanup
   NAME                                READY   STATUS      RESTARTS   AGE
   cluster-cleanup-job-<xx>        	0/1     Completed   0          8m35s
   cluster-cleanup-job-<yy>     		0/1     Completed   0          8m35s
   cluster-cleanup-job-<zz>     		0/1     Completed   0          8m35s

5. Confirm that the directory /var/lib/rook is now empty. This directory will be empty only if the uninstall.ocs.openshift.io/cleanup-policy annotation was set to delete(default).

   $ for i in $(oc get node -l cluster.ocs.openshift.io/openshift-storage= -o jsonpath='{ .items[*].metadata.name }'); do oc debug node/${i} -- chroot /host  ls -l /var/lib/rook; done

6. If encryption was enabled at the time of install, remove dm-crypt managed device-mapper mapping from OSD devices on all the OpenShift Container Storage nodes.

   1. Create a debug pod and chroot to the host on the storage node.

      $ oc debug node/<node name>
      $ chroot /host

   2. Get Device names and make note of the OpenShift Container Storage devices.

      $ dmsetup ls
      ocs-deviceset-0-data-0-57snx-block-dmcrypt (253:1)

   3. Remove the mapped device.

      $ cryptsetup luksClose --debug --verbose ocs-deviceset-0-data-0-57snx-block-dmcrypt

      Note

      If the above command gets stuck due to insufficient privileges, run the following commands:

      • Press CTRL+Z to exit the above command.

      • Find PID of the process which was stuck.

        $ ps -ef | grep crypt

      • Terminate the process using kill command.

        $ kill -9 <PID>

      • Verify that the device name is removed.

        $ dmsetup ls

7. Delete the namespace and wait till the deletion is complete. You will need to switch to another project if openshift-storage is the active project.

   For example:

   $ oc project default
   $ oc delete project openshift-storage --wait=true --timeout=5m

   The project is deleted if the following command returns a NotFound error.

   $ oc get project openshift-storage

   Note

   While uninstalling OpenShift Container Storage, if namespace is not deleted completely and remains in Terminating state, perform the steps in Troubleshooting and deleting remaining resources during Uninstall to identify objects that are blocking the namespace from being terminated.

8. Unlabel the storage nodes.

   $ oc label nodes  --all cluster.ocs.openshift.io/openshift-storage-
   $ oc label nodes  --all topology.rook.io/rack-

9. Remove the OpenShift Container Storage taint if the nodes were tainted.

   $ oc adm taint nodes --all node.ocs.openshift.io/storage-

10. Confirm all PVs provisioned using OpenShift Container Storage are deleted. If there is any PV left in the Released state, delete it.

    $ oc get pv
    $ oc delete pv <pv name>

11. Delete the Multicloud Object Gateway storageclass.

    $ oc delete storageclass openshift-storage.noobaa.io --wait=true --timeout=5m

12. Remove CustomResourceDefinitions.

    $ oc delete crd backingstores.noobaa.io bucketclasses.noobaa.io cephblockpools.ceph.rook.io cephclusters.ceph.rook.io cephfilesystems.ceph.rook.io cephnfses.ceph.rook.io cephobjectstores.ceph.rook.io cephobjectstoreusers.ceph.rook.io noobaas.noobaa.io ocsinitializations.ocs.openshift.io storageclusters.ocs.openshift.io cephclients.ceph.rook.io cephobjectrealms.ceph.rook.io cephobjectzonegroups.ceph.rook.io cephobjectzones.ceph.rook.io cephrbdmirrors.ceph.rook.io --wait=true --timeout=5m

13. Optional: To ensure that the vault keys are deleted permanently you need to manually delete the metadata associated with the vault key.

    Note

    Execute this step only if Vault Key/Value (KV) secret engine API, version 2 is used for cluster-wide encryption with Key Management System (KMS) since the vault keys are marked as deleted and not permanently deleted during the uninstallation of OpenShift Container Storage. You can always restore it later if required.

    1. List the keys in the vault.

       $ vault kv list <backend_path>

       <backend_path>

       Is the path in the vault where the encryption keys are stored.

       For example:

       $ vault kv list kv-v2

       Example output:

       Keys
       -----
       NOOBAA_ROOT_SECRET_PATH/
       rook-ceph-osd-encryption-key-ocs-deviceset-thin-0-data-0m27q8
       rook-ceph-osd-encryption-key-ocs-deviceset-thin-1-data-0sq227
       rook-ceph-osd-encryption-key-ocs-deviceset-thin-2-data-0xzszb

    2. List the metadata associated with the vault key.

       $ vault kv get kv-v2/<key>

       For the Multicloud Object Gateway (MCG) key:

       $ vault kv get kv-v2/NOOBAA_ROOT_SECRET_PATH/<key>

       <key>

       Is the encryption key.

       For Example:

       $ vault kv get kv-v2/rook-ceph-osd-encryption-key-ocs-deviceset-thin-0-data-0m27q8

       Example output:

       ====== Metadata ======
       Key              Value
       ---              -----
       created_time     2021-06-23T10:06:30.650103555Z
       deletion_time    2021-06-23T11:46:35.045328495Z
       destroyed        false
       version          1

    3. Delete the metadata.

       $ vault kv metadata delete kv-v2/<key>

       For the MCG key:

       $ vault kv metadata delete kv-v2/NOOBAA_ROOT_SECRET_PATH/<key>

       <key>

       Is the encryption key.

       For Example:

       $ vault kv metadata delete kv-v2/rook-ceph-osd-encryption-key-ocs-deviceset-thin-0-data-0m27q8

       Example output:

       Success! Data deleted (if it existed) at: kv-v2/metadata/rook-ceph-osd-encryption-key-ocs-deviceset-thin-0-data-0m27q8

    4. Repeat these steps to delete the metadata associated with all the vault keys.

14. To ensure that OpenShift Container Storage is uninstalled completely, on the OpenShift Container Platform Web Console,

    1. Click Home → Overview to access the dashboard.
    2. Verify that the Persistent Storage and Object Service tabs no longer appear next to the Cluster tab.

Procedure

1. List the pods and PVCs that are currently running in the openshift-monitoring namespace.

   $ oc get pod,pvc -n openshift-monitoring
   NAME                           READY   STATUS    RESTARTS   AGE
   pod/alertmanager-main-0         3/3     Running   0          8d
   pod/alertmanager-main-1         3/3     Running   0          8d
   pod/alertmanager-main-2         3/3     Running   0          8d
   pod/cluster-monitoring-
   operator-84457656d-pkrxm        1/1     Running   0          8d
   pod/grafana-79ccf6689f-2ll28    2/2     Running   0          8d
   pod/kube-state-metrics-
   7d86fb966-rvd9w                 3/3     Running   0          8d
   pod/node-exporter-25894         2/2     Running   0          8d
   pod/node-exporter-4dsd7         2/2     Running   0          8d
   pod/node-exporter-6p4zc         2/2     Running   0          8d
   pod/node-exporter-jbjvg         2/2     Running   0          8d
   pod/node-exporter-jj4t5         2/2     Running   0          6d18h
   pod/node-exporter-k856s         2/2     Running   0          6d18h
   pod/node-exporter-rf8gn         2/2     Running   0          8d
   pod/node-exporter-rmb5m         2/2     Running   0          6d18h
   pod/node-exporter-zj7kx         2/2     Running   0          8d
   pod/openshift-state-metrics-
   59dbd4f654-4clng                3/3     Running   0          8d
   pod/prometheus-adapter-
   5df5865596-k8dzn                1/1     Running   0          7d23h
   pod/prometheus-adapter-
   5df5865596-n2gj9                1/1     Running   0          7d23h
   pod/prometheus-k8s-0            6/6     Running   1          8d
   pod/prometheus-k8s-1            6/6     Running   1          8d
   pod/prometheus-operator-
   55cfb858c9-c4zd9                1/1     Running   0          6d21h
   pod/telemeter-client-
   78fc8fc97d-2rgfp                3/3     Running   0          8d
   
   NAME                                                              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                  AGE
   persistentvolumeclaim/my-alertmanager-claim-alertmanager-main-0   Bound    pvc-0d519c4f-15a5-11ea-baa0-026d231574aa   40Gi       RWO            ocs-storagecluster-ceph-rbd   8d
   persistentvolumeclaim/my-alertmanager-claim-alertmanager-main-1   Bound    pvc-0d5a9825-15a5-11ea-baa0-026d231574aa   40Gi       RWO            ocs-storagecluster-ceph-rbd   8d
   persistentvolumeclaim/my-alertmanager-claim-alertmanager-main-2   Bound    pvc-0d6413dc-15a5-11ea-baa0-026d231574aa   40Gi       RWO            ocs-storagecluster-ceph-rbd   8d
   persistentvolumeclaim/my-prometheus-claim-prometheus-k8s-0        Bound    pvc-0b7c19b0-15a5-11ea-baa0-026d231574aa   40Gi       RWO            ocs-storagecluster-ceph-rbd   8d
   persistentvolumeclaim/my-prometheus-claim-prometheus-k8s-1        Bound    pvc-0b8aed3f-15a5-11ea-baa0-026d231574aa   40Gi       RWO            ocs-storagecluster-ceph-rbd   8d

2. Edit the monitoring configmap.

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

3. Remove any config sections that reference the OpenShift Container Storage storage classes as shown in the following example and save it.

   Before editing

   . . . apiVersion: v1 data: config.yaml: | alertmanagerMain: volumeClaimTemplate: metadata: name: my-alertmanager-claim spec: resources: requests: storage: 40Gi storageClassName: ocs-storagecluster-ceph-rbd prometheusK8s: volumeClaimTemplate: metadata: name: my-prometheus-claim spec: resources: requests: storage: 40Gi storageClassName: ocs-storagecluster-ceph-rbd kind: ConfigMap metadata: creationTimestamp: "2019-12-02T07:47:29Z" name: cluster-monitoring-config namespace: openshift-monitoring resourceVersion: "22110" selfLink: /api/v1/namespaces/openshift-monitoring/configmaps/cluster-monitoring-config uid: fd6d988b-14d7-11ea-84ff-066035b9efa8 . . .

   After editing

   . . . apiVersion: v1 data: config.yaml: | kind: ConfigMap metadata: creationTimestamp: "2019-11-21T13:07:05Z" name: cluster-monitoring-config namespace: openshift-monitoring resourceVersion: "404352" selfLink: /api/v1/namespaces/openshift-monitoring/configmaps/cluster-monitoring-config uid: d12c796a-0c5f-11ea-9832-063cd735b81c . . .

   In this example, alertmanagerMain and prometheusK8s monitoring components are using the OpenShift Container Storage PVCs.

4. Delete relevant PVCs. Make sure you delete all the PVCs that are consuming the storage classes.

   $ oc delete -n openshift-monitoring pvc <pvc-name> --wait=true --timeout=5m

Procedure

1. Edit the configs.imageregistry.operator.openshift.io object and remove the content in the storage section.

   $ oc edit configs.imageregistry.operator.openshift.io

   Before editing

   . . . storage: pvc: claim: registry-cephfs-rwx-pvc . . .

   After editing

   . . . storage: . . .

   In this example, the PVC is called registry-cephfs-rwx-pvc, which is now safe to delete.

2. Delete the PVC.

   $ oc delete pvc <pvc-name> -n openshift-image-registry --wait=true --timeout=5m

Procedure

1. Remove the ClusterLogging instance in the namespace.

   $ oc delete clusterlogging instance -n openshift-logging --wait=true --timeout=5m

   The PVCs in the openshift-logging namespace are now safe to delete.

2. Delete PVCs.

   $ oc delete pvc <pvc-name> -n openshift-logging --wait=true --timeout=5m

Procedure

1. Log in to OpenShift Web Console.
2. Click Storage → Storage Classes.
3. Click Create Storage Class.
4. Enter the storage class Name and Description.
5. Select either Delete or Retain for the Reclaim Policy. By default, Delete is selected.
6. Select RBD Provisioner which is the plugin used for provisioning the persistent volumes.

7. You can either create a new pool or use an existing one.

   Create a new pool

   1. Enter a name for the pool.
   2. Choose 2-way-Replication or 3-way-Replication as the Data Protection Policy.

   3. Select Enable compression if you need to compress the data.

      Enabling compression can impact application performance and might prove ineffective when data to be written is already compressed or encrypted. Data written before enabling compression will not be compressed.

   4. Click Create to create the storage pool.
   5. Click Finish after the pool is created.
   6. Click Create to create the storage class.

   Use an existing pool

   1. Choose a pool from the list.
   2. Click Create to create the storage class with the selected pool.

Procedure

1. Navigate to Storage → Storage Classes.
2. Click Create Storage Class.
3. Enter the storage class Name and Description.
4. Select either Delete or Retain for the Reclaim Policy. By default, Delete is selected.
5. Select RBD Provisioner openshift-storage.rbd.csi.ceph.com which is the plugin used for provisioning the persistent volumes.
6. Select Storage Pool where the volume data will be stored.

7. Select Enable Encryption checkbox.

   1. Key Management Service Provider is set to Vault by default.
   2. Enter Vault Service Name, host Address of Vault server ('https://<hostname or ip>'), and Port number.

   3. Expand Advanced Settings to enter certificate details.

      1. Enter the key value secret path in Backend Path that is dedicated and unique to OpenShift Container Storage.
      2. (Optional) Enter TLS Server Name and Vault Enterprise Namespace.
      3. Provide CA Certificate, Client Certificate and Client Private Key by uploading the respective PEM encoded certificate file.
      4. Click Save.
   4. Click Connect.
8. Review external key management service Connection details. To modify the information, click Change connection details and edit the fields.
9. Click Create.

10. Edit the configmap to add the VAULT_BACKEND parameter if the Hashicorp Vault setup does not allow automatic detection of the Key/Value (KV) secret engine API version used by the backend path.

    Note

    VAULT_BACKEND is an optional parameter that is added to the configmap to specify the version of the KV secret engine API associated with the backend path. Ensure that the value matches the KV secret engine API version that is set for the backend path, otherwise it might result in a failure during persistent volume claim (PVC) creation.

    1. Identify the encryptionKMSID being used by the newly created storage class.

       1. On the OpenShift Web Console, navigate to Storage → Storage Classes.
       2. Click the Storage class name → YAML tab.

       3. Capture the encryptionKMSID being used by the storage class.

          Example:

          encryptionKMSID: 1-vault

    2. On the OpenShift Web Console, navigate to Workloads → ConfigMaps.
    3. To view the KMS connection details, click csi-kms-connection-details.

    4. Edit the configmap.

       1. Click Action menu (⋮) → Edit ConfigMap.

       2. Add the VAULT_BACKEND parameter depending on the backend that is configured for the previously identified encryptionKMSID.

          You can assign kv for KV secret engine API, version 1 and kv-v2 for KV secret engine API, version 2 as the VAULT_BACKEND parameter.

          Example:

          kind: ConfigMap
          apiVersion: v1
          metadata:
            name: csi-kms-connection-details
          [...]
          data:
            1-vault: >-
          
              {
                "KMS_PROVIDER": "vaulttokens",
                "KMS_SERVICE_NAME": "vault",
                [...]
                "VAULT_BACKEND": "kv-v2"
              }

       3. Click Save.

Procedure

1. Create a Persistent Volume Claim for the Image Registry to use.

   1. In the OpenShift Web Console, click Storage → Persistent Volume Claims.
   2. Set the Project to openshift-image-registry.

   3. Click Create Persistent Volume Claim.

      1. From the list of available storage classes retrieved above, specify the Storage Class with the provisioner openshift-storage.cephfs.csi.ceph.com.
      2. Specify the Persistent Volume Claim Name, for example, ocs4registry.
      3. Specify an Access Mode of Shared Access (RWX).
      4. Specify a Size of at least 100 GB.

      5. Click Create.

         Wait until the status of the new Persistent Volume Claim is listed as Bound.

2. Configure the cluster’s Image Registry to use the new Persistent Volume Claim.

   1. Click Administration → Custom Resource Definitions.
   2. Click the Config custom resource definition associated with the imageregistry.operator.openshift.io group.
   3. Click the Instances tab.
   4. Beside the cluster instance, click the Action Menu (⋮) → Edit Config.

   5. Add the new Persistent Volume Claim as persistent storage for the Image Registry.

      1. Add the following under spec:, replacing the existing storage: section if necessary.

           storage:
             pvc:
               claim: <new-pvc-name>

         For example:

           storage:
             pvc:
               claim: ocs4registry

      2. Click Save.

3. Verify that the new configuration is being used.

   1. Click Workloads → Pods.
   2. Set the Project to openshift-image-registry.
   3. Verify that the new image-registry-* pod appears with a status of Running, and that the previous image-registry-* pod terminates.
   4. Click the new image-registry-* pod to view pod details.
   5. Scroll down to Volumes and verify that the registry-storage volume has a Type that matches your new Persistent Volume Claim, for example, ocs4registry.

Procedure

1. In the OpenShift Web Console, go to Workloads → Config Maps.
2. Set the Project dropdown to openshift-monitoring.
3. Click Create Config Map.

4. Define a new cluster-monitoring-config Config Map using the following example.

   Replace the content in angle brackets (<, >) with your own values, for example, retention: 24h or storage: 40Gi.

   Replace the storageClassName with the storageclass that uses the provisioner openshift-storage.rbd.csi.ceph.com. In the example given below the name of the storageclass is ocs-storagecluster-ceph-rbd.

   Example cluster-monitoring-config Config Map

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
         prometheusK8s:
           retention: <time to retain monitoring files, e.g. 24h>
           volumeClaimTemplate:
             metadata:
               name: ocs-prometheus-claim
             spec:
               storageClassName: ocs-storagecluster-ceph-rbd
               resources:
                 requests:
                   storage: <size of claim, e.g. 40Gi>
         alertmanagerMain:
           volumeClaimTemplate:
             metadata:
               name: ocs-alertmanager-claim
             spec:
               storageClassName: ocs-storagecluster-ceph-rbd
               resources:
                 requests:
                   storage: <size of claim, e.g. 40Gi>

5. Click Create to save and create the Config Map.

Verification steps

1. Verify that the Persistent Volume Claims are bound to the pods.

   1. Go to Storage → Persistent Volume Claims.
   2. Set the Project dropdown to openshift-monitoring.

   3. Verify that 5 Persistent Volume Claims are visible with a state of Bound, attached to three alertmanager-main-* pods, and two prometheus-k8s-* pods.

      Monitoring storage created and bound

      

2. Verify that the new alertmanager-main-* pods appear with a state of Running.

   1. Go to Workloads → Pods.
   2. Click the new alertmanager-main-* pods to view the pod details.

   3. Scroll down to Volumes and verify that the volume has a Type, ocs-alertmanager-claim that matches one of your new Persistent Volume Claims, for example, ocs-alertmanager-claim-alertmanager-main-0.

      Persistent Volume Claims attached to alertmanager-main-* pod

      

3. Verify that the new prometheus-k8s-* pods appear with a state of Running.

   1. Click the new prometheus-k8s-* pods to view the pod details.

   2. Scroll down to Volumes and verify that the volume has a Type, ocs-prometheus-claim that matches one of your new Persistent Volume Claims, for example, ocs-prometheus-claim-prometheus-k8s-0.

      Persistent Volume Claims attached to prometheus-k8s-* pod

      

Procedure

1. Log in to the OpenShift Web Console.
2. Click on Operators → Installed Operators.
3. Click OpenShift Container Storage Operator.
4. Click Storage Cluster tab.
5. The visible list should have only one item. Click (⋮) on the far right to extend the options menu.
6. Select Add Capacity from the options menu.

7. Select the Storage Class.

   Set the storage class to standard if you are using the default storage class that uses HDD. However, if you created a storage class to use SSD based disks for better performance, you need to select that storage class.

   The Raw Capacity field shows the size set during storage class creation. The total amount of storage consumed is three times this amount, because OpenShift Container Storage uses a replica count of 3.

8. Click Add and wait for the cluster state to change to Ready.

Procedure

1. Enable access to the Multicloud Object Gateway console.

   Perform the following steps once on the cluster :

   1. Create a cluster-admins group.

      # oc adm groups new cluster-admins

   2. Bind the group to the cluster-admin role.

      # oc adm policy add-cluster-role-to-group cluster-admin cluster-admins

2. Add or remove users from the cluster-admins group to control access to the Multicloud Object Gateway console.

   • To add a set of users to the cluster-admins group :

     # oc adm groups add-users cluster-admins <user-name> <user-name> <user-name>...

     where <user-name> is the name of the user to be added.

     Note

     If you are adding a set of users to the cluster-admins group, you do not need to bind the newly added users to the cluster-admin role to allow access to the OpenShift Container Storage dashboard.

   • To remove a set of users from the cluster-admins group :

     # oc adm groups remove-users cluster-admins <user-name> <user-name> <user-name>...

     where <user-name> is the name of the user to be removed.

Verification steps

1. On the OpenShift Web Console, login as a user with access permission to Multicloud Object Gateway Console.
2. Navigate to Home → Overview → Object Service tab → select the Multicloud Object Gateway link .
3. On the Multicloud Object Gateway Console, login as the same user with access permission.
4. Click Allow selected permissions.

1. Connect your providers to the Multicloud Object Gateway.
2. Create a namespace resource for each of your providers so they can be added to a namespace bucket.
3. Add your namespace resources to a namespace bucket and configure the bucket to read from and write to the appropriate namespace resources.

Procedure

1. Create a Persistent Volume Claim (PVC) for the application to use.

   1. In OpenShift Web Console, click Storage → Persistent Volume Claims.
   2. Set the Project for the application pod.

   3. Click Create Persistent Volume Claim.

      1. Specify a Storage Class provided by OpenShift Container Storage.
      2. Specify the PVC Name, for example, myclaim.
      3. Select the required Access Mode.
      4. Specify a Size as per application requirement.
      5. Click Create and wait until the PVC is in Bound status.

2. Configure a new or existing application pod to use the new PVC.

   • For a new application pod, perform the following steps:

     1. Click Workloads →Pods.
     2. Create a new application pod.

     3. Under the spec: section, add volume: section to add the new PVC as a volume for the application pod.

        volumes:
          - name: <volume_name>
            persistentVolumeClaim:
              claimName: <pvc_name>

        For example:

        volumes:
          - name: mypd
            persistentVolumeClaim:
              claimName: myclaim

   • For an existing application pod, perform the following steps:

     1. Click Workloads →Deployment Configs.
     2. Search for the required deployment config associated with the application pod.
     3. Click on its Action menu (⋮) → Edit Deployment Config.

     4. Under the spec: section, add volume: section to add the new PVC as a volume for the application pod and click Save.

        volumes:
          - name: <volume_name>
            persistentVolumeClaim:
              claimName: <pvc_name>

        For example:

        volumes:
          - name: mypd
            persistentVolumeClaim:
              claimName: myclaim

3. Verify that the new configuration is being used.

   1. Click Workloads → Pods.
   2. Set the Project for the application pod.
   3. Verify that the application pod appears with a status of Running.
   4. Click the application pod name to view pod details.
   5. Scroll down to Volumes section and verify that the volume has a Type that matches your new Persistent Volume Claim, for example, myclaim.

Procedure

1. Log in to OpenShift Web Console.
2. Click Storage → Persistent Volume Claims
3. Search for the required PVC name by using the Filter textbox. You can also filter the list of PVCs by Name or Label to narrow down the list
4. Check the Status column corresponding to the required PVC.
5. Click the required Name to view the PVC details.

Procedure

1. Log in to OpenShift Web Console.
2. Click Home → Overview → Persistent Storage
3. Locate the Inventory card to see the number of PVCs with errors.
4. Click Storage → Persistent Volume Claims
5. Search for the required PVC using the Filter textbox.
6. Click on the PVC name and navigate to Events
7. Address the events as required or as directed.

Procedure

1. Click Storage → Persistent Volume Claims from the OpenShift Web Console.

2. To create a clone, do one of the following:

   • Beside the desired PVC, click Action menu (⋮) → Clone PVC.
   • Click on the PVC that you want to clone and click Actions → Clone PVC.
3. Enter a Name for the clone.

4. Select the access mode of your choice.

   Important

   The ReadOnlyMany (ROX) access mode is a Developer Preview feature and is subject to Developer Preview support limitations. Developer Preview releases are not intended to be run in production environments and are not supported through the Red Hat Customer Portal case management system. If you need assistance with ReadOnlyMany feature, reach out to the ocs-devpreview@redhat.com mailing list and a member of the Red Hat Development Team will assist you as quickly as possible based on availability and work schedules. See Creating a clone or restoring a snapshot with the new readonly access mode to use the ROX access mode.

5. Click Clone. You are redirected to the new PVC details page.

6. Wait for the cloned PVC status to become Bound.

   The cloned PVC is now available to be consumed by the pods. This cloned PVC is independent of its dataSource PVC.

Procedure

1. Log in to OpenShift Web Console and click Compute → Nodes.
2. Identify the node that needs to be replaced. Take a note of its Machine Name.

3. Mark the node as unschedulable using the following command:

   $ oc adm cordon <node_name>

4. Drain the node using the following command:

   $ oc adm drain <node_name> --force --delete-local-data --ignore-daemonsets

   Important

   This activity may take at least 5-10 minutes or more. Ceph errors generated during this period are temporary and are automatically resolved when the new node is labeled and functional.

5. Click Compute → Machines. Search for the required machine.
6. Besides the required machine, click the Action menu (⋮) → Delete Machine.
7. Click Delete to confirm the machine deletion. A new machine is automatically created.

8. Wait for new machine to start and transition into Running state.

   Important

   This activity may take at least 5-10 minutes or more.

9. Click Compute → Nodes, confirm if the new node is in Ready state.

10. Apply the OpenShift Container Storage label to the new node using any one of the following:

    From User interface

    1. For the new node, click Action Menu (⋮) → Edit Labels
    2. Add cluster.ocs.openshift.io/openshift-storage and click Save.

    From Command line interface

    • Execute the following command to apply the OpenShift Container Storage label to the new node:

      $ oc label node <new_node_name> cluster.ocs.openshift.io/openshift-storage=""

Verification steps

1. Execute the following command and verify that the new node is present in the output:

   $ oc get nodes --show-labels | grep cluster.ocs.openshift.io/openshift-storage= |cut -d' ' -f1

2. Click Workloads → Pods, confirm that at least the following pods on the new node are in Running state:

   • csi-cephfsplugin-*
   • csi-rbdplugin-*
3. Verify that all other required OpenShift Container Storage pods are in Running state.

4. Verify that new OSD pods are running on the replacement node.

   $ oc get pods -o wide -n openshift-storage| egrep -i new-node-name | egrep osd

5. (Optional) If cluster-wide encryption is enabled on the cluster, verify that the new OSD devices are encrypted.

   1. For each of the new nodes identified in previous step, do the following:

      1. Create a debug pod and open a chroot environment for the selected host(s).

         $ oc debug node/<node name>
         $ chroot /host

      2. Run “lsblk” and check for the “crypt” keyword beside the ocs-deviceset name(s)

         $ lsblk

6. If verification steps fail, contact Red Hat Support.

Procedure

1. Log in to OpenShift Web Console and click Compute → Nodes.
2. Identify the faulty node and click on its Machine Name.
3. Click Actions → Edit Annotations, and click Add More.
4. Add machine.openshift.io/exclude-node-draining and click Save.
5. Click Actions → Delete Machine, and click Delete.

6. A new machine is automatically created, wait for new machine to start.

   Important

   This activity may take at least 5-10 minutes or more. Ceph errors generated during this period are temporary and are automatically resolved when the new node is labeled and functional.

7. Click Compute → Nodes, confirm if the new node is in Ready state.

8. Apply the OpenShift Container Storage label to the new node using any one of the following:

   From the web user interface

   1. For the new node, click Action Menu (⋮) → Edit Labels
   2. Add cluster.ocs.openshift.io/openshift-storage and click Save.

   From the command line interface

   • Execute the following command to apply the OpenShift Container Storage label to the new node:

     $ oc label node <new_node_name> cluster.ocs.openshift.io/openshift-storage=""

9. [Optional]: If the failed Google Cloud instance is not removed automatically, terminate the instance from Google Cloud console.

Verification steps

1. Execute the following command and verify that the new node is present in the output:

   $ oc get nodes --show-labels | grep cluster.ocs.openshift.io/openshift-storage= |cut -d' ' -f1

2. Click Workloads → Pods, confirm that at least the following pods on the new node are in Running state:

   • csi-cephfsplugin-*
   • csi-rbdplugin-*
3. Verify that all other required OpenShift Container Storage pods are in Running state.

4. Verify that new OSD pods are running on the replacement node.

   $ oc get pods -o wide -n openshift-storage| egrep -i new-node-name | egrep osd

5. (Optional) If cluster-wide encryption is enabled on the cluster, verify that the new OSD devices are encrypted.

   1. For each of the new nodes identified in previous step, do the following:

      1. Create a debug pod and open a chroot environment for the selected host(s).

         $ oc debug node/<node name>
         $ chroot /host

      2. Run “lsblk” and check for the “crypt” keyword beside the ocs-deviceset name(s)

         $ lsblk

6. If verification steps fail, contact Red Hat Support.

1. Update OpenShift Container Platform according to the Updating clusters documentation for OpenShift Container Platform.

2. Update OpenShift Container Storage.

   1. To prepare a disconnected environment for updates, see Operators guide to using Operator Lifecycle Manager on restricted networks to be able to update Openshift Container Storage as well as Local Storage Operator when in use.

   2. Update the OpenShift Container Storage operator, using the appropriate process for your setup:

      • Update OpenShift Container Storage in internal mode

1. Configure authentication for an alternative registry.
2. Build and mirror the Red Hat operator catalog.
3. Creating Operator imageContentSourcePolicy
4. Updating redhat-operator catalogsource