Deploying and managing OpenShift Container Storage using Red Hat OpenStack Platform

Procedure

1. Click Workloads → Pods from the left pane of the OpenShift Web Console.

2. Select openshift-storage from the Project drop down list.

   For more information on the expected number of pods for each component and how it varies depending on the number of nodes, see Table 1.1, “Pods corresponding to OpenShift Container storage cluster”.

3. Verify that the following pods are in running and completed state by clicking on the Running and the Completed tabs:

   Table 1.1. Pods corresponding to OpenShift Container storage cluster

   Component	Corresponding pods
   OpenShift Container Storage Operator	ocs-operator-* (1 pod on any worker node) ocs-metrics-exporter-*
   Rook-ceph Operator	rook-ceph-operator-* (1 pod on any worker node)
   Multicloud Object Gateway	noobaa-operator-* (1 pod on any worker node) noobaa-core-* (1 pod on any storage node) noobaa-db-* (1 pod on any storage node) noobaa-endpoint-* (1 pod on any storage node)
   MON	rook-ceph-mon-* (3 pods distributed across storage nodes)
   MGR	rook-ceph-mgr-* (1 pod on any storage node)
   MDS	rook-ceph-mds-ocs-storagecluster-cephfilesystem-* (2 pods distributed across storage nodes)
   CSI	cephfs csi-cephfsplugin-* (1 pod on each worker node) csi-cephfsplugin-provisioner-* (2 pods distributed across worker nodes) rbd csi-rbdplugin-* (1 pod on each worker node) csi-rbdplugin-provisioner-* (2 pods distributed across worker nodes)
   rook-ceph-crashcollector	rook-ceph-crashcollector-* (1 pod on each storage node)
   OSD	rook-ceph-osd-* (1 pod for each device) rook-ceph-osd-prepare-ocs-deviceset-* (1 pod for each device)

1. Delete the volume snapshots that are using OpenShift Container Storage.

   1. List the volume snapshots from all the namespaces.

      $ oc get volumesnapshot --all-namespaces

   2. From the output of the previous command, identify and delete the volume snapshots that are using OpenShift Container Storage.

      $ oc delete volumesnapshot <VOLUME-SNAPSHOT-NAME> -n <NAMESPACE>

2. Delete PVCs and OBCs that are using OpenShift Container Storage.

   In the default uninstall mode (graceful), the uninstaller waits till all the PVCs and OBCs that use OpenShift Container Storage are deleted.

   If you wish to delete the Storage Cluster without deleting the PVCs beforehand, you may set the uninstall mode annotation to "forced" and skip this step. Doing so will result in orphan PVCs and OBCs in the system.

   1. Delete OpenShift Container Platform monitoring stack PVCs using OpenShift Container Storage.

      See Section 1.4.1.1, “Removing monitoring stack from OpenShift Container Storage”

   2. Delete OpenShift Container Platform Registry PVCs using OpenShift Container Storage.

      See Section 1.4.1.2, “Removing OpenShift Container Platform registry from OpenShift Container Storage”

   3. Delete OpenShift Container Platform logging PVCs using OpenShift Container Storage.

      See Section 1.4.1.3, “Removing the cluster logging operator from OpenShift Container Storage”

   4. Delete other PVCs and OBCs provisioned using OpenShift Container Storage.

      • Given below is a sample script to identify the PVCs and OBCs provisioned using OpenShift Container Storage. The script ignores the PVCs that are used internally by Openshift Container Storage.

        #!/bin/bash
        
        RBD_PROVISIONER="openshift-storage.rbd.csi.ceph.com"
        CEPHFS_PROVISIONER="openshift-storage.cephfs.csi.ceph.com"
        NOOBAA_PROVISIONER="openshift-storage.noobaa.io/obc"
        RGW_PROVISIONER="openshift-storage.ceph.rook.io/bucket"
        
        NOOBAA_DB_PVC="noobaa-db"
        NOOBAA_BACKINGSTORE_PVC="noobaa-default-backing-store-noobaa-pvc"
        
        # Find all the OCS StorageClasses
        OCS_STORAGECLASSES=$(oc get storageclasses | grep -e "$RBD_PROVISIONER" -e "$CEPHFS_PROVISIONER" -e "$NOOBAA_PROVISIONER" -e "$RGW_PROVISIONER" | awk '{print $1}')
        
        # List PVCs in each of the StorageClasses
        for SC in $OCS_STORAGECLASSES
        do
                echo "======================================================================"
                echo "$SC StorageClass PVCs and OBCs"
                echo "======================================================================"
                oc get pvc  --all-namespaces --no-headers 2>/dev/null | grep $SC | grep -v -e "$NOOBAA_DB_PVC" -e "$NOOBAA_BACKINGSTORE_PVC"
                oc get obc  --all-namespaces --no-headers 2>/dev/null | grep $SC
                echo
        done

        Note

        Omit RGW_PROVISIONER for cloud platforms.

      • Delete the OBCs.

        $ oc delete obc <obc name> -n <project name>

      • Delete the PVCs.

        $ oc delete pvc <pvc name> -n <project-name>

        Note

        Ensure that you have removed any custom backing stores, bucket classes, etc., created in the cluster.

3. Delete the Storage Cluster object and wait for the removal of the associated resources.

   $ oc delete -n openshift-storage storagecluster --all --wait=true

4. Check for cleanup pods if the uninstall.ocs.openshift.io/cleanup-policy was set to delete(default) and ensure that their status is Completed.

   $ oc get pods -n openshift-storage | grep -i cleanup
   NAME                                READY   STATUS      RESTARTS   AGE
   cluster-cleanup-job-<xx>        	0/1     Completed   0          8m35s
   cluster-cleanup-job-<yy>     		0/1     Completed   0          8m35s
   cluster-cleanup-job-<zz>     		0/1     Completed   0          8m35s

5. Confirm that the directory /var/lib/rook is now empty. This directory will be empty only if the uninstall.ocs.openshift.io/cleanup-policy annotation was set to delete(default).

   $ for i in $(oc get node -l cluster.ocs.openshift.io/openshift-storage= -o jsonpath='{ .items[*].metadata.name }'); do oc debug node/${i} -- chroot /host  ls -l /var/lib/rook; done

6. If encryption was enabled at the time of install, remove dm-crypt managed device-mapper mapping from OSD devices on all the OpenShift Container Storage nodes.

   1. Create a debug pod and chroot to the host on the storage node.

      $ oc debug node/<node name>
      $ chroot /host

   2. Get Device names and make note of the OpenShift Container Storage devices.

      $ dmsetup ls
      ocs-deviceset-0-data-0-57snx-block-dmcrypt (253:1)

   3. Remove the mapped device.

      $ cryptsetup luksClose --debug --verbose ocs-deviceset-0-data-0-57snx-block-dmcrypt

      If the above command gets stuck due to insufficient privileges, run the following commands:

      • Press CTRL+Z to exit the above command.

      • Find PID of the cryptsetup process which was stuck.

        $ ps

        Example output:

        PID     TTY    TIME     CMD
        778825   ?     00:00:00 cryptsetup

        Take a note of the PID number to kill. In this example, PID is 778825.

      • Terminate the process using kill command.

        $ kill -9 <PID>

      • Verify that the device name is removed.

        $ dmsetup ls

7. Delete the namespace and wait till the deletion is complete. You will need to switch to another project if openshift-storage is the active project.

   For example:

   $ oc project default
   $ oc delete project openshift-storage --wait=true --timeout=5m

   The project is deleted if the following command returns a NotFound error.

   $ oc get project openshift-storage

   Note

   While uninstalling OpenShift Container Storage, if namespace is not deleted completely and remains in Terminating state, perform the steps in Troubleshooting and deleting remaining resources during Uninstall to identify objects that are blocking the namespace from being terminated.

8. Unlabel the storage nodes.

   $ oc label nodes  --all cluster.ocs.openshift.io/openshift-storage-
   $ oc label nodes  --all topology.rook.io/rack-

9. Remove the OpenShift Container Storage taint if the nodes were tainted.

   $ oc adm taint nodes --all node.ocs.openshift.io/storage-

10. Confirm all PVs provisioned using OpenShift Container Storage are deleted. If there is any PV left in the Released state, delete it.

    $ oc get pv
    $ oc delete pv <pv name>

11. Delete the Multicloud Object Gateway storageclass.

    $ oc delete storageclass openshift-storage.noobaa.io --wait=true --timeout=5m

12. Remove CustomResourceDefinitions.

    $ oc delete crd backingstores.noobaa.io bucketclasses.noobaa.io cephblockpools.ceph.rook.io cephclusters.ceph.rook.io cephfilesystems.ceph.rook.io cephnfses.ceph.rook.io cephobjectstores.ceph.rook.io cephobjectstoreusers.ceph.rook.io noobaas.noobaa.io ocsinitializations.ocs.openshift.io storageclusters.ocs.openshift.io cephclients.ceph.rook.io cephobjectrealms.ceph.rook.io cephobjectzonegroups.ceph.rook.io cephobjectzones.ceph.rook.io cephrbdmirrors.ceph.rook.io --wait=true --timeout=5m

13. To ensure that OpenShift Container Storage is uninstalled completely, on the OpenShift Container Platform Web Console,

    1. Click Home → Overview to access the dashboard.
    2. Verify that the Persistent Storage and Object Service tabs no longer appear next to the Cluster tab.

1. Click Workloads → Pods from the left pane of the OpenShift Web Console.

2. Select openshift-storage from the Project drop down list.

   For more information on the expected number of pods for each component and how it varies depending on the number of nodes, see Table 2.1, “Pods corresponding to OpenShift Container Storage components”

3. Verify that the following pods are in running state:

   Table 2.1. Pods corresponding to OpenShift Container Storage components

   Component	Corresponding pods
   OpenShift Container Storage Operator	ocs-operator-* (1 pod on any worker node) ocs-metrics-exporter-*
   Rook-ceph Operator	rook-ceph-operator-* (1 pod on any worker node)
   Multicloud Object Gateway	noobaa-operator-* (1 pod on any worker node) noobaa-core-* (1 pod on any worker node) noobaa-db-* (1 pod on any worker node) noobaa-endpoint-* (1 pod on any worker node)
   CSI	cephfs csi-cephfsplugin-* (1 pod on each worker node) csi-cephfsplugin-provisioner-* (2 pods distributed across worker nodes) Note If an MDS is not deployed in the external cluster, the csi-cephfsplugin pods will not be created. rbd csi-rbdplugin-* (1 pod on each worker node) csi-rbdplugin-provisioner-* (2 pods distributed across worker nodes)

1. Delete the volume snapshots that are using OpenShift Container Storage.

   1. List the volume snapshots from all the namespaces

      $ oc get volumesnapshot --all-namespaces

   2. From the output of the previous command, identify and delete the volume snapshots that are using OpenShift Container Storage.

      $ oc delete volumesnapshot <VOLUME-SNAPSHOT-NAME> -n <NAMESPACE>

2. Delete PVCs and OBCs that are using OpenShift Container Storage.

   In the default uninstall mode (graceful), the uninstaller waits till all the PVCs and OBCs that use OpenShift Container Storage are deleted.

   If you wish to delete the Storage Cluster without deleting the PVCs beforehand, you may set the uninstall mode annotation to "forced" and skip this step. Doing so will result in orphan PVCs and OBCs in the system.

   1. Delete OpenShift Container Platform monitoring stack PVCs using OpenShift Container Storage.

      See Section 1.4.1.1, “Removing monitoring stack from OpenShift Container Storage”

   2. Delete OpenShift Container Platform Registry PVCs using OpenShift Container Storage.

      See Section 1.4.1.2, “Removing OpenShift Container Platform registry from OpenShift Container Storage”

   3. Delete OpenShift Container Platform logging PVCs using OpenShift Container Storage.

      See Section 1.4.1.3, “Removing the cluster logging operator from OpenShift Container Storage”

   4. Delete other PVCs and OBCs provisioned using OpenShift Container Storage.

      • Given below is a sample script to identify the PVCs and OBCs provisioned using OpenShift Container Storage. The script ignores the PVCs and OBCs that are used internally by Openshift Container Storage.

        #!/bin/bash
        
        RBD_PROVISIONER="openshift-storage.rbd.csi.ceph.com"
        CEPHFS_PROVISIONER="openshift-storage.cephfs.csi.ceph.com"
        NOOBAA_PROVISIONER="openshift-storage.noobaa.io/obc"
        RGW_PROVISIONER="openshift-storage.ceph.rook.io/bucket"
        
        NOOBAA_DB_PVC="noobaa-db"
        NOOBAA_BACKINGSTORE_PVC="noobaa-default-backing-store-noobaa-pvc"
        
        # Find all the OCS StorageClasses
        OCS_STORAGECLASSES=$(oc get storageclasses | grep -e "$RBD_PROVISIONER" -e "$CEPHFS_PROVISIONER" -e "$NOOBAA_PROVISIONER" -e "$RGW_PROVISIONER" | awk '{print $1}')
        
        # List PVCs in each of the StorageClasses
        for SC in $OCS_STORAGECLASSES
        do
                echo "======================================================================"
                echo "$SC StorageClass PVCs and OBCs"
                echo "======================================================================"
                oc get pvc  --all-namespaces --no-headers 2>/dev/null | grep $SC | grep -v -e "$NOOBAA_DB_PVC" -e "$NOOBAA_BACKINGSTORE_PVC"
                oc get obc  --all-namespaces --no-headers 2>/dev/null | grep $SC
                echo
        done

      • Delete the OBCs.

        $ oc delete obc <obc name> -n <project name>

      • Delete the PVCs.

        $ oc delete pvc <pvc name> -n <project-name>

        Ensure that you have removed any custom backing stores, bucket classes, etc., created in the cluster.

3. Delete the Storage Cluster object and wait for the removal of the associated resources.

   $ oc delete -n openshift-storage storagecluster --all --wait=true

4. Delete the namespace and wait until the deletion is complete. You will need to switch to another project if openshift-storage is the active project.

   For example:

   $ oc project default
   $ oc delete project openshift-storage --wait=true --timeout=5m

   The project is deleted if the following command returns a NotFound error.

   $ oc get project openshift-storage

   Note

   While uninstalling OpenShift Container Storage, if the namespace is not deleted completely and remains in Terminating state, perform the steps in Troubleshooting and deleting remaining resources during Uninstall to identify objects that are blocking the namespace from being terminated.

5. Confirm all PVs provisioned using OpenShift Container Storage are deleted. If there is any PV left in the Released state, delete it.

   $ oc get pv
   $ oc delete pv <pv name>

6. Delete the Multicloud Object Gateway storageclass.

   $ oc delete storageclass openshift-storage.noobaa.io --wait=true --timeout=5m

7. Remove CustomResourceDefinitions.

   $ oc delete crd backingstores.noobaa.io bucketclasses.noobaa.io cephblockpools.ceph.rook.io cephclusters.ceph.rook.io cephfilesystems.ceph.rook.io cephnfses.ceph.rook.io cephobjectstores.ceph.rook.io cephobjectstoreusers.ceph.rook.io noobaas.noobaa.io ocsinitializations.ocs.openshift.io storageclusters.ocs.openshift.io cephclients.ceph.rook.io cephobjectrealms.ceph.rook.io cephobjectzonegroups.ceph.rook.io cephobjectzones.ceph.rook.io cephrbdmirrors.ceph.rook.io --wait=true --timeout=5m

8. To ensure that OpenShift Container Storage is uninstalled completely, on the OpenShift Container Platform Web Console,

   1. Click Home → Overview to access the dashboard.
   2. Verify that the Persistent Storage and Object Service tabs no longer appear next to the Cluster tab.

Procedure

1. List the pods and PVCs that are currently running in the openshift-monitoring namespace.

   $ oc get pod,pvc -n openshift-monitoring
   NAME                           READY   STATUS    RESTARTS   AGE
   pod/alertmanager-main-0         3/3     Running   0          8d
   pod/alertmanager-main-1         3/3     Running   0          8d
   pod/alertmanager-main-2         3/3     Running   0          8d
   pod/cluster-monitoring-
   operator-84457656d-pkrxm        1/1     Running   0          8d
   pod/grafana-79ccf6689f-2ll28    2/2     Running   0          8d
   pod/kube-state-metrics-
   7d86fb966-rvd9w                 3/3     Running   0          8d
   pod/node-exporter-25894         2/2     Running   0          8d
   pod/node-exporter-4dsd7         2/2     Running   0          8d
   pod/node-exporter-6p4zc         2/2     Running   0          8d
   pod/node-exporter-jbjvg         2/2     Running   0          8d
   pod/node-exporter-jj4t5         2/2     Running   0          6d18h
   pod/node-exporter-k856s         2/2     Running   0          6d18h
   pod/node-exporter-rf8gn         2/2     Running   0          8d
   pod/node-exporter-rmb5m         2/2     Running   0          6d18h
   pod/node-exporter-zj7kx         2/2     Running   0          8d
   pod/openshift-state-metrics-
   59dbd4f654-4clng                3/3     Running   0          8d
   pod/prometheus-adapter-
   5df5865596-k8dzn                1/1     Running   0          7d23h
   pod/prometheus-adapter-
   5df5865596-n2gj9                1/1     Running   0          7d23h
   pod/prometheus-k8s-0            6/6     Running   1          8d
   pod/prometheus-k8s-1            6/6     Running   1          8d
   pod/prometheus-operator-
   55cfb858c9-c4zd9                1/1     Running   0          6d21h
   pod/telemeter-client-
   78fc8fc97d-2rgfp                3/3     Running   0          8d
   
   NAME                                                              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                  AGE
   persistentvolumeclaim/my-alertmanager-claim-alertmanager-main-0   Bound    pvc-0d519c4f-15a5-11ea-baa0-026d231574aa   40Gi       RWO            ocs-external-storagecluster-ceph-rbd   8d
   persistentvolumeclaim/my-alertmanager-claim-alertmanager-main-1   Bound    pvc-0d5a9825-15a5-11ea-baa0-026d231574aa   40Gi       RWO            ocs-external-storagecluster-ceph-rbd   8d
   persistentvolumeclaim/my-alertmanager-claim-alertmanager-main-2   Bound    pvc-0d6413dc-15a5-11ea-baa0-026d231574aa   40Gi       RWO            ocs-external-storagecluster-ceph-rbd   8d
   persistentvolumeclaim/my-prometheus-claim-prometheus-k8s-0        Bound    pvc-0b7c19b0-15a5-11ea-baa0-026d231574aa   40Gi       RWO            ocs-external-storagecluster-ceph-rbd   8d
   persistentvolumeclaim/my-prometheus-claim-prometheus-k8s-1        Bound    pvc-0b8aed3f-15a5-11ea-baa0-026d231574aa   40Gi       RWO            ocs-external-storagecluster-ceph-rbd   8d

2. Edit the monitoring configmap.

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

3. Remove any config sections that reference the OpenShift Container Storage storage classes as shown in the following example and save it.

   Before editing

   . . . apiVersion: v1 data: config.yaml: | alertmanagerMain: volumeClaimTemplate: metadata: name: my-alertmanager-claim spec: resources: requests: storage: 40Gi storageClassName: ocs-external-storagecluster-ceph-rbd prometheusK8s: volumeClaimTemplate: metadata: name: my-prometheus-claim spec: resources: requests: storage: 40Gi storageClassName: ocs-external-storagecluster-ceph-rbd kind: ConfigMap metadata: creationTimestamp: "2019-12-02T07:47:29Z" name: cluster-monitoring-config namespace: openshift-monitoring resourceVersion: "22110" selfLink: /api/v1/namespaces/openshift-monitoring/configmaps/cluster-monitoring-config uid: fd6d988b-14d7-11ea-84ff-066035b9efa8 . . .

   After editing

   . . . apiVersion: v1 data: config.yaml: | kind: ConfigMap metadata: creationTimestamp: "2019-11-21T13:07:05Z" name: cluster-monitoring-config namespace: openshift-monitoring resourceVersion: "404352" selfLink: /api/v1/namespaces/openshift-monitoring/configmaps/cluster-monitoring-config uid: d12c796a-0c5f-11ea-9832-063cd735b81c . . .

   In this example, alertmanagerMain and prometheusK8s monitoring components are using the OpenShift Container Storage PVCs.

4. List the pods consuming the PVC.

   In this example, the alertmanagerMain and prometheusK8s pods that were consuming the PVCs are in the Terminating state. You can delete the PVCs once these pods are no longer using OpenShift Container Storage PVC.

   $ oc get pod,pvc -n openshift-monitoring
   NAME                                               READY   STATUS      RESTARTS AGE
   pod/alertmanager-main-0                            3/3     Terminating   0      10h
   pod/alertmanager-main-1                            3/3     Terminating   0      10h
   pod/alertmanager-main-2                            3/3     Terminating   0      10h
   pod/cluster-monitoring-operator-84cd9df668-zhjfn   1/1     Running       0      18h
   pod/grafana-5db6fd97f8-pmtbf                       2/2     Running       0      10h
   pod/kube-state-metrics-895899678-z2r9q             3/3     Running       0      10h
   pod/node-exporter-4njxv                            2/2     Running       0      18h
   pod/node-exporter-b8ckz                            2/2     Running       0      11h
   pod/node-exporter-c2vp5                            2/2     Running       0      18h
   pod/node-exporter-cq65n                            2/2     Running       0      18h
   pod/node-exporter-f5sm7                            2/2     Running       0      11h
   pod/node-exporter-f852c                            2/2     Running       0      18h
   pod/node-exporter-l9zn7                            2/2     Running       0      11h
   pod/node-exporter-ngbs8                            2/2     Running       0      18h
   pod/node-exporter-rv4v9                            2/2     Running       0      18h
   pod/openshift-state-metrics-77d5f699d8-69q5x       3/3     Running       0      10h
   pod/prometheus-adapter-765465b56-4tbxx             1/1     Running       0      10h
   pod/prometheus-adapter-765465b56-s2qg2             1/1     Running       0      10h
   pod/prometheus-k8s-0                               6/6     Terminating   1      9m47s
   pod/prometheus-k8s-1                               6/6     Terminating   1      9m47s
   pod/prometheus-operator-cbfd89f9-ldnwc             1/1     Running       0      43m
   pod/telemeter-client-7b5ddb4489-2xfpz              3/3     Running       0      10h
   
   NAME                                                      STATUS  VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                  AGE
   persistentvolumeclaim/ocs-alertmanager-claim-alertmanager-main-0   Bound    pvc-2eb79797-1fed-11ea-93e1-0a88476a6a64   40Gi       RWO            ocs-external-storagecluster-ceph-rbd   19h
   persistentvolumeclaim/ocs-alertmanager-claim-alertmanager-main-1   Bound    pvc-2ebeee54-1fed-11ea-93e1-0a88476a6a64   40Gi       RWO            ocs-external-storagecluster-ceph-rbd   19h
   persistentvolumeclaim/ocs-alertmanager-claim-alertmanager-main-2   Bound    pvc-2ec6a9cf-1fed-11ea-93e1-0a88476a6a64   40Gi       RWO            ocs-external-storagecluster-ceph-rbd   19h
   persistentvolumeclaim/ocs-prometheus-claim-prometheus-k8s-0        Bound    pvc-3162a80c-1fed-11ea-93e1-0a88476a6a64   40Gi       RWO            ocs-external-storagecluster-ceph-rbd   19h
   persistentvolumeclaim/ocs-prometheus-claim-prometheus-k8s-1        Bound    pvc-316e99e2-1fed-11ea-93e1-0a88476a6a64   40Gi       RWO            ocs-external-storagecluster-ceph-rbd   19h

5. Delete relevant PVCs. Make sure you delete all the PVCs that are consuming the storage classes.

   $ oc delete -n openshift-monitoring pvc <pvc-name> --wait=true --timeout=5m

Procedure

1. Edit the configs.imageregistry.operator.openshift.io object and remove the content in the storage section.

   $ oc edit configs.imageregistry.operator.openshift.io

   Before editing

   . . . storage: pvc: claim: registry-cephfs-rwx-pvc . . .

   After editing

   . . . storage: emptyDir: {} . . .

   In this example, the PVC is called registry-cephfs-rwx-pvc, which is now safe to delete.

2. Delete the PVC.

   $ oc delete pvc <pvc-name> -n openshift-image-registry --wait=true --timeout=5m

Procedure

1. Remove the ClusterLogging instance in the namespace.

   $ oc delete clusterlogging instance -n openshift-logging --wait=true --timeout=5m

   The PVCs in the openshift-logging namespace are now safe to delete.

2. Delete PVCs.

   $ oc delete pvc <pvc-name> -n openshift-logging --wait=true --timeout=5m

Procedure

1. Click Administration → Custom Resource Definitions from the left pane of the OpenShift Web Console.
2. On the Custom Resource Definitions page, click ClusterLogging.
3. On the Custom Resource Definition Overview page, select View Instances from the Actions menu or click the Instances Tab.

4. On the Cluster Logging page, click Create Cluster Logging.

   You might have to refresh the page to load the data.

5. In the YAML, replace the storageClassName with the storageclass that uses the provisioner openshift-storage.rbd.csi.ceph.com. In the example given below the name of the storageclass is ocs-storagecluster-ceph-rbd:

   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
     namespace: "openshift-logging"
   spec:
     managementState: "Managed"
     logStore:
       type: "elasticsearch"
       elasticsearch:
         nodeCount: 3
         storage:
           storageClassName: ocs-storagecluster-ceph-rbd
           size: 200G # Change as per your requirement
         redundancyPolicy: "SingleRedundancy"
     visualization:
       type: "kibana"
       kibana:
         replicas: 1
     curation:
       type: "curator"
       curator:
         schedule: "30 3 * * *"
     collection:
       logs:
         type: "fluentd"
         fluentd: {}

   If you have tainted the OpenShift Container Storage nodes, you must add toleration to enable scheduling of the daemonset pods for logging.

   spec:
   [...]
     collection:
       logs:
         fluentd:
           tolerations:
           - effect: NoSchedule
             key: node.ocs.openshift.io/storage
             value: 'true'
         type: fluentd

6. Click Save.

Verification steps

1. Verify that the Persistent Volume Claims are bound to the elasticsearch pods.

   1. Go to Storage → Persistent Volume Claims.
   2. Set the Project dropdown to openshift-logging.

   3. Verify that Persistent Volume Claims are visible with a state of Bound, attached to elasticsearch-* pods.

      Figure 4.1. Cluster logging created and bound

      

2. Verify that the new cluster logging is being used.

   1. Click Workload → Pods.
   2. Set the Project to openshift-logging.
   3. Verify that the new elasticsearch-* pods appear with a state of Running.
   4. Click the new elasticsearch-* pod to view pod details.
   5. Scroll down to Volumes and verify that the elasticsearch volume has a Type that matches your new Persistent Volume Claim, for example, elasticsearch-elasticsearch-cdm-9r624biv-3.
   6. Click the Persistent Volume Claim name and verify the storage class name in the PersistenVolumeClaim Overview page.

1. Execute the following command and verify that the new node is present in the output:

   $ oc get nodes --show-labels | grep cluster.ocs.openshift.io/openshift-storage= |cut -d' ' -f1

2. Click Workloads → Pods, confirm that at least the following pods on the new node are in Running state:

   • csi-cephfsplugin-*
   • csi-rbdplugin-*

Procedure

1. Click Operators → Installed Operators from the left pane of the OpenShift Web Console to view the installed operators.
2. Click OpenShift Container Storage Operator.

3. On the OpenShift Container Storage Operator page, scroll right and click the Backing Store tab.

   Figure 8.1. OpenShift Container Storage Operator page with backing store tab

   

4. Click Create Backing Store.

   Figure 8.2. Create Backing Store page

   

5. On the Create New Backing Store page, perform the following:

   1. Enter a Backing Store Name.
   2. Select a Provider.
   3. Select a Region.
   4. Enter an Endpoint. This is optional.

   5. Select a Secret from drop down list, or create your own secret. Optionally, you can Switch to Credentials view which lets you fill in the required secrets.

      For more information on creating an OCP secret, see the section Creating the secret in the Openshift Container Platform documentation.

      Each backingstore requires a different secret. For more information on creating the secret for a particular backingstore, see the Section 8.3.2, “Adding storage resources for hybrid or Multicloud using the MCG command line interface” and follow the procedure for the addition of storage resources using a YAML.

      Note

      This menu is relevant for all providers except Google Cloud and local PVC.

   6. Enter Target bucket. The target bucket is a container storage that is hosted on the remote cloud service. It allows you to create a connection that tells MCG that it can use this bucket for the system.
6. Click Create Backing Store.

Verification steps

1. Click Operators → Installed Operators.
2. Click OpenShift Container Storage Operator.
3. Search for the new backing store or click Backing Store tab to view all the backing stores.

Procedure

1. From the Multicloud Object Gateway (MCG) command-line interface, run the following NooBaa command:

   noobaa backingstore create s3-compatible rgw-resource --access-key=<RGW ACCESS KEY> --secret-key=<RGW SECRET KEY> --target-bucket=<bucket-name> --endpoint=<RGW endpoint>

   1. To get the <RGW ACCESS KEY> and <RGW SECRET KEY>, run the following command using your RGW user secret name:

      oc get secret <RGW USER SECRET NAME> -o yaml

   2. Decode the access key ID and the access key from Base64 and keep them.
   3. Replace <RGW USER ACCESS KEY> and <RGW USER SECRET ACCESS KEY> with the appropriate, decoded data from the previous step.
   4. Replace <bucket-name> with an existing RGW bucket name. This argument tells Multicloud Object Gateway which bucket to use as a target bucket for its backing store, and subsequently, data storage and administration.

   5. To get the <RGW endpoint>, see Accessing the RADOS Object Gateway S3 endpoint.

      The output will be similar to the following:

      INFO[0001] ✅ Exists: NooBaa "noobaa"
      INFO[0002] ✅ Created: BackingStore "rgw-resource"
      INFO[0002] ✅ Created: Secret "backing-store-secret-rgw-resource"

1. Create a CephObjectStore user. This also creates a secret containing the RGW credentials:

   apiVersion: ceph.rook.io/v1
   kind: CephObjectStoreUser
   metadata:
     name: <RGW-Username>
     namespace: openshift-storage
   spec:
     store: ocs-storagecluster-cephobjectstore
     displayName: "<Display-name>"

   1. Replace <RGW-Username> and <Display-name> with a unique username and display name.

2. Apply the following YAML for an S3-Compatible backing store:

   apiVersion: noobaa.io/v1alpha1
   kind: BackingStore
   metadata:
     finalizers:
     - noobaa.io/finalizer
     labels:
       app: noobaa
     name: <backingstore-name>
     namespace: openshift-storage
   spec:
     s3Compatible:
       endpoint: <RGW endpoint>
       secret:
         name: <backingstore-secret-name>
         namespace: openshift-storage
       signatureVersion: v4
       targetBucket: <RGW-bucket-name>
     type: s3-compatible

   1. Replace <backingstore-secret-name> with the name of the secret that was created with CephObjectStore in the previous step.
   2. Replace <bucket-name> with an existing RGW bucket name. This argument tells Multicloud Object Gateway which bucket to use as a target bucket for its backing store, and subsequently, data storage and administration.
   3. To get the <RGW endpoint>, see Accessing the RADOS Object Gateway S3 endpoint.

Procedure

1. In your OpenShift Storage console, navigate to Overview → Object Service → select the Multicloud Object Gateway link:

   

2. Select the Resources tab in the left, highlighted below. From the list that populates, select Add Cloud Resource:

   

3. Select Add new connection:

   

4. Select the relevant native cloud provider or S3 compatible option and fill in the details:

   

5. Select the newly created connection and map it to the existing bucket:

   
6. Repeat these steps to create as many backing stores as needed.

Procedure

1. Click Operators → Installed Operators from the left pane of the OpenShift Web Console to view the installed operators.
2. Click OpenShift Container Storage Operator.

3. On the OpenShift Container Storage Operator page, scroll right and click the Bucket Class tab.

   Figure 8.3. OpenShift Container Storage Operator page with Bucket Class tab

   
4. Click Create Bucket Class.

5. On the Create new Bucket Class page, perform the following:

   1. Enter a Bucket Class Name and click Next.

      Figure 8.4. Create Bucket Class page

      

   2. In Placement Policy, select Tier 1 - Policy Type and click Next. You can choose either one of the options as per your requirements.

      • Spread allows spreading of the data across the chosen resources.
      • Mirror allows full duplication of the data across the chosen resources.

      • Click Add Tier to add another policy tier.

        Figure 8.5. Tier 1 - Policy Type selection page

        

   3. Select atleast one Backing Store resource from the available list if you have selected Tier 1 - Policy Type as Spread and click Next. Alternatively, you can also create a new backing store.

      Figure 8.6. Tier 1 - Backing Store selection page

      

1. Review and confirm Bucket Class settings.

   Figure 8.7. Bucket class settings review page

   
2. Click Create Bucket Class.

Verification steps

1. Click Operators → Installed Operators.
2. Click OpenShift Container Storage Operator.
3. Search for the new Bucket Class or click Bucket Class tab to view all the Bucket Classes.

Procedure

1. In the OpenShift Console, click Home → Overview and click the Object Service tab.
2. Click Multicloud Object Gateway and log in if prompted.
3. Click Accounts and select an account to add the connection to.
4. Click My Connections.

5. Click Add Connection.

   1. Enter a Connection Name.
   2. Your cloud provider is shown in the Service dropdown by default. Change the selection to use a different provider.
   3. Your cloud provider’s default endpoint is shown in the Endpoint field by default. Enter an alternative endpoint if required.
   4. Enter your Access Key for this cloud provider.
   5. Enter your Secret Key for this cloud provider.
   6. Click Save.

Procedure

1. In the OpenShift Console, click Home → Overview and click on the Object Service tab.
2. Click Multicloud Storage Gateway and log in if prompted.
3. Click Resources, and click the Namespace Resources tab.

4. Click Create Namespace Resource.

   1. In Target Connection, select the connection to be used for this namespace’s storage provider.

      If you need to add a new connection, click Add New Connection and enter your provider details; see Section 8.4.1, “Adding provider connections to the Multicloud Object Gateway” for more information.

   2. In Target Bucket, select the name of the bucket to use as a target.
   3. Enter a Resource Name for your namespace resource.
   4. Click Create.

Procedure

1. In the OpenShift Console, click Home → Overview and click the Object Service tab.
2. Click Multicloud Object Gateway and log in if prompted.
3. Click Buckets, and click on the Namespace Buckets tab.

4. Click Create Namespace Bucket.

   1. On the Choose Name tab, specify a Name for the namespace bucket and click Next.

   2. On the Set Placement tab:

      1. Under Read Policy, select the checkbox for each namespace resource that the namespace bucket should read data from.
      2. Under Write Policy, specify which namespace resource the namespace bucket should write data to.
      3. Click Next.
   3. Do not make changes on the Set Caching Policy tab in a production environment. This tab is provided as a Development Preview and is subject to support limitations.
   4. Click Create.

1. From the MCG command-line interface, run the following command to create a bucket class with a mirroring policy:

   $ noobaa bucketclass create mirror-to-aws --backingstores=azure-resource,aws-resource --placement Mirror

2. Set the newly created bucket class to a new bucket claim, generating a new bucket that will be mirrored between two locations:

   $ noobaa obc create  mirrored-bucket --bucketclass=mirror-to-aws

1. Apply the following YAML. This YAML is a hybrid example that mirrors data between local Ceph storage and AWS:

   apiVersion: noobaa.io/v1alpha1
   kind: BucketClass
   metadata:
     name: hybrid-class
     labels:
       app: noobaa
   spec:
     placementPolicy:
       tiers:
         - tier:
             mirrors:
               - mirror:
                   spread:
                     - cos-east-us
               - mirror:
                   spread:
                     - noobaa-test-bucket-for-ocp201907291921-11247_resource

2. Add the following lines to your standard Object Bucket Claim (OBC):

   additionalConfig:
     bucketclass: mirror-to-aws

   For more information about OBCs, see Section 8.7, “Object Bucket Claim”.

1. In your OpenShift Storage console, navigate to Overview → Object Service → select the Multicloud Object Gateway link:

   

2. Click the buckets icon on the left side. You will see a list of your buckets:

   
3. Click the bucket you want to update.

4. Click Edit Tier 1 Resources:

   

5. Select Mirror and check the relevant resources you want to use for this bucket. In the following example, we mirror data between on prem Ceph RGW to AWS:

   
6. Click Save.

1. Create the bucket policy in JSON format. See the following example:

   {
       "Version": "NewVersion",
       "Statement": [
           {
               "Sid": "Example",
               "Effect": "Allow",
               "Principal": [
                       "john.doe@example.com"
               ],
               "Action": [
                   "s3:GetObject"
               ],
               "Resource": [
                   "arn:aws:s3:::john_bucket"
               ]
           }
       ]
   }

   There are many available elements for bucket policies. For details on these elements and examples of how they can be used, see AWS Access Policy Language Overview.

   For more examples of bucket policies, see AWS Bucket Policy Examples.

   Instructions for creating S3 users can be found in Section 8.6.3, “Creating an AWS S3 user in the Multicloud Object Gateway”.

2. Using AWS S3 client, use the put-bucket-policy command to apply the bucket policy to your S3 bucket:

   # aws --endpoint ENDPOINT --no-verify-ssl s3api put-bucket-policy --bucket MyBucket --policy BucketPolicy

   Replace ENDPOINT with the S3 endpoint

   Replace MyBucket with the bucket to set the policy on

   Replace BucketPolicy with the bucket policy JSON file

   Add --no-verify-ssl if you are using the default self signed certificates

   For example:

   # aws --endpoint https://s3-openshift-storage.apps.gogo44.noobaa.org --no-verify-ssl s3api put-bucket-policy -bucket MyBucket --policy file://BucketPolicy

   For more information on the put-bucket-policy command, see the AWS CLI Command Reference for put-bucket-policy.

Procedure

1. In your OpenShift Storage console, navigate to Overview → Object Service → select the Multicloud Object Gateway link:

   

2. Under the Accounts tab, click Create Account:

   

3. Select S3 Access Only, provide the Account Name, for example, john.doe@example.com. Click Next:

   

4. Select S3 default placement, for example, noobaa-default-backing-store. Select Buckets Permissions. A specific bucket or all buckets can be selected. Click Create:

   

Procedure

1. Add the following lines to your application YAML:

   apiVersion: objectbucket.io/v1alpha1
   kind: ObjectBucketClaim
   metadata:
     name: <obc-name>
   spec:
     generateBucketName: <obc-bucket-name>
     storageClassName: openshift-storage.noobaa.io

   These lines are the Object Bucket Claim itself.

   1. Replace <obc-name> with the a unique Object Bucket Claim name.
   2. Replace <obc-bucket-name> with a unique bucket name for your Object Bucket Claim.

2. You can add more lines to the YAML file to automate the use of the Object Bucket Claim. The example below is the mapping between the bucket claim result, which is a configuration map with data and a secret with the credentials. This specific job will claim the Object Bucket from NooBaa, which will create a bucket and an account.

   apiVersion: batch/v1
   kind: Job
   metadata:
     name: testjob
   spec:
     template:
       spec:
         restartPolicy: OnFailure
         containers:
           - image: <your application image>
             name: test
             env:
               - name: BUCKET_NAME
                 valueFrom:
                   configMapKeyRef:
                     name: <obc-name>
                     key: BUCKET_NAME
               - name: BUCKET_HOST
                 valueFrom:
                   configMapKeyRef:
                     name: <obc-name>
                     key: BUCKET_HOST
               - name: BUCKET_PORT
                 valueFrom:
                   configMapKeyRef:
                     name: <obc-name>
                     key: BUCKET_PORT
               - name: AWS_ACCESS_KEY_ID
                 valueFrom:
                   secretKeyRef:
                     name: <obc-name>
                     key: AWS_ACCESS_KEY_ID
               - name: AWS_SECRET_ACCESS_KEY
                 valueFrom:
                   secretKeyRef:
                     name: <obc-name>
                     key: AWS_SECRET_ACCESS_KEY

   1. Replace all instances of <obc-name> with your Object Bucket Claim name.
   2. Replace <your application image> with your application image.

3. Apply the updated YAML file:

   # oc apply -f <yaml.file>

   1. Replace <yaml.file> with the name of your YAML file.

4. To view the new configuration map, run the following:

   # oc get cm <obc-name>

   1. Replace obc-name with the name of your Object Bucket Claim.

      You can expect the following environment variables in the output:

      • BUCKET_HOST - Endpoint to use in the application

      • BUCKET_PORT - The port available for the application

        • The port is related to the BUCKET_HOST. For example, if the BUCKET_HOST is https://my.example.com, and the BUCKET_PORT is 443, the endpoint for the object service would be https://my.example.com:443.
      • BUCKET_NAME - Requested or generated bucket name
      • AWS_ACCESS_KEY_ID - Access key that is part of the credentials
      • AWS_SECRET_ACCESS_KEY - Secret access key that is part of the credentials

Procedure

1. Use the command-line interface to generate the details of a new bucket and credentials. Run the following command:

   # noobaa obc create <obc-name> -n openshift-storage

   Replace <obc-name> with a unique Object Bucket Claim name, for example, myappobc.

   Additionally, you can use the --app-namespace option to specify the namespace where the Object Bucket Claim configuration map and secret will be created, for example, myapp-namespace.

   Example output:

   INFO[0001] ✅ Created: ObjectBucketClaim "test21obc"

   The MCG command-line-interface has created the necessary configuration and has informed OpenShift about the new OBC.

2. Run the following command to view the Object Bucket Claim:

   # oc get obc -n openshift-storage

   Example output:

   NAME        STORAGE-CLASS                 PHASE   AGE
   test21obc   openshift-storage.noobaa.io   Bound   38s

3. Run the following command to view the YAML file for the new Object Bucket Claim:

   # oc get obc test21obc -o yaml -n openshift-storage

   Example output:

   apiVersion: objectbucket.io/v1alpha1
   kind: ObjectBucketClaim
   metadata:
     creationTimestamp: "2019-10-24T13:30:07Z"
     finalizers:
     - objectbucket.io/finalizer
     generation: 2
     labels:
       app: noobaa
       bucket-provisioner: openshift-storage.noobaa.io-obc
       noobaa-domain: openshift-storage.noobaa.io
     name: test21obc
     namespace: openshift-storage
     resourceVersion: "40756"
     selfLink: /apis/objectbucket.io/v1alpha1/namespaces/openshift-storage/objectbucketclaims/test21obc
     uid: 64f04cba-f662-11e9-bc3c-0295250841af
   spec:
     ObjectBucketName: obc-openshift-storage-test21obc
     bucketName: test21obc-933348a6-e267-4f82-82f1-e59bf4fe3bb4
     generateBucketName: test21obc
     storageClassName: openshift-storage.noobaa.io
   status:
     phase: Bound

4. Inside of your openshift-storage namespace, you can find the configuration map and the secret to use this Object Bucket Claim. The CM and the secret have the same name as the Object Bucket Claim. To view the secret:

   # oc get -n openshift-storage secret test21obc -o yaml

   Example output:

   Example output:
   apiVersion: v1
   data:
     AWS_ACCESS_KEY_ID: c0M0R2xVanF3ODR3bHBkVW94cmY=
     AWS_SECRET_ACCESS_KEY: Wi9kcFluSWxHRzlWaFlzNk1hc0xma2JXcjM1MVhqa051SlBleXpmOQ==
   kind: Secret
   metadata:
     creationTimestamp: "2019-10-24T13:30:07Z"
     finalizers:
     - objectbucket.io/finalizer
     labels:
       app: noobaa
       bucket-provisioner: openshift-storage.noobaa.io-obc
       noobaa-domain: openshift-storage.noobaa.io
     name: test21obc
     namespace: openshift-storage
     ownerReferences:
     - apiVersion: objectbucket.io/v1alpha1
       blockOwnerDeletion: true
       controller: true
       kind: ObjectBucketClaim
       name: test21obc
       uid: 64f04cba-f662-11e9-bc3c-0295250841af
     resourceVersion: "40751"
     selfLink: /api/v1/namespaces/openshift-storage/secrets/test21obc
     uid: 65117c1c-f662-11e9-9094-0a5305de57bb
   type: Opaque

   The secret gives you the S3 access credentials.

5. To view the configuration map:

   # oc get -n openshift-storage cm test21obc -o yaml

   Example output:

   apiVersion: v1
   data:
     BUCKET_HOST: 10.0.171.35
     BUCKET_NAME: test21obc-933348a6-e267-4f82-82f1-e59bf4fe3bb4
     BUCKET_PORT: "31242"
     BUCKET_REGION: ""
     BUCKET_SUBREGION: ""
   kind: ConfigMap
   metadata:
     creationTimestamp: "2019-10-24T13:30:07Z"
     finalizers:
     - objectbucket.io/finalizer
     labels:
       app: noobaa
       bucket-provisioner: openshift-storage.noobaa.io-obc
       noobaa-domain: openshift-storage.noobaa.io
     name: test21obc
     namespace: openshift-storage
     ownerReferences:
     - apiVersion: objectbucket.io/v1alpha1
       blockOwnerDeletion: true
       controller: true
       kind: ObjectBucketClaim
       name: test21obc
       uid: 64f04cba-f662-11e9-bc3c-0295250841af
     resourceVersion: "40752"
     selfLink: /api/v1/namespaces/openshift-storage/configmaps/test21obc
     uid: 651c6501-f662-11e9-9094-0a5305de57bb

   The configuration map contains the S3 endpoint information for your application.

Procedure

1. Log into the OpenShift Web Console.
2. On the left navigation bar, click Storage → Object Bucket Claims.

3. Click Create Object Bucket Claim:

   

4. Enter a name for your object bucket claim and select the appropriate storage class based on your deployment, internal or external, from the dropdown menu:

   Internal mode

   

   The following storage classes, which were created after deployment, are available for use:

   • ocs-storagecluster-ceph-rgw uses the Ceph Object Gateway (RGW)
   • openshift-storage.noobaa.io uses the Multicloud Object Gateway

   External mode

   

   The following storage classes, which were created after deployment, are available for use:

   • ocs-external-storagecluster-ceph-rgw uses the Ceph Object Gateway (RGW)

   • openshift-storage.noobaa.io uses the Multicloud Object Gateway

     Note

     The RGW OBC storage class is only available with fresh installations of OpenShift Container Storage version 4.5. It does not apply to clusters upgraded from previous OpenShift Container Storage releases.

5. Click Create.

   Once you create the OBC, you are redirected to its detail page:

   

Procedure

1. In the Multicloud Object Gateway user interface, from the Overview page, click Add Storage Resources:

   

2. In the window, click Deploy Kubernetes Pool:

   

3. In the Create Pool step create the target pool for the future installed nodes.

   

4. In the Configure step, configure the number of requested pods and the size of each PV. For each new pod, one PV is be created.

   
5. In the Review step, you can find the details of the new pool and select the deployment method you wish to use: local or external deployment. If local deployment is selected, the Kubernetes nodes will deploy within the cluster. If external deployment is selected, you will be provided with a YAML file to run externally.

6. All nodes will be assigned to the pool you chose in the first step, and can be found under Resources → Storage resources → Resource name:

   

Procedure

1. Log in to the OpenShift Container Platform cluster using the cluster-admin role.

2. Locate your auth.json file.

   This file is generated when you use podman or docker to log in to a registry. It is located in one of the following locations:

   • ~/.docker/auth.json
   • /run/user/<UID>/containers/auth.json
   • /var/run/containers/<UID>/auth.json

3. Obtain your unique Red Hat registry pull secret and paste it into your auth.json file. It will look something like this.

   {
       "auths": {
           "cloud.openshift.com": {
               "auth": "*****************",
               "email": "user@example.com"
           },
           "quay.io": {
               "auth": "*****************",
               "email": "user@example.com"
           },
           "registry.connect.redhat.com": {
               "auth": "*****************",
               "email": "user@example.com"
           },
           "registry.redhat.io": {
               "auth": "*****************",
               "email": "user@example.com"
           }
       }
     }

4. Export environment variables with the appropriate details for your setup.

   $ export AUTH_FILE="<location_of_auth.json>"
   $ export MIRROR_REGISTRY_DNS="<your_registry_url>:<port>"

5. Use podman to log in to the mirror registry and store the credentials in the ${AUTH_FILE}.

   $ podman login ${MIRROR_REGISTRY_DNS} --tls-verify=false --authfile ${AUTH_FILE}

   This adds the mirror registry to the auth.json file.

   {
       "auths": {
           "cloud.openshift.com": {
               "auth": "*****************",
               "email": "user@example.com"
           },
           "quay.io": {
               "auth": "*****************",
               "email": "user@example.com"
           },
           "registry.connect.redhat.com": {
               "auth": "*****************",
               "email": "user@example.com"
           },
           "registry.redhat.io": {
               "auth": "*****************",
               "email": "user@example.com"
           },
           "<mirror_registry>": {
               "auth": "*****************",
           }
       }
     }

Procedure

1. Build the catalog for redhat-operators.

   Set --from to the ose-operator-registry base image using the tag that matches the target OpenShift Container Platform cluster major and minor version.

   $ oc adm catalog build --appregistry-org redhat-operators \
     --from=registry.redhat.io/openshift4/ose-operator-registry:v4.6 \
     --to=${MIRROR_REGISTRY_DNS}/olm/redhat-operators:v2 \
     --registry-config=${AUTH_FILE} \
     --filter-by-os="linux/amd64" --insecure

2. Mirror the catalog for redhat-operators.

   This is a long operation and can take 1-5 hours. Make sure there is 100 GB available disk space on the mirror host.

   $ oc adm catalog mirror ${MIRROR_REGISTRY_DNS}/olm/redhat-operators:v2 \
   ${MIRROR_REGISTRY_DNS} --registry-config=${AUTH_FILE} --insecure

Procedure

1. Check the content of this file for the mirrors mapping shown as follows:

   spec:
     repositoryDigestMirrors:
       - mirrors:
         - <your_registry>/ocs4
         source: registry.redhat.io/ocs4
       - mirrors:
         - <your_registry>/rhceph
         source: registry.redhat.io/rhceph
       - mirrors:
         - <your_registry>/openshift4
         source: registry.redhat.io/openshift4
       - mirrors:
         - <your_registry>/rhscl
         source: registry.redhat.io/rhscl

2. Add any missing entries to the end of the imageContentSourcePolicy.yaml file.

3. Apply the imageContentSourcePolicy.yaml file to the cluster.

   $ oc apply -f ./[output dir]/imageContentSourcePolicy.yaml

   Once the Image Content Source Policy is updated, all the nodes (master, infra, and workers) in the cluster need to be updated and rebooted. This process is automatically handled through the Machine Config Pool operator and take up to 30 minutes although the exact elapsed time might vary based on the number of nodes in your OpenShift cluster. You can monitor the update process by using the oc get mcp command or the oc get node command.

Procedure

1. Recreate a CatalogSource object that references the catalog image for Red Hat operators.

   Note

   Make sure you have mirrored the correct catalog source with the correct version (that is, v2).

   Save the following in a redhat-operator-catalogsource.yaml file, remembering to replace <your_registry> with your mirror registry URL:

   apiVersion: operators.coreos.com/v1alpha1
   kind: CatalogSource
   metadata:
     name: redhat-operators
     namespace: openshift-marketplace
   spec:
     sourceType: grpc
     icon:
       base64data: PHN2ZyBpZD0iTGF5ZXJfMSIgZGF0YS1uYW1lPSJMYXllciAxIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHZpZXdCb3g9IjAgMCAxOTIgMTQ1Ij48ZGVmcz48c3R5bGU+LmNscy0xe2ZpbGw6I2UwMDt9PC9zdHlsZT48L2RlZnM+PHRpdGxlPlJlZEhhdC1Mb2dvLUhhdC1Db2xvcjwvdGl0bGU+PHBhdGggZD0iTTE1Ny43Nyw2Mi42MWExNCwxNCwwLDAsMSwuMzEsMy40MmMwLDE0Ljg4LTE4LjEsMTcuNDYtMzAuNjEsMTcuNDZDNzguODMsODMuNDksNDIuNTMsNTMuMjYsNDIuNTMsNDRhNi40Myw2LjQzLDAsMCwxLC4yMi0xLjk0bC0zLjY2LDkuMDZhMTguNDUsMTguNDUsMCwwLDAtMS41MSw3LjMzYzAsMTguMTEsNDEsNDUuNDgsODcuNzQsNDUuNDgsMjAuNjksMCwzNi40My03Ljc2LDM2LjQzLTIxLjc3LDAtMS4wOCwwLTEuOTQtMS43My0xMC4xM1oiLz48cGF0aCBjbGFzcz0iY2xzLTEiIGQ9Ik0xMjcuNDcsODMuNDljMTIuNTEsMCwzMC42MS0yLjU4LDMwLjYxLTE3LjQ2YTE0LDE0LDAsMCwwLS4zMS0zLjQybC03LjQ1LTMyLjM2Yy0xLjcyLTcuMTItMy4yMy0xMC4zNS0xNS43My0xNi42QzEyNC44OSw4LjY5LDEwMy43Ni41LDk3LjUxLjUsOTEuNjkuNSw5MCw4LDgzLjA2LDhjLTYuNjgsMC0xMS42NC01LjYtMTcuODktNS42LTYsMC05LjkxLDQuMDktMTIuOTMsMTIuNSwwLDAtOC40MSwyMy43Mi05LjQ5LDI3LjE2QTYuNDMsNi40MywwLDAsMCw0Mi41Myw0NGMwLDkuMjIsMzYuMywzOS40NSw4NC45NCwzOS40NU0xNjAsNzIuMDdjMS43Myw4LjE5LDEuNzMsOS4wNSwxLjczLDEwLjEzLDAsMTQtMTUuNzQsMjEuNzctMzYuNDMsMjEuNzdDNzguNTQsMTA0LDM3LjU4LDc2LjYsMzcuNTgsNTguNDlhMTguNDUsMTguNDUsMCwwLDEsMS41MS03LjMzQzIyLjI3LDUyLC41LDU1LC41LDc0LjIyYzAsMzEuNDgsNzQuNTksNzAuMjgsMTMzLjY1LDcwLjI4LDQ1LjI4LDAsNTYuNy0yMC40OCw1Ni43LTM2LjY1LDAtMTIuNzItMTEtMjcuMTYtMzAuODMtMzUuNzgiLz48L3N2Zz4=
       mediatype: image/svg+xml
     image: <your_registry>/olm/redhat-operators:v2
     displayName: Redhat Operators Catalog
     publisher: Red Hat

2. Create a catalogsource using the redhat-operator-catalogsource.yaml file:

   $ oc apply -f redhat-operator-catalogsource.yaml

3. Verify that the new redhat-operator pod is running.

   $ oc get pod -n openshift-marketplace | grep redhat-operators

Procedure

1. Log in to OpenShift Web Console.
2. Click Operators → Installed Operators
3. Select the openshift-storage project.
4. Click the OpenShift Container Storage operator name.
5. Click the Subscription tab and click the link under Approval.
6. Select Automatic (default) and click Save.

7. Perform one of the following depending on the Upgrade Status:

   • Upgrade Status shows requires approval.

     Note

     Upgrade status shows requires approval if the new OpenShift Container Storage version is already detected in the channel, and approval strategy was changed from Manual to Automatic at the time of update.

     1. Click on the Install Plan link.
     2. On the InstallPlan Details page, click Preview Install Plan.
     3. Review the install plan and click Approve.
     4. Wait for the Status to change from Unknown to Created.
     5. Click Operators → Installed Operators
     6. Select the openshift-storage project.
     7. Wait for the Status to change to Up to date

   • Upgrade Status does not show requires approval:

     1. Wait for the update to initiate. This may take up to 20 minutes.
     2. Click Operators → Installed Operators
     3. Select the openshift-storage project.
     4. Wait for the Status to change to Up to date

Verification steps

1. Click Overview → Persistent Storage tab and in the Status card confirm that the OCS Cluster and Data Resiliency has a green tick mark indicating it is healthy.
2. Click Overview → Object Service tab and in the Status card confirm that both the Object Service and Data Resiliency are in Ready state (green tick) indicating it is healthy.

3. Click Operators → Installed Operators → OpenShift Container Storage Operator. Under Storage Cluster, verify that the cluster service status is Ready.

   Note

   Once updated from OpenShift Container Storage version 4.5 to 4.6, the Version field here will still display 4.5. This is because the ocs-operator does not update the string represented in this field.

4. Ensure that all OpenShift Container Storage Pods, including the operator pods, are in Running state in the openshift-storage namespace.

   To view the state of the pods, click Workloads → Pods. Select openshift-storage from the Project drop down list.

5. If verification steps fail, contact Red Hat Support.

Procedure

1. Log in to OpenShift Web Console.
2. Click Operators → Installed Operators
3. Select the openshift-storage project.
4. Click the OpenShift Container Storage operator name.
5. Click the Subscription tab and click the link under Approval.
6. Select Manual and click Save.
7. Wait for the Upgrade Status to change to Upgrading.
8. If the Upgrade Status shows requires approval, click on requires approval.
9. On the InstallPlan Details page, click Preview Install Plan.
10. Review the install plan and click Approve.
11. Wait for the Status to change from Unknown to Created.
12. Click Operators → Installed Operators
13. Select the openshift-storage project.
14. Wait for the Status to change to Up to date

Verification steps

1. Click Overview → Persistent Storage tab and in the Status card confirm that the OCS Cluster and Data Resiliency has a green tick mark indicating it is healthy.
2. Click Overview → Object Service tab and in the Status card confirm that both the Object Service and Data Resiliency are in Ready state (green tick) indicating it is healthy.

3. Click Operators → Installed Operators → OpenShift Container Storage Operator. Under Storage Cluster, verify that the cluster service status is Ready.

   Note

   Once updated from OpenShift Container Storage version 4.5 to 4.6, the Version field here will still display 4.5. This is because the ocs-operator does not update the string represented in this field.

4. Ensure that all OpenShift Container Storage Pods, including the operator pods, are in Running state in the openshift-storage namespace.

   To view the state of the pods, click Workloads → Pods from the left pane of the OpenShift Web Console. Select openshift-storage from the Project drop down list.

5. If verification steps fail, contact Red Hat Support.