Managing OpenShift Container Storage

Procedure

1. Click Administration → Custom Resource Definitions from the left pane of the OpenShift Web Console.
2. On the Custom Resource Definitions page, click ClusterLogging.
3. On the Custom Resource Definition Overview page, select View Instances from the Actions menu or click the Instances Tab.

4. On the Cluster Logging page, click Create Cluster Logging.

   You might have to refresh the page to load the data.

5. In the YAML, replace the storageClassName with the storageclass that uses the provisioner openshift-storage.rbd.csi.ceph.com. In the example given below the name of the storageclass is ocs-storagecluster-ceph-rbd:

   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
     namespace: "openshift-logging"
   spec:
     managementState: "Managed"
     logStore:
       type: "elasticsearch"
       elasticsearch:
         nodeCount: 3
         storage:
           storageClassName: ocs-storagecluster-ceph-rbd
           size: 200G
         redundancyPolicy: "SingleRedundancy"
     visualization:
       type: "kibana"
       kibana:
         replicas: 1
     curation:
       type: "curator"
       curator:
         schedule: "30 3 * * *"
     collection:
       logs:
         type: "fluentd"
         fluentd: {}

   Copy to Clipboard Toggle word wrap
6. Click Save.

Verification steps

1. Verify that the Persistent Volume Claims are bound to the elasticsearch pods.

   1. Go to Storage → Persistent Volume Claims.
   2. Set the Project dropdown to openshift-logging.

   3. Verify that Persistent Volume Claims are visible with a state of Bound, attached to elasticsearch-* pods.

      Figure 2.1. Cluster logging created and bound

      

      View larger image

      

2. Verify that the new cluster logging is being used.

   1. Click Workload → Pods.
   2. Set the Project to openshift-logging.
   3. Verify that the new elasticsearch-* pods appear with a state of Running.
   4. Click the new elasticsearch-* pod to view pod details.
   5. Scroll down to Volumes and verify that the elasticsearch volume has a Type that matches your new Persistent Volume Claim, for example, elasticsearch-elasticsearch-cdm-9r624biv-3.
   6. Click the Persistent Volume Claim name and verify the storage class name in the PersistenVolumeClaim Overview page.

Procedure

1. Navigate to the OpenShift Web Console.
2. Click on Operators on the left navigation bar.
3. Select Installed Operators.

4. In the window, click OpenShift Container Storage Operator:

   

   View larger image

   

5. In the top navigation bar, scroll right and click Storage Cluster tab.

   

   View larger image

   
6. The visible list should have only one item. Click (⋮) on the far right to extend the options menu.

7. Select Add Capacity from the options menu.

   

   View larger image

   

   From this dialog box, you can set the requested additional capacity and the storage class. Add capacity will show the capacity selected at the time of installation and will allow to add the capacity only in this increment. On AWS, the storage class should be set to gp2. On VMware, the storage class should be set to thin.

   Note

   The effectively provisioned capacity will be three times as much as what you see in the Raw Capacity field because OpenShift Container Storage uses a replica count of 3.

8. Once you are done with your setting, click Add. You might need to wait a couple of minutes for the storage cluster to reach Ready state.

Verification steps

1. Navigate to Overview → Persistent Storage tab, then check the Capacity breakdown card.

   

   View larger image

   
2. Note that the capacity increases based on your selections.

Procedure

1. To add storage capacity to OpenShift Container Platform nodes with OpenShift Container Storage installed, you need to

   1. Find the unique by-id identifier for available devices that you want to add, that is, a minimum of one device per worker node. You can follow the procedure for finding available storage devices in the respective deployment guide.

      Note

      Make sure you perform this process for all the existing nodes (minimum of 3) for which you want to add storage.

   2. Add the unique device ID to the LocalVolume custom resource (CR).

      $ oc edit -n local-storage localvolume local-block

      Copy to Clipboard Toggle word wrap

      Example output:

      spec:
        logLevel: Normal
        managementState: Managed
        nodeSelector:
          nodeSelectorTerms:
          - matchExpressions:
            - key: cluster.ocs.openshift.io/openshift-storage
              operator: In
              values:
              - ""
        storageClassDevices:
        - devicePaths:
          - /dev/disk/by-id/nvme-INTEL_SSDPE2KX010T7_PHLF733402P51P0GGN
          - /dev/disk/by-id/nvme-INTEL_SSDPE2KX010T7_PHLF733402LM1P0GGN
          - /dev/disk/by-id/nvme-INTEL_SSDPE2KX010T7_PHLF733402M21P0GGN
          - /dev/disk/by-id/nvme-INTEL_SSDPE2KX010T7_PHLF733402B71P0GGN    # newly  added device by-id
          - /dev/disk/by-id/nvme-INTEL_SSDPE2KX010T7_PHLF733402A31P0GGN    # newly  added device by-id
          - /dev/disk/by-id/nvme-INTEL_SSDPE2KX010T7_PHLF733402Q71P0GGN    # newly  added device by-id
          storageClassName: localblock
          volumeMode: Block

      Copy to Clipboard Toggle word wrap

      Make sure to save the changes after editing the CR.

      Example output:

      localvolume.local.storage.openshift.io/local-block edited

      Copy to Clipboard Toggle word wrap

      You can see in this CR that new devices using by-id have been added. Each new device maps to one NVMe device on the three worker nodes.

      • nvme-INTEL_SSDPE2KX010T7_PHLF733402B71P0GGN
      • nvme-INTEL_SSDPE2KX010T7_PHLF733402A31P0GGN
      • nvme-INTEL_SSDPE2KX010T7_PHLF733402Q71P0GGN

2. Display the newly created PVs with storageclass name used in localVolume CR.

   $ oc get pv | grep localblock | grep Available

   Copy to Clipboard Toggle word wrap

   Example output:

   local-pv-5ee61dcc   931Gi   RWO     Delete  Available   localblock      2m35s
   local-pv-b1fa607a   931Gi   RWO     Delete  Available   localblock      2m27s
   local-pv-e971c51d   931Gi   RWO     Delete  Available   localblock      2m22s
   ...

   Copy to Clipboard Toggle word wrap

   There are three more available PVs of same size which will be used for new OSDs.

3. Navigate to the OpenShift Web Console.
4. Click on Operators on the left navigation bar.
5. Select Installed Operators.

6. In the window, click OpenShift Container Storage Operator:

   

   View larger image

   

7. In the top navigation bar, scroll right and click Storage Cluster tab.

   

   View larger image

   
8. The visible list should have only one item. Click (⋮) on the far right to extend the options menu.

9. Select Add Capacity from the options menu.

   

   View larger image

   

   From this dialog box, set the Storage Class name to the name used in the localVolume CR. Available Capacity displayed is based on the local disks available in storage class.

10. Once you are done with your setting, click Add. You might need to wait a couple of minutes for the storage cluster to reach Ready state.

11. Verify that the new OSDs and their corresponding new PVCs are created.

    $ oc get -n openshift-storage pods -l app=rook-ceph-osd

    Copy to Clipboard Toggle word wrap

    Example output:

    NAME                               READY   STATUS    RESTARTS   AGE
    rook-ceph-osd-0-77c4fdb758-qshw4   1/1     Running   0          1h
    rook-ceph-osd-1-8645c5fbb6-656ks   1/1     Running   0          1h
    rook-ceph-osd-2-86895b854f-r4gt6   1/1     Running   0          1h
    rook-ceph-osd-3-dc7f787dd-gdnsz    1/1     Running   0          10m
    rook-ceph-osd-4-554b5c46dd-hbf9t   1/1     Running   0          10m
    rook-ceph-osd-5-5cf94c4448-k94j6   1/1     Running   0          10m

    Copy to Clipboard Toggle word wrap

    In the above example, osd-3, osd-4, and osd-5 are the newly added pods to the OpenShift Container Storage cluster.

    $ oc get pvc -n openshift-storage |grep localblock

    Copy to Clipboard Toggle word wrap

    Example output:

    ocs-deviceset-0-0-qc29m   Bound    local-pv-fc5562d3    931Gi   RWO  localblock  1h
    ocs-deviceset-0-1-qdmrl   Bound    local-pv-b1fa607a    931Gi   RWO  localblock  10m
    ocs-deviceset-1-0-mpwmk   Bound    local-pv-58cdd0bc    931Gi   RWO  localblock  1h
    ocs-deviceset-1-1-85892   Bound    local-pv-e971c51d    931Gi   RWO  localblock  10m
    ocs-deviceset-2-0-rll47   Bound    local-pv-29d8ad8d    931Gi   RWO  localblock  1h
    ocs-deviceset-2-1-cgth2   Bound    local-pv-5ee61dcc    931Gi   RWO  localblock  10m

    Copy to Clipboard Toggle word wrap

    In the above example, we see three new PVCs are created.

Verification steps

1. Navigate to Overview → Persistent Storage tab, then check the Capacity breakdown card.

   

   View larger image

   

   Note that the capacity increases based on your selections.

   Important

   OpenShift Container Storage does not support cluster reduction either by reducing OSDs or reducing nodes.

1. Execute the following command and verify that the new node is present in the output:

   $ oc get nodes --show-labels | grep cluster.ocs.openshift.io/openshift-storage= |cut -d' ' -f1

   Copy to Clipboard Toggle word wrap

2. Click Workloads → Pods, confirm that at least the following pods on the new node are in Running state:

   • csi-cephfsplugin-*
   • csi-rbdplugin-*

Procedure

1. Click Operators → Installed Operators from the left pane of the OpenShift Web Console to view the installed operators.
2. Click OpenShift Container Storage Operator.

3. On the OpenShift Container Storage Operator page, scroll right and click the Backing Store tab.

   Figure 7.1. OpenShift Container Storage Operator page with backing store tab

   

   View larger image

   

4. Click Create Backing Store.

   Figure 7.2. Create Backing Store page

   

   View larger image

   

5. On the Create New Backing Store page, perform the following:

   1. Enter a Backing Store Name.
   2. Select a Provider.
   3. Select a Region.
   4. Enter an Endpoint. This is optional.

   5. Select a Secret from drop down list, or create your own secret. Optionally, you can Switch to Credentials view which lets you fill in the required secrets.

      For more information on creating an OCP secret, see the section Creating the secret in the Openshift Container Platform documentation.

      Each backingstore requires a different secret. For more information on creating the secret for a particular backingstore, see the Section 7.4.2, “Adding storage resources for hybrid or Multicloud using the MCG command line interface” and follow the procedure for the addition of storage resources using a YAML.

      Note

      This menu is relevant for all providers except Google Cloud and local PVC.

   6. Enter Target bucket. The target bucket is a container storage that is hosted on the remote cloud service. It allows you to create a connection that tells MCG that it can use this bucket for the system.
6. Click Create Backing Store.

Verification steps

1. Click Operators → Installed Operators.
2. Click OpenShift Container Storage Operator.
3. Search for the new backing store or click Backing Store tab to view all the backing stores.

Procedure

1. From the Multicloud Object Gateway (MCG) command-line interface, run the following NooBaa command:

   noobaa backingstore create s3-compatible rgw-resource --access-key=<RGW ACCESS KEY> --secret-key=<RGW SECRET KEY> --target-bucket=<bucket-name> --endpoint=http://rook-ceph-rgw-ocs-storagecluster-cephobjectstore.openshift-storage.svc.cluster.local:80

   Copy to Clipboard Toggle word wrap

   1. To get the <RGW ACCESS KEY> and <RGW SECRET KEY>, run the following command using your RGW user secret name:

      oc get secret <RGW USER SECRET NAME> -o yaml

      Copy to Clipboard Toggle word wrap
   2. Decode the access key ID and the access key from Base64 and keep them.
   3. Replace <RGW USER ACCESS KEY> and <RGW USER SECRET ACCESS KEY> with the appropriate, decoded data from the previous step.

   4. Replace <bucket-name> with an existing RGW bucket name. This argument tells Multicloud Object Gateway which bucket to use as a target bucket for its backing store, and subsequently, data storage and administration.

      The output will be similar to the following:

      INFO[0001] ✅ Exists: NooBaa "noobaa"
      INFO[0002] ✅ Created: BackingStore "rgw-resource"
      INFO[0002] ✅ Created: Secret "backing-store-secret-rgw-resource"

      Copy to Clipboard Toggle word wrap

1. Create a CephObjectStore user. This also creates a secret containing the RGW credentials:

   apiVersion: ceph.rook.io/v1
   kind: CephObjectStoreUser
   metadata:
     name: <RGW-Username>
     namespace: openshift-storage
   spec:
     store: ocs-storagecluster-cephobjectstore
     displayName: "<Display-name>"

   Copy to Clipboard Toggle word wrap
   1. Replace <RGW-Username> and <Display-name> with a unique username and display name.

2. Apply the following YAML for an S3-Compatible backing store:

   apiVersion: noobaa.io/v1alpha1
   kind: BackingStore
   metadata:
     finalizers:
     - noobaa.io/finalizer
     labels:
       app: noobaa
     name: <backingstore-name>
     namespace: openshift-storage
   spec:
     s3Compatible:
       endpoint: http://rook-ceph-rgw-ocs-storagecluster-cephobjectstore.openshift-storage.svc.cluster.local:80
       secret:
         name: <backingstore-secret-name>
         namespace: openshift-storage
       signatureVersion: v4
       targetBucket: <RGW-bucket-name>
     type: s3-compatible

   Copy to Clipboard Toggle word wrap
   1. Replace <backingstore-secret-name> with the name of the secret that was created with CephObjectStore in the previous step.
   2. Replace <bucket-name> with an existing RGW bucket name. This argument tells Multicloud Object Gateway which bucket to use as a target bucket for its backing store, and subsequently, data storage and administration.

Procedure

1. In your OpenShift Storage console, navigate to Overview → Object Service → select the noobaa link:

   

   View larger image

   

2. Select the Resources tab in the left, highlighted below. From the list that populates, select Add Cloud Resource:

   

   View larger image

   

3. Select Add new connection:

   

   View larger image

   

4. Select the relevant native cloud provider or S3 compatible option and fill in the details:

   

   View larger image

   

5. Select the newly created connection and map it to the existing bucket:

   

   View larger image

   
6. Repeat these steps to create as many backing stores as needed.

Procedure

1. Click Operators → Installed Operators from the left pane of the OpenShift Web Console to view the installed operators.
2. Click OpenShift Container Storage Operator.

3. On the OpenShift Container Storage Operator page, scroll right and click the Bucket Class tab.

   Figure 7.3. OpenShift Container Storage Operator page with Bucket Class tab

   

   View larger image

   
4. Click Create Bucket Class.

5. On the Create new Bucket Class page, perform the following:

   1. Enter a Bucket Class Name and click Next.

      Figure 7.4. Create Bucket Class page

      

      View larger image

      

   2. In Placement Policy, select Tier 1 - Policy Type and click Next. You can choose either one of the options as per your requirements.

      • Spread allows spreading of the data across the chosen resources.
      • Mirror allows full duplication of the data across the chosen resources.

      • Click Add Tier to add another policy tier.

        Figure 7.5. Tier 1 - Policy Type selection page

        

        View larger image

        

   3. Select atleast one Backing Store resource from the available list if you have selected Tier 1 - Policy Type as Spread and click Next. Alternatively, you can also create a new backing store.

      Figure 7.6. Tier 1 - Backing Store selection page

      

      View larger image

      

1. Review and confirm Bucket Class settings.

   Figure 7.7. Bucket class settings review page

   

   View larger image

   
2. Click Create Bucket Class.

Verification steps

1. Click Operators → Installed Operators.
2. Click OpenShift Container Storage Operator.
3. Search for the new Bucket Class or click Bucket Class tab to view all the Bucket Classes.

1. From the MCG command-line interface, run the following command to create a bucket class with a mirroring policy:

   $ noobaa bucketclass create mirror-to-aws --backingstores=azure-resource,aws-resource --placement Mirror

   Copy to Clipboard Toggle word wrap

2. Set the newly created bucket class to a new bucket claim, generating a new bucket that will be mirrored between two locations:

   $ noobaa obc create  mirrored-bucket --bucketclass=mirror-to-aws

   Copy to Clipboard Toggle word wrap

1. Apply the following YAML. This YAML is a hybrid example that mirrors data between local Ceph storage and AWS:

   apiVersion: noobaa.io/v1alpha1
   kind: BucketClass
   metadata:
     name: hybrid-class
     labels:
       app: noobaa
   spec:
     placementPolicy:
       tiers:
         - tier:
             mirrors:
               - mirror:
                   spread:
                     - cos-east-us
               - mirror:
                   spread:
                     - noobaa-test-bucket-for-ocp201907291921-11247_resource

   Copy to Clipboard Toggle word wrap

2. Add the following lines to your standard Object Bucket Claim (OBC):

   additionalConfig:
     bucketclass: mirror-to-aws

   Copy to Clipboard Toggle word wrap

   For more information about OBCs, see Section 7.7, “Object Bucket Claim”.

1. In your OpenShift Storage console, navigate to Overview → Object Service → select the noobaa link:

   

   View larger image

   

2. Click the buckets icon on the left side. You will see a list of your buckets:

   

   View larger image

   
3. Click the bucket you want to update.

4. Click Edit Tier 1 Resources:

   

   View larger image

   

5. Select Mirror and check the relevant resources you want to use for this bucket. In the following example, we mirror data between on prem Ceph RGW to AWS:

   

   View larger image

   
6. Click Save.

1. Create the bucket policy in JSON format. See the following example:

   {
       "Version": "NewVersion",
       "Statement": [
           {
               "Sid": "Example",
               "Effect": "Allow",
               "Principal": [
                       "john.doe@example.com"
               ],
               "Action": [
                   "s3:GetObject"
               ],
               "Resource": [
                   "arn:aws:s3:::john_bucket"
               ]
           }
       ]
   }

   Copy to Clipboard Toggle word wrap

   There are many available elements for bucket policies. For details on these elements and examples of how they can be used, see AWS Access Policy Language Overview.

   For more examples of bucket policies, see AWS Bucket Policy Examples.

   Instructions for creating S3 users can be found in Section 7.6.3, “Creating an AWS S3 user in the Multicloud Object Gateway”.

2. Using AWS S3 client, use the put-bucket-policy command to apply the bucket policy to your S3 bucket:

   # aws --endpoint ENDPOINT --no-verify-ssl s3api put-bucket-policy --bucket MyBucket --policy BucketPolicy

   Copy to Clipboard Toggle word wrap

   Replace ENDPOINT with the S3 endpoint

   Replace MyBucket with the bucket to set the policy on

   Replace BucketPolicy with the bucket policy JSON file

   Add --no-verify-ssl if you are using the default self signed certificates

   For example:

   # aws --endpoint https://s3-openshift-storage.apps.gogo44.noobaa.org --no-verify-ssl s3api put-bucket-policy -bucket MyBucket --policy file://BucketPolicy

   Copy to Clipboard Toggle word wrap

   For more information on the put-bucket-policy command, see the AWS CLI Command Reference for put-bucket-policy.

Procedure

1. In your OpenShift Storage console, navigate to Overview → Object Service → select the noobaa link:

   

   View larger image

   

2. Under the Accounts tab, click Create Account:

   

   View larger image

   

3. Select S3 Access Only, provide the Account Name, for example, john.doe@example.com. Click Next:

   

   View larger image

   

4. Select S3 default placement, for example, noobaa-default-backing-store. Select Buckets Permissions. A specific bucket or all buckets can be selected. Click Create:

   

   View larger image

   

Procedure

1. Add the following lines to your application YAML:

   apiVersion: objectbucket.io/v1alpha1
   kind: ObjectBucketClaim
   metadata:
     name: <obc-name>
   spec:
     generateBucketName: <obc-bucket-name>
     storageClassName: openshift-storage.noobaa.io

   Copy to Clipboard Toggle word wrap

   These lines are the Object Bucket Claim itself.

   1. Replace <obc-name> with the a unique Object Bucket Claim name.
   2. Replace <obc-bucket-name> with a unique bucket name for your Object Bucket Claim.

2. You can add more lines to the YAML file to automate the use of the Object Bucket Claim. The example below is the mapping between the bucket claim result, which is a configuration map with data and a secret with the credentials. This specific job will claim the Object Bucket from NooBaa, which will create a bucket and an account.

   apiVersion: batch/v1
   kind: Job
   metadata:
     name: testjob
   spec:
     template:
       spec:
         restartPolicy: OnFailure
         containers:
           - image: <your application image>
             name: test
             env:
               - name: BUCKET_NAME
                 valueFrom:
                   configMapKeyRef:
                     name: <obc-name>
                     key: BUCKET_NAME
               - name: BUCKET_HOST
                 valueFrom:
                   configMapKeyRef:
                     name: <obc-name>
                     key: BUCKET_HOST
               - name: BUCKET_PORT
                 valueFrom:
                   configMapKeyRef:
                     name: <obc-name>
                     key: BUCKET_PORT
               - name: AWS_ACCESS_KEY_ID
                 valueFrom:
                   secretKeyRef:
                     name: <obc-name>
                     key: AWS_ACCESS_KEY_ID
               - name: AWS_SECRET_ACCESS_KEY
                 valueFrom:
                   secretKeyRef:
                     name: <obc-name>
                     key: AWS_SECRET_ACCESS_KEY

   Copy to Clipboard Toggle word wrap
   1. Replace all instances of <obc-name> with your Object Bucket Claim name.
   2. Replace <your application image> with your application image.

3. Apply the updated YAML file:

   # oc apply -f <yaml.file>

   Copy to Clipboard Toggle word wrap
   1. Replace <yaml.file> with the name of your YAML file.

4. To view the new configuration map, run the following:

   # oc get cm <obc-name> -o yaml

   Copy to Clipboard Toggle word wrap

   1. Replace obc-name with the name of your Object Bucket Claim.

      You can expect the following environment variables in the output:

      • BUCKET_HOST - Endpoint to use in the application

      • BUCKET_PORT - The port available for the application

        • The port is related to the BUCKET_HOST. For example, if the BUCKET_HOST is https://my.example.com, and the BUCKET_PORT is 443, the endpoint for the object service would be https://my.example.com:443.
      • BUCKET_NAME - Requested or generated bucket name
      • AWS_ACCESS_KEY_ID - Access key that is part of the credentials
      • AWS_SECRET_ACCESS_KEY - Secret access key that is part of the credentials

Procedure

1. Use the command-line interface to generate the details of a new bucket and credentials. Run the following command:

   # noobaa obc create <obc-name> -n openshift-storage

   Copy to Clipboard Toggle word wrap

   Replace <obc-name> with a unique Object Bucket Claim name, for example, myappobc.

   Additionally, you can use the --app-namespace option to specify the namespace where the Object Bucket Claim configuration map and secret will be created, for example, myapp-namespace.

   Example output:

   INFO[0001] ✅ Created: ObjectBucketClaim "test21obc"

   Copy to Clipboard Toggle word wrap

   The MCG command-line-interface has created the necessary configuration and has informed OpenShift about the new OBC.

2. Run the following command to view the Object Bucket Claim:

   # oc get obc -n openshift-storage

   Copy to Clipboard Toggle word wrap

   Example output:

   NAME        STORAGE-CLASS                 PHASE   AGE
   test21obc   openshift-storage.noobaa.io   Bound   38s

   Copy to Clipboard Toggle word wrap

3. Run the following command to view the YAML file for the new Object Bucket Claim:

   # oc get obc test21obc -o yaml -n openshift-storage

   Copy to Clipboard Toggle word wrap

   Example output:

   apiVersion: objectbucket.io/v1alpha1
   kind: ObjectBucketClaim
   metadata:
     creationTimestamp: "2019-10-24T13:30:07Z"
     finalizers:
     - objectbucket.io/finalizer
     generation: 2
     labels:
       app: noobaa
       bucket-provisioner: openshift-storage.noobaa.io-obc
       noobaa-domain: openshift-storage.noobaa.io
     name: test21obc
     namespace: openshift-storage
     resourceVersion: "40756"
     selfLink: /apis/objectbucket.io/v1alpha1/namespaces/openshift-storage/objectbucketclaims/test21obc
     uid: 64f04cba-f662-11e9-bc3c-0295250841af
   spec:
     ObjectBucketName: obc-openshift-storage-test21obc
     bucketName: test21obc-933348a6-e267-4f82-82f1-e59bf4fe3bb4
     generateBucketName: test21obc
     storageClassName: openshift-storage.noobaa.io
   status:
     phase: Bound

   Copy to Clipboard Toggle word wrap

4. Inside of your openshift-storage namespace, you can find the configuration map and the secret to use this Object Bucket Claim. The CM and the secret have the same name as the Object Bucket Claim. To view the secret:

   # oc get -n openshift-storage secret test21obc -o yaml

   Copy to Clipboard Toggle word wrap

   Example output:

   Example output:
   apiVersion: v1
   data:
     AWS_ACCESS_KEY_ID: c0M0R2xVanF3ODR3bHBkVW94cmY=
     AWS_SECRET_ACCESS_KEY: Wi9kcFluSWxHRzlWaFlzNk1hc0xma2JXcjM1MVhqa051SlBleXpmOQ==
   kind: Secret
   metadata:
     creationTimestamp: "2019-10-24T13:30:07Z"
     finalizers:
     - objectbucket.io/finalizer
     labels:
       app: noobaa
       bucket-provisioner: openshift-storage.noobaa.io-obc
       noobaa-domain: openshift-storage.noobaa.io
     name: test21obc
     namespace: openshift-storage
     ownerReferences:
     - apiVersion: objectbucket.io/v1alpha1
       blockOwnerDeletion: true
       controller: true
       kind: ObjectBucketClaim
       name: test21obc
       uid: 64f04cba-f662-11e9-bc3c-0295250841af
     resourceVersion: "40751"
     selfLink: /api/v1/namespaces/openshift-storage/secrets/test21obc
     uid: 65117c1c-f662-11e9-9094-0a5305de57bb
   type: Opaque

   Copy to Clipboard Toggle word wrap

   The secret gives you the S3 access credentials.

5. To view the configuration map:

   # oc get -n openshift-storage cm test21obc -o yaml

   Copy to Clipboard Toggle word wrap

   Example output:

   apiVersion: v1
   data:
     BUCKET_HOST: 10.0.171.35
     BUCKET_NAME: test21obc-933348a6-e267-4f82-82f1-e59bf4fe3bb4
     BUCKET_PORT: "31242"
     BUCKET_REGION: ""
     BUCKET_SUBREGION: ""
   kind: ConfigMap
   metadata:
     creationTimestamp: "2019-10-24T13:30:07Z"
     finalizers:
     - objectbucket.io/finalizer
     labels:
       app: noobaa
       bucket-provisioner: openshift-storage.noobaa.io-obc
       noobaa-domain: openshift-storage.noobaa.io
     name: test21obc
     namespace: openshift-storage
     ownerReferences:
     - apiVersion: objectbucket.io/v1alpha1
       blockOwnerDeletion: true
       controller: true
       kind: ObjectBucketClaim
       name: test21obc
       uid: 64f04cba-f662-11e9-bc3c-0295250841af
     resourceVersion: "40752"
     selfLink: /api/v1/namespaces/openshift-storage/configmaps/test21obc
     uid: 651c6501-f662-11e9-9094-0a5305de57bb

   Copy to Clipboard Toggle word wrap

   The configuration map contains the S3 endpoint information for your application.

Procedure

1. Log into the OpenShift Web Console.
2. On the left navigation bar, click Storage → Object Bucket Claims.

3. Click Create Object Bucket Claim:

   

   View larger image

   

4. Enter a name for your object bucket claim and select the appropriate storage class based on your deployment, internal or external, from the dropdown menu:

   Internal mode

   

   View larger image

   

   The following storage classes, which were created after deployment, are available for use:

   • ocs-storagecluster-ceph-rgw uses the Ceph Object Gateway (RGW)
   • openshift-storage.noobaa.io uses the Multicloud Object Gateway

   External mode

   

   View larger image

   

   The following storage classes, which were created after deployment, are available for use:

   • ocs-external-storagecluster-ceph-rgw uses the Ceph Object Gateway (RGW)

   • openshift-storage.noobaa.io uses the Multicloud Object Gateway

     Note

     The RGW OBC storage class is only available with fresh installations of OpenShift Container Storage version 4.5. It does not apply to clusters upgraded from previous OpenShift Container Storage releases.

5. Click Create.

   Once you create the OBC, you are redirected to its detail page:

   

   View larger image

   

Procedure

1. On the left navigation bar, click Storage → Object Bucket Claims.
2. Click the action menu (⋮) next to the OBC you created.

3. From the drop down menu, select Attach to Deployment.

   

   View larger image

   

4. Select the desired deployment from the Deployment Name list, then click Attach:

   

   View larger image

   

1. Log into the OpenShift Web Console.

2. On the left navigation bar, click Storage → Object Buckets:

   

   View larger image

   

   You can also navigate to the details page of a specific OBC and click the Resource link to view the object buckets for that OBC.

3. Select the object bucket you want to see details for. You are navigated to the object bucket’s details page:

   

   View larger image

   

Procedure

1. On the left navigation bar, click Storage → Object Bucket Claims.

2. click on the action menu (⋮) next to the Object Bucket Claim you want to delete.

   

   View larger image

   

3. Select Delete Object Bucket Claim from menu.

   

   View larger image

   
4. Click Delete.

Procedure

1. In the Multicloud Object Gateway user interface, from the Overview page, click Add Storage Resources:

   

   View larger image

   

2. In the window, click Deploy Kubernetes Pool:

   

   View larger image

   

3. In the Create Pool step create the target pool for the future installed nodes.

   

   View larger image

   

4. In the Configure step, configure the number of requested pods and the size of each PV. For each new pod, one PV is be created.

   

   View larger image

   
5. In the Review step, you can find the details of the new pool and select the deployment method you wish to use: local or external deployment. If local deployment is selected, the Kubernetes nodes will deploy within the cluster. If external deployment is selected, you will be provided with a YAML file to run externally.

6. All nodes will be assigned to the pool you chose in the first step, and can be found under Resources → Storage resources → Resource name:

   

   View larger image

   

Procedure

1. Identify the node that needs to be replaced.

2. Mark the node as unschedulable using the following command:

   $ oc adm cordon <node_name>

   Copy to Clipboard Toggle word wrap

3. Drain the node using the following command:

   $ oc adm drain <node_name> --force --delete-local-data --ignore-daemonsets

   Copy to Clipboard Toggle word wrap
   Important

   This activity may take at least 5-10 minutes or more. Ceph errors generated during this period are temporary and are automatically resolved when the new node is labeled and functional.

4. Delete the node using the following command:

   $ oc delete nodes <node_name>

   Copy to Clipboard Toggle word wrap
5. Create a new AWS machine instance with the required infrastructure. See Platform requirements.
6. Create a new OpenShift Container Platform node using the new AWS machine instance.

7. Check for certificate signing requests (CSRs) related to OpenShift Container Platform that are in Pending state:

   $ oc get csr

   Copy to Clipboard Toggle word wrap

8. Approve all required OpenShift Container Platform CSRs for the new node:

   $ oc adm certificate approve <Certificate_Name>

   Copy to Clipboard Toggle word wrap
9. Click Compute → Nodes, confirm if the new node is in Ready state.

10. Apply the OpenShift Container Storage label to the new node using any one of the following:

    From User interface

    1. For the new node, click Action Menu (⋮) → Edit Labels
    2. Add cluster.ocs.openshift.io/openshift-storage and click Save.

    From Command line interface

    • Execute the following command to apply the OpenShift Container Storage label to the new node:

      $ oc label node <new_node_name> cluster.ocs.openshift.io/openshift-storage=""

      Copy to Clipboard Toggle word wrap

Verification steps

1. Execute the following command and verify that the new node is present in the output:

   $ oc get nodes --show-labels | grep cluster.ocs.openshift.io/openshift-storage= |cut -d' ' -f1

   Copy to Clipboard Toggle word wrap

2. Click Workloads → Pods, confirm that at least the following pods on the new node are in Running state:

   • csi-cephfsplugin-*
   • csi-rbdplugin-*
3. Verify that all other required OpenShift Container Storage pods are in Running state.
4. If verification steps fail, kindly contact Red Hat Support.

Procedure

1. Log in to OpenShift Web Console and click Compute → Nodes.
2. Identify the node that needs to be replaced. Take a note of its Machine Name.

3. Mark the node as unschedulable using the following command:

   $ oc adm cordon <node_name>

   Copy to Clipboard Toggle word wrap

4. Drain the node using the following command:

   $ oc adm drain <node_name> --force --delete-local-data --ignore-daemonsets

   Copy to Clipboard Toggle word wrap
   Important

   This activity may take at least 5-10 minutes or more. Ceph errors generated during this period are temporary and are automatically resolved when the new node is labeled and functional.

5. Click Compute → Machines. Search for the required machine.
6. Besides the required machine, click the Action menu (⋮) → Delete Machine.
7. Click Delete to confirm the machine deletion. A new machine is automatically created.

8. Wait for new machine to start and transition into Running state.

   Important

   This activity may take at least 5-10 minutes or more.

9. Click Compute → Nodes, confirm if the new node is in Ready state.

10. Apply the OpenShift Container Storage label to the new node using any one of the following:

    From User interface

    1. For the new node, click Action Menu (⋮) → Edit Labels
    2. Add cluster.ocs.openshift.io/openshift-storage and click Save.

    From Command line interface

    • Execute the following command to apply the OpenShift Container Storage label to the new node:

      $ oc label node <new_node_name> cluster.ocs.openshift.io/openshift-storage=""

      Copy to Clipboard Toggle word wrap

Verification steps

1. Execute the following command and verify that the new node is present in the output:

   $ oc get nodes --show-labels | grep cluster.ocs.openshift.io/openshift-storage= |cut -d' ' -f1

   Copy to Clipboard Toggle word wrap

2. Click Workloads → Pods, confirm that at least the following pods on the new node are in Running state:

   • csi-cephfsplugin-*
   • csi-rbdplugin-*
3. Verify that all other required OpenShift Container Storage pods are in Running state.
4. If verification steps fail, kindly contact Red Hat Support.

Procedure

1. Identify the AWS machine instance of the node that needs to be replaced.
2. Log in to AWS and terminate the identified AWS machine instance.
3. Create a new AWS machine instance with the required infrastructure. See platform requirements.
4. Create a new OpenShift Container Platform node using the new AWS machine instance.

5. Check for certificate signing requests (CSRs) related to OpenShift Container Platform that are in Pending state:

   $ oc get csr

   Copy to Clipboard Toggle word wrap

6. Approve all required OpenShift Container Platform CSRs for the new node:

   $ oc adm certificate approve <Certificate_Name>

   Copy to Clipboard Toggle word wrap
7. Click Compute → Nodes, confirm if the new node is in Ready state.

8. Apply the OpenShift Container Storage label to the new node using any one of the following:

   From User interface

   1. For the new node, click Action Menu (⋮) → Edit Labels
   2. Add cluster.ocs.openshift.io/openshift-storage and click Save.

   From Command line interface

   • Execute the following command to apply the OpenShift Container Storage label to the new node:

     $ oc label node <new_node_name> cluster.ocs.openshift.io/openshift-storage=""

     Copy to Clipboard Toggle word wrap

Verification steps

1. Execute the following command and verify that the new node is present in the output:

   $ oc get nodes --show-labels | grep cluster.ocs.openshift.io/openshift-storage= |cut -d' ' -f1

   Copy to Clipboard Toggle word wrap

2. Click Workloads → Pods, confirm that at least the following pods on the new node are in Running state:

   • csi-cephfsplugin-*
   • csi-rbdplugin-*
3. Verify that all other required OpenShift Container Storage pods are in Running state.
4. If verification steps fail, contact Red Hat Support.

Procedure

1. Log in to OpenShift Web Console and click Compute → Nodes.
2. Identify the faulty node and click on its Machine Name.
3. Click Actions → Edit Annotations, and click Add More.
4. Add machine.openshift.io/exclude-node-draining and click Save.
5. Click Actions → Delete Machine, and click Delete.

6. A new machine is automatically created, wait for new machine to start.

   Important

   This activity may take at least 5-10 minutes or more. Ceph errors generated during this period are temporary and are automatically resolved when the new node is labeled and functional.

7. Click Compute → Nodes, confirm if the new node is in Ready state.

8. Apply the OpenShift Container Storage label to the new node using any one of the following:

   From User interface

   1. For the new node, click Action Menu (⋮) → Edit Labels
   2. Add cluster.ocs.openshift.io/openshift-storage and click Save.

   From Command line interface

   • Execute the following command to apply the OpenShift Container Storage label to the new node:

     $ oc label node <new_node_name> cluster.ocs.openshift.io/openshift-storage=""

     Copy to Clipboard Toggle word wrap

9. Execute the following command and verify that the new node is present in the output:

   $ oc get nodes --show-labels | grep cluster.ocs.openshift.io/openshift-storage= |cut -d' ' -f1

   Copy to Clipboard Toggle word wrap

10. Click Workloads → Pods, confirm that at least the following pods on the new node are in Running state:

    • csi-cephfsplugin-*
    • csi-rbdplugin-*
11. Verify that all other required OpenShift Container Storage pods are in Running state.
12. If verification steps fail, kindly contact Red Hat Support.

Procedure

1. Identify the node and its VM that needs to be replaced.

2. Mark the node as unschedulable using the following command:

   $ oc adm cordon <node_name>

   Copy to Clipboard Toggle word wrap

3. Drain the node using the following command:

   $ oc adm drain <node_name> --force --delete-local-data --ignore-daemonsets

   Copy to Clipboard Toggle word wrap
   Important

   This activity may take at least 5-10 minutes or more. Ceph errors generated during this period are temporary and are automatically resolved when the new node is labeled and functional.

4. Delete the node using the following command:

   $ oc delete nodes <node_name>

   Copy to Clipboard Toggle word wrap

5. Log in to vSphere and terminate the identified VM.

   Important

   VM should be deleted only from the inventory and not from the disk.

6. Create a new VM on vSphere with the required infrastructure. See Platform requirements.
7. Create a new OpenShift Container Platform worker node using the new VM.

8. Check for certificate signing requests (CSRs) related to OpenShift Container Platform that are in Pending state:

   $ oc get csr

   Copy to Clipboard Toggle word wrap

9. Approve all required OpenShift Container Platform CSRs for the new node:

   $ oc adm certificate approve <Certificate_Name>

   Copy to Clipboard Toggle word wrap
10. Click Compute → Nodes, confirm if the new node is in Ready state.

11. Apply the OpenShift Container Storage label to the new node using any one of the following:

    From User interface

    1. For the new node, click Action Menu (⋮) → Edit Labels
    2. Add cluster.ocs.openshift.io/openshift-storage and click Save.

    From Command line interface

    • Execute the following command to apply the OpenShift Container Storage label to the new node:

      $ oc label node <new_node_name> cluster.ocs.openshift.io/openshift-storage=""

      Copy to Clipboard Toggle word wrap

Verification steps

1. Execute the following command and verify that the new node is present in the output:

   $ oc get nodes --show-labels | grep cluster.ocs.openshift.io/openshift-storage= |cut -d' ' -f1

   Copy to Clipboard Toggle word wrap

2. Click Workloads → Pods, confirm that at least the following pods on the new node are in Running state:

   • csi-cephfsplugin-*
   • csi-rbdplugin-*
3. Verify that all other required OpenShift Container Storage pods are in Running state.
4. If verification steps fail, kindly contact Red Hat Support.

Procedure

1. Identify the node and its VM that needs to be replaced.

2. Delete the node using the following command:

   $ oc delete nodes <node_name>

   Copy to Clipboard Toggle word wrap

3. Log in to vSphere and terminate the identified VM.

   Important

   VM should be deleted only from the inventory and not from the disk.

4. Create a new VM on vSphere with the required infrastructure. See Platform requirements.
5. Create a new OpenShift Container Platform worker node using the new VM.

6. Check for certificate signing requests (CSRs) related to OpenShift Container Platform that are in Pending state:

   $ oc get csr

   Copy to Clipboard Toggle word wrap

7. Approve all required OpenShift Container Platform CSRs for the new node:

   $ oc adm certificate approve <Certificate_Name>

   Copy to Clipboard Toggle word wrap
8. Click Compute → Nodes, confirm if the new node is in Ready state.

9. Apply the OpenShift Container Storage label to the new node using any one of the following:

   From User interface

   1. For the new node, click Action Menu (⋮) → Edit Labels
   2. Add cluster.ocs.openshift.io/openshift-storage and click Save.

   From Command line interface

   • Execute the following command to apply the OpenShift Container Storage label to the new node:

     $ oc label node <new_node_name> cluster.ocs.openshift.io/openshift-storage=""

     Copy to Clipboard Toggle word wrap