Chapter 7. Multicloud Object Gateway

Procedure

1. Click Operators Installed Operators from the left pane of the OpenShift Web Console to view the installed operators.
2. On the Installed Operator page, select openshift-storage from the Project drop down list to switch to the openshift-storage project.
3. Click OpenShift Container Storage Operator.

4. On the OpenShift Container Storage Operator page, scroll right and click the Backing Store tab.

   Figure 7.1. OpenShift Container Storage Operator page with backing store tab

   

5. Click Create Backing Store.

   Figure 7.2. Create Backing Store page

   

6. On the Create New Backing Store page, perform the following:

   1. Make sure the Namespace is set to openshift-storage.
   2. Enter a Backing Store Name.
   3. Select Provider.
   4. Select Region.
   5. Enter Endpoint. This is optional.

   6. Select Secret from drop down list, or create your own secret. Optionally, you can Switch to Credentials view which lets you fill in the required secrets.

      Note

      This menu is relevant for all providers except Google Cloud and local PVC.

   7. Enter Target bucket. The target bucket is a container storage that is hosted on the remote cloud service. It allows you to create a connection that tells NooBaa that it can use this bucket for the system.
7. Click Create Backing Store.

Verification steps

1. Click Operators Installed Operators.
2. Click OpenShift Container Storage Operator.
3. Search for the new backing store.

Procedure

1. From the Multicloud Object Gateway command-line interface, run the following NooBaa command:

   noobaa backingstore create s3-compatible rgw-resource --access-key=<RGW ACCESS KEY> --secret-key=<RGW SECRET KEY> --target-bucket=<bucket-name> --endpoint=http://rook-ceph-rgw-ocs-storagecluster-cephobjectstore.openshift-storage.svc.cluster.local:80

   1. To get the <RGW ACCESS KEY> and <RGW SECRET KEY>, run the following command using your RGW user secret name:

      oc get secret <RGW USER SECRET NAME> -o yaml

   2. Decode the access key ID and the access key from Base64 and keep them.
   3. Replace <RGW USER ACCESS KEY> and <RGW USER SECRET ACCESS KEY> with the appropriate, decoded data from the previous step.

   4. Replace <bucket-name> with an existing RGW bucket name. This argument tells Multicloud Object Gateway which bucket to use as a target bucket for its backing store, and subsequently, data storage and administration.

      The output will be similar to the following:

      INFO[0001] ✅ Exists: NooBaa "noobaa"
      INFO[0002] ✅ Created: BackingStore "rgw-resource"
      INFO[0002] ✅ Created: Secret "backing-store-secret-rgw-resource"

1. Create a CephObjectStore user. This also creates a secret containing the RGW credentials:

   apiVersion: ceph.rook.io/v1
   kind: CephObjectStoreUser
   metadata:
     name: <RGW-Username>
     namespace: openshift-storage
   spec:
     store: ocs-storagecluster-cephobjectstore
     displayName: "<Display-name>"

   1. Replace <RGW-Username> and <Display-name> with a unique username and display name.

2. Apply the following YAML for an S3-Compatible backing store:

   apiVersion: noobaa.io/v1alpha1
   kind: BackingStore
   metadata:
     finalizers:
     - noobaa.io/finalizer
     labels:
       app: noobaa
     name: <backingstore-name>
     namespace: openshift-storage
   spec:
     s3Compatible:
       endpoint: http://rook-ceph-rgw-ocs-storagecluster-cephobjectstore.openshift-storage.svc.cluster.local:80
       secret:
         name: <backingstore-secret-name>
         namespace: openshift-storage
       signatureVersion: v4
       targetBucket: <RGW-bucket-name>
     type: s3-compatible

   1. Replace <backingstore-secret-name> with the name of the secret that was created with CephObjectStore in the previous step.
   2. Replace <bucket-name> with an existing RGW bucket name. This argument tells Multicloud Object Gateway which bucket to use as a target bucket for its backing store, and subsequently, data storage and administration.

Procedure

1. In your OpenShift Storage console, navigate to Dashboards OCS Object Service select the noobaa link:

   

2. Select the Resources tab in the left, highlighted below. From the list that populates, select Add Cloud Resource:

   

3. Select Add new connection:

   

4. Select the relevant native cloud provider or S3 compatible option and fill in the details:

   

5. Select the newly created connection and map it to the existing bucket:

   
6. Repeat these steps to create as many backing stores as needed.

Procedure

1. Click Operators Installed Operators from the left pane of the OpenShift Web Console to view the installed operators.
2. Click OpenShift Container Storage Operator.

3. On the OpenShift Container Storage Operator page, scroll right and click the Bucket Class tab.

   Figure 7.3. OpenShift Container Storage Operator page with Bucket Class tab

   
4. Click Create Bucket Class.

5. On the Create new Bucket Class page, perform the following:

   1. Enter a Bucket Class Name and click Next.

      Figure 7.4. Create Bucket Class page

      

   2. In Placement Policy, select Tier 1 - Policy Type and click Next. You can choose either one of the options as per your requirements.

      • Spread allows spreading of the data across the chosen resources.
      • Mirror allows full duplication of the data across the chosen resources.

      • Click Add Tier to add another policy tier.

        Figure 7.5. Tier 1 - Policy Type selection page

        

   3. Select atleast one Backing Store resource from the available list if you have selected Tier 1 - Policy Type as Spread and click Next. Alternatively, you can also create a new backing store.

      Figure 7.6. Tier 1 - Backing Store selection page

      

1. Review and confirm Bucket Class settings.

   Figure 7.7. Bucket class settings review page

   
2. Click Create Bucket Class.

Verification steps

1. Click Operators Installed Operators.
2. Click OpenShift Container Storage Operator.
3. Search for the new Bucket Class or click Bucket Class tab to view all the Bucket Classes.

1. From the MCG command-line interface, run the following command to create a bucket class with a mirroring policy:

   $ noobaa bucketclass create mirror-to-aws --backingstores=azure-resource,aws-resource --placement Mirror

2. Set the newly created bucket class to a new bucket claim, generating a new bucket that will be mirrored between two locations:

   $ noobaa obc create  mirrored-bucket --bucketclass=mirror-to-aws

1. Apply the following YAML. This YAML is a hybrid example that mirrors data between local Ceph storage and AWS:

   apiVersion: noobaa.io/v1alpha1
   kind: BucketClass
   metadata:
     name: hybrid-class
     labels:
       app: noobaa
   spec:
     placementPolicy:
       tiers:
         - tier:
             mirrors:
               - mirror:
                   spread:
                     - cos-east-us
               - mirror:
                   spread:
                     - noobaa-test-bucket-for-ocp201907291921-11247_resource

2. Add the following lines to your standard Object Bucket Claim (OBC):

   additionalConfig:
     bucketclass: mirror-to-aws

   For more information about OBCs, see Section 7.6, “Object Bucket Claim”.

1. In your OpenShift Storage console, navigate to Dashboards OCS Object Service select the noobaa link:

   

2. Click the buckets icon on the left side. You will see a list of your buckets:

   
3. Click the bucket you want to update.

4. Click Edit Tier 1 Resources:

   

5. Select Mirror and check the relevant resources you want to use for this bucket. In the following example, we mirror data between on prem Ceph RGW to AWS:

   
6. Click Save.

1. Create the bucket policy in JSON format. See the following example:

   {
       "Version": "NewVersion",
       "Statement": [
           {
               "Sid": "Example",
               "Effect": "Allow",
               "Principal": [
                       "john.doe@example.com"
               ],
               "Action": [
                   "s3:GetObject"
               ],
               "Resource": [
                   "arn:aws:s3:::john_bucket"
               ]
           }
       ]
   }

   There are many available elements for bucket policies. For details on these elements and examples of how they can be used, see AWS Access Policy Language Overview.

   For more examples of bucket policies, see AWS Bucket Policy Examples.

   Instructions for creating S3 users can be found in Section 7.5.3, “Creating an AWS S3 user in the Multicloud Object Gateway”.

2. Using AWS S3 client, use the put-bucket-policy command to apply the bucket policy to your S3 bucket:

   # aws s3api put-bucket-policy --bucket MyBucket --policy BucketPolicy

   Replace MyBucket with the bucket to set the policy on

   Replace BucketPolicy with the bucket policy JSON file

   For example:

   # aws s3api put-bucket-policy --bucket MyBucket --policy file://BucketPolicy

   For more information on the put-bucket-policy command, see the AWS CLI Command Reference for put-bucket-policy.

Procedure

1. In your OpenShift Storage console, navigate to Dashboards OCS Object Service select the noobaa link:

   

2. Under the Accounts tab, click Create Account:

   

3. Select S3 Access Only, provide the Account Name, for example, john.doe@example.com. Click Next:

   

4. Select S3 default placement, for example, noobaa-default-backing-store. Select Buckets Permissions. A specific bucket or all buckets can be selected. Click Create:

   

Procedure

1. Add the following lines to your application YAML:

   apiVersion: objectbucket.io/v1alpha1
   kind: ObjectBucketClaim
   metadata:
     name: <obc-name>
   spec:
     generateBucketName: <obc-bucket-name>
     storageClassName: noobaa

   These lines are the Object Bucket Claim itself.

   1. Replace <obc-name> with the a unique Object Bucket Claim name.
   2. Replace <obc-bucket-name> with a unique bucket name for your Object Bucket Claim.

2. You can add more lines to the YAML file to automate the use of the Object Bucket Claim. The example below is the mapping between the bucket claim result, which is a configuration map with data and a secret with the credentials. This specific job will claim the Object Bucket from NooBaa, which will create a bucket and an account.

   apiVersion: batch/v1
   kind: Job
   metadata:
     name: testjob
   spec:
     template:
       spec:
         restartPolicy: OnFailure
         containers:
           - image: <your application image>
             name: test
             env:
               - name: BUCKET_NAME
                 valueFrom:
                   configMapKeyRef:
                     name: <obc-name>
                     key: BUCKET_NAME
               - name: BUCKET_HOST
                 valueFrom:
                   configMapKeyRef:
                     name: <obc-name>
                     key: BUCKET_HOST
               - name: BUCKET_PORT
                 valueFrom:
                   configMapKeyRef:
                     name: <obc-name>
                     key: BUCKET_PORT
               - name: AWS_ACCESS_KEY_ID
                 valueFrom:
                   secretKeyRef:
                     name: <obc-name>
                     key: AWS_ACCESS_KEY_ID
               - name: AWS_SECRET_ACCESS_KEY
                 valueFrom:
                   secretKeyRef:
                     name: <obc-name>
                     key: AWS_SECRET_ACCESS_KEY

   1. Replace all instances of <obc-name> with your Object Bucket Claim name.
   2. Replace <your application image> with your application image.

3. Apply the updated YAML file:

   # oc apply -f <yaml.file>

   1. Replace <yaml.file> with the name of your YAML file.

4. To view the new configuration map, run the following:

   # oc get cm <obc-name>

   1. Replace obc-name with the name of your Object Bucket Claim.

      You can expect the following environment variables in the output:

      • BUCKET_HOST - Endpoint to use in the application

      • BUCKET_PORT - The port available for the application

        • The port is related to the BUCKET_HOST. For example, if the BUCKET_HOST is https://my.example.com, and the BUCKET_PORT is 443, the endpoint for the object service would be https://my.example.com:443.
      • BUCKET_NAME - Requested or generated bucket name
      • AWS_ACCESS_KEY_ID - Access key that is part of the credentials
      • AWS_SECRET_ACCESS_KEY - Secret access key that is part of the credentials

Procedure

1. Use the command-line interface to generate the details of a new bucket and credentials. Run the following command:

   # noobaa obc create <obc-name> -n openshift-storage

   Replace <obc-name> with a unique Object Bucket Claim name, for example, myappobc.

   Additionally, you can use the --app-namespace option to specify the namespace where the Object Bucket Claim configuration map and secret will be created, for example, myapp-namespace.

   Example output:

   INFO[0001] ✅ Created: ObjectBucketClaim "test21obc"

   The MCG command-line-interface has created the necessary configuration and has informed OpenShift about the new OBC.

2. Run the following command to view the Object Bucket Claim:

   # oc get obc -n openshift-storage

   Example output:

   NAME        STORAGE-CLASS                 PHASE   AGE
   test21obc   openshift-storage.noobaa.io   Bound   38s

3. Run the following command to view the YAML file for the new Object Bucket Claim:

   # oc get obc test21obc -o yaml -n openshift-storage

   Example output:

   apiVersion: objectbucket.io/v1alpha1
   kind: ObjectBucketClaim
   metadata:
     creationTimestamp: "2019-10-24T13:30:07Z"
     finalizers:
     - objectbucket.io/finalizer
     generation: 2
     labels:
       app: noobaa
       bucket-provisioner: openshift-storage.noobaa.io-obc
       noobaa-domain: openshift-storage.noobaa.io
     name: test21obc
     namespace: openshift-storage
     resourceVersion: "40756"
     selfLink: /apis/objectbucket.io/v1alpha1/namespaces/openshift-storage/objectbucketclaims/test21obc
     uid: 64f04cba-f662-11e9-bc3c-0295250841af
   spec:
     ObjectBucketName: obc-openshift-storage-test21obc
     bucketName: test21obc-933348a6-e267-4f82-82f1-e59bf4fe3bb4
     generateBucketName: test21obc
     storageClassName: openshift-storage.noobaa.io
   status:
     phase: Bound

4. Inside of your openshift-storage namespace, you can find the configuration map and the secret to use this Object Bucket Claim. The CM and the secret have the same name as the Object Bucket Claim. To view the secret:

   # oc get -n openshift-storage secret test21obc -o yaml

   Example output:

   Example output:
   apiVersion: v1
   data:
     AWS_ACCESS_KEY_ID: c0M0R2xVanF3ODR3bHBkVW94cmY=
     AWS_SECRET_ACCESS_KEY: Wi9kcFluSWxHRzlWaFlzNk1hc0xma2JXcjM1MVhqa051SlBleXpmOQ==
   kind: Secret
   metadata:
     creationTimestamp: "2019-10-24T13:30:07Z"
     finalizers:
     - objectbucket.io/finalizer
     labels:
       app: noobaa
       bucket-provisioner: openshift-storage.noobaa.io-obc
       noobaa-domain: openshift-storage.noobaa.io
     name: test21obc
     namespace: openshift-storage
     ownerReferences:
     - apiVersion: objectbucket.io/v1alpha1
       blockOwnerDeletion: true
       controller: true
       kind: ObjectBucketClaim
       name: test21obc
       uid: 64f04cba-f662-11e9-bc3c-0295250841af
     resourceVersion: "40751"
     selfLink: /api/v1/namespaces/openshift-storage/secrets/test21obc
     uid: 65117c1c-f662-11e9-9094-0a5305de57bb
   type: Opaque

   The secret gives you the S3 access credentials.

5. To view the configuration map:

   # oc get -n openshift-storage cm test21obc -o yaml

   Example output:

   apiVersion: v1
   data:
     BUCKET_HOST: 10.0.171.35
     BUCKET_NAME: test21obc-933348a6-e267-4f82-82f1-e59bf4fe3bb4
     BUCKET_PORT: "31242"
     BUCKET_REGION: ""
     BUCKET_SUBREGION: ""
   kind: ConfigMap
   metadata:
     creationTimestamp: "2019-10-24T13:30:07Z"
     finalizers:
     - objectbucket.io/finalizer
     labels:
       app: noobaa
       bucket-provisioner: openshift-storage.noobaa.io-obc
       noobaa-domain: openshift-storage.noobaa.io
     name: test21obc
     namespace: openshift-storage
     ownerReferences:
     - apiVersion: objectbucket.io/v1alpha1
       blockOwnerDeletion: true
       controller: true
       kind: ObjectBucketClaim
       name: test21obc
       uid: 64f04cba-f662-11e9-bc3c-0295250841af
     resourceVersion: "40752"
     selfLink: /api/v1/namespaces/openshift-storage/configmaps/test21obc
     uid: 651c6501-f662-11e9-9094-0a5305de57bb

   The configuration map contains the S3 endpoint information for your application.

Procedure

1. Log into the OpenShift Web Console.
2. On the left navigation bar, click Storage Object Bucket Claims.

3. In the following window, click Create Object Bucket Claim:

   

4. In the following window, enter a name for your object bucket claim, and select the appropriate storage class and bucket class from the dropdown menus:

   

5. Click Create.

   Once the OBC is created, you will be redirected to its detail page:

   

6. Once you’ve created the OBC, you can attach it to a deployment.

   1. On the left navigation bar, click Storage Object Bucket Claims.
   2. Click the action menu (⋮) next to the OBC you created.

   3. From the drop down menu, select Attach to Deployment.

      

   4. In the following window, select the desired deployment from the drop down menu, then click Attach:

      

Procedure

1. In the Mult-Cloud Object Gateway user interface, from the Overview page, click Add Storage Resources:

   

2. In the window, click Deploy Kubernetes Pool:

   

3. In the Create Pool step create the target pool for the future installed nodes.

   

4. In the Configure step, configure the number of requested pods and the size of each PV. For each new pod, one PV is be created.

   
5. In the Review step, you can find the details of the new pool and select the deployment method you wish to use: local or external deployment. If local deployment is selected, the Kubernetes nodes will deploy within the cluster. If external deployment is selected, you will be provided with a YAML file to run externally.

6. All nodes will be assigned to the pool you chose in the first step, and can be found under Resources Storage resources Resource name:

   

1. Section 7.7.3.1, “Create an external NooBaa Daemon pool on Cluster B”
2. Section 7.7.3.2, “Adding Cluster B as a backing store to Cluster A”
3. Section 7.7.3.3, “Create a federated bucket on Cluster A using Cluster B’s NooBaa system”
4. Section 7.7.3.4, “Assign regions for the locality optimization”
5. Section 7.7.3.5, “Managing the clusters”