Managing hybrid and multicloud resources

1. Create a secret with the credentials:

   apiVersion: v1
   kind: Secret
   metadata:
     name: <backingstore-secret-name>
     namespace: openshift-storage
   type: Opaque
   data:
     AWS_ACCESS_KEY_ID: <AWS ACCESS KEY ID ENCODED IN BASE64>
     AWS_SECRET_ACCESS_KEY: <AWS SECRET ACCESS KEY ENCODED IN BASE64>

   <AWS ACCESS KEY> and <AWS SECRET ACCESS KEY>

   Supply and encode your own AWS access key ID and secret access key using Base64, and use the results for <AWS ACCESS KEY ID ENCODED IN BASE64> and <AWS SECRET ACCESS KEY ENCODED IN BASE64>.

   <backingstore-secret-name>

   The name of the backingstore secret created in the previous step.

2. Apply the following YAML for a specific backing store:

   apiVersion: noobaa.io/v1alpha1
   kind: BackingStore
   metadata:
     finalizers:
     - noobaa.io/finalizer
     labels:
       app: noobaa
     name: bs
     namespace: openshift-storage
   spec:
     awsS3:
       secret:
         name: <backingstore-secret-name>
         namespace: openshift-storage
       targetBucket: <bucket-name>
     type: aws-s3

   <bucket-name>

   The existing AWS bucket name.

   <backingstore-secret-name>

   The name of the backingstore secret created in the previous step.

1. From the MCG command-line interface, run the following command:

   noobaa backingstore create ibm-cos <backingstore_name> --access-key=<IBM ACCESS KEY> --secret-key=<IBM SECRET ACCESS KEY> --endpoint=<IBM COS ENDPOINT> --target-bucket <bucket-name> -n openshift-storage

   <backingstore_name>

   The name of the backingstore.

   <IBM ACCESS KEY>, <IBM SECRET ACCESS KEY>, and <IBM COS ENDPOINT>

   An IBM access key ID, secret access key and the appropriate regional endpoint that corresponds to the location of the existing IBM bucket.

   To generate the above keys on IBM cloud, you must include HMAC credentials while creating the service credentials for your target bucket.

   <bucket-name>

   An existing IBM bucket name. This argument indicates MCG about the bucket to use as a target bucket for its backing store, and subsequently, data storage and administration.

   The output will be similar to the following:

   INFO[0001] ✅ Exists: NooBaa "noobaa"
   INFO[0002] ✅ Created: BackingStore "ibm-resource"
   INFO[0002] ✅ Created: Secret "backing-store-secret-ibm-resource"

1. Create a secret with the credentials:

   apiVersion: v1
   kind: Secret
   metadata:
     name: <backingstore-secret-name>
     namespace: openshift-storage
   type: Opaque
   data:
     IBM_COS_ACCESS_KEY_ID: <IBM COS ACCESS KEY ID ENCODED IN BASE64>
     IBM_COS_SECRET_ACCESS_KEY: <IBM COS SECRET ACCESS KEY ENCODED IN BASE64>

   <IBM COS ACCESS KEY ID ENCODED IN BASE64> and <IBM COS SECRET ACCESS KEY ENCODED IN BASE64>

   Provide and encode your own IBM COS access key ID and secret access key using Base64, and use the results in place of these attributes respectively.

   <backingstore-secret-name>

   The name of the backingstore secret.

2. Apply the following YAML for a specific backing store:

   apiVersion: noobaa.io/v1alpha1
   kind: BackingStore
   metadata:
     finalizers:
     - noobaa.io/finalizer
     labels:
       app: noobaa
     name: bs
     namespace: openshift-storage
   spec:
     ibmCos:
       endpoint: <endpoint>
       secret:
         name: <backingstore-secret-name>
         namespace: openshift-storage
       targetBucket: <bucket-name>
     type: ibm-cos

   <bucket-name>

   an existing IBM COS bucket name. This argument indicates to MCG about the bucket to use as a target bucket for its backingstore, and subsequently, data storage and administration.

   <endpoint>

   A regional endpoint that corresponds to the location of the existing IBM bucket name. This argument indicates to MCG about the endpoint to use for its backingstore, and subsequently, data storage and administration.

   <backingstore-secret-name>

   The name of the secret created in the previous step.

1. Create a secret with the credentials:

   apiVersion: v1
   kind: Secret
   metadata:
     name: <backingstore-secret-name>
   type: Opaque
   data:
     AccountName: <AZURE ACCOUNT NAME ENCODED IN BASE64>
     AccountKey: <AZURE ACCOUNT KEY ENCODED IN BASE64>

   <AZURE ACCOUNT NAME ENCODED IN BASE64> and <AZURE ACCOUNT KEY ENCODED IN BASE64>

   Supply and encode your own Azure Account Name and Account Key using Base64, and use the results in place of these attributes respectively.

   <backingstore-secret-name>

   A unique name of backingstore secret.

2. Apply the following YAML for a specific backing store:

   apiVersion: noobaa.io/v1alpha1
   kind: BackingStore
   metadata:
     finalizers:
     - noobaa.io/finalizer
     labels:
       app: noobaa
     name: bs
     namespace: openshift-storage
   spec:
     azureBlob:
       secret:
         name: <backingstore-secret-name>
         namespace: openshift-storage
       targetBlobContainer: <blob-container-name>
     type: azure-blob

   <blob-container-name>

   An existing Azure blob container name. This argument indicates to the MCG about the bucket to use as a target bucket for its backingstore, and subsequently, data storage and administration.

   <backingstore-secret-name>

   with the name of the secret created in the previous step.

1. Create a secret with the credentials:

   apiVersion: v1
   kind: Secret
   metadata:
     name: <backingstore-secret-name>
   type: Opaque
   data:
     GoogleServiceAccountPrivateKeyJson: <GCP PRIVATE KEY ENCODED IN BASE64>

   <GCP PRIVATE KEY ENCODED IN BASE64>

   Provide and encode your own GCP service account private key using Base64, and use the results for this attribute.

   <backingstore-secret-name>

   A unique name of the backingstore secret.

2. Apply the following YAML for a specific backing store:

   apiVersion: noobaa.io/v1alpha1
   kind: BackingStore
   metadata:
     finalizers:
     - noobaa.io/finalizer
     labels:
       app: noobaa
     name: bs
     namespace: openshift-storage
   spec:
     googleCloudStorage:
       secret:
         name: <backingstore-secret-name>
         namespace: openshift-storage
       targetBucket: <target bucket>
     type: google-cloud-storage

   <target bucket>

   An existing Google storage bucket. This argument indicates to the MCG about the bucket to use as a target bucket for its backing store, and subsequently, data storage dfdand administration.

   <backingstore-secret-name>

   The name of the secret created in the previous step.

Procedure

1. Create a secret with the credentials:

   apiVersion: v1
   kind: Secret
   metadata:
     name: <namespacestore-secret-name>
     type: Opaque
   data:
     AWS_ACCESS_KEY_ID: <AWS ACCESS KEY ID ENCODED IN BASE64>
     AWS_SECRET_ACCESS_KEY: <AWS SECRET ACCESS KEY ENCODED IN BASE64>

   where <namespacestore-secret-name> is a unique NamespaceStore name.

   You must provide and encode your own AWS access key ID and secret access key using Base64, and use the results in place of <AWS ACCESS KEY ID ENCODED IN BASE64> and <AWS SECRET ACCESS KEY ENCODED IN BASE64>.

2. Create a NamespaceStore resource using OpenShift custom resource definitions (CRDs).

   A NamespaceStore represents underlying storage to be used as a read or write target for the data in the MCG namespace buckets.

   To create a NamespaceStore resource, apply the following YAML:

   apiVersion: noobaa.io/v1alpha1
   kind: NamespaceStore
   metadata:
     finalizers:
     - noobaa.io/finalizer
     labels:
       app: noobaa
     name: <resource-name>
     namespace: openshift-storage
   spec:
     awsS3:
       secret:
         name: <namespacestore-secret-name>
         namespace: <namespace-secret>
       targetBucket: <target-bucket>
     type: aws-s3

   <resource-name>

   The name you want to give to the resource.

   <namespacestore-secret-name>

   The secret created in the previous step.

   <namespace-secret>

   The namespace where the secret can be found.

   <target-bucket>

   The target bucket you created for the NamespaceStore.

3. Create a namespace bucket class that defines a namespace policy for the namespace buckets. The namespace policy requires a type of either single or multi.

   • A namespace policy of type single requires the following configuration:

     apiVersion: noobaa.io/v1alpha1
     kind: BucketClass
     metadata:
       labels:
         app: noobaa
       name: <my-bucket-class>
       namespace: openshift-storage
     spec:
       namespacePolicy:
         type:
         single:
           resource: <resource>

     <my-bucket-class>

     The unique namespace bucket class name.

     <resource>

     The name of a single NamespaceStore that defines the read and write target of the namespace bucket.

   • A namespace policy of type multi requires the following configuration:

     apiVersion: noobaa.io/v1alpha1
     
     kind: BucketClass
     metadata:
       labels:
         app: noobaa
       name: <my-bucket-class>
       namespace: openshift-storage
     spec:
       namespacePolicy:
         type: Multi
         multi:
           writeResource: <write-resource>
           readResources:
           - <read-resources>
           - <read-resources>

     <my-bucket-class>

     A unique bucket class name.

     <write-resource>

     The name of a single NamespaceStore that defines the write target of the namespace bucket.

     <read-resources>

     A list of the names of the NamespaceStores that defines the read targets of the namespace bucket.

4. Create a bucket using an Object Bucket Class (OBC) resource that uses the bucket class defined in the earlier step using the following YAML:

   apiVersion: objectbucket.io/v1alpha1
   kind: ObjectBucketClaim
   metadata:
     name: <resource-name>
     namespace: openshift-storage
   spec:
     generateBucketName: <my-bucket>
     storageClassName: openshift-storage.noobaa.io
     additionalConfig:
       bucketclass: <my-bucket-class>

   <resource-name>

   The name you want to give to the resource.

   <my-bucket>

   The name you want to give to the bucket.

   <my-bucket-class>

   The bucket class created in the previous step.

Procedure

1. Create a secret with the credentials:

   apiVersion: v1
   kind: Secret
   metadata:
     name: <namespacestore-secret-name>
     type: Opaque
   data:
     IBM_COS_ACCESS_KEY_ID: <IBM COS ACCESS KEY ID ENCODED IN BASE64>
     IBM_COS_SECRET_ACCESS_KEY: <IBM COS SECRET ACCESS KEY ENCODED IN BASE64>

   <namespacestore-secret-name>

   A unique NamespaceStore name.

   You must provide and encode your own IBM COS access key ID and secret access key using Base64, and use the results in place of <IBM COS ACCESS KEY ID ENCODED IN BASE64> and <IBM COS SECRET ACCESS KEY ENCODED IN BASE64>.

2. Create a NamespaceStore resource using OpenShift custom resource definitions (CRDs).

   A NamespaceStore represents underlying storage to be used as a read or write target for the data in the MCG namespace buckets.

   To create a NamespaceStore resource, apply the following YAML:

   apiVersion: noobaa.io/v1alpha1
   kind: NamespaceStore
   metadata:
     finalizers:
     - noobaa.io/finalizer
     labels:
       app: noobaa
     name: bs
     namespace: openshift-storage
   spec:
     s3Compatible:
       endpoint: <IBM COS ENDPOINT>
       secret:
         name: <namespacestore-secret-name>
         namespace: <namespace-secret>
       signatureVersion: v2
       targetBucket: <target-bucket>
     type: ibm-cos

   <IBM COS ENDPOINT>

   The appropriate IBM COS endpoint.

   <namespacestore-secret-name>

   The secret created in the previous step.

   <namespace-secret>

   The namespace where the secret can be found.

   <target-bucket>

   The target bucket you created for the NamespaceStore.

3. Create a namespace bucket class that defines a namespace policy for the namespace buckets. The namespace policy requires a type of either single or multi.

   • The namespace policy of type single requires the following configuration:

     apiVersion: noobaa.io/v1alpha1
     kind: BucketClass
     metadata:
       labels:
         app: noobaa
       name: <my-bucket-class>
       namespace: openshift-storage
     spec:
       namespacePolicy:
         type:
         single:
           resource: <resource>

     <my-bucket-class>

     The unique namespace bucket class name.

     <resource>

     The name of a single NamespaceStore that defines the read and write target of the namespace bucket.

   • The namespace policy of type multi requires the following configuration:

     apiVersion: noobaa.io/v1alpha1
     kind: BucketClass
     metadata:
       labels:
         app: noobaa
       name: <my-bucket-class>
       namespace: openshift-storage
     spec:
       namespacePolicy:
         type: Multi
         multi:
           writeResource: <write-resource>
           readResources:
           - <read-resources>
           - <read-resources>

     <my-bucket-class>

     The unique bucket class name.

     <write-resource>

     The name of a single NamespaceStore that defines the write target of the namespace bucket.

     <read-resources>

     A list of the NamespaceStores names that defines the read targets of the namespace bucket.

4. To create a bucket using an Object Bucket Class (OBC) resource that uses the bucket class defined in the previous step, apply the following YAML:

   apiVersion: objectbucket.io/v1alpha1
   kind: ObjectBucketClaim
   metadata:
     name: <resource-name>
     namespace: openshift-storage
   spec:
     generateBucketName: <my-bucket>
     storageClassName: openshift-storage.noobaa.io
     additionalConfig:
       bucketclass: <my-bucket-class>

   <resource-name>

   The name you want to give to the resource.

   <my-bucket>

   The name you want to give to the bucket.

   <my-bucket-class>

   The bucket class created in the previous step.

   After the OBC is provisioned by the operator, a bucket is created in the MCG, and the operator creates a Secret and ConfigMap with the same name and in the same namespace as that of the OBC.

Procedure

1. In the MCG command-line interface, create a NamespaceStore resource.

   A NamespaceStore represents an underlying storage to be used as a read or write target for the data in MCG namespace buckets.

   $ noobaa namespacestore create aws-s3 <namespacestore> --access-key <AWS ACCESS KEY> --secret-key <AWS SECRET ACCESS KEY> --target-bucket <bucket-name> -n openshift-storage

   <namespacestore>

   The name of the NamespaceStore.

   <AWS ACCESS KEY> and <AWS SECRET ACCESS KEY>

   The AWS access key ID and secret access key you created for this purpose.

   <bucket-name>

   The existing AWS bucket name. This argument tells the MCG which bucket to use as a target bucket for its backing store, and subsequently, data storage and administration.

2. Create a namespace bucket class that defines a namespace policy for the namespace buckets. The namespace policy can be either single or multi.

   • To create a namespace bucket class with a namespace policy of type single:

     $ noobaa bucketclass create namespace-bucketclass single <my-bucket-class> --resource <resource> -n openshift-storage

     <resource-name>

     The name you want to give the resource.

     <my-bucket-class>

     A unique bucket class name.

     <resource>

     A single namespace-store that defines the read and write target of the namespace bucket.

   • To create a namespace bucket class with a namespace policy of type multi:

     $ noobaa bucketclass create namespace-bucketclass multi <my-bucket-class> --write-resource <write-resource> --read-resources <read-resources> -n openshift-storage

     <resource-name>

     The name you want to give the resource.

     <my-bucket-class>

     A unique bucket class name.

     <write-resource>

     A single namespace-store that defines the write target of the namespace bucket.

     <read-resources>s

     A list of namespace-stores separated by commas that defines the read targets of the namespace bucket.

3. Create a bucket using an Object Bucket Class (OBC) resource that uses the bucket class defined in the previous step.

   $ noobaa obc create my-bucket-claim -n openshift-storage --app-namespace my-app --bucketclass <custom-bucket-class>

   <bucket-name>

   A bucket name of your choice.

   <custom-bucket-class>

   The name of the bucket class created in the previous step.

   After the OBC is provisioned by the operator, a bucket is created in the MCG, and the operator creates a Secret and a ConfigMap with the same name and in the same namespace as that of the OBC.

Procedure

1. In the MCG command-line interface, create a NamespaceStore resource.

   A NamespaceStore represents an underlying storage to be used as a read or write target for the data in the MCG namespace buckets.

   $ noobaa namespacestore create ibm-cos <namespacestore> --endpoint <IBM COS ENDPOINT> --access-key <IBM ACCESS KEY> --secret-key <IBM SECRET ACCESS KEY> --target-bucket <bucket-name> -n openshift-storage

   <namespacestore>

   The name of the NamespaceStore.

   <IBM ACCESS KEY>, <IBM SECRET ACCESS KEY>, <IBM COS ENDPOINT>

   An IBM access key ID, secret access key, and the appropriate regional endpoint that corresponds to the location of the existing IBM bucket.

   <bucket-name>

   An existing IBM bucket name. This argument tells the MCG which bucket to use as a target bucket for its backing store, and subsequently, data storage and administration.

2. Create a namespace bucket class that defines a namespace policy for the namespace buckets. The namespace policy requires a type of either single or multi.

   • To create a namespace bucket class with a namespace policy of type single:

     $ noobaa bucketclass create namespace-bucketclass single <my-bucket-class> --resource <resource> -n openshift-storage

     <resource-name>

     The name you want to give the resource.

     <my-bucket-class>

     A unique bucket class name.

     <resource>

     A single NamespaceStore that defines the read and write target of the namespace bucket.

   • To create a namespace bucket class with a namespace policy of type multi:

     $ noobaa bucketclass create namespace-bucketclass multi <my-bucket-class> --write-resource <write-resource> --read-resources <read-resources> -n openshift-storage

     <resource-name>

     The name you want to give the resource.

     <my-bucket-class>

     A unique bucket class name.

     <write-resource>

     A single NamespaceStore that defines the write target of the namespace bucket.

     <read-resources>

     A comma-separated list of NamespaceStores that defines the read targets of the namespace bucket.

3. Create a bucket using an Object Bucket Class (OBC) resource that uses the bucket class defined in the earlier step.

   $ noobaa obc create my-bucket-claim -n openshift-storage --app-namespace my-app --bucketclass <custom-bucket-class>

   <bucket-name>

   A bucket name of your choice.

   <custom-bucket-class>

   The name of the bucket class created in the previous step.

Procedure

1. Log into the OpenShift Web Console.
2. Click Storage → Object Storage.
3. Click the NamespaceStore tab to create NamespaceStore resources to be used in the namespace bucket.
4. Click Create namespacestore.
5. Enter a name for the NamespaceStore.
6. Choose Filesystem as the provider.
7. Choose the Persistent volume claim.

8. Enter a folder name.

   If the folder name exists, then that folder is used to create the NamespaceStore or else a folder with that name is created.

9. Click Create.
10. Verify the NamespaceStore is in the Ready state.

Procedure

1. Display the application namespace with scc:

   $ oc get ns <application_namespace> -o yaml | grep scc

   <application_namespace>

   Specify the name of the application namespace.

   For example:

   $ oc get ns testnamespace -o yaml | grep scc
   
   openshift.io/sa.scc.mcs: s0:c26,c5
   openshift.io/sa.scc.supplemental-groups: 1000660000/10000
   openshift.io/sa.scc.uid-range: 1000660000/10000

2. Navigate into the application namespace:

   $ oc project <application_namespace>

   For example:

   $ oc project testnamespace

3. Ensure that a ReadWriteMany (RWX) PVC is mounted on the pod that you want to consume from the noobaa S3 endpoint using the MCG NSFS feature:

   $ oc get pvc
   
   NAME                                               STATUS VOLUME
   CAPACITY ACCESS MODES STORAGECLASS              AGE
   cephfs-write-workload-generator-no-cache-pv-claim  Bound  pvc-aa58fb91-c3d2-475b-bbee-68452a613e1a
   10Gi     RWX          ocs-storagecluster-cephfs 12s

   $ oc get pod
   
   NAME                                                READY   STATUS              RESTARTS   AGE
   cephfs-write-workload-generator-no-cache-1-cv892    1/1     Running             0          11s

4. Check the mount point of the Persistent Volume (PV) inside your pod.

   1. Get the volume name of the PV from the pod:

      $ oc get pods <pod_name> -o jsonpath='{.spec.volumes[]}'

      <pod_name>

      Specify the name of the pod.

      For example:

      $ oc get pods cephfs-write-workload-generator-no-cache-1-cv892 -o jsonpath='{.spec.volumes[]}'
      
      {"name":"app-persistent-storage","persistentVolumeClaim":{"claimName":"cephfs-write-workload-generator-no-cache-pv-claim"}}

      In this example, the name of the volume for the PVC is cephfs-write-workload-generator-no-cache-pv-claim.

   2. List all the mounts in the pod, and check for the mount point of the volume that you identified in the previous step:

      $ oc get pods <pod_name> -o jsonpath='{.spec.containers[].volumeMounts}'

      For example:

      $ oc get pods cephfs-write-workload-generator-no-cache-1-cv892 -o jsonpath='{.spec.containers[].volumeMounts}'
      
      [{"mountPath":"/mnt/pv","name":"app-persistent-storage"},{"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount","name":"kube-api-access-8tnc5","readOnly":true}]

5. Confirm the mount point of the RWX PV in your pod:

   $ oc exec -it <pod_name> -- df <mount_path>

   <mount_path>

   Specify the path to the mount point that you identified in the previous step.

   For example:

   $ oc exec -it cephfs-write-workload-generator-no-cache-1-cv892 -- df /mnt/pv
   
   main
   Filesystem
   1K-blocks Used Available  Use%  Mounted on
   172.30.202.87:6789,172.30.120.254:6789,172.30.77.247:6789:/volumes/csi/csi-vol-cc416d9e-dbf3-11ec-b286-0a580a810213/edcfe4d5-bdcb-4b8e-8824-8a03ad94d67c
   10485760  0    10485760   0%    /mnt/pv

6. Ensure that the UID and SELinux labels are the same as the ones that the legacy namespace uses:

   $ oc exec -it <pod_name> -- ls -latrZ <mount_path>

   For example:

   $ oc exec -it cephfs-write-workload-generator-no-cache-1-cv892 -- ls -latrZ /mnt/pv/
   
   total 567
   drwxrwxrwx. 3 root       root system_u:object_r:container_file_t:s0:c26,c5      2 May 25 06:35 .
   -rw-r--r--. 1 1000660000 root system_u:object_r:container_file_t:s0:c26,c5 580138 May 25 06:35 fs_write_cephfs-write-workload-generator-no-cache-1-cv892-data.log
   drwxrwxrwx. 3 root       root system_u:object_r:container_file_t:s0:c26,c5     30 May 25 06:35 ..

7. Get the information of the legacy application RWX PV that you want to make accessible from the openshift-storage namespace:

   $ oc get pv | grep <pv_name>

   <pv_name>

   Specify the name of the PV.

   For example:

   $ oc get pv | grep pvc-aa58fb91-c3d2-475b-bbee-68452a613e1a
   
   pvc-aa58fb91-c3d2-475b-bbee-68452a613e1a   10Gi       RWX            Delete           Bound    testnamespace/cephfs-write-workload-generator-no-cache-pv-claim   ocs-storagecluster-cephfs              47s

8. Ensure that the PVC from the legacy application is accessible from the openshift-storage namespace so that one or more noobaa-endpoint pods can access the PVC.

   1. Find the values of the subvolumePath and volumeHandle from the volumeAttributes. You can get these values from the YAML description of the legacy application PV:

      $ oc get pv <pv_name> -o yaml

      For example:

      $ oc get pv pvc-aa58fb91-c3d2-475b-bbee-68452a613e1a -o yaml
      
      apiVersion: v1
      kind: PersistentVolume
      metadata:
        annotations:
          pv.kubernetes.io/provisioned-by: openshift-storage.cephfs.csi.ceph.com
        creationTimestamp: "2022-05-25T06:27:49Z"
        finalizers:
        - kubernetes.io/pv-protection
        name: pvc-aa58fb91-c3d2-475b-bbee-68452a613e1a
        resourceVersion: "177458"
        uid: 683fa87b-5192-4ccf-af2f-68c6bcf8f500
      spec:
        accessModes:
        - ReadWriteMany
        capacity:
          storage: 10Gi
        claimRef:
          apiVersion: v1
          kind: PersistentVolumeClaim
          name: cephfs-write-workload-generator-no-cache-pv-claim
          namespace: testnamespace
          resourceVersion: "177453"
          uid: aa58fb91-c3d2-475b-bbee-68452a613e1a
        csi:
          controllerExpandSecretRef:
            name: rook-csi-cephfs-provisioner
            namespace: openshift-storage
          driver: openshift-storage.cephfs.csi.ceph.com
          nodeStageSecretRef:
            name: rook-csi-cephfs-node
            namespace: openshift-storage
          volumeAttributes:
            clusterID: openshift-storage
            fsName: ocs-storagecluster-cephfilesystem
            storage.kubernetes.io/csiProvisionerIdentity: 1653458225664-8081-openshift-storage.cephfs.csi.ceph.com
            subvolumeName: csi-vol-cc416d9e-dbf3-11ec-b286-0a580a810213
            subvolumePath: /volumes/csi/csi-vol-cc416d9e-dbf3-11ec-b286-0a580a810213/edcfe4d5-bdcb-4b8e-8824-8a03ad94d67c
          volumeHandle: 0001-0011-openshift-storage-0000000000000001-cc416d9e-dbf3-11ec-b286-0a580a810213
        persistentVolumeReclaimPolicy: Delete
        storageClassName: ocs-storagecluster-cephfs
        volumeMode: Filesystem
      status:
        phase: Bound

   2. Use the subvolumePath and volumeHandle values that you identified in the previous step to create a new PV and PVC object in the openshift-storage namespace that points to the same CephFS volume as the legacy application PV:

      Example YAML file:

      $ cat << EOF >> pv-openshift-storage.yaml
      apiVersion: v1
      kind: PersistentVolume
      metadata:
        name: cephfs-pv-legacy-openshift-storage
      spec:
        storageClassName: ""
        accessModes:
        - ReadWriteMany
        capacity:
          storage: 10Gi     1
        csi:
          driver: openshift-storage.cephfs.csi.ceph.com
          nodeStageSecretRef:
            name: rook-csi-cephfs-node
            namespace: openshift-storage
          volumeAttributes:
          # Volume Attributes can be copied from the Source testnamespace PV
            "clusterID": "openshift-storage"
            "fsName": "ocs-storagecluster-cephfilesystem"
            "staticVolume": "true"
          # rootpath is the subvolumePath: you copied from the Source testnamespace PV
            "rootPath": /volumes/csi/csi-vol-cc416d9e-dbf3-11ec-b286-0a580a810213/edcfe4d5-bdcb-4b8e-8824-8a03ad94d67c
          volumeHandle: 0001-0011-openshift-storage-0000000000000001-cc416d9e-dbf3-11ec-b286-0a580a810213-clone   2
        persistentVolumeReclaimPolicy: Retain
        volumeMode: Filesystem
      ---
      apiVersion: v1
      kind: PersistentVolumeClaim
      metadata:
        name: cephfs-pvc-legacy
        namespace: openshift-storage
      spec:
        storageClassName: ""
        accessModes:
        - ReadWriteMany
        resources:
          requests:
            storage: 10Gi     3
        volumeMode: Filesystem
        # volumeName should be same as PV name
        volumeName: cephfs-pv-legacy-openshift-storage
      EOF

      1

      The storage capacity of the PV that you are creating in the openshift-storage namespace must be the same as the original PV.

      2

      The volume handle for the target PV that you create in openshift-storage needs to have a different handle than the original application PV, for example, add -clone at the end of the volume handle.

      3

      The storage capacity of the PVC that you are creating in the openshift-storage namespace must be the same as the original PVC.

   3. Create the PV and PVC in the openshift-storage namespace using the YAML file specified in the previous step:

      $ oc create -f <YAML_file>

      <YAML_file>

      Specify the name of the YAML file.

      For example:

      $ oc create -f pv-openshift-storage.yaml
      
      persistentvolume/cephfs-pv-legacy-openshift-storage created
      persistentvolumeclaim/cephfs-pvc-legacy created

   4. Ensure that the PVC is available in the openshift-storage namespace:

      $ oc get pvc -n openshift-storage
      
      NAME                                  STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                  AGE
      cephfs-pvc-legacy                     Bound    cephfs-pv-legacy-openshift-storage         10Gi       RWX                                          14s

   5. Navigate into the openshift-storage project:

      $ oc project openshift-storage
      
      Now using project "openshift-storage" on server "https://api.cluster-5f6ng.5f6ng.sandbox65.opentlc.com:6443".

   6. Create the NSFS namespacestore:

      $ noobaa namespacestore create nsfs <nsfs_namespacestore> --pvc-name='<cephfs_pvc_name>' --fs-backend='CEPH_FS'

      <nsfs_namespacestore>

      Specify the name of the NSFS namespacestore.

      <cephfs_pvc_name>

      Specify the name of the CephFS PVC in the openshift-storage namespace.

      For example:

      $ noobaa namespacestore create nsfs legacy-namespace --pvc-name='cephfs-pvc-legacy' --fs-backend='CEPH_FS'

   7. Ensure that the noobaa-endpoint pod restarts and that it successfully mounts the PVC at the NSFS namespacestore, for example, /nsfs/legacy-namespace mountpoint:

      $ oc exec -it <noobaa_endpoint_pod_name> -- df -h /nsfs/<nsfs_namespacestore>

      <noobaa_endpoint_pod_name>

      Specify the name of the noobaa-endpoint pod.

      For example:

      $ oc exec -it noobaa-endpoint-5875f467f5-546c6 -- df -h /nsfs/legacy-namespace
      
      Filesystem                                                                                                                                                Size  Used Avail Use% Mounted on
      172.30.202.87:6789,172.30.120.254:6789,172.30.77.247:6789:/volumes/csi/csi-vol-cc416d9e-dbf3-11ec-b286-0a580a810213/edcfe4d5-bdcb-4b8e-8824-8a03ad94d67c   10G     0   10G   0% /nsfs/legacy-namespace

   8. Create a MCG user account:

      $ noobaa account create <user_account> --full_permission --allow_bucket_create=true --new_buckets_path='/' --nsfs_only=true --nsfs_account_config=true --gid <gid_number> --uid <uid_number> --default_resource='legacy-namespace'

      <user_account>

      Specify the name of the MCG user account.

      <gid_number>

      Specify the GID number.

      <uid_number>

      Specify the UID number.

      Important

      Use the same UID and GID as that of the legacy application. You can find it from the previous output.

      For example:

      $ noobaa account create leguser --full_permission --allow_bucket_create=true --new_buckets_path='/' --nsfs_only=true --nsfs_account_config=true --gid 0 --uid 1000660000 --default_resource='legacy-namespace'

   9. Create a MCG bucket.

      1. Create a dedicated folder for S3 inside the NSFS share on the CephFS PV and PVC of the legacy application pod:

         $ oc exec -it <pod_name> -- mkdir <mount_path>/nsfs

         For example:

         $ oc exec -it cephfs-write-workload-generator-no-cache-1-cv892 -- mkdir /mnt/pv/nsfs

      2. Create the MCG bucket using the nsfs/ path:

         $ noobaa api bucket_api create_bucket '{
           "name": "<bucket_name>",
           "namespace":{
             "write_resource": { "resource": "<nsfs_namespacestore>", "path": "nsfs/" },
             "read_resources": [ { "resource": "<nsfs_namespacestore>", "path": "nsfs/" }]
           }
         }'

         For example:

         $ noobaa api bucket_api create_bucket '{
           "name": "legacy-bucket",
           "namespace":{
             "write_resource": { "resource": "legacy-namespace", "path": "nsfs/" },
             "read_resources": [ { "resource": "legacy-namespace", "path": "nsfs/" }]
           }
         }'

   10. Check the SELinux labels of the folders residing in the PVCs in the legacy application and openshift-storage namespaces:

       $ oc exec -it <noobaa_endpoint_pod_name> -n openshift-storage -- ls -ltraZ /nsfs/<nsfs_namespacstore>

       For example:

       $ oc exec -it noobaa-endpoint-5875f467f5-546c6 -n openshift-storage -- ls -ltraZ /nsfs/legacy-namespace
       
       total 567
       drwxrwxrwx. 3 root       root system_u:object_r:container_file_t:s0:c0,c26      2 May 25 06:35 .
       -rw-r--r--. 1 1000660000 root system_u:object_r:container_file_t:s0:c0,c26 580138 May 25 06:35 fs_write_cephfs-write-workload-generator-no-cache-1-cv892-data.log
       drwxrwxrwx. 3 root       root system_u:object_r:container_file_t:s0:c0,c26     30 May 25 06:35 ..

       $ oc exec -it <pod_name> -- ls -latrZ <mount_path>

       For example:

       $ oc exec -it cephfs-write-workload-generator-no-cache-1-cv892 -- ls -latrZ /mnt/pv/
       
       total 567
       drwxrwxrwx. 3 root       root system_u:object_r:container_file_t:s0:c26,c5      2 May 25 06:35 .
       -rw-r--r--. 1 1000660000 root system_u:object_r:container_file_t:s0:c26,c5 580138 May 25 06:35 fs_write_cephfs-write-workload-generator-no-cache-1-cv892-data.log
       drwxrwxrwx. 3 root       root system_u:object_r:container_file_t:s0:c26,c5     30 May 25 06:35 ..

       In these examples, you can see that the SELinux labels are not the same which results in permission denied or access issues.

9. Ensure that the legacy application and openshift-storage pods use the same SELinux labels on the files.

   You can do this in one of the following ways:

   • Section 5.4.3.1, “Changing the default SELinux label on the legacy application project to match the one in the openshift-storage project”.
   • Section 5.4.3.2, “Modifying the SELinux label only for the deployment config that has the pod which mounts the legacy application PVC”.

10. Delete the NSFS namespacestore:

    1. Delete the MCG bucket:

       $ noobaa bucket delete <bucket_name>

       For example:

       $ noobaa bucket delete legacy-bucket

    2. Delete the MCG user account:

       $ noobaa account delete <user_account>

       For example:

       $ noobaa account delete leguser

    3. Delete the NSFS namespacestore:

       $ noobaa namespacestore delete <nsfs_namespacestore>

       For example:

       $ noobaa namespacestore delete legacy-namespace

11. Delete the PV and PVC:

    Important

    Before you delete the PV and PVC, ensure that the PV has a retain policy configured.

    $ oc delete pv <cephfs_pv_name>

    $ oc delete pvc <cephfs_pvc_name>

    <cephfs_pv_name>

    Specify the CephFS PV name of the legacy application.

    <cephfs_pvc_name>

    Specify the CephFS PVC name of the legacy application.

    For example:

    $ oc delete pv cephfs-pv-legacy-openshift-storage

    $ oc delete pvc cephfs-pvc-legacy

Procedure

1. Get the account name.

   For listing the accounts, run the following command:

   $ noobaa account list

   Example output:

   NAME           ALLOWED_BUCKETS   DEFAULT_RESOURCE               PHASE   AGE
   account-test   [*]               noobaa-default-backing-store   Ready   14m17s
   test2          [first.bucket]    noobaa-default-backing-store   Ready   3m12s

   Alternatively, run the oc get noobaaaccount command from the terminal:

   $ oc get noobaaaccount

   Example output:

   NAME           PHASE   AGE
   account-test   Ready   15m
   test2          Ready   3m59s

2. To regenerate the noobaa account S3 credentials, run the following command:

   $ noobaa account regenerate <noobaa_account_name> [options]

   $ noobaa account regenerate
   FATA[0000] ❌ Missing expected arguments: <noobaa-account-name>
   
   Usage:
       noobaa account regenerate <noobaa-account-name> [flags] [options]
   
   Use "noobaa options" for a list of global command-line options (applies to all commands).

3. Once you run the noobaa account regenerate command it will prompt a warning that says "This will invalidate all connections between S3 clients and NooBaa which are connected using the current credentials.", and ask for confirmation:

   Example:

   $ noobaa account regenerate account-test

   Example output:

   INFO[0000] You are about to regenerate an account's security credentials.
   INFO[0000] This will invalidate all connections between S3 clients and NooBaa which are connected using the current credentials.
   INFO[0000] are you sure? y/n

4. On approving, it will regenerate the credentials and eventually print them:

   INFO[0015] ✅ Exists: Secret "noobaa-account-account-test"
   Connection info:
   AWS_ACCESS_KEY_ID      : ***
   AWS_SECRET_ACCESS_KEY  : ***

Procedure

1. To get the OBC name, run the following command:

   $ noobaa obc list

   Example output:

   NAMESPACE   NAME       BUCKET-NAME                                     STORAGE-CLASS       BUCKET-CLASS                  PHASE
   default     obc-test   obc-test-35800e50-8978-461f-b7e0-7793080e26ba   default.noobaa.io   noobaa-default-bucket-class   Bound

   Alternatively, run the oc get obc command from the terminal:

   $ oc get obc

   Example output:

   NAME       STORAGE-CLASS       PHASE   AGE
   obc-test   default.noobaa.io   Bound   38s

2. To regenerate the noobaa OBC S3 credentials, run the following command:

   $ noobaa obc regenerate <bucket_claim_name> [options]

   $ noobaa obc regenerate
   FATA[0000] ❌ Missing expected arguments: <bucket-claim-name>
   
   Usage:
      noobaa obc regenerate <bucket-claim-name> [flags] [options]
   
   Use "noobaa options" for a list of global command-line options (applies to all commands).

3. Once you run the noobaa obc regenerate command it will prompt a warning that says "This will invalidate all connections between the S3 clients and noobaa which are connected using the current credentials.", and ask for confirmation:

   Example:

   $ noobaa obc regenerate obc-test

   Example output:

   INFO[0000] You are about to regenerate an OBC's security credentials.
   INFO[0000] This will invalidate all connections between S3 clients and NooBaa which are connected using the current credentials.
   INFO[0000] are you sure? y/n

4. On approving, it will regenerate the credentials and eventually print them:

   INFO[0022] ✅ RPC: bucket.read_bucket() Response OK: took 95.4ms
   
   ObjectBucketClaim info:
     Phase                  : Bound
     ObjectBucketClaim      : kubectl get -n default objectbucketclaim obc-test
     ConfigMap              : kubectl get -n default configmap obc-test
     Secret                 : kubectl get -n default secret obc-test
     ObjectBucket           : kubectl get objectbucket obc-default-obc-test
     StorageClass           : kubectl get storageclass default.noobaa.io
     BucketClass            : kubectl get -n default bucketclass noobaa-default-bucket-class
   
   Connection info:
    BUCKET_HOST            : s3.default.svc
    BUCKET_NAME            : obc-test-35800e50-8978-461f-b7e0-7793080e26ba
       BUCKET_PORT            : 443
       AWS_ACCESS_KEY_ID      : ***
       AWS_SECRET_ACCESS_KEY  : ***
   
   Shell commands:
     AWS S3 Alias           : alias s3='AWS_ACCESS_KEY_ID=***
   AWS_SECRET_ACCESS_KEY=*** aws s3 --no-verify-ssl --endpoint-url ***'
   
   Bucket status:
     Name                   : obc-test-35800e50-8978-461f-b7e0-7793080e26ba
     Type                   : REGULAR
     Mode                   : OPTIMAL
     ResiliencyStatus       : OPTIMAL
     QuotaStatus            : QUOTA_NOT_SET
     Num Objects            : 0
     Data Size              : 0.000 B
     Data Size Reduced      : 0.000 B
     Data Space Avail       : 13.261 GB
     Num Objects Avail      : 9007199254740991

Procedure

1. Apply the following YAML:

   apiVersion: noobaa.io/v1alpha1
   kind: BucketClass
   metadata:
     labels:
       app: <desired-app-label>
     name: <desired-bucketclass-name>
     namespace: <desired-namespace>
   spec:
     placementPolicy:
       tiers:
       - backingstores:
         - <backingstore>
         placement: Spread
     replicationPolicy: [{ "rule_id": "<rule id>", "destination_bucket": "first.bucket", "filter": {"prefix": "<object name prefix>"}}]

   This YAML is an example that creates a placement bucket class. Each Object bucket claim (OBC) object that is uploaded to the bucket is filtered based on the prefix and is replicated to first.bucket.

   <desired-app-label>

   Specify a label for the app.

   <desired-bucketclass-name>

   Specify the bucket class name.

   <desired-namespace>

   Specify the namespace in which the bucket class gets created.

   <backingstore>

   Specify the name of a backingstore. You can pass many backingstores.

   "rule_id"

   Specify the ID number of the rule, for example, `{"rule_id": "rule-1"}.

   "destination_bucket"

   Specify the name of the destination bucket, for example, {"destination_bucket": "first.bucket"}.

   "prefix"

   Is optional. The prefix of the object keys gets replicated. You can leave it empty, for example, {"prefix": ""}.

Procedure

1. In the OpenShift Web Console, navigate to Storage → Object Storage → Object Bucket Claims.
2. Click Create ObjectBucketClaim.
3. Enter the name of ObjectBucketName and select StorageClass and BucketClass.
4. Select the Enable replication check box to enable replication.
5. In the Replication policy section, select the Optimize replication using event logs checkbox.

6. Enter the name of the bucket that will contain the logs under Event log Bucket.

   If the logs are not stored in the root of the bucket, provide the full path without s3://

7. Enter a prefix to replicate only the objects whose name begins with the given prefix.

Procedure

1. Create a secret with credentials to be used by the namespacestores.

   apiVersion: v1
   kind: Secret
   metadata:
     name: <namespacestore-secret-name>
   type: Opaque
   data:
       TenantID: <AZURE TENANT ID ENCODED IN BASE64>
       ApplicationID: <AZURE APPLICATIOM ID ENCODED IN BASE64>
       ApplicationSecret: <AZURE APPLICATION SECRET ENCODED IN BASE64>
       LogsAnalyticsWorkspaceID: <AZURE LOG ANALYTICS WORKSPACE ID ENCODED IN BASE64>
       AccountName: <AZURE ACCOUNT NAME ENCODED IN BASE64>
       AccountKey: <AZURE ACCOUNT KEY ENCODED IN BASE64>

2. Create a NamespaceStore backed by a container created in Azure.

   For more information, see Adding a namespace bucket using the OpenShift Container Platform user interface.

3. Create a new Namespace-Bucketclass and OBC that utilizes it.
4. Check the object bucket name by looking in the YAML of target OBC, or by listing all S3 buckets, for example, - s3 ls.

5. Use the following template to apply an Azure replication policy on your source OBC by adding the following in its YAML, under .spec:

   replicationPolicy:'{"rules":[ {"rule_id":"ID goes here", "sync_deletions": "<true or false>"", "destination_bucket":object bucket name"}
    ], "log_replication_info":{"endpoint_type":"AZURE"}}'

   sync_deletion

   Specify a boolean value, true or false.

   destination_bucket

   Make sure to use the name of the object bucket, and not the claim. The name can be retrieved using the s3 ls command, or by looking for the value in an OBC’s YAML.

Verification steps

1. Write objects to the source bucket.
2. Wait until MCG replicates them.
3. Delete the objects from the source bucket.
4. Verify the objects were removed from the target bucket.

Procedure

1. In the OpenShift Web Console, navigate to Storage → Object Storage → Object Bucket Claims.
2. Click Create new Object bucket claim.
3. (Optional) In the Replication rules section, select the Sync deletion checkbox for each rule separately.

4. Enter the name of the bucket that will contain the logs under Event log Bucket.

   If the logs are not stored in the root of the bucket, provide the full path without s3://

5. Enter a prefix to replicate only the objects whose name begins with the given prefix.

Procedure

1. Create a data bucket where you can upload the objects.

   nb bucket create data.bucket

2. Create a log bucket where you want to store the logs for bucket operations by using the following command:

   nb bucket create log.bucket

3. Configure bucket logging on data bucket with log bucket in one of the following ways:

   • Using the NooBaa API

     nb api bucket_api put_bucket_logging '{
        "name": "data.bucket",
        "log_bucket": "log.bucket",
        "log_prefix": "data-bucket-logs"
     }'

   • Using the S3 API

     alias s3api_alias='AWS_ACCESS_KEY_ID=$NOOBAA_ACCESS_KEY AWS_SECRET_ACCESS_KEY=$NOOBAA_SECRET_KEY aws --endpoint https://localhost:10443 --no-verify-ssl s3api'

     1. Create a file called setlogging.json in the following format:

        {
          "LoggingEnabled": {
             "TargetBucket": "<log-bucket-name>",
             "TargetPrefix": "<prefix/empty-string>"
          }
        }

     2. Run the following command:

        s3api_alias put-bucket-logging --endpoint <ep> --bucket <source-bucket> --bucket-logging-status file://setlogging.json --no-verify-ssl

4. Verify if the bucket logging is set for the data bucket in one of the following ways:

   • Using the NooBaa API

     nb api bucket_api get_bucket_logging '{
        "name": "data.bucket"
     }'

   • Using the S3 API

     s3api_alias get-bucket-logging --no-verify-ssl --endpoint <ep> --bucket <source-bucket>

     The S3 operations can take up to 24 hours to get recorded in the logs bucket. The following example shows the recorded logs and how to download them:

     Example

     s3_alias cp s3://logs.bucket/data-bucket-logs/logs.bucket.bucket_data-bucket-logs_1719230150.log - | tail -n 2
     
     Jun 24 14:00:02 10-XXX-X-XXX.sts.openshift-storage.svc.cluster.local  {"noobaa_bucket_logging":"true","op":"GET","bucket_owner":"operator@noobaa.io","source_bucket":"data.bucket","object_key":"/data.bucket?list-type=2&prefix=data-bucket-logs&delimiter=%2F&encoding-type=url","log_bucket":"logs.bucket","remote_ip":"100.XX.X.X","request_uri":"/data.bucket?list-type=2&prefix=data-bucket-logs&delimiter=%2F&encoding-type=url","request_id":"luv2XXXX-ctyg2k-12gs"} Jun 24 14:00:06 10-XXX-X-XXX.s3.openshift-storage.svc.cluster.local  {"noobaa_bucket_logging":"true","op":"PUT","bucket_owner":"operator@noobaa.io","source_bucket":"data.bucket","object_key":"/data.bucket/B69EC83F-0177-44D8-A8D1-4A10C5A5AB0F.file","log_bucket":"logs.bucket","remote_ip":"100.XX.X.X","request_uri":"/data.bucket/B69EC83F-0177-44D8-A8D1-4A10C5A5AB0F.file","request_id":"luv2XXXX-9syea5-x5z"}

5. (Optional) To disable bucket logging, use the following command:

   nb api bucket_api delete_bucket_logging '{
      "name": "data.bucket"
   }'