Chapter 4. Managing namespace buckets
Namespace buckets let you connect data repositories on different providers together, so that you can interact with all of your data through a single unified view. Add the object bucket associated with each provider to the namespace bucket, and access your data through the namespace bucket to see all of your object buckets at once. This lets you write to your preferred storage provider while reading from multiple other storage providers, greatly reducing the cost of migrating to a new storage provider.
A namespace bucket can only be used if its write
target is available and functional.
4.1. Amazon S3 API endpoints for objects in namespace buckets
You can interact with objects in the namespace buckets using the Amazon Simple Storage Service (S3) API.
Red Hat OpenShift Data Foundation 4.6 onwards supports the following namespace bucket operations:
See the Amazon S3 API reference documentation for the most up-to-date information about these operations and how to use them.
Additional resources
4.2. Adding a namespace bucket using the Multicloud Object Gateway CLI and YAML
For more information about namespace buckets, see Managing namespace buckets.
Depending on the type of your deployment and whether you want to use YAML or the Multicloud Object Gateway (MCG) CLI, choose one of the following procedures to add a namespace bucket:
4.2.1. Adding an AWS S3 namespace bucket using YAML
Prerequisites
- Openshift Container Platform with OpenShift Data Foundation operator installed.
Access to the Multicloud Object Gateway (MCG).
For information, see Chapter 2, Accessing the Multicloud Object Gateway with your applications.
Procedure
Create a secret with the credentials:
apiVersion: v1 kind: Secret metadata: name: <namespacestore-secret-name> type: Opaque data: AWS_ACCESS_KEY_ID: <AWS ACCESS KEY ID ENCODED IN BASE64> AWS_SECRET_ACCESS_KEY: <AWS SECRET ACCESS KEY ENCODED IN BASE64>
where
<namespacestore-secret-name>
is a unique NamespaceStore name.You must provide and encode your own AWS access key ID and secret access key using
Base64
, and use the results in place of<AWS ACCESS KEY ID ENCODED IN BASE64>
and<AWS SECRET ACCESS KEY ENCODED IN BASE64>
.Create a NamespaceStore resource using OpenShift custom resource definitions (CRDs).
A NamespaceStore represents underlying storage to be used as a
read
orwrite
target for the data in the MCG namespace buckets.To create a NamespaceStore resource, apply the following YAML:
apiVersion: noobaa.io/v1alpha1 kind: NamespaceStore metadata: finalizers: - noobaa.io/finalizer labels: app: noobaa name: <resource-name> namespace: openshift-storage spec: awsS3: secret: name: <namespacestore-secret-name> namespace: <namespace-secret> targetBucket: <target-bucket> type: aws-s3
<resource-name>
- The name you want to give to the resource.
<namespacestore-secret-name>
- The secret created in the previous step.
<namespace-secret>
- The namespace where the secret can be found.
<target-bucket>
- The target bucket you created for the NamespaceStore.
Create a namespace bucket class that defines a namespace policy for the namespace buckets. The namespace policy requires a type of either
single
ormulti
.A namespace policy of type
single
requires the following configuration:apiVersion: noobaa.io/v1alpha1 kind: BucketClass metadata: labels: app: noobaa name: <my-bucket-class> namespace: openshift-storage spec: namespacePolicy: type: single: resource: <resource>
<my-bucket-class>
- The unique namespace bucket class name.
<resource>
- The name of a single NamespaceStore that defines the read and write target of the namespace bucket.
A namespace policy of type
multi
requires the following configuration:apiVersion: noobaa.io/v1alpha1 kind: BucketClass metadata: labels: app: noobaa name: <my-bucket-class> namespace: openshift-storage spec: namespacePolicy: type: Multi multi: writeResource: <write-resource> readResources: - <read-resources> - <read-resources>
<my-bucket-class>
- A unique bucket class name.
<write-resource>
-
The name of a single NamespaceStore that defines the
write
target of the namespace bucket. <read-resources>
-
A list of the names of the NamespaceStores that defines the
read
targets of the namespace bucket.
Create a bucket using an Object Bucket Class (OBC) resource that uses the bucket class defined in the earlier step using the following YAML:
apiVersion: objectbucket.io/v1alpha1 kind: ObjectBucketClaim metadata: name: <resource-name> namespace: openshift-storage spec: generateBucketName: <my-bucket> storageClassName: openshift-storage.noobaa.io additionalConfig: bucketclass: <my-bucket-class>
<resource-name>
- The name you want to give to the resource.
<my-bucket>
- The name you want to give to the bucket.
<my-bucket-class>
- The bucket class created in the previous step.
After the OBC is provisioned by the operator, a bucket is created in the MCG, and the operator creates a Secret
and ConfigMap
with the same name and in the same namespace as that of the OBC.
4.2.2. Adding an IBM COS namespace bucket using YAML
Prerequisites
- Openshift Container Platform with OpenShift Data Foundation operator installed.
- Access to the Multicloud Object Gateway (MCG), see Chapter 2, Accessing the Multicloud Object Gateway with your applications.
Procedure
Create a secret with the credentials:
apiVersion: v1 kind: Secret metadata: name: <namespacestore-secret-name> type: Opaque data: IBM_COS_ACCESS_KEY_ID: <IBM COS ACCESS KEY ID ENCODED IN BASE64> IBM_COS_SECRET_ACCESS_KEY: <IBM COS SECRET ACCESS KEY ENCODED IN BASE64>
<namespacestore-secret-name>
A unique NamespaceStore name.
You must provide and encode your own IBM COS access key ID and secret access key using
Base64
, and use the results in place of<IBM COS ACCESS KEY ID ENCODED IN BASE64>
and<IBM COS SECRET ACCESS KEY ENCODED IN BASE64>
.
Create a NamespaceStore resource using OpenShift custom resource definitions (CRDs).
A NamespaceStore represents underlying storage to be used as a
read
orwrite
target for the data in the MCG namespace buckets.To create a NamespaceStore resource, apply the following YAML:
apiVersion: noobaa.io/v1alpha1 kind: NamespaceStore metadata: finalizers: - noobaa.io/finalizer labels: app: noobaa name: bs namespace: openshift-storage spec: s3Compatible: endpoint: <IBM COS ENDPOINT> secret: name: <namespacestore-secret-name> namespace: <namespace-secret> signatureVersion: v2 targetBucket: <target-bucket> type: ibm-cos
<IBM COS ENDPOINT>
- The appropriate IBM COS endpoint.
<namespacestore-secret-name>
- The secret created in the previous step.
<namespace-secret>
- The namespace where the secret can be found.
<target-bucket>
- The target bucket you created for the NamespaceStore.
Create a namespace bucket class that defines a namespace policy for the namespace buckets. The namespace policy requires a type of either
single
ormulti
.The namespace policy of type
single
requires the following configuration:apiVersion: noobaa.io/v1alpha1 kind: BucketClass metadata: labels: app: noobaa name: <my-bucket-class> namespace: openshift-storage spec: namespacePolicy: type: single: resource: <resource>
<my-bucket-class>
- The unique namespace bucket class name.
<resource>
-
The name of a single NamespaceStore that defines the
read
andwrite
target of the namespace bucket.
The namespace policy of type
multi
requires the following configuration:apiVersion: noobaa.io/v1alpha1 kind: BucketClass metadata: labels: app: noobaa name: <my-bucket-class> namespace: openshift-storage spec: namespacePolicy: type: Multi multi: writeResource: <write-resource> readResources: - <read-resources> - <read-resources>
<my-bucket-class>
- The unique bucket class name.
<write-resource>
- The name of a single NamespaceStore that defines the write target of the namespace bucket.
<read-resources>
-
A list of the NamespaceStores names that defines the
read
targets of the namespace bucket.
To create a bucket using an Object Bucket Class (OBC) resource that uses the bucket class defined in the previous step, apply the following YAML:
apiVersion: objectbucket.io/v1alpha1 kind: ObjectBucketClaim metadata: name: <resource-name> namespace: openshift-storage spec: generateBucketName: <my-bucket> storageClassName: openshift-storage.noobaa.io additionalConfig: bucketclass: <my-bucket-class>
<resource-name>
- The name you want to give to the resource.
<my-bucket>
- The name you want to give to the bucket.
<my-bucket-class>
The bucket class created in the previous step.
After the OBC is provisioned by the operator, a bucket is created in the MCG, and the operator creates a
Secret
andConfigMap
with the same name and in the same namespace as that of the OBC.
4.2.3. Adding an AWS S3 namespace bucket using the Multicloud Object Gateway CLI
Prerequisites
- Openshift Container Platform with OpenShift Data Foundation operator installed.
- Access to the Multicloud Object Gateway (MCG), see Chapter 2, Accessing the Multicloud Object Gateway with your applications.
- Download the MCG command-line interface:
# subscription-manager repos --enable=rh-odf-4-for-rhel-8-x86_64-rpms # yum install mcg
Specify the appropriate architecture for enabling the repositories using subscription manager. For instance, in case of IBM Z use the following command:
# subscription-manager repos --enable=rh-odf-4-for-rhel-8-s390x-rpms
Alternatively, you can install the MCG package from the OpenShift Data Foundation RPMs found here https://access.redhat.com/downloads/content/547/ver=4/rhel---8/4/x86_64/package.
Choose the correct Product Variant according to your architecture.
Procedure
In the MCG command-line interface, create a NamespaceStore resource.
A NamespaceStore represents an underlying storage to be used as a
read
orwrite
target for the data in MCG namespace buckets.$ noobaa namespacestore create aws-s3 <namespacestore> --access-key <AWS ACCESS KEY> --secret-key <AWS SECRET ACCESS KEY> --target-bucket <bucket-name> -n openshift-storage
<namespacestore>
- The name of the NamespaceStore.
<AWS ACCESS KEY>
and<AWS SECRET ACCESS KEY>
- The AWS access key ID and secret access key you created for this purpose.
<bucket-name>
- The existing AWS bucket name. This argument tells the MCG which bucket to use as a target bucket for its backing store, and subsequently, data storage and administration.
Create a namespace bucket class that defines a namespace policy for the namespace buckets. The namespace policy can be either
single
ormulti
.To create a namespace bucket class with a namespace policy of type
single
:$ noobaa bucketclass create namespace-bucketclass single <my-bucket-class> --resource <resource> -n openshift-storage
<resource-name>
- The name you want to give the resource.
<my-bucket-class>
- A unique bucket class name.
<resource>
-
A single namespace-store that defines the
read
andwrite
target of the namespace bucket.
To create a namespace bucket class with a namespace policy of type
multi
:$ noobaa bucketclass create namespace-bucketclass multi <my-bucket-class> --write-resource <write-resource> --read-resources <read-resources> -n openshift-storage
<resource-name>
- The name you want to give the resource.
<my-bucket-class>
- A unique bucket class name.
<write-resource>
-
A single namespace-store that defines the
write
target of the namespace bucket. <read-resources>s
-
A list of namespace-stores separated by commas that defines the
read
targets of the namespace bucket.
Create a bucket using an Object Bucket Class (OBC) resource that uses the bucket class defined in the previous step.
$ noobaa obc create my-bucket-claim -n openshift-storage --app-namespace my-app --bucketclass <custom-bucket-class>
<bucket-name>
- A bucket name of your choice.
<custom-bucket-class>
- The name of the bucket class created in the previous step.
After the OBC is provisioned by the operator, a bucket is created in the MCG, and the operator creates a
Secret
and aConfigMap
with the same name and in the same namespace as that of the OBC.
4.2.4. Adding an IBM COS namespace bucket using the Multicloud Object Gateway CLI
Prerequisites
- Openshift Container Platform with OpenShift Data Foundation operator installed.
- Access to the Multicloud Object Gateway (MCG), see Chapter 2, Accessing the Multicloud Object Gateway with your applications.
Download the MCG command-line interface:
# subscription-manager repos --enable=rh-odf-4-for-rhel-8-x86_64-rpms # yum install mcg
NoteSpecify the appropriate architecture for enabling the repositories using subscription manager.
- For IBM Power, use the following command:
# subscription-manager repos --enable=rh-odf-4-for-rhel-8-ppc64le-rpms
- For IBM Z, use the following command:
# subscription-manager repos --enable=rh-odf-4-for-rhel-8-s390x-rpms
Alternatively, you can install the MCG package from the OpenShift Data Foundation RPMs found here https://access.redhat.com/downloads/content/547/ver=4/rhel---8/4/x86_64/package.
NoteChoose the correct Product Variant according to your architecture.
Procedure
In the MCG command-line interface, create a NamespaceStore resource.
A NamespaceStore represents an underlying storage to be used as a
read
orwrite
target for the data in the MCG namespace buckets.$ noobaa namespacestore create ibm-cos <namespacestore> --endpoint <IBM COS ENDPOINT> --access-key <IBM ACCESS KEY> --secret-key <IBM SECRET ACCESS KEY> --target-bucket <bucket-name> -n openshift-storage
<namespacestore>
- The name of the NamespaceStore.
<IBM ACCESS KEY>
,<IBM SECRET ACCESS KEY>
,<IBM COS ENDPOINT>
- An IBM access key ID, secret access key, and the appropriate regional endpoint that corresponds to the location of the existing IBM bucket.
<bucket-name>
- An existing IBM bucket name. This argument tells the MCG which bucket to use as a target bucket for its backing store, and subsequently, data storage and administration.
Create a namespace bucket class that defines a namespace policy for the namespace buckets. The namespace policy requires a type of either
single
ormulti
.To create a namespace bucket class with a namespace policy of type
single
:$ noobaa bucketclass create namespace-bucketclass single <my-bucket-class> --resource <resource> -n openshift-storage
<resource-name>
- The name you want to give the resource.
<my-bucket-class>
- A unique bucket class name.
<resource>
-
A single NamespaceStore that defines the
read
andwrite
target of the namespace bucket.
To create a namespace bucket class with a namespace policy of type
multi
:$ noobaa bucketclass create namespace-bucketclass multi <my-bucket-class> --write-resource <write-resource> --read-resources <read-resources> -n openshift-storage
<resource-name>
- The name you want to give the resource.
<my-bucket-class>
- A unique bucket class name.
<write-resource>
-
A single NamespaceStore that defines the
write
target of the namespace bucket. <read-resources>
-
A comma-separated list of NamespaceStores that defines the
read
targets of the namespace bucket.
Create a bucket using an Object Bucket Class (OBC) resource that uses the bucket class defined in the earlier step.
$ noobaa obc create my-bucket-claim -n openshift-storage --app-namespace my-app --bucketclass <custom-bucket-class>
<bucket-name>
- A bucket name of your choice.
<custom-bucket-class>
- The name of the bucket class created in the previous step.
After the OBC is provisioned by the operator, a bucket is created in the MCG, and the operator creates a Secret
and ConfigMap
with the same name and in the same namespace as that of the OBC.
4.3. Adding a namespace bucket using the OpenShift Container Platform user interface
You can add namespace buckets using the OpenShift Container Platform user interface. For information about namespace buckets, see Managing namespace buckets.
Prerequisites
- Openshift Container Platform with OpenShift Data Foundation operator installed.
- Access to the Multicloud Object Gateway (MCG).
Procedure
- Log into the OpenShift Web Console.
-
Click Storage
Data Foundation. Click the Namespace Store tab to create a
namespacestore
resources to be used in the namespace bucket.- Click Create namespace store.
- Enter a namespacestore name.
- Choose a provider.
- Choose a region.
- Either select an existing secret, or click Switch to credentials to create a secret by entering a secret key and secret access key.
- Choose a target bucket.
- Click Create.
- Verify that the namespacestore is in the Ready state.
- Repeat these steps until you have the desired amount of resources.
Click the Bucket Class tab
Create a new Bucket Class. - Select the Namespace radio button.
- Enter a Bucket Class name.
- (Optional) Add description.
- Click Next.
- Choose a namespace policy type for your namespace bucket, and then click Next.
Select the target resources.
- If your namespace policy type is Single, you need to choose a read resource.
- If your namespace policy type is Multi, you need to choose read resources and a write resource.
- If your namespace policy type is Cache, you need to choose a Hub namespace store that defines the read and write target of the namespace bucket.
- Click Next.
- Review your new bucket class, and then click Create Bucketclass.
- On the BucketClass page, verify that your newly created resource is in the Created phase.
-
In the OpenShift Web Console, click Storage
Data Foundation. - In the Status card, click Storage System and click the storage system link from the pop up that appears.
-
In the Object tab, click Multicloud Object Gateway
Buckets Namespace Buckets tab . Click Create Namespace Bucket.
- On the Choose Name tab, specify a name for the namespace bucket and click Next.
On the Set Placement tab:
- Under Read Policy, select the checkbox for each namespace resource created in the earlier step that the namespace bucket should read data from.
- If the namespace policy type you are using is Multi, then Under Write Policy, specify which namespace resource the namespace bucket should write data to.
- Click Next.
- Click Create.
Verification steps
- Verify that the namespace bucket is listed with a green check mark in the State column, the expected number of read resources, and the expected write resource name.
4.4. Sharing legacy application data with cloud native application using S3 protocol
Many legacy applications use file systems to share data sets. You can access and share the legacy data in the file system by using the S3 operations. To share data you need to do the following:
- Export the pre-existing file system datasets, that is, RWX volume such as Ceph FileSystem (CephFS) or create a new file system datasets using the S3 protocol.
- Access file system datasets from both file system and S3 protocol.
- Configure S3 accounts and map them to the existing or a new file system unique identifiers (UIDs) and group identifiers (GIDs).
4.4.1. Creating a NamespaceStore to use a file system
Prerequisites
- Openshift Container Platform with OpenShift Data Foundation operator installed.
- Access to the Multicloud Object Gateway (MCG).
Procedure
- Log into the OpenShift Web Console.
-
Click Storage
Data Foundation. - Click the NamespaceStore tab to create NamespaceStore resources to be used in the namespace bucket.
- Click Create namespacestore.
- Enter a name for the NamespaceStore.
- Choose Filesystem as the provider.
- Choose the Persistent volume claim.
Enter a folder name.
If the folder name exists, then that folder is used to create the NamespaceStore or else a folder with that name is created.
- Click Create.
- Verify the NamespaceStore is in the Ready state.
4.4.2. Creating accounts with NamespaceStore filesystem configuration
You can either create a new account with NamespaceStore filesystem configuration or convert an existing normal account into a NamespaceStore filesystem account by editing the YAML.
You cannot remove a NamespaceStore filesystem configuration from an account.
Prerequisites
Download the Multicloud Object Gateway (MCG) command-line interface:
# subscription-manager repos --enable=rh-odf-4-for-rhel-8-x86_64-rpms # yum install mcg
Procedure
Create a new account with NamespaceStore filesystem configuration using the MCG command-line interface.
$ noobaa account create <noobaa-account-name> [flags]
For example:
$ noobaa account create testaccount --nsfs_account_config --gid 10001 --uid 10001 –default_resource fs_namespacestore
allow_bucket_create
Indicates whether the account is allowed to create new buckets. Supported values are
true
orfalse
. Default value istrue
.default_resource
The NamespaceStore resource on which the new buckets will be created when using the S3 CreateBucket operation. The NamespaceStore must be backed by an RWX (ReadWriteMany) persistent volume claim (PVC).
new_buckets_path
The filesystem path where directories corresponding to new buckets will be created. The path is inside the filesystem of NamespaceStore filesystem PVCs where new directories are created to act as the filesystem mapping of newly created object bucket classes.
nsfs_account_config
A mandatory field that indicates if the account is used for NamespaceStore filesystem.
nsfs_only
Indicates whether the account is used only for NamespaceStore filesystem or not. Supported values are true or
false
. Default value isfalse
. If it is set to 'true', it limits you from accessing other types of buckets.uid
The user ID of the filesystem to which the MCG account will be mapped and it is used to access and manage data on the filesystem
gid
The group ID of the filesystem to which the MCG account will be mapped and it is used to access and manage data on the filesystem
The MCG system sends a response with the account configuration and its S3 credentials:
# NooBaaAccount spec: allow_bucket_creation: true default_resource: noobaa-default-namespace-store Nsfs_account_config: gid: 10001 new_buckets_path: / nsfs_only: true uid: 10001 INFO[0006] ✅ Exists: Secret "noobaa-account-testaccount" Connection info: AWS_ACCESS_KEY_ID : <aws-access-key-id> AWS_SECRET_ACCESS_KEY : <aws-secret-access-key>
You can list all the custom resource definition (CRD) based accounts by using the following command:
$ noobaa account list NAME DEFAULT_RESOURCE PHASE AGE testaccount noobaa-default-backing-store Ready 1m17s
If you are interested in a particular account, you can read its custom resource definition (CRD) directly by the account name:
$ oc get noobaaaccount/testaccount -o yaml spec: allow_bucket_creation: true default_resource: noobaa-default-namespace-store nsfs_account_config: gid: 10001 new_buckets_path: / nsfs_only: true uid: 10001
4.4.3. Accessing legacy application data from the openshift-storage namespace
When using the Multicloud Object Gateway (MCG) NamespaceStore filesystem (NSFS) feature, you need to have the Persistent Volume Claim (PVC) where the data resides in the openshift-storage
namespace. In almost all cases, the data you need to access is not in the openshift-storage
namespace, but in the namespace that the legacy application uses.
In order to access data stored in another namespace, you need to create a PVC in the openshift-storage
namespace that points to the same CephFS volume that the legacy application uses.
Procedure
Display the application namespace with
scc
:$ oc get ns <application_namespace> -o yaml | grep scc
- <application_namespace>
Specify the name of the application namespace.
For example:
$ oc get ns testnamespace -o yaml | grep scc openshift.io/sa.scc.mcs: s0:c26,c5 openshift.io/sa.scc.supplemental-groups: 1000660000/10000 openshift.io/sa.scc.uid-range: 1000660000/10000
Navigate into the application namespace:
$ oc project <application_namespace>
For example:
$ oc project testnamespace
Ensure that a ReadWriteMany (RWX) PVC is mounted on the pod that you want to consume from the noobaa S3 endpoint using the MCG NSFS feature:
$ oc get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE cephfs-write-workload-generator-no-cache-pv-claim Bound pvc-aa58fb91-c3d2-475b-bbee-68452a613e1a 10Gi RWX ocs-storagecluster-cephfs 12s
$ oc get pod NAME READY STATUS RESTARTS AGE cephfs-write-workload-generator-no-cache-1-cv892 1/1 Running 0 11s
Check the mount point of the Persistent Volume (PV) inside your pod.
Get the volume name of the PV from the pod:
$ oc get pods <pod_name> -o jsonpath='{.spec.volumes[]}'
- <pod_name>
Specify the name of the pod.
For example:
$ oc get pods cephfs-write-workload-generator-no-cache-1-cv892 -o jsonpath='{.spec.volumes[]}' {"name":"app-persistent-storage","persistentVolumeClaim":{"claimName":"cephfs-write-workload-generator-no-cache-pv-claim"}}
In this example, the name of the volume for the PVC is
cephfs-write-workload-generator-no-cache-pv-claim
.
List all the mounts in the pod, and check for the mount point of the volume that you identified in the previous step:
$ oc get pods <pod_name> -o jsonpath='{.spec.containers[].volumeMounts}'
For example:
$ oc get pods cephfs-write-workload-generator-no-cache-1-cv892 -o jsonpath='{.spec.containers[].volumeMounts}' [{"mountPath":"/mnt/pv","name":"app-persistent-storage"},{"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount","name":"kube-api-access-8tnc5","readOnly":true}]
Confirm the mount point of the RWX PV in your pod:
$ oc exec -it <pod_name> -- df <mount_path>
<mount_path>
Specify the path to the mount point that you identified in the previous step.
For example:
$ oc exec -it cephfs-write-workload-generator-no-cache-1-cv892 -- df /mnt/pv main Filesystem 1K-blocks Used Available Use% Mounted on 172.30.202.87:6789,172.30.120.254:6789,172.30.77.247:6789:/volumes/csi/csi-vol-cc416d9e-dbf3-11ec-b286-0a580a810213/edcfe4d5-bdcb-4b8e-8824-8a03ad94d67c 10485760 0 10485760 0% /mnt/pv
Ensure that the UID and SELinux labels are the same as the ones that the legacy namespace uses:
$ oc exec -it <pod_name> -- ls -latrZ <mount_path>
For example:
$ oc exec -it cephfs-write-workload-generator-no-cache-1-cv892 -- ls -latrZ /mnt/pv/ total 567 drwxrwxrwx. 3 root root system_u:object_r:container_file_t:s0:c26,c5 2 May 25 06:35 . -rw-r--r--. 1 1000660000 root system_u:object_r:container_file_t:s0:c26,c5 580138 May 25 06:35 fs_write_cephfs-write-workload-generator-no-cache-1-cv892-data.log drwxrwxrwx. 3 root root system_u:object_r:container_file_t:s0:c26,c5 30 May 25 06:35 ..
Get the information of the legacy application RWX PV that you want to make accessible from the
openshift-storage
namespace:$ oc get pv | grep <pv_name>
<pv_name>
Specify the name of the PV.
For example:
$ oc get pv | grep pvc-aa58fb91-c3d2-475b-bbee-68452a613e1a pvc-aa58fb91-c3d2-475b-bbee-68452a613e1a 10Gi RWX Delete Bound testnamespace/cephfs-write-workload-generator-no-cache-pv-claim ocs-storagecluster-cephfs 47s
Ensure that the PVC from the legacy application is accessible from the
openshift-storage
namespace so that one or more noobaa-endpoint pods can access the PVC.Find the values of the
subvolumePath
andvolumeHandle
from thevolumeAttributes
. You can get these values from the YAML description of the legacy application PV:$ oc get pv <pv_name> -o yaml
For example:
$ oc get pv pvc-aa58fb91-c3d2-475b-bbee-68452a613e1a -o yaml apiVersion: v1 kind: PersistentVolume metadata: annotations: pv.kubernetes.io/provisioned-by: openshift-storage.cephfs.csi.ceph.com creationTimestamp: "2022-05-25T06:27:49Z" finalizers: - kubernetes.io/pv-protection name: pvc-aa58fb91-c3d2-475b-bbee-68452a613e1a resourceVersion: "177458" uid: 683fa87b-5192-4ccf-af2f-68c6bcf8f500 spec: accessModes: - ReadWriteMany capacity: storage: 10Gi claimRef: apiVersion: v1 kind: PersistentVolumeClaim name: cephfs-write-workload-generator-no-cache-pv-claim namespace: testnamespace resourceVersion: "177453" uid: aa58fb91-c3d2-475b-bbee-68452a613e1a csi: controllerExpandSecretRef: name: rook-csi-cephfs-provisioner namespace: openshift-storage driver: openshift-storage.cephfs.csi.ceph.com nodeStageSecretRef: name: rook-csi-cephfs-node namespace: openshift-storage volumeAttributes: clusterID: openshift-storage fsName: ocs-storagecluster-cephfilesystem storage.kubernetes.io/csiProvisionerIdentity: 1653458225664-8081-openshift-storage.cephfs.csi.ceph.com subvolumeName: csi-vol-cc416d9e-dbf3-11ec-b286-0a580a810213 subvolumePath: /volumes/csi/csi-vol-cc416d9e-dbf3-11ec-b286-0a580a810213/edcfe4d5-bdcb-4b8e-8824-8a03ad94d67c volumeHandle: 0001-0011-openshift-storage-0000000000000001-cc416d9e-dbf3-11ec-b286-0a580a810213 persistentVolumeReclaimPolicy: Delete storageClassName: ocs-storagecluster-cephfs volumeMode: Filesystem status: phase: Bound
Use the
subvolumePath
andvolumeHandle
values that you identified in the previous step to create a new PV and PVC object in theopenshift-storage
namespace that points to the same CephFS volume as the legacy application PV:Example YAML file:
$ cat << EOF >> pv-openshift-storage.yaml apiVersion: v1 kind: PersistentVolume metadata: name: cephfs-pv-legacy-openshift-storage spec: storageClassName: "" accessModes: - ReadWriteMany capacity: storage: 10Gi 1 csi: driver: openshift-storage.cephfs.csi.ceph.com nodeStageSecretRef: name: rook-csi-cephfs-node namespace: openshift-storage volumeAttributes: # Volume Attributes can be copied from the Source testnamespace PV "clusterID": "openshift-storage" "fsName": "ocs-storagecluster-cephfilesystem" "staticVolume": "true" # rootpath is the subvolumePath: you copied from the Source testnamespace PV "rootPath": /volumes/csi/csi-vol-cc416d9e-dbf3-11ec-b286-0a580a810213/edcfe4d5-bdcb-4b8e-8824-8a03ad94d67c volumeHandle: 0001-0011-openshift-storage-0000000000000001-cc416d9e-dbf3-11ec-b286-0a580a810213-clone 2 persistentVolumeReclaimPolicy: Retain volumeMode: Filesystem --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: cephfs-pvc-legacy namespace: openshift-storage spec: storageClassName: "" accessModes: - ReadWriteMany resources: requests: storage: 10Gi 3 volumeMode: Filesystem # volumeName should be same as PV name volumeName: cephfs-pv-legacy-openshift-storage EOF
- 1
- The storage capacity of the PV that you are creating in the
openshift-storage
namespace must be the same as the original PV. - 2
- The volume handle for the target PV that you create in
openshift-storage
needs to have a different handle than the original application PV, for example, add-clone
at the end of the volume handle. - 3
- The storage capacity of the PVC that you are creating in the
openshift-storage
namespace must be the same as the original PVC.
Create the PV and PVC in the
openshift-storage
namespace using the YAML file specified in the previous step:$ oc create -f <YAML_file>
<YAML_file>
Specify the name of the YAML file.
For example:
$ oc create -f pv-openshift-storage.yaml persistentvolume/cephfs-pv-legacy-openshift-storage created persistentvolumeclaim/cephfs-pvc-legacy created
Ensure that the PVC is available in the
openshift-storage
namespace:$ oc get pvc -n openshift-storage NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE cephfs-pvc-legacy Bound cephfs-pv-legacy-openshift-storage 10Gi RWX 14s
Navigate into the
openshift-storage
project:$ oc project openshift-storage Now using project "openshift-storage" on server "https://api.cluster-5f6ng.5f6ng.sandbox65.opentlc.com:6443".
Create the NSFS namespacestore:
$ noobaa namespacestore create nsfs <nsfs_namespacestore> --pvc-name='<cephfs_pvc_name>' --fs-backend='CEPH_FS'
<nsfs_namespacestore>
- Specify the name of the NSFS namespacestore.
<cephfs_pvc_name>
Specify the name of the CephFS PVC in the
openshift-storage
namespace.For example:
$ noobaa namespacestore create nsfs legacy-namespace --pvc-name='cephfs-pvc-legacy' --fs-backend='CEPH_FS'
Ensure that the noobaa-endpoint pod restarts and that it successfully mounts the PVC at the NSFS namespacestore, for example,
/nsfs/legacy-namespace
mountpoint:$ oc exec -it <noobaa_endpoint_pod_name> -- df -h /nsfs/<nsfs_namespacestore>
<noobaa_endpoint_pod_name>
Specify the name of the noobaa-endpoint pod.
For example:
$ oc exec -it noobaa-endpoint-5875f467f5-546c6 -- df -h /nsfs/legacy-namespace Filesystem Size Used Avail Use% Mounted on 172.30.202.87:6789,172.30.120.254:6789,172.30.77.247:6789:/volumes/csi/csi-vol-cc416d9e-dbf3-11ec-b286-0a580a810213/edcfe4d5-bdcb-4b8e-8824-8a03ad94d67c 10G 0 10G 0% /nsfs/legacy-namespace
Create a MCG user account:
$ noobaa account create <user_account> --allow_bucket_create=true --new_buckets_path='/' --nsfs_only=true --nsfs_account_config=true --gid <gid_number> --uid <uid_number> --default_resource='legacy-namespace'
<user_account>
- Specify the name of the MCG user account.
<gid_number>
- Specify the GID number.
<uid_number>
Specify the UID number.
ImportantUse the same
UID
andGID
as that of the legacy application. You can find it from the previous output.For example:
$ noobaa account create leguser --allow_bucket_create=true --new_buckets_path='/' --nsfs_only=true --nsfs_account_config=true --gid 0 --uid 1000660000 --default_resource='legacy-namespace'
Create a MCG bucket.
Create a dedicated folder for S3 inside the NSFS share on the CephFS PV and PVC of the legacy application pod:
$ oc exec -it <pod_name> -- mkdir <mount_path>/nsfs
For example:
$ oc exec -it cephfs-write-workload-generator-no-cache-1-cv892 -- mkdir /mnt/pv/nsfs
Create the MCG bucket using the
nsfs/
path:$ noobaa api bucket_api create_bucket '{ "name": "<bucket_name>", "namespace":{ "write_resource": { "resource": "<nsfs_namespacestore>", "path": "nsfs/" }, "read_resources": [ { "resource": "<nsfs_namespacestore>", "path": "nsfs/" }] } }'
For example:
$ noobaa api bucket_api create_bucket '{ "name": "legacy-bucket", "namespace":{ "write_resource": { "resource": "legacy-namespace", "path": "nsfs/" }, "read_resources": [ { "resource": "legacy-namespace", "path": "nsfs/" }] } }'
Check the SELinux labels of the folders residing in the PVCs in the legacy application and
openshift-storage
namespaces:$ oc exec -it <noobaa_endpoint_pod_name> -n openshift-storage -- ls -ltraZ /nsfs/<nsfs_namespacstore>
For example:
$ oc exec -it noobaa-endpoint-5875f467f5-546c6 -n openshift-storage -- ls -ltraZ /nsfs/legacy-namespace total 567 drwxrwxrwx. 3 root root system_u:object_r:container_file_t:s0:c0,c26 2 May 25 06:35 . -rw-r--r--. 1 1000660000 root system_u:object_r:container_file_t:s0:c0,c26 580138 May 25 06:35 fs_write_cephfs-write-workload-generator-no-cache-1-cv892-data.log drwxrwxrwx. 3 root root system_u:object_r:container_file_t:s0:c0,c26 30 May 25 06:35 ..
$ oc exec -it <pod_name> -- ls -latrZ <mount_path>
For example:
$ oc exec -it cephfs-write-workload-generator-no-cache-1-cv892 -- ls -latrZ /mnt/pv/ total 567 drwxrwxrwx. 3 root root system_u:object_r:container_file_t:s0:c26,c5 2 May 25 06:35 . -rw-r--r--. 1 1000660000 root system_u:object_r:container_file_t:s0:c26,c5 580138 May 25 06:35 fs_write_cephfs-write-workload-generator-no-cache-1-cv892-data.log drwxrwxrwx. 3 root root system_u:object_r:container_file_t:s0:c26,c5 30 May 25 06:35 ..
In these examples, you can see that the SELinux labels are not the same which results in permission denied or access issues.
Ensure that the legacy application and
openshift-storage
pods use the same SELinux labels on the files.You can do this in one of the following ways:
Delete the NSFS namespacestore:
Delete the MCG bucket:
$ noobaa bucket delete <bucket_name>
For example:
$ noobaa bucket delete legacy-bucket
Delete the MCG user account:
$ noobaa account delete <user_account>
For example:
$ noobaa account delete leguser
Delete the NSFS namespacestore:
$ noobaa namespacestore delete <nsfs_namespacestore>
For example:
$ noobaa namespacestore delete legacy-namespace
Delete the PV and PVC:
ImportantBefore you delete the PV and PVC, ensure that the PV has a retain policy configured.
$ oc delete pv <cephfs_pv_name>
$ oc delete pvc <cephfs_pvc_name>
<cephfs_pv_name>
- Specify the CephFS PV name of the legacy application.
<cephfs_pvc_name>
Specify the CephFS PVC name of the legacy application.
For example:
$ oc delete pv cephfs-pv-legacy-openshift-storage
$ oc delete pvc cephfs-pvc-legacy
4.4.3.1. Changing the default SELinux label on the legacy application project to match the one in the openshift-storage project
Display the current
openshift-storage
namespace withsa.scc.mcs
:$ oc get ns openshift-storage -o yaml | grep sa.scc.mcs openshift.io/sa.scc.mcs: s0:c26,c0
Edit the legacy application namespace, and modify the
sa.scc.mcs
with the value from thesa.scc.mcs
of theopenshift-storage
namespace:$ oc edit ns <appplication_namespace>
For example:
$ oc edit ns testnamespace
$ oc get ns <application_namespace> -o yaml | grep sa.scc.mcs
For example:
$ oc get ns testnamespace -o yaml | grep sa.scc.mcs openshift.io/sa.scc.mcs: s0:c26,c0
-
Restart the legacy application pod. A relabel of all the files take place and now the SELinux labels match with the
openshift-storage
deployment.
4.4.3.2. Modifying the SELinux label only for the deployment config that has the pod which mounts the legacy application PVC
Create a new
scc
with theMustRunAs
andseLinuxOptions
options, with the Multi Category Security (MCS) that theopenshift-storage
project uses.Example YAML file:
$ cat << EOF >> scc.yaml allowHostDirVolumePlugin: false allowHostIPC: false allowHostNetwork: false allowHostPID: false allowHostPorts: false allowPrivilegeEscalation: true allowPrivilegedContainer: false allowedCapabilities: null apiVersion: security.openshift.io/v1 defaultAddCapabilities: null fsGroup: type: MustRunAs groups: - system:authenticated kind: SecurityContextConstraints metadata: annotations: name: restricted-pvselinux priority: null readOnlyRootFilesystem: false requiredDropCapabilities: - KILL - MKNOD - SETUID - SETGID runAsUser: type: MustRunAsRange seLinuxContext: seLinuxOptions: level: s0:c26,c0 type: MustRunAs supplementalGroups: type: RunAsAny users: [] volumes: - configMap - downwardAPI - emptyDir - persistentVolumeClaim - projected - secret EOF
$ oc create -f scc.yaml
Create a service account for the deployment and add it to the newly created
scc
.Create a service account:
$ oc create serviceaccount <service_account_name>
- <service_account_name>`
Specify the name of the service account.
For example:
$ oc create serviceaccount testnamespacesa
Add the service account to the newly created
scc
:$ oc adm policy add-scc-to-user restricted-pvselinux -z <service_account_name>
For example:
$ oc adm policy add-scc-to-user restricted-pvselinux -z testnamespacesa
Patch the legacy application deployment so that it uses the newly created service account. This allows you to specify the SELinux label in the deployment:
$ oc patch dc/<pod_name> '{"spec":{"template":{"spec":{"serviceAccountName": "<service_account_name>"}}}}'
For example:
$ oc patch dc/cephfs-write-workload-generator-no-cache --patch '{"spec":{"template":{"spec":{"serviceAccountName": "testnamespacesa"}}}}'
Edit the deployment to specify the security context to use at the SELinux label in the deployment configuration:
$ oc edit dc <pod_name> -n <application_namespace>
Add the following lines:
spec: template: metadata: securityContext: seLinuxOptions: Level: <security_context_value>
<security_context_value>
You can find this value when you execute the command to create a dedicated folder for S3 inside the NSFS share, on the CephFS PV and PVC of the legacy application pod.
For example:
$ oc edit dc cephfs-write-workload-generator-no-cache -n testnamespace
spec: template: metadata: securityContext: seLinuxOptions: level: s0:c26,c0
Ensure that the security context to be used at the SELinux label in the deployment configuration is specified correctly:
$ oc get dc <pod_name> -n <application_namespace> -o yaml | grep -A 2 securityContext
For example"
$ oc get dc cephfs-write-workload-generator-no-cache -n testnamespace -o yaml | grep -A 2 securityContext securityContext: seLinuxOptions: level: s0:c26,c0
The legacy application is restarted and begins using the same SELinux labels as the
openshift-storage
namespace.