Chapter 6. Using Container Storage Interface (CSI)

Procedure

1. Create the

   Pod

   object definition and save it to a file.

2. Embed the CSI inline ephemeral volume in the file.

   my-csi-app.yaml

   kind: Pod
   apiVersion: v1
   metadata:
     name: my-csi-app
   spec:
     containers:
       - name: my-frontend
         image: busybox
         volumeMounts:
         - mountPath: "/data"
           name: my-csi-inline-vol
         command: [ "sleep", "1000000" ]
     volumes: 

   1

   
       - name: my-csi-inline-vol
         csi:
           driver: inline.storage.kubernetes.io
           volumeAttributes:
             foo: bar

   1

   The name of the volume that is used by pods.

3. Create the object definition file that you saved in the previous step.

   $ oc create -f my-csi-app.yaml

Procedure

1. Grant a given service account RBAC permissions to use the

   SharedSecret

   CR instance in its pod by using

   oc apply

   with YAML content:
   Note

   Currently,

   kubectl

   and

   oc

   have hard-coded special case logic restricting the

   use

   verb to roles centered around pod security. Therefore, you cannot use

   oc create role …​

   to create the role needed for consuming

   SharedSecret

   CR instances.

   $ oc apply -f - <<EOF
   apiVersion: rbac.authorization.k8s.io/v1
   kind: Role
   metadata:
     name: shared-resource-my-share
     namespace: my-namespace
   rules:
     - apiGroups:
         - sharedresource.openshift.io
       resources:
         - sharedsecrets
       resourceNames:
         - my-share
       verbs:
         - use
   EOF

2. Create the

   RoleBinding

   associated with the role by using the

   oc

   command:

   $ oc create rolebinding shared-resource-my-share --role=shared-resource-my-share --serviceaccount=my-namespace:builder

3. Access the

   SharedSecret

   CR instance from a pod:

   $ oc apply -f - <<EOF
   kind: Pod
   apiVersion: v1
   metadata:
     name: my-app
     namespace: my-namespace
   spec:
     serviceAccountName: default
   
   # containers omitted …. Follow standard use of ‘volumeMounts’ for referencing your shared resource volume
   
       volumes:
       - name: my-csi-volume
         csi:
           readOnly: true
           driver: csi.sharedresource.openshift.io
           volumeAttributes:
             sharedSecret: my-share
   
   EOF

Procedure

1. Create a

   SharedConfigMap

   CR instance for the config map that you want to share across namespaces in the cluster:

   $ oc apply -f - <<EOF
   apiVersion: sharedresource.openshift.io/v1alpha1
   kind: SharedConfigMap
   metadata:
     name: my-share
   spec:
     configMapRef:
       name: <name of configmap>
       namespace: <namespace of configmap>
   EOF

Procedure

1. Grant a given service account RBAC permissions to use the

   SharedConfigMap

   CR instance in its pod by using

   oc apply

   with YAML content.
   Note

   Currently,

   kubectl

   and

   oc

   have hard-coded special case logic restricting the

   use

   verb to roles centered around pod security. Therefore, you cannot use

   oc create role …​

   to create the role needed for consuming a

   SharedConfigMap

   CR instance.

   $ oc apply -f - <<EOF
   apiVersion: rbac.authorization.k8s.io/v1
   kind: Role
   metadata:
     name: shared-resource-my-share
     namespace: my-namespace
   rules:
     - apiGroups:
         - sharedresource.openshift.io
       resources:
         - sharedconfigmaps
       resourceNames:
         - my-share
       verbs:
         - use
   EOF

2. Create the

   RoleBinding

   associated with the role by using the

   oc

   command:

   oc create rolebinding shared-resource-my-share --role=shared-resource-my-share --serviceaccount=my-namespace:builder

3. Access the

   SharedConfigMap

   CR instance from a pod:

   $ oc apply -f - <<EOF
   kind: Pod
   apiVersion: v1
   metadata:
     name: my-app
     namespace: my-namespace
   spec:
     serviceAccountName: default
   
   # containers omitted …. Follow standard use of ‘volumeMounts’ for referencing your shared resource volume
   
       volumes:
       - name: my-csi-volume
         csi:
           readOnly: true
           driver: csi.sharedresource.openshift.io
           volumeAttributes:
             sharedConfigMap: my-share
   
   EOF

1. Create a file with the

   VolumeSnapshotClass

   object described by the following YAML:

   volumesnapshotclass.yaml

   apiVersion: snapshot.storage.k8s.io/v1
   kind: VolumeSnapshotClass
   metadata:
     name: csi-hostpath-snap
   driver: hostpath.csi.k8s.io 

   1

   
   deletionPolicy: Delete

   1

   The name of the CSI driver that is used to create snapshots of this VolumeSnapshotClass object. The name must be the same as the Provisioner field of the storage class that is responsible for the PVC that is being snapshotted.

   Note

   Depending on the driver that you used to configure persistent storage, additional parameters might be required. You can also use an existing

   VolumeSnapshotClass

   object.

2. Create the object you saved in the previous step by entering the following command:

   $ oc create -f volumesnapshotclass.yaml

3. Create a

   VolumeSnapshot

   object:

   volumesnapshot-dynamic.yaml

   apiVersion: snapshot.storage.k8s.io/v1
   kind: VolumeSnapshot
   metadata:
     name: mysnap
   spec:
     volumeSnapshotClassName: csi-hostpath-snap 

   1

   
     source:
       persistentVolumeClaimName: myclaim 

   2

   1

   The request for a particular class by the volume snapshot. If the volumeSnapshotClassName setting is absent and there is a default volume snapshot class, a snapshot is created with the default volume snapshot class name. But if the field is absent and no default volume snapshot class exists, then no snapshot is created.

   2

   The name of the PersistentVolumeClaim object bound to a persistent volume. This defines what you want to create a snapshot of. Required for dynamically provisioning a snapshot.

4. Create the object you saved in the previous step by entering the following command:

   $ oc create -f volumesnapshot-dynamic.yaml

1. Provide a value for the

   volumeSnapshotContentName

   parameter as the source for the snapshot, in addition to defining volume snapshot class as shown above.

   volumesnapshot-manual.yaml

   apiVersion: snapshot.storage.k8s.io/v1
   kind: VolumeSnapshot
   metadata:
     name: snapshot-demo
   spec:
     source:
       volumeSnapshotContentName: mycontent 

   1

   1

   The volumeSnapshotContentName parameter is required for pre-provisioned snapshots.

2. Create the object you saved in the previous step by entering the following command:

   $ oc create -f volumesnapshot-manual.yaml

1. To display details about the volume snapshot that was created, enter the following command:

   $ oc describe volumesnapshot mysnap

   The following example displays details about the

   mysnap

   volume snapshot:

   volumesnapshot.yaml

   apiVersion: snapshot.storage.k8s.io/v1
   kind: VolumeSnapshot
   metadata:
     name: mysnap
   spec:
     source:
       persistentVolumeClaimName: myclaim
     volumeSnapshotClassName: csi-hostpath-snap
   status:
     boundVolumeSnapshotContentName: snapcontent-1af4989e-a365-4286-96f8-d5dcd65d78d6 

   1

   
     creationTime: "2020-01-29T12:24:30Z" 

   2

   
     readyToUse: true 

   3

   
     restoreSize: 500Mi

   1

   The pointer to the actual storage content that was created by the controller.

   2

   The time when the snapshot was created. The snapshot contains the volume content that was available at this indicated time.

   3

   If the value is set to true, the snapshot can be used to restore as a new PVC.
   If the value is set to false, the snapshot was created. However, the storage back end needs to perform additional tasks to make the snapshot usable so that it can be restored as a new volume. For example, Amazon Elastic Block Store data might be moved to a different, less expensive location, which can take several minutes.

2. To verify that the volume snapshot was created, enter the following command:

   $ oc get volumesnapshotcontent

   The pointer to the actual content is displayed. If the

   boundVolumeSnapshotContentName

   field is populated, a

   VolumeSnapshotContent

   object exists and the snapshot was created.

3. To verify that the snapshot is ready, confirm that the

   VolumeSnapshot

   object has

   readyToUse: true

   .

Procedure

1. Specify the deletion policy that you require in the

   VolumeSnapshotClass

   object, as shown in the following example:

   volumesnapshotclass.yaml

   apiVersion: snapshot.storage.k8s.io/v1
   kind: VolumeSnapshotClass
   metadata:
     name: csi-hostpath-snap
   driver: hostpath.csi.k8s.io
   deletionPolicy: Delete 

   1

   1

   When deleting the volume snapshot, if the Delete value is set, the underlying snapshot is deleted along with the VolumeSnapshotContent object. If the Retain value is set, both the underlying snapshot and VolumeSnapshotContent object remain.
   If the Retain value is set and the VolumeSnapshot object is deleted without deleting the corresponding VolumeSnapshotContent object, the content remains. The snapshot itself is also retained in the storage back end.

2. Delete the volume snapshot by entering the following command:

   $ oc delete volumesnapshot <volumesnapshot_name>

   Example output

   volumesnapshot.snapshot.storage.k8s.io "mysnapshot" deleted

3. If the deletion policy is set to

   Retain

   , delete the volume snapshot content by entering the following command:

   $ oc delete volumesnapshotcontent <volumesnapshotcontent_name>

4. Optional: If the

   VolumeSnapshot

   object is not successfully deleted, enter the following command to remove any finalizers for the leftover resource so that the delete operation can continue:
   Important

   Only remove the finalizers if you are confident that there are no existing references from either persistent volume claims or volume snapshot contents to the

   VolumeSnapshot

   object. Even with the

   --force

   option, the delete operation does not delete snapshot objects until all finalizers are removed.

   $ oc patch -n $PROJECT volumesnapshot/$NAME --type=merge -p '{"metadata": {"finalizers":null}}'

   Example output

   volumesnapshotclass.snapshot.storage.k8s.io "csi-ocs-rbd-snapclass" deleted

   The finalizers are removed and the volume snapshot is deleted.

Procedure

1. Specify a

   VolumeSnapshot

   data source on a PVC as shown in the following:

   pvc-restore.yaml

   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: myclaim-restore
   spec:
     storageClassName: csi-hostpath-sc
     dataSource:
       name: mysnap 

   1

   
       kind: VolumeSnapshot 

   2

   
       apiGroup: snapshot.storage.k8s.io 

   3

   
     accessModes:
       - ReadWriteOnce
     resources:
       requests:
         storage: 1Gi

   1

   Name of the VolumeSnapshot object representing the snapshot to use as source.

   2

   Must be set to the VolumeSnapshot value.

   3

   Must be set to the snapshot.storage.k8s.io value.

2. Create a PVC by entering the following command:

   $ oc create -f pvc-restore.yaml

3. Verify that the restored PVC has been created by entering the following command:

   $ oc get pvc

   A new PVC such as

   myclaim-restore

   is displayed.

1. Create and save a file with the

   PersistentVolumeClaim

   object described by the following YAML:

   pvc-clone.yaml

   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: pvc-1-clone
     namespace: mynamespace
   spec:
     storageClassName: csi-cloning 

   1

   
     accessModes:
       - ReadWriteOnce
     resources:
       requests:
         storage: 5Gi
     dataSource:
       kind: PersistentVolumeClaim
       name: pvc-1

   1

   The name of the storage class that provisions the storage back end. The default storage class can be used and storageClassName can be omitted in the spec.

2. Create the object you saved in the previous step by running the following command:

   $ oc create -f pvc-clone.yaml

   A new PVC

   pvc-1-clone

   is created.

3. Verify that the volume clone was created and is ready by running the following command:

   $ oc get pvc pvc-1-clone

   The

   pvc-1-clone

   shows that it is

   Bound

   .

   You are now ready to use the newly cloned PVC to configure a pod.

4. Create and save a file with the

   Pod

   object described by the YAML. For example:

   kind: Pod
   apiVersion: v1
   metadata:
     name: mypod
   spec:
     containers:
       - name: myfrontend
         image: dockerfile/nginx
         volumeMounts:
         - mountPath: "/var/www/html"
           name: mypd
     volumes:
       - name: mypd
         persistentVolumeClaim:
           claimName: pvc-1-clone 

   1

   1

   The cloned PVC created during the CSI volume cloning operation.

   The created

   Pod

   object is now ready to consume, clone, snapshot, or delete your cloned PVC independently of its original

   dataSource

   PVC.

1. Log in to the web console.
2. Click Administration > CustomResourceDefinitions.
3. On the CustomResourceDefinitions page, type

   clustercsidriver

   to find the

   ClusterCSIDriver

   object.
4. Click ClusterCSIDriver, and then click the Instances tab.
5. Click the name of the desired instance, and then click the YAML tab.

6. Add the

   spec.storageClassState

   field with a value of

   Managed

   ,

   Unmanaged

   , or

   Removed

   .

   Example

   ...
   spec:
     driverConfig:
       driverType: ''
     logLevel: Normal
     managementState: Managed
     observedConfig: null
     operatorLogLevel: Normal
     storageClassState: Unmanaged 

   1

   
   ...

   1

   spec.storageClassState field set to "Unmanaged"

7. Click Save.

1. List the storage classes:

   $ oc get storageclass

   Example output

   NAME                 TYPE
   gp3 (default)        kubernetes.io/aws-ebs 

   1

   
   standard             kubernetes.io/aws-ebs

   1

   (default) indicates the default storage class.

2. Make the desired storage class the default.

   For the desired storage class, set the

   storageclass.kubernetes.io/is-default-class

   annotation to

   true

   by running the following command:

   $ oc patch storageclass standard -p '{"metadata": {"annotations": {"storageclass.kubernetes.io/is-default-class": "true"}}}'

   Note

   You can have multiple default storage classes for a short time. However, you should ensure that only one default storage class exists eventually.

   With multiple default storage classes present, any persistent volume claim (PVC) requesting the default storage class (

   pvc.spec.storageClassName

   =nil) gets the most recently created default storage class, regardless of the default status of that storage class, and the administrator receives an alert in the alerts dashboard that there are multiple default storage classes,

   MultipleDefaultStorageClasses

   .

3. Remove the default storage class setting from the old default storage class.

   For the old default storage class, change the value of the

   storageclass.kubernetes.io/is-default-class

   annotation to

   false

   by running the following command:

   $ oc patch storageclass gp3 -p '{"metadata": {"annotations": {"storageclass.kubernetes.io/is-default-class": "false"}}}'

4. Verify the changes:

   $ oc get storageclass

   Example output

   NAME                 TYPE
   gp3                  kubernetes.io/aws-ebs
   standard (default)   kubernetes.io/aws-ebs

1. If you are using AWS EFS with AWS Secure Token Service (STS), obtain a role Amazon Resource Name (ARN) for STS. This is required for installing the AWS EFS CSI Driver Operator.
2. Install the AWS EFS CSI Driver Operator.
3. Install the AWS EFS CSI Driver.

1. Set up the environment:

   1. Configure environment variables by running the following commands:

      export CLUSTER_NAME="<CLUSTER_NAME>" 

      1

      
      export AWS_REGION="<AWS_REGION>" 

      2

      
      export AWS_ACCOUNT_A_ID="<ACCOUNT_A_ID>" 

      3

      
      export AWS_ACCOUNT_B_ID="<ACCOUNT_B_ID>" 

      4

      
      export AWS_ACCOUNT_A_VPC_CIDR="<VPC_A_CIDR>" 

      5

      
      export AWS_ACCOUNT_B_VPC_CIDR="<VPC_B_CIDR>" 

      6

      
      export AWS_ACCOUNT_A_VPC_ID="<VPC_A_ID>" 

      7

      
      export AWS_ACCOUNT_B_VPC_ID="<VPC_B_ID>" 

      8

      
      export SCRATCH_DIR="<WORKING_DIRECTORY>" 

      9

      
      export CSI_DRIVER_NAMESPACE="openshift-cluster-csi-drivers" 

      10

      
      export AWS_PAGER="" 

      11

      1

      Cluster name of choice.

      2

      AWS region of choice.

      3

      AWS Account A ID.

      4

      AWS Account B ID.

      5

      CIDR range of VPC in Account A.

      6

      CIDR range of VPC in Account B.

      7

      VPC ID in Account A (cluster)

      8

      VPC ID in Account B (EFS cross account)

      9

      Any writeable directory of choice to use to store temporary files.

      10

      If your driver is installed in a non-default namespace, change this value.

      11

      Makes AWS CLI output everything directly to stdout.

   2. Create the working directory by running the following command:

      mkdir -p $SCRATCH_DIR

   3. Verify cluster connectivity by running the following command in the OpenShift Container Platform CLI:

      $ oc whoami

   4. Determine the OpenShift Container Platform cluster type and set node selector:

      The EFS cross account feature requires assigning AWS IAM policies to nodes running EFS CSI controller pods. However, this is not consistent for every OpenShift Container Platform type.

      • If your cluster is deployed as a Hosted Control Plane (HyperShift), set the

        NODE_SELECTOR

        environment variable to hold the worker node label by running the following command:

        export NODE_SELECTOR=node-role.kubernetes.io/worker

      • For all other OpenShift Container Platform types, set the

        NODE_SELECTOR

        environment variable to hold the master node label by running the following command:

        export NODE_SELECTOR=node-role.kubernetes.io/master

   5. Configure AWS CLI profiles as environment variables for account switching by running the following commands:

      export AWS_ACCOUNT_A="<ACCOUNT_A_NAME>"
      export AWS_ACCOUNT_B="<ACCOUNT_B_NAME>"

   6. Ensure that your AWS CLI is configured with JSON output format as the default for both accounts by running the following commands:

      export AWS_DEFAULT_PROFILE=${AWS_ACCOUNT_A}
      aws configure get output
      export AWS_DEFAULT_PROFILE=${AWS_ACCOUNT_B}
      aws configure get output

      If the preceding commands return:

      • No value: The default output format is already set to JSON and no changes are required.
      • Any value: Reconfigure your AWS CLI to use JSON format. For information about changing output formats, see Setting the output format in the AWS CLI in the AWS documentation.

   7. Unset

      AWS_PROFILE

      in your shell to prevent conflicts with

      AWS_DEFAULT_PROFILE

      by running the following command:

      unset AWS_PROFILE

2. Configure the AWS Account B IAM roles and policies:

   1. Switch to your Account B profile by running the following command:

      export AWS_DEFAULT_PROFILE=${AWS_ACCOUNT_B}

   2. Define the IAM role name for the EFS CSI Driver Operator by running the following command:

      export ACCOUNT_B_ROLE_NAME=${CLUSTER_NAME}-cross-account-aws-efs-csi-operator

   3. Create the IAM trust policy file by running the following command:

      cat <<EOF > $SCRATCH_DIR/AssumeRolePolicyInAccountB.json
      {
          "Version": "2012-10-17",
          "Statement": [
              {
                  "Effect": "Allow",
                  "Principal": {
                      "AWS": "arn:aws:iam::${AWS_ACCOUNT_A_ID}:root"
                  },
                  "Action": "sts:AssumeRole",
                  "Condition": {}
              }
          ]
      }
      EOF

   4. Create the IAM role for the EFS CSI Driver Operator by running the following command:

      ACCOUNT_B_ROLE_ARN=$(aws iam create-role \
        --role-name "${ACCOUNT_B_ROLE_NAME}" \
        --assume-role-policy-document file://$SCRATCH_DIR/AssumeRolePolicyInAccountB.json \
        --query "Role.Arn" --output text) \
      && echo $ACCOUNT_B_ROLE_ARN

   5. Create the IAM policy file by running the following command:

      cat << EOF > $SCRATCH_DIR/EfsPolicyInAccountB.json
      {
          "Version": "2012-10-17",
          "Statement": [
              {
                  "Sid": "VisualEditor0",
                  "Effect": "Allow",
                  "Action": [
                      "ec2:DescribeNetworkInterfaces",
                      "ec2:DescribeSubnets"
                  ],
                  "Resource": "*"
              },
              {
                  "Sid": "VisualEditor1",
                  "Effect": "Allow",
                  "Action": [
                      "elasticfilesystem:DescribeMountTargets",
                      "elasticfilesystem:DeleteAccessPoint",
                      "elasticfilesystem:ClientMount",
                      "elasticfilesystem:DescribeAccessPoints",
                      "elasticfilesystem:ClientWrite",
                      "elasticfilesystem:ClientRootAccess",
                      "elasticfilesystem:DescribeFileSystems",
                      "elasticfilesystem:CreateAccessPoint",
                      "elasticfilesystem:TagResource"
                  ],
                  "Resource": "*"
              }
          ]
      }
      EOF

   6. Create the IAM policy by running the following command:

      ACCOUNT_B_POLICY_ARN=$(aws iam create-policy --policy-name "${CLUSTER_NAME}-efs-csi-policy" \
         --policy-document file://$SCRATCH_DIR/EfsPolicyInAccountB.json \
         --query 'Policy.Arn' --output text) \
      && echo ${ACCOUNT_B_POLICY_ARN}

   7. Attach the policy to the role by running the following command:

      aws iam attach-role-policy \
         --role-name "${ACCOUNT_B_ROLE_NAME}" \
         --policy-arn "${ACCOUNT_B_POLICY_ARN}"

3. Configure the AWS Account A IAM roles and policies:

   1. Switch to your Account A profile by running the following command:

      export AWS_DEFAULT_PROFILE=${AWS_ACCOUNT_A}

   2. Create the IAM policy document by running the following command:

      cat << EOF > $SCRATCH_DIR/AssumeRoleInlinePolicyPolicyInAccountA.json
      {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Effect": "Allow",
            "Action": "sts:AssumeRole",
            "Resource": "${ACCOUNT_B_ROLE_ARN}"
          }
        ]
      }
      EOF

   3. In AWS Account A, attach the AWS-managed policy "AmazonElasticFileSystemClientFullAccess" to the OpenShift Container Platform cluster master role by running the following command:

      EFS_CLIENT_FULL_ACCESS_BUILTIN_POLICY_ARN=arn:aws:iam::aws:policy/AmazonElasticFileSystemClientFullAccess
      declare -A ROLE_SEEN
      for NODE in $(oc get nodes --selector="${NODE_SELECTOR}" -o jsonpath='{.items[*].metadata.name}'); do
          INSTANCE_PROFILE=$(aws ec2 describe-instances \
              --filters "Name=private-dns-name,Values=${NODE}" \
              --query 'Reservations[].Instances[].IamInstanceProfile.Arn' \
              --output text | awk -F'/' '{print $NF}' | xargs)
          MASTER_ROLE_ARN=$(aws iam get-instance-profile \
              --instance-profile-name "${INSTANCE_PROFILE}" \
              --query 'InstanceProfile.Roles[0].Arn' \
              --output text | xargs)
          MASTER_ROLE_NAME=$(echo "${MASTER_ROLE_ARN}" | awk -F'/' '{print $NF}' | xargs)
          echo "Checking role: '${MASTER_ROLE_NAME}'"
          if [[ -n "${ROLE_SEEN[$MASTER_ROLE_NAME]:-}" ]]; then
              echo "Already processed role: '${MASTER_ROLE_NAME}', skipping."
              continue
          fi
          ROLE_SEEN["$MASTER_ROLE_NAME"]=1
          echo "Assigning policy ${EFS_CLIENT_FULL_ACCESS_BUILTIN_POLICY_ARN} to role ${MASTER_ROLE_NAME}"
          aws iam attach-role-policy --role-name "${MASTER_ROLE_NAME}" --policy-arn "${EEFS_CLIENT_FULL_ACCESS_BUILTIN_POLICY_ARN}"
      done

4. Attach the policy to the IAM entity to allow role assumption:

   This step depends on your cluster configuration. In both of the following scenarios, the EFS CSI Driver Operator uses an entity to authenticate to AWS, and this entity must be granted permission to assume roles in Account B.

   If your cluster:

   • Does not have STS enabled: The EFS CSI Driver Operator uses an IAM User entity for AWS authentication. Continue with the step "Attach policy to IAM User to allow role assumption".
   • Has STS enabled: The EFS CSI Driver Operator uses an IAM role entity for AWS authentication. Continue with the step "Attach policy to IAM Role to allow role assumption".

5. Attach policy to IAM User to allow role assumption

   1. Identify the IAM User used by the EFS CSI Driver Operator by running the following command:

      EFS_CSI_DRIVER_OPERATOR_USER=$(oc -n openshift-cloud-credential-operator get credentialsrequest/openshift-aws-efs-csi-driver -o json | jq -r '.status.providerStatus.user')

   2. Attach the policy to the IAM user by running the following command:

      aws iam put-user-policy \
          --user-name "${EFS_CSI_DRIVER_OPERATOR_USER}"  \
          --policy-name efs-cross-account-inline-policy \
          --policy-document file://$SCRATCH_DIR/AssumeRoleInlinePolicyPolicyInAccountA.json

6. Attach the policy to the IAM role to allow role assumption:

   1. Identify the IAM role name currently used by the EFS CSI Driver Operator by running the following command:

      EFS_CSI_DRIVER_OPERATOR_ROLE=$(oc -n ${CSI_DRIVER_NAMESPACE} get secret/aws-efs-cloud-credentials -o jsonpath='{.data.credentials}' | base64 -d | grep role_arn | cut -d'/' -f2) && echo ${EFS_CSI_DRIVER_OPERATOR_ROLE}

   2. Attach the policy to the IAM role used by the EFS CSI Driver Operator by running the following command:

       aws iam put-role-policy \
          --role-name "${EFS_CSI_DRIVER_OPERATOR_ROLE}"  \
          --policy-name efs-cross-account-inline-policy \
          --policy-document file://$SCRATCH_DIR/AssumeRoleInlinePolicyPolicyInAccountA.json

7. Configure VPC peering:

   1. Initiate a peering request from Account A to Account B by running the following command:

      export AWS_DEFAULT_PROFILE=${AWS_ACCOUNT_A}
      PEER_REQUEST_ID=$(aws ec2 create-vpc-peering-connection --vpc-id "${AWS_ACCOUNT_A_VPC_ID}" --peer-vpc-id "${AWS_ACCOUNT_B_VPC_ID}" --peer-owner-id "${AWS_ACCOUNT_B_ID}" --query VpcPeeringConnection.VpcPeeringConnectionId --output text)

   2. Accept the peering request from Account B by running the following command:

      export AWS_DEFAULT_PROFILE=${AWS_ACCOUNT_B}
      aws ec2 accept-vpc-peering-connection --vpc-peering-connection-id "${PEER_REQUEST_ID}"

   3. Retrieve the route table IDs for Account A and add routes to the Account B VPC by running the following command:

      export AWS_DEFAULT_PROFILE=${AWS_ACCOUNT_A}
      for NODE in $(oc get nodes --selector=node-role.kubernetes.io/worker | tail -n +2 | awk '{print $1}')
      do
          SUBNET=$(aws ec2 describe-instances --filters "Name=private-dns-name,Values=$NODE" --query 'Reservations[*].Instances[*].NetworkInterfaces[*].SubnetId' | jq -r '.[0][0][0]')
          echo SUBNET is ${SUBNET}
          ROUTE_TABLE_ID=$(aws ec2 describe-route-tables --filters "Name=association.subnet-id,Values=${SUBNET}" --query 'RouteTables[*].RouteTableId' | jq -r '.[0]')
          echo Route table ID is $ROUTE_TABLE_ID
          aws ec2 create-route --route-table-id ${ROUTE_TABLE_ID} --destination-cidr-block ${AWS_ACCOUNT_B_VPC_CIDR} --vpc-peering-connection-id ${PEER_REQUEST_ID}
      done

   4. Retrieve the route table IDs for Account B and add routes to the Account A VPC by running the following command:

      export AWS_DEFAULT_PROFILE=${AWS_ACCOUNT_B}
      for ROUTE_TABLE_ID in $(aws ec2 describe-route-tables   --filters "Name=vpc-id,Values=${AWS_ACCOUNT_B_VPC_ID}"   --query "RouteTables[].RouteTableId" | jq -r '.[]')
      do
          echo Route table ID is $ROUTE_TABLE_ID
          aws ec2 create-route --route-table-id ${ROUTE_TABLE_ID} --destination-cidr-block ${AWS_ACCOUNT_A_VPC_CIDR} --vpc-peering-connection-id ${PEER_REQUEST_ID}
      done

8. Configure security groups in Account B to allow NFS traffic from Account A to EFS:

   1. Switch to your Account B profile by running the following command:

      export AWS_DEFAULT_PROFILE=${AWS_ACCOUNT_B}

   2. Configure the VPC security groups for EFS access by running the following command:

      SECURITY_GROUP_ID=$(aws ec2 describe-security-groups --filters Name=vpc-id,Values="${AWS_ACCOUNT_B_VPC_ID}" | jq -r '.SecurityGroups[].GroupId')
      aws ec2 authorize-security-group-ingress \
       --group-id "${SECURITY_GROUP_ID}" \
       --protocol tcp \
       --port 2049 \
       --cidr "${AWS_ACCOUNT_A_VPC_CIDR}" | jq .

9. Create a region-wide EFS filesystem in Account B:

   1. Switch to your Account B profile by running the following command:

      export AWS_DEFAULT_PROFILE=${AWS_ACCOUNT_B}

   2. Create a region-wide EFS file system by running the following command:

      CROSS_ACCOUNT_FS_ID=$(aws efs create-file-system --creation-token efs-token-1 \
      --region ${AWS_REGION} \
      --encrypted | jq -r '.FileSystemId') \
      && echo $CROSS_ACCOUNT_FS_ID

   3. Configure region-wide mount targets for EFS by running the following command:

      for SUBNET in $(aws ec2 describe-subnets \
        --filters "Name=vpc-id,Values=${AWS_ACCOUNT_B_VPC_ID}" \
        --region ${AWS_REGION} \
        | jq -r '.Subnets.[].SubnetId'); do \
          MOUNT_TARGET=$(aws efs create-mount-target --file-system-id ${CROSS_ACCOUNT_FS_ID} \
          --subnet-id ${SUBNET} \
          --region ${AWS_REGION} \
          | jq -r '.MountTargetId'); \
          echo ${MOUNT_TARGET}; \
      done

      This creates a mount point in each subnet of your VPC.

10. Configure the EFS Operator for cross-account access:

    1. Define custom names for the secret and storage class that you will create in subsequent steps by running the following command:

       export SECRET_NAME=my-efs-cross-account
       export STORAGE_CLASS_NAME=efs-sc-cross

    2. Create a secret that references the role ARN in Account B by running the following command in the OpenShift Container Platform CLI:

       oc create secret generic ${SECRET_NAME} -n ${CSI_DRIVER_NAMESPACE} --from-literal=awsRoleArn="${ACCOUNT_B_ROLE_ARN}"

    3. Grant the CSI driver controller access to the newly created secret by running the following commands in the OpenShift Container Platform CLI:

       oc -n ${CSI_DRIVER_NAMESPACE} create role access-secrets --verb=get,list,watch --resource=secrets
       oc -n ${CSI_DRIVER_NAMESPACE} create rolebinding --role=access-secrets default-to-secrets --serviceaccount=${CSI_DRIVER_NAMESPACE}:aws-efs-csi-driver-controller-sa

    4. Create a new storage class that references the EFS ID from Account B and the secret created previously by running the following command in the OpenShift Container Platform CLI:

       cat << EOF | oc apply -f -
       kind: StorageClass
       apiVersion: storage.k8s.io/v1
       metadata:
         name: ${STORAGE_CLASS_NAME}
       provisioner: efs.csi.aws.com
       parameters:
         provisioningMode: efs-ap
         fileSystemId: ${CROSS_ACCOUNT_FS_ID}
         directoryPerms: "700"
         gidRangeStart: "1000"
         gidRangeEnd: "2000"
         basePath: "/dynamic_provisioning"
         csi.storage.k8s.io/provisioner-secret-name: ${SECRET_NAME}
         csi.storage.k8s.io/provisioner-secret-namespace: ${CSI_DRIVER_NAMESPACE}
       EOF

1. Log in to the web console.
2. Stop all applications that use AWS EFS PVs.

3. Delete all AWS EFS PVs:

   1. Click Storage PersistentVolumeClaims.
   2. Select each PVC that is in use by the AWS EFS CSI Driver Operator, click the drop-down menu on the far right of the PVC, and then click Delete PersistentVolumeClaims.

4. Uninstall the AWS EFS CSI driver:

   Note

   Before you can uninstall the Operator, you must remove the CSI driver first.

   1. Click Administration CustomResourceDefinitions ClusterCSIDriver.
   2. On the Instances tab, for efs.csi.aws.com, on the far left side, click the drop-down menu, and then click Delete ClusterCSIDriver.
   3. When prompted, click Delete.

5. Uninstall the AWS EFS CSI Operator:

   1. Click Operators Installed Operators.
   2. On the Installed Operators page, scroll or type AWS EFS CSI into the Search by name box to find the Operator, and then click it.
   3. On the upper, right of the Installed Operators > Operator details page, click Actions Uninstall Operator.

   4. When prompted on the Uninstall Operator window, click the Uninstall button to remove the Operator from the namespace. Any applications deployed by the Operator on the cluster need to be cleaned up manually.

      After uninstalling, the AWS EFS CSI Driver Operator is no longer listed in the Installed Operators section of the web console.

1. Create a storage class designating the storage account type using a YAML file similar to the following:

   $ oc create -f - << EOF
   apiVersion: storage.k8s.io/v1
   kind: StorageClass
   metadata:
     name: <storage-class> 

   1

   
   provisioner: disk.csi.azure.com
   parameters:
     skuName: <storage-class-account-type> 

   2

   
   reclaimPolicy: Delete
   volumeBindingMode: WaitForFirstConsumer
   allowVolumeExpansion: true
   EOF

   1

   Storage class name.

   2

   Storage account type. This corresponds to your Azure storage account SKU tier:`Standard_LRS`, Premium_LRS, StandardSSD_LRS, UltraSSD_LRS, Premium_ZRS, StandardSSD_ZRS, PremiumV2_LRS.

   Note

   For PremiumV2_LRS, specify

   cachingMode: None

   in

   storageclass.parameters

   .

2. Ensure that the storage class was created by listing the storage classes:

   $ oc get storageclass

   Example output

   $ oc get storageclass
   NAME                    PROVISIONER          RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
   azurefile-csi           file.csi.azure.com   Delete          Immediate              true                   68m
   managed-csi (default)   disk.csi.azure.com   Delete          WaitForFirstConsumer   true                   68m
   sc-prem-zrs             disk.csi.azure.com   Delete          WaitForFirstConsumer   true                   4m25s 

   1

   1

   New storage class with storage account type.

1. If you have not yet created a secret for the Azure storage account, create it now:

   This secret must contain the Azure Storage Account name and key with the following very specific format with two key-value pairs:

   • azurestorageaccountname

     : <storage_account_name>

   • azurestorageaccountkey

     : <account_key>

     To create a secret named azure-secret, run the following command:

     oc create secret generic azure-secret  -n <namespace_name> --type=Opaque --from-literal=azurestorageaccountname="<storage_account_name>" --from-literal=azurestorageaccountkey="<account_key>" 

     1

     2

     1

     Set <namespace_name> to the namespace where the PV is consumed.

     2

     Provide your values for <storage_account_name> and <account_key>.

2. Create a PV by using the following example YAML file:

   Example PV YAML file

   apiVersion: v1
   kind: PersistentVolume
   metadata:
     annotations:
       pv.kubernetes.io/provisioned-by: file.csi.azure.com
     name: pv-azurefile
   spec:
     capacity:
       storage: 5Gi 

   1

   
     accessModes:
       - ReadWriteMany 

   2

   
     persistentVolumeReclaimPolicy: Retain 

   3

   
     storageClassName: <sc-name> 

   4

   
     mountOptions:
       - dir_mode=0777  

   5

   
       - file_mode=0777
       - uid=0
       - gid=0
       - cache=strict  

   6

   
       - nosharesock  

   7

   
       - actimeo=30  

   8

   
       - nobrl  

   9

   
     csi:
       driver: file.csi.azure.com
       volumeHandle: "{resource-group-name}#{account-name}#{file-share-name}" 

   10

   
       volumeAttributes:
         shareName: EXISTING_FILE_SHARE_NAME  

   11

   
       nodeStageSecretRef:
         name: azure-secret 

   12

   
         namespace: <my-namespace> 

   13

   1

   Volume size.

   2

   Access mode. Defines the read-write and mount permissions. For more information, under Additional resources, see Access modes.

   3

   Reclaim policy. Tells the cluster what to do with the volume after it is released. Accepted values are Retain, Recycle, or Delete.

   4

   Storage class name. This name is used by the PVC to bind to this specific PV. For static provisioning, a StorageClass object does not need to exist, but the name in the PV and PVC must match.

   5

   Modify this permission if you want to enhance the security.

   6

   Cache mode. Accepted values are none, strict, and loose. The default is strict.

   7

   Use to reduce the probability of a reconnect race.

   8

   The time (in seconds) that the CIFS client caches attributes of a file or directory before it requests attribute information from a server.

   9

   Disables sending byte range lock requests to the server, and for applications which have challenges with POSIX locks.

   10

   Ensure that volumeHandle is unique across the cluster. The resource-group-name is the Azure resource group where the storage account resides.

   11

   File share name. Use only the file share name; do not use full path.

   12

   Provide the name of the secret created in step 1 of this procedure. In this example, it is azure-secret.

   13

   The namespace that the secret was created in. This must be the namespace where the PV is consumed.

3. Create a PVC that references the PV using the following example file:

   Example PVC YAML file

   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: <pvc-name> 

   1

   
     namespace: <my-namespace> 

   2

   
   spec:
     volumeName: pv-azurefile 

   3

   
     storageClassName: <sc-name> 

   4

   
     accessModes:
       - ReadWriteMany 

   5

   
     resources:
       requests:
         storage: 5Gi 

   6

   1

   PVC name.

   2

   Namespace for the PVC.

   3

   The name of the PV that you created in the previous step.

   4

   Storage class name. This name is used by the PVC to bind to this specific PV. For static provisioning, a StorageClass object does not need to exist, but the name in the PV and PVC must match.

   5

   Access mode. Defines the requested read-write access for the PVC. Claims use the same conventions as volumes when requesting storage with specific access modes. For more information, under Additional resources, see Access modes.

   6

   PVC size.

4. Ensure that the PVC is created and in

   Bound

   status after a while by running the following command:

   $ oc get pvc <pvc-name> 

   1

   1

   The name of your PVC.

   Example output

   NAME       STATUS    VOLUME         CAPACITY   ACCESS MODES   STORAGECLASS   AGE
   pvc-name   Bound     pv-azurefile   5Gi        ReadWriteMany  my-sc          7m2s

1. Create a storage class with the Cloud KMS key. The following example enables dynamic provisioning of encrypted volumes:

   apiVersion: storage.k8s.io/v1
   kind: StorageClass
   metadata:
     name: csi-gce-pd-cmek
   provisioner: pd.csi.storage.gke.io
   volumeBindingMode: "WaitForFirstConsumer"
   allowVolumeExpansion: true
   parameters:
     type: pd-standard
     disk-encryption-kms-key: projects/<key-project-id>/locations/<location>/keyRings/<key-ring>/cryptoKeys/<key> 

   1

   1

   This field must be the resource identifier for the key that will be used to encrypt new disks. Values are case-sensitive. For more information about providing key ID values, see Retrieving a resource’s ID and Getting a Cloud KMS resource ID.

   Note

   You cannot add the

   disk-encryption-kms-key

   parameter to an existing storage class. However, you can delete the storage class and recreate it with the same name and a different set of parameters. If you do this, the provisioner of the existing class must be

   pd.csi.storage.gke.io

   .

2. Deploy the storage class on your OpenShift Container Platform cluster using the

   oc

   command:

   $ oc describe storageclass csi-gce-pd-cmek

   Example output

   Name:                  csi-gce-pd-cmek
   IsDefaultClass:        No
   Annotations:           None
   Provisioner:           pd.csi.storage.gke.io
   Parameters:            disk-encryption-kms-key=projects/key-project-id/locations/location/keyRings/ring-name/cryptoKeys/key-name,type=pd-standard
   AllowVolumeExpansion:  true
   MountOptions:          none
   ReclaimPolicy:         Delete
   VolumeBindingMode:     WaitForFirstConsumer
   Events:                none

3. Create a file named

   pvc.yaml

   that matches the name of your storage class object that you created in the previous step:

   kind: PersistentVolumeClaim
   apiVersion: v1
   metadata:
     name: podpvc
   spec:
     accessModes:
       - ReadWriteOnce
     storageClassName: csi-gce-pd-cmek
     resources:
       requests:
         storage: 6Gi

   Note

   If you marked the new storage class as default, you can omit the

   storageClassName

   field.

4. Apply the PVC on your cluster:

   $ oc apply -f pvc.yaml

5. Get the status of your PVC and verify that it is created and bound to a newly provisioned PV:

   $ oc get pvc

   Example output

   NAME      STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS     AGE
   podpvc    Bound     pvc-e36abf50-84f3-11e8-8538-42010a800002   10Gi       RWO            csi-gce-pd-cmek  9s

   Note

   If your storage class has the

   volumeBindingMode

   field set to

   WaitForFirstConsumer

   , you must create a pod to use the PVC before you can verify it.

1. Log in to the web console.

2. Enable the Filestore API in the GCE project by running the following command:

   $ gcloud services enable file.googleapis.com  --project <my_gce_project> 

   1

   1

   Replace <my_gce_project> with your Google Cloud project.

   You can also do this using Google Cloud web console.

3. Install the GCP Filestore CSI Operator:

   1. Click Operators OperatorHub.
   2. Locate the GCP Filestore CSI Operator by typing GCP Filestore in the filter box.
   3. Click the GCP Filestore CSI Driver Operator button.
   4. On the GCP Filestore CSI Driver Operator page, click Install.

   5. On the Install Operator page, ensure that:

      • All namespaces on the cluster (default) is selected.
      • Installed Namespace is set to openshift-cluster-csi-drivers.

   6. Click Install.

      After the installation finishes, the GCP Filestore CSI Operator is listed in the Installed Operators section of the web console.

4. Install the GCP Filestore CSI Driver:

   1. Click administration CustomResourceDefinitions ClusterCSIDriver.

   2. On the Instances tab, click Create ClusterCSIDriver.

      Use the following YAML file:

      apiVersion: operator.openshift.io/v1
      kind: ClusterCSIDriver
      metadata:
          name: filestore.csi.storage.gke.io
      spec:
        managementState: Managed

   3. Click Create.

   4. Wait for the following Conditions to change to a "true" status:

      • GCPFilestoreDriverCredentialsRequestControllerAvailable
      • GCPFilestoreDriverNodeServiceControllerAvailable
      • GCPFilestoreDriverControllerServiceControllerAvailable

1. Create a storage class using the following example YAML file:

   Example YAML file

   kind: StorageClass
   apiVersion: storage.k8s.io/v1
   metadata:
     name: filestore-csi
   provisioner: filestore.csi.storage.gke.io
   parameters:
     network: network-name 

   1

   
   allowVolumeExpansion: true
   volumeBindingMode: WaitForFirstConsumer

   1

   Specify the name of the GCP virtual private cloud (VPC) network where Filestore instances should be created in.

2. Specify the name of the VPC network where Filestore instances should be created in.

   It is recommended to specify the VPC network that the Filestore instances should be created in. If no VPC network is specified, the Container Storage Interface (CSI) driver tries to create the instances in the default VPC network of the project. On IPI installations, the VPC network name is typically the cluster name with the suffix "-network". However, on UPI installations, the VPC network name can be any value chosen by the user.

   You can find out the VPC network name by inspecting the

   MachineSets

   objects with the following command:

   $ oc -n openshift-machine-api get machinesets -o yaml | grep "network:"
               - network: gcp-filestore-network
   (...)

   In this example, the VPC network name in this cluster is "gcp-filestore-network".

1. List all PVCs that were created using the storage class

   filestore-csi

   :

   $ oc get pvc -o json -A | jq -r '.items[] | select(.spec.storageClassName == "filestore-csi")

2. Delete all of the PVCs listed by the previous command:

   $ oc delete <pvc-name> 

   1

   1

   Replace <pvc-name> with the name of any PVC that you need to delete.