Chapter 5. Using Container Storage Interface (CSI)

Procedure

1. Log in to the web console.
2. Click Administration > CustomResourceDefinitions.
3. On the CustomResourceDefinitions page, type clustercsidriver to find the ClusterCSIDriver object.
4. Click ClusterCSIDriver, and then click the Instances tab.
5. Click the name of the desired instance, and then click the YAML tab.

6. Add the spec.storageClassState field with a value of Managed, Unmanaged, or Removed.

   Example

   ...
   spec:
     driverConfig:
       driverType: ''
     logLevel: Normal
     managementState: Managed
     observedConfig: null
     operatorLogLevel: Normal
     storageClassState: Unmanaged
   ...

   For this example, spec.storageClassState field is set to "Unmanaged".

7. Click Save.

Procedure

1. List the storage classes by running the following command:

   $ oc get storageclass

   Example output

   NAME                 TYPE
   gp3 (default)        ebs.csi.aws.com
   standard             ebs.csi.aws.com

   The text (default) indicates the default storage class. In this example gp3 is the current default storage class.

2. Make the required storage class the default.

   For the required storage class, set the storageclass.kubernetes.io/is-default-class annotation to true by running the following command:

   $ oc patch storageclass standard -p '{"metadata": {"annotations": {"storageclass.kubernetes.io/is-default-class": "true"}}}'

   Note

   You can have many default storage classes for a short time. However, you must ensure that only one default storage class exists eventually.

   With many default storage classes present, any persistent volume claim (PVC) requesting the default storage class (pvc.spec.storageClassName=nil) gets the most recently created default storage class, regardless of the default status of that storage class. The administrator receives an alert in the alerts dashboard that there are many default storage classes, MultipleDefaultStorageClasses.

3. Remove the default storage class setting from the old default storage class.

   For the old default storage class, change the value of the storageclass.kubernetes.io/is-default-class annotation to false by running the following command:

   $ oc patch storageclass gp3 -p '{"metadata": {"annotations": {"storageclass.kubernetes.io/is-default-class": "false"}}}'

4. Verify the changes by running the following command:

   $ oc get storageclass

   Example output

   NAME                 TYPE
   gp3                  ebs.csi.aws.com
   standard (default)   ebs.csi.aws.com

   The standard storage class is now the default.

Procedure

1. If you are using AWS EFS with AWS Secure Token Service (STS), obtain a role Amazon Resource Name (ARN) for STS. This is required for installing the AWS EFS CSI Driver Operator.
2. Install the AWS EFS CSI Driver Operator.
3. Install the AWS EFS CSI Driver.

Procedure

1. Set up the environment:

   1. Configure environment variables by running the following commands:

      export CLUSTER_NAME="<CLUSTER_NAME>"
      export AWS_REGION="<AWS_REGION>"
      export AWS_ACCOUNT_A_ID="<ACCOUNT_A_ID>"
      export AWS_ACCOUNT_B_ID="<ACCOUNT_B_ID>"
      export AWS_ACCOUNT_A_VPC_CIDR="<VPC_A_CIDR>"
      export AWS_ACCOUNT_B_VPC_CIDR="<VPC_B_CIDR>"
      export AWS_ACCOUNT_A_VPC_ID="<VPC_A_ID>"
      export AWS_ACCOUNT_B_VPC_ID="<VPC_B_ID>"
      export SCRATCH_DIR="<WORKING_DIRECTORY>"
      export CSI_DRIVER_NAMESPACE="openshift-cluster-csi-drivers"
      export AWS_PAGER=""

      • <CLUSTER_NAME>: Cluster name of choice.
      • <AWS_REGION>: AWS region of choice.
      • <ACCOUNT_A_ID>: AWS Account A ID.
      • <ACCOUNT_B_ID>: AWS Account B ID.
      • <VPC_A_CIDR>: CIDR range of VPC in Account A.
      • <VPC_B_CIDR>: CIDR range of VPC in Account B.
      • <VPC_A_ID>: VPC ID in Account A (cluster)
      • <VPC_B_ID>: VPC ID in Account B (EFS cross account)
      • <WORKING_DIRECTORY>: Any writeable directory of choice to use to store temporary files.
      • CSI_DRIVER_NAMESPACE: If your driver is installed in a non-default namespace, change this value.
      • AWS_PAGER: Makes AWS CLI output everything directly to stdout.

   2. Create the working directory by running the following command:

      mkdir -p $SCRATCH_DIR

   3. Verify cluster connectivity by running the following command in the Red Hat OpenShift Service on AWS CLI:

      $ oc whoami

   4. Determine the Red Hat OpenShift Service on AWS cluster type and set node selector:

      The EFS cross account feature requires assigning AWS IAM policies to nodes running EFS CSI controller pods. However, this is not consistent for every Red Hat OpenShift Service on AWS type.

      • If your cluster is deployed as a Hosted Control Plane (HyperShift), set the NODE_SELECTOR environment variable to hold the worker node label by running the following command:

        export NODE_SELECTOR=node-role.kubernetes.io/worker

      • For all other Red Hat OpenShift Service on AWS types, set the NODE_SELECTOR environment variable to hold the master node label by running the following command:

        export NODE_SELECTOR=node-role.kubernetes.io/master

   5. Configure AWS CLI profiles as environment variables for account switching by running the following commands:

      export AWS_ACCOUNT_A="<ACCOUNT_A_NAME>"
      export AWS_ACCOUNT_B="<ACCOUNT_B_NAME>"

   6. Ensure that your AWS CLI is configured with JSON output format as the default for both accounts by running the following commands:

      export AWS_DEFAULT_PROFILE=${AWS_ACCOUNT_A}
      aws configure get output
      export AWS_DEFAULT_PROFILE=${AWS_ACCOUNT_B}
      aws configure get output

      If the preceding commands return:

      • No value: The default output format is already set to JSON and no changes are required.
      • Any value: Reconfigure your AWS CLI to use JSON format. For information about changing output formats, see Setting the output format in the AWS CLI in the AWS documentation.

   7. Unset AWS_PROFILE in your shell to prevent conflicts with AWS_DEFAULT_PROFILE by running the following command:

      unset AWS_PROFILE

2. Configure the AWS Account B IAM roles and policies:

   1. Switch to your Account B profile by running the following command:

      export AWS_DEFAULT_PROFILE=${AWS_ACCOUNT_B}

   2. Define the IAM role name for the EFS CSI Driver Operator by running the following command:

      export ACCOUNT_B_ROLE_NAME=${CLUSTER_NAME}-cross-account-aws-efs-csi-operator

   3. Create the IAM trust policy file by running the following command:

      cat <<EOF > $SCRATCH_DIR/AssumeRolePolicyInAccountB.json
      {
          "Version": "2012-10-17",
          "Statement": [
              {
                  "Effect": "Allow",
                  "Principal": {
                      "AWS": "arn:aws:iam::${AWS_ACCOUNT_A_ID}:root"
                  },
                  "Action": "sts:AssumeRole",
                  "Condition": {}
              }
          ]
      }
      EOF

   4. Create the IAM role for the EFS CSI Driver Operator by running the following command:

      ACCOUNT_B_ROLE_ARN=$(aws iam create-role \
        --role-name "${ACCOUNT_B_ROLE_NAME}" \
        --assume-role-policy-document file://$SCRATCH_DIR/AssumeRolePolicyInAccountB.json \
        --query "Role.Arn" --output text) \
      && echo $ACCOUNT_B_ROLE_ARN

   5. Create the IAM policy file by running the following command:

      cat << EOF > $SCRATCH_DIR/EfsPolicyInAccountB.json
      {
          "Version": "2012-10-17",
          "Statement": [
              {
                  "Sid": "VisualEditor0",
                  "Effect": "Allow",
                  "Action": [
                      "ec2:DescribeNetworkInterfaces",
                      "ec2:DescribeSubnets"
                  ],
                  "Resource": "*"
              },
              {
                  "Sid": "VisualEditor1",
                  "Effect": "Allow",
                  "Action": [
                      "elasticfilesystem:DescribeMountTargets",
                      "elasticfilesystem:DeleteAccessPoint",
                      "elasticfilesystem:ClientMount",
                      "elasticfilesystem:DescribeAccessPoints",
                      "elasticfilesystem:ClientWrite",
                      "elasticfilesystem:ClientRootAccess",
                      "elasticfilesystem:DescribeFileSystems",
                      "elasticfilesystem:CreateAccessPoint",
                      "elasticfilesystem:TagResource"
                  ],
                  "Resource": "*"
              }
          ]
      }
      EOF

   6. Create the IAM policy by running the following command:

      ACCOUNT_B_POLICY_ARN=$(aws iam create-policy --policy-name "${CLUSTER_NAME}-efs-csi-policy" \
         --policy-document file://$SCRATCH_DIR/EfsPolicyInAccountB.json \
         --query 'Policy.Arn' --output text) \
      && echo ${ACCOUNT_B_POLICY_ARN}

   7. Attach the policy to the role by running the following command:

      aws iam attach-role-policy \
         --role-name "${ACCOUNT_B_ROLE_NAME}" \
         --policy-arn "${ACCOUNT_B_POLICY_ARN}"

3. Configure the AWS Account A IAM roles and policies:

   1. Switch to your Account A profile by running the following command:

      export AWS_DEFAULT_PROFILE=${AWS_ACCOUNT_A}

   2. Create the IAM policy document by running the following command:

      cat << EOF > $SCRATCH_DIR/AssumeRoleInlinePolicyPolicyInAccountA.json
      {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Effect": "Allow",
            "Action": "sts:AssumeRole",
            "Resource": "${ACCOUNT_B_ROLE_ARN}"
          }
        ]
      }
      EOF

   3. In AWS Account A, attach the AWS-managed policy "AmazonElasticFileSystemClientFullAccess" to the Red Hat OpenShift Service on AWS cluster master role by running the following command:

      EFS_CLIENT_FULL_ACCESS_BUILTIN_POLICY_ARN=arn:aws:iam::aws:policy/AmazonElasticFileSystemClientFullAccess
      declare -A ROLE_SEEN
      for NODE in $(oc get nodes --selector="${NODE_SELECTOR}" -o jsonpath='{.items[*].metadata.name}'); do
          INSTANCE_PROFILE=$(aws ec2 describe-instances \
              --filters "Name=private-dns-name,Values=${NODE}" \
              --query 'Reservations[].Instances[].IamInstanceProfile.Arn' \
              --output text | awk -F'/' '{print $NF}' | xargs)
          MASTER_ROLE_ARN=$(aws iam get-instance-profile \
              --instance-profile-name "${INSTANCE_PROFILE}" \
              --query 'InstanceProfile.Roles[0].Arn' \
              --output text | xargs)
          MASTER_ROLE_NAME=$(echo "${MASTER_ROLE_ARN}" | awk -F'/' '{print $NF}' | xargs)
          echo "Checking role: '${MASTER_ROLE_NAME}'"
          if [[ -n "${ROLE_SEEN[$MASTER_ROLE_NAME]:-}" ]]; then
              echo "Already processed role: '${MASTER_ROLE_NAME}', skipping."
              continue
          fi
          ROLE_SEEN["$MASTER_ROLE_NAME"]=1
          echo "Assigning policy ${EFS_CLIENT_FULL_ACCESS_BUILTIN_POLICY_ARN} to role ${MASTER_ROLE_NAME}"
          aws iam attach-role-policy --role-name "${MASTER_ROLE_NAME}" --policy-arn "${EEFS_CLIENT_FULL_ACCESS_BUILTIN_POLICY_ARN}"
      done

4. Attach the policy to the IAM entity to allow role assumption:

   This step depends on your cluster configuration. In both of the following scenarios, the EFS CSI Driver Operator uses an entity to authenticate to AWS, and this entity must be granted permission to assume roles in Account B.

   If your cluster:

   • Does not have STS enabled: The EFS CSI Driver Operator uses an IAM User entity for AWS authentication. Continue with the step "Attach policy to IAM User to allow role assumption".
   • Has STS enabled: The EFS CSI Driver Operator uses an IAM role entity for AWS authentication. Continue with the step "Attach policy to IAM Role to allow role assumption".

5. Attach policy to IAM User to allow role assumption

   1. Identify the IAM User used by the EFS CSI Driver Operator by running the following command:

      EFS_CSI_DRIVER_OPERATOR_USER=$(oc -n openshift-cloud-credential-operator get credentialsrequest/openshift-aws-efs-csi-driver -o json | jq -r '.status.providerStatus.user')

   2. Attach the policy to the IAM user by running the following command:

      aws iam put-user-policy \
          --user-name "${EFS_CSI_DRIVER_OPERATOR_USER}"  \
          --policy-name efs-cross-account-inline-policy \
          --policy-document file://$SCRATCH_DIR/AssumeRoleInlinePolicyPolicyInAccountA.json

6. Attach the policy to the IAM role to allow role assumption:

   1. Identify the IAM role name currently used by the EFS CSI Driver Operator by running the following command:

      EFS_CSI_DRIVER_OPERATOR_ROLE=$(oc -n ${CSI_DRIVER_NAMESPACE} get secret/aws-efs-cloud-credentials -o jsonpath='{.data.credentials}' | base64 -d | grep role_arn | cut -d'/' -f2) && echo ${EFS_CSI_DRIVER_OPERATOR_ROLE}

   2. Attach the policy to the IAM role used by the EFS CSI Driver Operator by running the following command:

       aws iam put-role-policy \
          --role-name "${EFS_CSI_DRIVER_OPERATOR_ROLE}"  \
          --policy-name efs-cross-account-inline-policy \
          --policy-document file://$SCRATCH_DIR/AssumeRoleInlinePolicyPolicyInAccountA.json

7. Configure VPC peering:

   1. Initiate a peering request from Account A to Account B by running the following command:

      export AWS_DEFAULT_PROFILE=${AWS_ACCOUNT_A}
      PEER_REQUEST_ID=$(aws ec2 create-vpc-peering-connection --vpc-id "${AWS_ACCOUNT_A_VPC_ID}" --peer-vpc-id "${AWS_ACCOUNT_B_VPC_ID}" --peer-owner-id "${AWS_ACCOUNT_B_ID}" --query VpcPeeringConnection.VpcPeeringConnectionId --output text)

   2. Accept the peering request from Account B by running the following command:

      export AWS_DEFAULT_PROFILE=${AWS_ACCOUNT_B}
      aws ec2 accept-vpc-peering-connection --vpc-peering-connection-id "${PEER_REQUEST_ID}"

   3. Retrieve the route table IDs for Account A and add routes to the Account B VPC by running the following command:

      export AWS_DEFAULT_PROFILE=${AWS_ACCOUNT_A}
      for NODE in $(oc get nodes --selector=node-role.kubernetes.io/worker | tail -n +2 | awk '{print $1}')
      do
          SUBNET=$(aws ec2 describe-instances --filters "Name=private-dns-name,Values=$NODE" --query 'Reservations[*].Instances[*].NetworkInterfaces[*].SubnetId' | jq -r '.[0][0][0]')
          echo SUBNET is ${SUBNET}
          ROUTE_TABLE_ID=$(aws ec2 describe-route-tables --filters "Name=association.subnet-id,Values=${SUBNET}" --query 'RouteTables[*].RouteTableId' | jq -r '.[0]')
          echo Route table ID is $ROUTE_TABLE_ID
          aws ec2 create-route --route-table-id ${ROUTE_TABLE_ID} --destination-cidr-block ${AWS_ACCOUNT_B_VPC_CIDR} --vpc-peering-connection-id ${PEER_REQUEST_ID}
      done

   4. Retrieve the route table IDs for Account B and add routes to the Account A VPC by running the following command:

      export AWS_DEFAULT_PROFILE=${AWS_ACCOUNT_B}
      for ROUTE_TABLE_ID in $(aws ec2 describe-route-tables   --filters "Name=vpc-id,Values=${AWS_ACCOUNT_B_VPC_ID}"   --query "RouteTables[].RouteTableId" | jq -r '.[]')
      do
          echo Route table ID is $ROUTE_TABLE_ID
          aws ec2 create-route --route-table-id ${ROUTE_TABLE_ID} --destination-cidr-block ${AWS_ACCOUNT_A_VPC_CIDR} --vpc-peering-connection-id ${PEER_REQUEST_ID}
      done

8. Configure security groups in Account B to allow NFS traffic from Account A to EFS:

   1. Switch to your Account B profile by running the following command:

      export AWS_DEFAULT_PROFILE=${AWS_ACCOUNT_B}

   2. Configure the VPC security groups for EFS access by running the following command:

      SECURITY_GROUP_ID=$(aws ec2 describe-security-groups --filters Name=vpc-id,Values="${AWS_ACCOUNT_B_VPC_ID}" | jq -r '.SecurityGroups[].GroupId')
      aws ec2 authorize-security-group-ingress \
       --group-id "${SECURITY_GROUP_ID}" \
       --protocol tcp \
       --port 2049 \
       --cidr "${AWS_ACCOUNT_A_VPC_CIDR}" | jq .

9. Create a region-wide EFS filesystem in Account B:

   1. Switch to your Account B profile by running the following command:

      export AWS_DEFAULT_PROFILE=${AWS_ACCOUNT_B}

   2. Create a region-wide EFS file system by running the following command:

      CROSS_ACCOUNT_FS_ID=$(aws efs create-file-system --creation-token efs-token-1 \
      --region ${AWS_REGION} \
      --encrypted | jq -r '.FileSystemId') \
      && echo $CROSS_ACCOUNT_FS_ID

   3. Configure region-wide mount targets for EFS by running the following command:

      for SUBNET in $(aws ec2 describe-subnets \
        --filters "Name=vpc-id,Values=${AWS_ACCOUNT_B_VPC_ID}" \
        --region ${AWS_REGION} \
        | jq -r '.Subnets.[].SubnetId'); do \
          MOUNT_TARGET=$(aws efs create-mount-target --file-system-id ${CROSS_ACCOUNT_FS_ID} \
          --subnet-id ${SUBNET} \
          --region ${AWS_REGION} \
          | jq -r '.MountTargetId'); \
          echo ${MOUNT_TARGET}; \
      done

      This creates a mount point in each subnet of your VPC.

10. Configure the EFS Operator for cross-account access:

    1. Define custom names for the secret and storage class that you will create in subsequent steps by running the following command:

       export SECRET_NAME=my-efs-cross-account
       export STORAGE_CLASS_NAME=efs-sc-cross

    2. Create a secret that references the role ARN in Account B by running the following command in the Red Hat OpenShift Service on AWS CLI:

       oc create secret generic ${SECRET_NAME} -n ${CSI_DRIVER_NAMESPACE} --from-literal=awsRoleArn="${ACCOUNT_B_ROLE_ARN}"

    3. Grant the CSI driver controller access to the newly created secret by running the following commands in the Red Hat OpenShift Service on AWS CLI:

       oc -n ${CSI_DRIVER_NAMESPACE} create role access-secrets --verb=get,list,watch --resource=secrets
       oc -n ${CSI_DRIVER_NAMESPACE} create rolebinding --role=access-secrets default-to-secrets --serviceaccount=${CSI_DRIVER_NAMESPACE}:aws-efs-csi-driver-controller-sa

    4. Create a new storage class that references the EFS ID from Account B and the secret created previously by running the following command in the Red Hat OpenShift Service on AWS CLI:

       cat << EOF | oc apply -f -
       kind: StorageClass
       apiVersion: storage.k8s.io/v1
       metadata:
         name: ${STORAGE_CLASS_NAME}
       provisioner: efs.csi.aws.com
       parameters:
         provisioningMode: efs-ap
         fileSystemId: ${CROSS_ACCOUNT_FS_ID}
         directoryPerms: "700"
         gidRangeStart: "1000"
         gidRangeEnd: "2000"
         basePath: "/dynamic_provisioning"
         csi.storage.k8s.io/provisioner-secret-name: ${SECRET_NAME}
         csi.storage.k8s.io/provisioner-secret-namespace: ${CSI_DRIVER_NAMESPACE}
       EOF

Procedure

1. Log in to the web console.
2. Stop all applications that use AWS EFS PVs.

3. Delete all AWS EFS PVs:

   1. Click Storage PersistentVolumeClaims.
   2. Select each PVC that is in use by the AWS EFS CSI Driver Operator, click the drop-down menu on the far right of the PVC, and then click Delete PersistentVolumeClaims.

4. Uninstall the AWS EFS CSI driver:

   Note

   Before you can uninstall the Operator, you must remove the CSI driver first.

   1. Click Administration CustomResourceDefinitions ClusterCSIDriver.
   2. On the Instances tab, for efs.csi.aws.com, on the far left side, click the drop-down menu, and then click Delete ClusterCSIDriver.
   3. When prompted, click Delete.

5. Uninstall the AWS EFS CSI Operator:

   1. Click Ecosystem Installed Operators.
   2. On the Installed Operators page, scroll or type AWS EFS CSI into the Search by name box to find the Operator, and then click it.
   3. On the upper, right of the Installed Operators > Operator details page, click Actions Uninstall Operator.

   4. When prompted on the Uninstall Operator window, click the Uninstall button to remove the Operator from the namespace. Any applications deployed by the Operator on the cluster need to be cleaned up manually.

      After uninstalling, the AWS EFS CSI Driver Operator is no longer listed in the Installed Operators section of the web console.