Chapter 5. Using Container Storage Interface (CSI)

Procedure

1. Create an IAM policy JSON file with the following content:

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Action": [
           "elasticfilesystem:DescribeAccessPoints",
           "elasticfilesystem:DescribeFileSystems",
           "elasticfilesystem:DescribeMountTargets",
           "ec2:DescribeAvailabilityZones",
           "elasticfilesystem:TagResource"
         ],
         "Resource": "*"
       },
       {
         "Effect": "Allow",
         "Action": [
           "elasticfilesystem:CreateAccessPoint"
         ],
         "Resource": "*",
         "Condition": {
           "StringLike": {
             "aws:RequestTag/efs.csi.aws.com/cluster": "true"
           }
         }
       },
       {
         "Effect": "Allow",
         "Action": "elasticfilesystem:DeleteAccessPoint",
         "Resource": "*",
         "Condition": {
           "StringEquals": {
             "aws:ResourceTag/efs.csi.aws.com/cluster": "true"
           }
         }
       }
     ]
   }

2. Create an IAM trust JSON file with the following content:

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Principal": {
           "Federated": "arn:aws:iam::<your_aws_account_ID>:oidc-provider/<openshift_oidc_provider>"  1
         },
         "Action": "sts:AssumeRoleWithWebIdentity",
         "Condition": {
           "StringEquals": {
             "<openshift_oidc_provider>:sub": [  2
               "system:serviceaccount:openshift-cluster-csi-drivers:aws-efs-csi-driver-operator",
               "system:serviceaccount:openshift-cluster-csi-drivers:aws-efs-csi-driver-controller-sa"
             ]
           }
         }
       }
     ]
   }

   1

   Specify your AWS account ID and the OpenShift OIDC provider endpoint.

   Obtain your AWS account ID by running the following command:

   $ aws sts get-caller-identity --query Account --output text

   Obtain the OpenShift OIDC endpoint by running the following command:

   $ openshift_oidc_provider=`oc get authentication.config.openshift.io cluster \
     -o json | jq -r .spec.serviceAccountIssuer | sed -e "s/^https:\/\///"`; \
     echo $openshift_oidc_provider

   2

   Specify the OpenShift OIDC endpoint again.

3. Create the IAM role:

   ROLE_ARN=$(aws iam create-role \
     --role-name "<your_cluster_name>-aws-efs-csi-operator" \
     --assume-role-policy-document file://<your_trust_file_name>.json \
     --query "Role.Arn" --output text); echo $ROLE_ARN

   Copy the role ARN. You will need it when you install the AWS EFS CSI Driver Operator.

4. Create the IAM policy:

   POLICY_ARN=$(aws iam create-policy \
     --policy-name "<your_cluster_name>-aws-efs-csi" \
     --policy-document file://<your_policy_file_name>.json \
     --query 'Policy.Arn' --output text); echo $POLICY_ARN

5. Attach the IAM policy to the IAM role:

   $ aws iam attach-role-policy \
     --role-name "<your_cluster_name>-aws-efs-csi-operator" \
     --policy-arn $POLICY_ARN

1. Log in to the web console.

2. Install the AWS EFS CSI Operator:

   1. Click Operators OperatorHub.
   2. Locate the AWS EFS CSI Operator by typing AWS EFS CSI in the filter box.
   3. Click the AWS EFS CSI Driver Operator button.
   Important

   Be sure to select the AWS EFS CSI Driver Operator and not the AWS EFS Operator. The AWS EFS Operator is a community Operator and is not supported by Red Hat.

   1. On the AWS EFS CSI Driver Operator page, click Install.

   2. On the Install Operator page, ensure that:

      • If you are using AWS EFS with AWS Secure Token Service (STS), in the role ARN field, enter the ARN role copied from the last step of the Obtaining a role Amazon Resource Name for Security Token Service procedure.
      • All namespaces on the cluster (default) is selected.
      • Installed Namespace is set to openshift-cluster-csi-drivers.

   3. Click Install.

      After the installation finishes, the AWS EFS CSI Operator is listed in the Installed Operators section of the web console.

Procedure

1. Click Administration CustomResourceDefinitions ClusterCSIDriver.
2. On the Instances tab, click Create ClusterCSIDriver.

3. Use the following YAML file:

   apiVersion: operator.openshift.io/v1
   kind: ClusterCSIDriver
   metadata:
       name: efs.csi.aws.com
   spec:
     managementState: Managed

4. Click Create.

5. Wait for the following Conditions to change to a "True" status:

   • AWSEFSDriverNodeServiceControllerAvailable
   • AWSEFSDriverControllerServiceControllerAvailable

1. Click Administration > CustomResourceDefinitions.
2. On the CustomResourceDefinitions page next to the Name dropdown box, type clustercsidriver.
3. Click CRD ClusterCSIDriver.
4. Click the YAML tab.

5. Under spec.aws.efsVolumeMetrics.state, set the value to RecursiveWalk.

   RecursiveWalk indicates that volume metrics collection in the AWS EFS CSI Driver is performed by recursively walking through the files in the volume.

   Example ClusterCSIDriver efs.csi.aws.com YAML file

   spec:
       driverConfig:
           driverType: AWS
           aws:
               efsVolumeMetrics:
                 state: RecursiveWalk
                 recursiveWalk:
                   refreshPeriodMinutes: 100
                   fsRateLimit: 10

6. Optional: To define how the recursive walk operates, you can also set the following fields:

   • refreshPeriodMinutes: Specifies the refresh frequency for volume metrics in minutes. If this field is left blank, a reasonable default is chosen, which is subject to change over time. The current default is 240 minutes. The valid range is 1 to 43,200 minutes.
   • fsRateLimit: Defines the rate limit for processing volume metrics in goroutines per file system. If this field is left blank, a reasonable default is chosen, which is subject to change over time. The current default is 5 goroutines. The valid range is 1 to 100 goroutines.
7. Click Save.

1. Edit ClusterCSIDriver by running the following command:

   $ oc edit clustercsidriver efs.csi.aws.com

2. Under spec.aws.efsVolumeMetrics.state, set the value to RecursiveWalk.

   RecursiveWalk indicates that volume metrics collection in the AWS EFS CSI Driver is performed by recursively walking through the files in the volume.

   Example ClusterCSIDriver efs.csi.aws.com YAML file

   spec:
       driverConfig:
           driverType: AWS
           aws:
               efsVolumeMetrics:
                 state: RecursiveWalk
                 recursiveWalk:
                   refreshPeriodMinutes: 100
                   fsRateLimit: 10

3. Optional: To define how the recursive walk operates, you can also set the following fields:

   • refreshPeriodMinutes: Specifies the refresh frequency for volume metrics in minutes. If this field is left blank, a reasonable default is chosen, which is subject to change over time. The current default is 240 minutes. The valid range is 1 to 43,200 minutes.
   • fsRateLimit: Defines the rate limit for processing volume metrics in goroutines per file system. If this field is left blank, a reasonable default is chosen, which is subject to change over time. The current default is 5 goroutines. The valid range is 1 to 100 goroutines.
4. Save the changes to the efs.csi.aws.com object.

1. Create a persistent volume claim (PVC) that uses the hyperdisk-specific storage class using the following example YAML file:

   Example PVC YAML file

   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: my-pvc
   spec:
     storageClassName: hyperdisk-sc 1
     accessModes:
     - ReadWriteOnce
     resources:
       requests:
         storage: 2048Gi 2

   1

   PVC references the the storage pool-specific storage class. In this example, hyperdisk-sc.

   2

   Target storage capacity of the hyperdisk-balanced volume. In this example, 2048Gi.

2. Create a deployment that uses the PVC that you just created. Using a deployment helps ensure that your application has access to the persistent storage even after the pod restarts and rescheduling:

   1. Ensure a node pool with the specified machine series is up and running before creating the deployment. Otherwise, the pod fails to schedule.

   2. Use the following example YAML file to create the deployment:

      Example deployment YAML file

      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: postgres
      spec:
        selector:
          matchLabels:
            app: postgres
        template:
          metadata:
            labels:
              app: postgres
          spec:
            nodeSelector:
              cloud.google.com/machine-family: n4 1
            containers:
            - name: postgres
              image: postgres:14-alpine
              args: [ "sleep", "3600" ]
              volumeMounts:
              - name: sdk-volume
                mountPath: /usr/share/data/
            volumes:
            - name: sdk-volume
              persistentVolumeClaim:
                claimName: my-pvc 2

      1

      Specifies the machine family. In this example, it is n4.

      2

      Specifies the name of the PVC created in the preceding step. In this example, it is my-pfc.

   3. Confirm that the deployment was successfully created by running the following command:

      $ oc get deployment

      Example output

      NAME       READY   UP-TO-DATE   AVAILABLE   AGE
      postgres   0/1     1            0           42s

      It might take a few minutes for hyperdisk instances to complete provisioning and display a READY status.

   4. Confirm that PVC my-pvc has been successfully bound to a persistent volume (PV) by running the following command:

      $ oc get pvc my-pvc

      Example output

      NAME          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS       VOLUMEATTRIBUTESCLASS  AGE
      my-pvc        Bound    pvc-1ff52479-4c81-4481-aa1d-b21c8f8860c6   2Ti        RWO            hyperdisk-sc       <unset>                2m24s

   5. Confirm the expected configuration of your hyperdisk-balanced disk:

      $ gcloud compute disks list

      Example output

      NAME                                        LOCATION        LOCATION_SCOPE  SIZE_GB  TYPE                STATUS
      instance-20240914-173145-boot               us-central1-a   zone            150      pd-standard         READY
      instance-20240914-173145-data-workspace     us-central1-a   zone            100      pd-balanced         READY
      c4a-rhel-vm                                 us-central1-a   zone            50       hyperdisk-balanced  READY 1

      1

      Hyperdisk-balanced disk.

   6. If using storage pools, check that the volume is provisioned as specified in your storage class and PVC by running the following command:

      $ gcloud compute storage-pools list-disks pool-us-east4-c --zone=us-east4-c

      Example output

      NAME                                      STATUS  PROVISIONED_IOPS  PROVISIONED_THROUGHPUT  SIZE_GB
      pvc-1ff52479-4c81-4481-aa1d-b21c8f8860c6  READY   3000              140                     2048

1. Obtain the project number:

   1. Obtain the project ID by running the following command:

      $ export PROJECT_ID=$(oc get infrastructure/cluster -o jsonpath='{.status.platformStatus.gcp.projectID}')

   2. Obtain the project number, using the project ID, by running the following command:

      $ gcloud projects describe $PROJECT_ID --format="value(projectNumber)"

2. Find the identity pool ID and the provider ID:

   During cluster installation, the names of these resources are provided to the Cloud Credential Operator utility (ccoctl) with the --name parameter. See "Creating GCP resources with the Cloud Credential Operator utility".

3. Create Workload Identity resources for the GCP Filestore Operator:

   1. Create a CredentialsRequest file using the following example file:

      Example Credentials Request YAML file

      apiVersion: cloudcredential.openshift.io/v1
      kind: CredentialsRequest
      metadata:
        name: openshift-gcp-filestore-csi-driver-operator
        namespace: openshift-cloud-credential-operator
        annotations:
          include.release.openshift.io/self-managed-high-availability: "true"
          include.release.openshift.io/single-node-developer: "true"
      spec:
        serviceAccountNames:
        - gcp-filestore-csi-driver-operator
        - gcp-filestore-csi-driver-controller-sa
        secretRef:
          name: gcp-filestore-cloud-credentials
          namespace: openshift-cluster-csi-drivers
        providerSpec:
          apiVersion: cloudcredential.openshift.io/v1
      	kind: GCPProviderSpec
          predefinedRoles:
          - roles/file.editor
          - roles/resourcemanager.tagUser
          skipServiceCheck: true

   2. Use the CredentialsRequest file to create a GCP service account by running the following command:

      $ ./ccoctl gcp create-service-accounts --name=<filestore-service-account> \1
        --workload-identity-pool=<workload-identity-pool> \2
        --workload-identity-provider=<workload-identity-provider> \3
        --project=<project-id> \4
        --credentials-requests-dir=/tmp/credreq 5

      1

      <filestore-service-account> is a user-chosen name.

      2

      <workload-identity-pool> comes from Step 2 above.

      3

      <workload-identity-provider> comes from Step 2 above.

      4

      <project-id> comes from Step 1.a above.

      5

      The name of directory where the CredentialsRequest file resides.

      Example output

      2025/02/10 17:47:39 Credentials loaded from gcloud CLI defaults
      2025/02/10 17:47:42 IAM service account filestore-service-account-openshift-gcp-filestore-csi-driver-operator created
      2025/02/10 17:47:44 Unable to add predefined roles to IAM service account, retrying...
      2025/02/10 17:47:59 Updated policy bindings for IAM service account filestore-service-account-openshift-gcp-filestore-csi-driver-operator
      2025/02/10 17:47:59 Saved credentials configuration to: /tmp/install-dir/ 1
      openshift-cluster-csi-drivers-gcp-filestore-cloud-credentials-credentials.yaml

      1

      The current directory.

   3. Find the service account email of the newly created service account by running the following command:

      $ cat /tmp/install-dir/manifests/openshift-cluster-csi-drivers-gcp-filestore-cloud-credentials-credentials.yaml | yq '.data["service_account.json"]' | base64 -d | jq '.service_account_impersonation_url'

      Example output

      https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/filestore-se-openshift-g-ch8cm@openshift-gce-devel.iam.gserviceaccount.com:generateAccessToken

      In this example output, the service account email is filestore-se-openshift-g-ch8cm@openshift-gce-devel.iam.gserviceaccount.com.

1. Log in to the OpenShift Cluster Manager.
2. Select your cluster.
3. Click Open console and log in with your credentials.

4. Enable the Filestore API in the GCE project by running the following command:

   $ gcloud services enable file.googleapis.com  --project <my_gce_project> 1

   1

   Replace <my_gce_project> with your Google Cloud project.

   You can also do this using Google Cloud web console.

5. Install the GCP Filestore CSI Operator:

   1. Click Operators OperatorHub.
   2. Locate the GCP Filestore CSI Operator by typing GCP Filestore in the filter box.
   3. Click the GCP Filestore CSI Driver Operator button.
   4. On the GCP Filestore CSI Driver Operator page, click Install.

   5. On the Install Operator page, ensure that:

      • All namespaces on the cluster (default) is selected.

      • Installed Namespace is set to openshift-cluster-csi-drivers.

        If using GCP Workload Identity, enter values for the following fields obtained from the procedure in Section Preparing to install the GCP Filestore CSI Driver Operator with Workload Identity:

      • GCP Project Number
      • GCP Pool ID
      • GCP Provider ID
      • GCP Service Account Email

   6. Click Install.

      After the installation finishes, the GCP Filestore CSI Operator is listed in the Installed Operators section of the web console.

6. Install the GCP Filestore CSI Driver:

   1. Click administration CustomResourceDefinitions ClusterCSIDriver.

   2. On the Instances tab, click Create ClusterCSIDriver.

      Use the following YAML file:

      apiVersion: operator.openshift.io/v1
      kind: ClusterCSIDriver
      metadata:
          name: filestore.csi.storage.gke.io
      spec:
        managementState: Managed

   3. Click Create.

   4. Wait for the following Conditions to change to a "true" status:

      • GCPFilestoreDriverCredentialsRequestControllerAvailable
      • GCPFilestoreDriverNodeServiceControllerAvailable
      • GCPFilestoreDriverControllerServiceControllerAvailable