Chapter 5. Developing Operators

Procedure

1. Update your Operator project’s ClusterServiceVersion (CSV) object:

   1. Ensure your Operator has RBAC permission to create CredentialsRequests objects:

      Example 5.1. Example clusterPermissions list

      # ...
      install:
        spec:
          clusterPermissions:
          - rules:
            - apiGroups:
              - "cloudcredential.openshift.io"
              resources:
              - credentialsrequests
              verbs:
              - create
              - delete
              - get
              - list
              - patch
              - update
              - watch

      Copy to Clipboard

   2. Add the following annotation to claim support for this method of CCO-based workflow with AWS STS:

      # ...
      metadata:
       annotations:
         features.operators.openshift.io/token-auth-aws: "true"

      Copy to Clipboard

2. Update your Operator project code:

   1. Get the role ARN from the environment variable set on the pod by the Subscription object. For example:

      // Get ENV var
      roleARN := os.Getenv("ROLEARN")
      setupLog.Info("getting role ARN", "role ARN = ", roleARN)
      webIdentityTokenPath := "/var/run/secrets/openshift/serviceaccount/token"

      Copy to Clipboard

   2. Ensure you have a CredentialsRequest object ready to be patched and applied. For example:

      Example 5.2. Example CredentialsRequest object creation

      import (
         minterv1 "github.com/openshift/cloud-credential-operator/pkg/apis/cloudcredential/v1"
         corev1 "k8s.io/api/core/v1"
         metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
      )
      
      var in = minterv1.AWSProviderSpec{
         StatementEntries: []minterv1.StatementEntry{
            {
               Action: []string{
                  "s3:*",
               },
               Effect:   "Allow",
               Resource: "arn:aws:s3:*:*:*",
            },
         },
      	STSIAMRoleARN: "<role_arn>",
      }
      
      var codec = minterv1.Codec
      var ProviderSpec, _ = codec.EncodeProviderSpec(in.DeepCopyObject())
      
      const (
         name      = "<credential_request_name>"
         namespace = "<namespace_name>"
      )
      
      var CredentialsRequestTemplate = &minterv1.CredentialsRequest{
         ObjectMeta: metav1.ObjectMeta{
             Name:      name,
             Namespace: "openshift-cloud-credential-operator",
         },
         Spec: minterv1.CredentialsRequestSpec{
            ProviderSpec: ProviderSpec,
            SecretRef: corev1.ObjectReference{
               Name:      "<secret_name>",
               Namespace: namespace,
            },
            ServiceAccountNames: []string{
               "<service_account_name>",
            },
            CloudTokenPath:   "",
         },
      }

      Copy to Clipboard

      Alternatively, if you are starting from a CredentialsRequest object in YAML form (for example, as part of your Operator project code), you can handle it differently:

      Example 5.3. Example CredentialsRequest object creation in YAML form

      // CredentialsRequest is a struct that represents a request for credentials
      type CredentialsRequest struct {
        APIVersion string `yaml:"apiVersion"`
        Kind       string `yaml:"kind"`
        Metadata   struct {
           Name      string `yaml:"name"`
           Namespace string `yaml:"namespace"`
        } `yaml:"metadata"`
        Spec struct {
           SecretRef struct {
              Name      string `yaml:"name"`
              Namespace string `yaml:"namespace"`
           } `yaml:"secretRef"`
           ProviderSpec struct {
              APIVersion     string `yaml:"apiVersion"`
              Kind           string `yaml:"kind"`
              StatementEntries []struct {
                 Effect   string   `yaml:"effect"`
                 Action   []string `yaml:"action"`
                 Resource string   `yaml:"resource"`
              } `yaml:"statementEntries"`
              STSIAMRoleARN   string `yaml:"stsIAMRoleARN"`
           } `yaml:"providerSpec"`
      
           // added new field
            CloudTokenPath   string `yaml:"cloudTokenPath"`
        } `yaml:"spec"`
      }
      
      // ConsumeCredsRequestAddingTokenInfo is a function that takes a YAML filename and two strings as arguments
      // It unmarshals the YAML file to a CredentialsRequest object and adds the token information.
      func ConsumeCredsRequestAddingTokenInfo(fileName, tokenString, tokenPath string) (*CredentialsRequest, error) {
        // open a file containing YAML form of a CredentialsRequest
        file, err := os.Open(fileName)
        if err != nil {
           return nil, err
        }
        defer file.Close()
      
        // create a new CredentialsRequest object
        cr := &CredentialsRequest{}
      
        // decode the yaml file to the object
        decoder := yaml.NewDecoder(file)
        err = decoder.Decode(cr)
        if err != nil {
           return nil, err
        }
      
        // assign the string to the existing field in the object
        cr.Spec.CloudTokenPath = tokenPath
      
        // return the modified object
        return cr, nil
      }

      Copy to Clipboard
      Note

      Adding a CredentialsRequest object to the Operator bundle is not currently supported.

   3. Add the role ARN and web identity token path to the credentials request and apply it during Operator initialization:

      Example 5.4. Example applying CredentialsRequest object during Operator initialization

      // apply CredentialsRequest on install
      credReq := credreq.CredentialsRequestTemplate
      credReq.Spec.CloudTokenPath = webIdentityTokenPath
      
      c := mgr.GetClient()
      if err := c.Create(context.TODO(), credReq); err != nil {
         if !errors.IsAlreadyExists(err) {
            setupLog.Error(err, "unable to create CredRequest")
            os.Exit(1)
         }
      }

      Copy to Clipboard

   4. Ensure your Operator can wait for a Secret object to show up from the CCO, as shown in the following example, which is called along with the other items you are reconciling in your Operator:

      Example 5.5. Example wait for Secret object

      // WaitForSecret is a function that takes a Kubernetes client, a namespace, and a v1 "k8s.io/api/core/v1" name as arguments
      // It waits until the secret object with the given name exists in the given namespace
      // It returns the secret object or an error if the timeout is exceeded
      func WaitForSecret(client kubernetes.Interface, namespace, name string) (*v1.Secret, error) {
        // set a timeout of 10 minutes
        timeout := time.After(10 * time.Minute) 

      1

      
      
        // set a polling interval of 10 seconds
        ticker := time.NewTicker(10 * time.Second)
      
        // loop until the timeout or the secret is found
        for {
           select {
           case <-timeout:
              // timeout is exceeded, return an error
              return nil, fmt.Errorf("timed out waiting for secret %s in namespace %s", name, namespace)
                 // add to this error with a pointer to instructions for following a manual path to a Secret that will work on STS
           case <-ticker.C:
              // polling interval is reached, try to get the secret
              secret, err := client.CoreV1().Secrets(namespace).Get(context.Background(), name, metav1.GetOptions{})
              if err != nil {
                 if errors.IsNotFound(err) {
                    // secret does not exist yet, continue waiting
                    continue
                 } else {
                    // some other error occurred, return it
                    return nil, err
                 }
              } else {
                 // secret is found, return it
                 return secret, nil
              }
           }
        }
      }

      Copy to Clipboard

      1

      The timeout value is based on an estimate of how fast the CCO might detect an added CredentialsRequest object and generate a Secret object. You might consider lowering the time or creating custom feedback for cluster administrators that could be wondering why the Operator is not yet accessing the cloud resources.

   5. Set up the AWS configuration by reading the secret created by the CCO from the credentials request and creating the AWS config file containing the data from that secret:

      Example 5.6. Example AWS configuration creation

      func SharedCredentialsFileFromSecret(secret *corev1.Secret) (string, error) {
         var data []byte
         switch {
         case len(secret.Data["credentials"]) > 0:
             data = secret.Data["credentials"]
         default:
             return "", errors.New("invalid secret for aws credentials")
         }
      
      
         f, err := ioutil.TempFile("", "aws-shared-credentials")
         if err != nil {
             return "", errors.Wrap(err, "failed to create file for shared credentials")
         }
         defer f.Close()
         if _, err := f.Write(data); err != nil {
             return "", errors.Wrapf(err, "failed to write credentials to %s", f.Name())
         }
         return f.Name(), nil
      }

      Copy to Clipboard
      Important

      The secret is assumed to exist, but your Operator code should wait and retry when using this secret to give time to the CCO to create the secret.

      Additionally, the wait period should eventually time out and warn users that the OpenShift Container Platform cluster version, and therefore the CCO, might be an earlier version that does not support the CredentialsRequest object workflow with STS detection. In such cases, instruct users that they must add a secret by using another method.

   6. Configure the AWS SDK session, for example:

      Example 5.7. Example AWS SDK session configuration

      sharedCredentialsFile, err := SharedCredentialsFileFromSecret(secret)
      if err != nil {
         // handle error
      }
      options := session.Options{
         SharedConfigState: session.SharedConfigEnable,
         SharedConfigFiles: []string{sharedCredentialsFile},
      }

      Copy to Clipboard

Procedure

1. Update your Operator project’s ClusterServiceVersion (CSV) object:

   1. Ensure your Operator has RBAC permission to create CredentialsRequests objects:

      Example 5.9. Example clusterPermissions list

      # ...
      install:
        spec:
          clusterPermissions:
          - rules:
            - apiGroups:
              - "cloudcredential.openshift.io"
              resources:
              - credentialsrequests
              verbs:
              - create
              - delete
              - get
              - list
              - patch
              - update
              - watch

      Copy to Clipboard

   2. Add the following annotation to claim support for this method of CCO-based workflow with Workload ID:

      # ...
      metadata:
       annotations:
         features.operators.openshift.io/token-auth-azure: "true"

      Copy to Clipboard

2. Update your Operator project code:

   1. Get the client ID, tenant ID, and subscription ID from the environment variables set on the pod by the Subscription object. For example:

      // Get ENV var
      clientID := os.Getenv("CLIENTID")
      tenantID := os.Getenv("TENANTID")
      subscriptionID := os.Getenv("SUBSCRIPTIONID")
      azureFederatedTokenFile := "/var/run/secrets/openshift/serviceaccount/token"

      Copy to Clipboard

   2. Ensure you have a CredentialsRequest object ready to be patched and applied.

      Note

      Adding a CredentialsRequest object to the Operator bundle is not currently supported.

   3. Add the Azure credentials information and web identity token path to the credentials request and apply it during Operator initialization:

      Example 5.10. Example applying CredentialsRequest object during Operator initialization

      // apply CredentialsRequest on install
      credReqTemplate.Spec.AzureProviderSpec.AzureClientID = clientID
      credReqTemplate.Spec.AzureProviderSpec.AzureTenantID = tenantID
      credReqTemplate.Spec.AzureProviderSpec.AzureRegion = "centralus"
      credReqTemplate.Spec.AzureProviderSpec.AzureSubscriptionID = subscriptionID
      credReqTemplate.CloudTokenPath = azureFederatedTokenFile
      
      c := mgr.GetClient()
      if err := c.Create(context.TODO(), credReq); err != nil {
          if !errors.IsAlreadyExists(err) {
              setupLog.Error(err, "unable to create CredRequest")
              os.Exit(1)
          }
      }

      Copy to Clipboard

   4. Ensure your Operator can wait for a Secret object to show up from the CCO, as shown in the following example, which is called along with the other items you are reconciling in your Operator:

      Example 5.11. Example wait for Secret object

      // WaitForSecret is a function that takes a Kubernetes client, a namespace, and a v1 "k8s.io/api/core/v1" name as arguments
      // It waits until the secret object with the given name exists in the given namespace
      // It returns the secret object or an error if the timeout is exceeded
      func WaitForSecret(client kubernetes.Interface, namespace, name string) (*v1.Secret, error) {
        // set a timeout of 10 minutes
        timeout := time.After(10 * time.Minute) 

      1

      
      
        // set a polling interval of 10 seconds
        ticker := time.NewTicker(10 * time.Second)
      
        // loop until the timeout or the secret is found
        for {
           select {
           case <-timeout:
              // timeout is exceeded, return an error
              return nil, fmt.Errorf("timed out waiting for secret %s in namespace %s", name, namespace)
                 // add to this error with a pointer to instructions for following a manual path to a Secret that will work on STS
           case <-ticker.C:
              // polling interval is reached, try to get the secret
              secret, err := client.CoreV1().Secrets(namespace).Get(context.Background(), name, metav1.GetOptions{})
              if err != nil {
                 if errors.IsNotFound(err) {
                    // secret does not exist yet, continue waiting
                    continue
                 } else {
                    // some other error occurred, return it
                    return nil, err
                 }
              } else {
                 // secret is found, return it
                 return secret, nil
              }
           }
        }
      }

      Copy to Clipboard

      1

      The timeout value is based on an estimate of how fast the CCO might detect an added CredentialsRequest object and generate a Secret object. You might consider lowering the time or creating custom feedback for cluster administrators that could be wondering why the Operator is not yet accessing the cloud resources.

   5. Read the secret created by the CCO from the CredentialsRequest object to authenticate with Azure and receive the necessary credentials.

Procedure

1. Update your Operator project’s ClusterServiceVersion (CSV) object:

   1. Ensure Operator deployment in the CSV has the following volumeMounts and volumes fields so that the Operator can assume the role with web identity:

      Example 5.12. Example volumeMounts and volumes fields

      # ...
            volumeMounts:
      
            - name: bound-sa-token
              mountPath: /var/run/secrets/openshift/serviceaccount
              readOnly: true
            volumes:
               # This service account token can be used to provide identity outside the cluster.
               - name: bound-sa-token
                 projected:
                   sources:
                   - serviceAccountToken:
                     path: token
                     audience: openshift

      Copy to Clipboard

   2. Ensure your Operator has RBAC permission to create CredentialsRequests objects:

      Example 5.13. Example clusterPermissions list

      # ...
      install:
        spec:
          clusterPermissions:
          - rules:
            - apiGroups:
              - "cloudcredential.openshift.io"
              resources:
              - credentialsrequests
              verbs:
              - create
              - delete
              - get
              - list
              - patch
              - update
              - watch

      Copy to Clipboard

   3. Add the following annotation to claim support for this method of CCO-based workflow with GCP Workload Identity:

      # ...
      metadata:
       annotations:
         features.operators.openshift.io/token-auth-gcp: "true"

      Copy to Clipboard

2. Update your Operator project code:

   1. Get the audience and the serviceAccountEmail values from the environment variables set on the pod by the subscription config:

       // Get ENV var
         audience := os.Getenv("AUDIENCE")
         serviceAccountEmail := os.Getenv("SERVICE_ACCOUNT_EMAIL")
         gcpIdentityTokenFile := "/var/run/secrets/openshift/serviceaccount/token"

      Copy to Clipboard

   2. Ensure you have a CredentialsRequest object ready to be patched and applied.

      Note

      Adding a CredentialsRequest object to the Operator bundle is not currently supported.

   3. Add the GCP Workload Identity variables to the credentials request and apply it during Operator initialization:

      Example 5.14. Example applying CredentialsRequest object during Operator initialization

      // apply CredentialsRequest on install
         credReqTemplate.Spec.GCPProviderSpec.Audience = audience
         credReqTemplate.Spec.GCPProviderSpec.ServiceAccountEmail = serviceAccountEmail
         credReqTemplate.CloudTokenPath = gcpIdentityTokenFile
      
      
         c := mgr.GetClient()
         if err := c.Create(context.TODO(), credReq); err != nil {
             if !errors.IsAlreadyExists(err) {
                 setupLog.Error(err, "unable to create CredRequest")
                 os.Exit(1)
             }
         }

      Copy to Clipboard

   4. Ensure your Operator can wait for a Secret object to show up from the CCO, as shown in the following example, which is called along with the other items you are reconciling in your Operator:

      Example 5.15. Example wait for Secret object

      // WaitForSecret is a function that takes a Kubernetes client, a namespace, and a v1 "k8s.io/api/core/v1" name as arguments
      // It waits until the secret object with the given name exists in the given namespace
      // It returns the secret object or an error if the timeout is exceeded
      func WaitForSecret(client kubernetes.Interface, namespace, name string) (*v1.Secret, error) {
        // set a timeout of 10 minutes
        timeout := time.After(10 * time.Minute) 

      1

      
      
        // set a polling interval of 10 seconds
        ticker := time.NewTicker(10 * time.Second)
      
        // loop until the timeout or the secret is found
        for {
           select {
           case <-timeout:
              // timeout is exceeded, return an error
              return nil, fmt.Errorf("timed out waiting for secret %s in namespace %s", name, namespace)
      // add to this error with a pointer to instructions for following a manual path to a Secret that will work
           case <-ticker.C:
              // polling interval is reached, try to get the secret
              secret, err := client.CoreV1().Secrets(namespace).Get(context.Background(), name, metav1.GetOptions{})
              if err != nil {
                 if errors.IsNotFound(err) {
                    // secret does not exist yet, continue waiting
                    continue
                 } else {
                    // some other error occurred, return it
                    return nil, err
                 }
              } else {
                 // secret is found, return it
                 return secret, nil
              }
           }
        }
      }

      Copy to Clipboard

      1

      The timeout value is based on an estimate of how fast the CCO might detect an added CredentialsRequest object and generate a Secret object. You might consider lowering the time or creating custom feedback for cluster administrators that could be wondering why the Operator is not yet accessing the cloud resources.

   5. Read the service_account.json field from the secret and use it to authenticate your GCP client:

      service_account_json := secret.StringData["service_account.json"]

      Copy to Clipboard