SupportConsoleDevelopersStart a trialContact

Chapter 19. Managing cloud provider credentials

19.1. About the Cloud Credential Operator

The Cloud Credential Operator (CCO) manages cloud provider credentials as custom resource definitions (CRDs). The CCO syncs on CredentialsRequest custom resources (CRs) to allow OpenShift Container Platform components to request cloud provider credentials with the specific permissions that are required for the cluster to run.

By setting different values for the credentialsMode parameter in the install-config.yaml file, the CCO can be configured to operate in several different modes. If no mode is specified, or the credentialsMode parameter is set to an empty string (""), the CCO operates in its default mode.

19.1.1. Modes

By setting different values for the credentialsMode parameter in the install-config.yaml file, the CCO can be configured to operate in mint, passthrough, or manual mode. These options provide transparency and flexibility in how the CCO uses cloud credentials to process CredentialsRequest CRs in the cluster, and allow the CCO to be configured to suit the security requirements of your organization. Not all CCO modes are supported for all cloud providers.

  • Mint: In mint mode, the CCO uses the provided admin-level cloud credential to create new credentials for components in the cluster with only the specific permissions that are required.
  • Passthrough: In passthrough mode, the CCO passes the provided cloud credential to the components that request cloud credentials.
  • Manual mode with long-term credentials for components: In manual mode, you can manage long-term cloud credentials instead of the CCO.
  • Manual mode with short-term credentials for components: For some providers, you can use the CCO utility (ccoctl) during installation to implement short-term credentials for individual components. These credentials are created and managed outside the OpenShift Container Platform cluster.
Table 19.1. CCO mode support matrix
Cloud providerMintPassthroughManual with long-term credentialsManual with short-term credentials

Alibaba Cloud

  

X [1]

 

Amazon Web Services (AWS)

X

X

X

X

Global Microsoft Azure

 

X

X

X

Microsoft Azure Stack Hub

  

X

 

Google Cloud Platform (GCP)

X

X

X

X

IBM Cloud®

  

X [1]

 

Nutanix

  

X [1]

 

Red Hat OpenStack Platform (RHOSP)

 

X

  

VMware vSphere

 

X

  
  1. This platform uses the ccoctl utility during installation to configure long-term credentials.

19.1.2. Determining the Cloud Credential Operator mode

For platforms that support using the CCO in multiple modes, you can determine what mode the CCO is configured to use by using the web console or the CLI.

Figure 19.1. Determining the CCO configuration

Decision tree showing how to determine the configured CCO credentials mode for your cluster.

19.1.2.1. Determining the Cloud Credential Operator mode by using the web console

You can determine what mode the Cloud Credential Operator (CCO) is configured to use by using the web console.

Note

Only Amazon Web Services (AWS), global Microsoft Azure, and Google Cloud Platform (GCP) clusters support multiple CCO modes.

Prerequisites

  • You have access to an OpenShift Container Platform account with cluster administrator permissions.

Procedure

  1. Log in to the OpenShift Container Platform web console as a user with the cluster-admin role.
  2. Navigate to Administration Cluster Settings.
  3. On the Cluster Settings page, select the Configuration tab.
  4. Under Configuration resource, select CloudCredential.
  5. On the CloudCredential details page, select the YAML tab.

  6. In the YAML block, check the value of spec.credentialsMode. The following values are possible, though not all are supported on all platforms:

    • '': The CCO is operating in the default mode. In this configuration, the CCO operates in mint or passthrough mode, depending on the credentials provided during installation.
    • Mint: The CCO is operating in mint mode.
    • Passthrough: The CCO is operating in passthrough mode.
    • Manual: The CCO is operating in manual mode.
    Important

    To determine the specific configuration of an AWS, GCP, or global Microsoft Azure cluster that has a spec.credentialsMode of '', Mint, or Manual, you must investigate further.

    AWS and GCP clusters support using mint mode with the root secret deleted.

    An AWS, GCP, or global Microsoft Azure cluster that uses manual mode might be configured to create and manage cloud credentials from outside of the cluster with AWS STS, GCP Workload Identity, or Microsoft Entra Workload ID. You can determine whether your cluster uses this strategy by examining the cluster Authentication object.

  7. AWS or GCP clusters that use the default ('') only: To determine whether the cluster is operating in mint or passthrough mode, inspect the annotations on the cluster root secret:

    1. Navigate to Workloads Secrets and look for the root secret for your cloud provider.

      Note

      Ensure that the Project dropdown is set to All Projects.

      PlatformSecret name

      AWS

      aws-creds

      GCP

      gcp-credentials

    2. To view the CCO mode that the cluster is using, click 1 annotation under Annotations, and check the value field. The following values are possible:

      • Mint: The CCO is operating in mint mode.
      • Passthrough: The CCO is operating in passthrough mode.

      If your cluster uses mint mode, you can also determine whether the cluster is operating without the root secret.

  8. AWS or GCP clusters that use mint mode only: To determine whether the cluster is operating without the root secret, navigate to Workloads Secrets and look for the root secret for your cloud provider.

    Note

    Ensure that the Project dropdown is set to All Projects.

    PlatformSecret name

    AWS

    aws-creds

    GCP

    gcp-credentials

    • If you see one of these values, your cluster is using mint or passthrough mode with the root secret present.
    • If you do not see these values, your cluster is using the CCO in mint mode with the root secret removed.

  9. AWS, GCP, or global Microsoft Azure clusters that use manual mode only: To determine whether the cluster is configured to create and manage cloud credentials from outside of the cluster, you must check the cluster Authentication object YAML values.

    1. Navigate to Administration Cluster Settings.
    2. On the Cluster Settings page, select the Configuration tab.
    3. Under Configuration resource, select Authentication.
    4. On the Authentication details page, select the YAML tab.

    5. In the YAML block, check the value of the .spec.serviceAccountIssuer parameter.

      • A value that contains a URL that is associated with your cloud provider indicates that the CCO is using manual mode with short-term credentials for components. These clusters are configured using the ccoctl utility to create and manage cloud credentials from outside of the cluster.
      • An empty value ('') indicates that the cluster is using the CCO in manual mode but was not configured using the ccoctl utility.

19.1.2.2. Determining the Cloud Credential Operator mode by using the CLI

You can determine what mode the Cloud Credential Operator (CCO) is configured to use by using the CLI.

Note

Only Amazon Web Services (AWS), global Microsoft Azure, and Google Cloud Platform (GCP) clusters support multiple CCO modes.

Prerequisites

  • You have access to an OpenShift Container Platform account with cluster administrator permissions.
  • You have installed the OpenShift CLI (oc).

Procedure

  1. Log in to oc on the cluster as a user with the cluster-admin role.

  2. To determine the mode that the CCO is configured to use, enter the following command: 

    $ oc get cloudcredentials cluster \
  -o=jsonpath={.spec.credentialsMode}

    The following output values are possible, though not all are supported on all platforms:

    • '': The CCO is operating in the default mode. In this configuration, the CCO operates in mint or passthrough mode, depending on the credentials provided during installation.
    • Mint: The CCO is operating in mint mode.
    • Passthrough: The CCO is operating in passthrough mode.
    • Manual: The CCO is operating in manual mode.
    Important

    To determine the specific configuration of an AWS, GCP, or global Microsoft Azure cluster that has a spec.credentialsMode of '', Mint, or Manual, you must investigate further.

    AWS and GCP clusters support using mint mode with the root secret deleted.

    An AWS, GCP, or global Microsoft Azure cluster that uses manual mode might be configured to create and manage cloud credentials from outside of the cluster with AWS STS, GCP Workload Identity, or Microsoft Entra Workload ID. You can determine whether your cluster uses this strategy by examining the cluster Authentication object.

  3. AWS or GCP clusters that use the default ('') only: To determine whether the cluster is operating in mint or passthrough mode, run the following command: 

    $ oc get secret <secret_name> \
  -n kube-system \
  -o jsonpath \
  --template '{ .metadata.annotations }'

    where <secret_name> is aws-creds for AWS or gcp-credentials for GCP.

    This command displays the value of the .metadata.annotations parameter in the cluster root secret object. The following output values are possible:

    • Mint: The CCO is operating in mint mode.
    • Passthrough: The CCO is operating in passthrough mode.

    If your cluster uses mint mode, you can also determine whether the cluster is operating without the root secret.

  4. AWS or GCP clusters that use mint mode only: To determine whether the cluster is operating without the root secret, run the following command: 

    $ oc get secret <secret_name> \
  -n=kube-system

    where <secret_name> is aws-creds for AWS or gcp-credentials for GCP.

    If the root secret is present, the output of this command returns information about the secret. An error indicates that the root secret is not present on the cluster.

  5. AWS, GCP, or global Microsoft Azure clusters that use manual mode only: To determine whether the cluster is configured to create and manage cloud credentials from outside of the cluster, run the following command: 

    $ oc get authentication cluster \
  -o jsonpath \
  --template='{ .spec.serviceAccountIssuer }'

    This command displays the value of the .spec.serviceAccountIssuer parameter in the cluster Authentication object.

    • An output of a URL that is associated with your cloud provider indicates that the CCO is using manual mode with short-term credentials for components. These clusters are configured using the ccoctl utility to create and manage cloud credentials from outside of the cluster.
    • An empty output indicates that the cluster is using the CCO in manual mode but was not configured using the ccoctl utility.

19.1.3. Default behavior

For platforms on which multiple modes are supported (AWS, Azure, and GCP), when the CCO operates in its default mode, it checks the provided credentials dynamically to determine for which mode they are sufficient to process CredentialsRequest CRs.

By default, the CCO determines whether the credentials are sufficient for mint mode, which is the preferred mode of operation, and uses those credentials to create appropriate credentials for components in the cluster. If the credentials are not sufficient for mint mode, it determines whether they are sufficient for passthrough mode. If the credentials are not sufficient for passthrough mode, the CCO cannot adequately process CredentialsRequest CRs.

If the provided credentials are determined to be insufficient during installation, the installation fails. For AWS, the installation program fails early in the process and indicates which required permissions are missing. Other providers might not provide specific information about the cause of the error until errors are encountered.

If the credentials are changed after a successful installation and the CCO determines that the new credentials are insufficient, the CCO puts conditions on any new CredentialsRequest CRs to indicate that it cannot process them because of the insufficient credentials.

To resolve insufficient credentials issues, provide a credential with sufficient permissions. If an error occurred during installation, try installing again. For issues with new CredentialsRequest CRs, wait for the CCO to try to process the CR again. As an alternative, you can configure your cluster to use a different CCO mode that is supported for your cloud provider.

19.1.4. Additional resources

19.2. The Cloud Credential Operator in mint mode

Mint mode is the default Cloud Credential Operator (CCO) credentials mode for OpenShift Container Platform on platforms that support it. Mint mode supports Amazon Web Services (AWS) and Google Cloud Platform (GCP) clusters.

19.2.1. Mint mode credentials management

For clusters that use the CCO in mint mode, the administrator-level credential is stored in the kube-system namespace. The CCO uses the admin credential to process the CredentialsRequest objects in the cluster and create users for components with limited permissions.

With mint mode, each cluster component has only the specific permissions it requires. The automatic, continuous reconciliation of cloud credentials in mint mode allows actions that require additional credentials or permissions, such as upgrading, to proceed.

Note

By default, mint mode requires storing the admin credential in the cluster kube-system namespace. If this approach does not meet the security requirements of your organization, you can remove the credential after installing the cluster.

19.2.1.1. Mint mode permissions requirements

When using the CCO in mint mode, ensure that the credential you provide meets the requirements of the cloud on which you are running or installing OpenShift Container Platform. If the provided credentials are not sufficient for mint mode, the CCO cannot create an IAM user.

The credential you provide for mint mode in Amazon Web Services (AWS) must have the following permissions:

Example 19.1. Required AWS permissions

  • iam:CreateAccessKey
  • iam:CreateUser
  • iam:DeleteAccessKey
  • iam:DeleteUser
  • iam:DeleteUserPolicy
  • iam:GetUser
  • iam:GetUserPolicy
  • iam:ListAccessKeys
  • iam:PutUserPolicy
  • iam:TagUser
  • iam:SimulatePrincipalPolicy

The credential you provide for mint mode in Google Cloud Platform (GCP) must have the following permissions:

Example 19.2. Required GCP permissions

  • resourcemanager.projects.get
  • serviceusage.services.list
  • iam.serviceAccountKeys.create
  • iam.serviceAccountKeys.delete
  • iam.serviceAccountKeys.list
  • iam.serviceAccounts.create
  • iam.serviceAccounts.delete
  • iam.serviceAccounts.get
  • iam.roles.create
  • iam.roles.get
  • iam.roles.list
  • iam.roles.undelete
  • iam.roles.update
  • resourcemanager.projects.getIamPolicy
  • resourcemanager.projects.setIamPolicy

19.2.1.2. Admin credentials root secret format

Each cloud provider uses a credentials root secret in the kube-system namespace by convention, which is then used to satisfy all credentials requests and create their respective secrets. This is done either by minting new credentials with mint mode, or by copying the credentials root secret with passthrough mode.

The format for the secret varies by cloud, and is also used for each CredentialsRequest secret.

Amazon Web Services (AWS) secret format

apiVersion: v1
kind: Secret
metadata:
  namespace: kube-system
  name: aws-creds
stringData:
  aws_access_key_id: <base64-encoded_access_key_id>
  aws_secret_access_key: <base64-encoded_secret_access_key>

Google Cloud Platform (GCP) secret format

apiVersion: v1
kind: Secret
metadata:
  namespace: kube-system
  name: gcp-credentials
stringData:
  service_account.json: <base64-encoded_service_account>

19.2.2. Maintaining cloud provider credentials

If your cloud provider credentials are changed for any reason, you must manually update the secret that the Cloud Credential Operator (CCO) uses to manage cloud provider credentials.

The process for rotating cloud credentials depends on the mode that the CCO is configured to use. After you rotate credentials for a cluster that is using mint mode, you must manually remove the component credentials that were created by the removed credential.

Prerequisites

  • Your cluster is installed on a platform that supports rotating cloud credentials manually with the CCO mode that you are using:

    • For mint mode, Amazon Web Services (AWS) and Google Cloud Platform (GCP) are supported.
  • You have changed the credentials that are used to interface with your cloud provider.
  • The new credentials have sufficient permissions for the mode CCO is configured to use in your cluster.

Procedure

  1. In the Administrator perspective of the web console, navigate to Workloads Secrets.

  2. In the table on the Secrets page, find the root secret for your cloud provider.

    PlatformSecret name

    AWS

    aws-creds

    GCP

    gcp-credentials

  3. Click the Options menu kebab in the same row as the secret and select Edit Secret.
  4. Record the contents of the Value field or fields. You can use this information to verify that the value is different after updating the credentials.
  5. Update the text in the Value field or fields with the new authentication information for your cloud provider, and then click Save.

  6. Delete each component secret that is referenced by the individual CredentialsRequest objects.

    1. Log in to the OpenShift Container Platform CLI as a user with the cluster-admin role.

    2. Get the names and namespaces of all referenced component secrets: 

      $ oc -n openshift-cloud-credential-operator get CredentialsRequest \
  -o json | jq -r '.items[] | select (.spec.providerSpec.kind=="<provider_spec>") | .spec.secretRef'

      where <provider_spec> is the corresponding value for your cloud provider:

      • AWS: AWSProviderSpec
      • GCP: GCPProviderSpec

      Partial example output for AWS

      {
  "name": "ebs-cloud-credentials",
  "namespace": "openshift-cluster-csi-drivers"
}
{
  "name": "cloud-credential-operator-iam-ro-creds",
  "namespace": "openshift-cloud-credential-operator"
}

    3. Delete each of the referenced component secrets: 

      $ oc delete secret <secret_name> \1
  -n <secret_namespace> 2
      1
      Specify the name of a secret.
      2
      Specify the namespace that contains the secret.

      Example deletion of an AWS secret

      $ oc delete secret ebs-cloud-credentials -n openshift-cluster-csi-drivers

      You do not need to manually delete the credentials from your provider console. Deleting the referenced component secrets will cause the CCO to delete the existing credentials from the platform and create new ones.

Verification

To verify that the credentials have changed:

  1. In the Administrator perspective of the web console, navigate to Workloads Secrets.
  2. Verify that the contents of the Value field or fields have changed.

19.2.3. Additional resources

19.3. The Cloud Credential Operator in passthrough mode

Passthrough mode is supported for Amazon Web Services (AWS), Microsoft Azure, Google Cloud Platform (GCP), Red Hat OpenStack Platform (RHOSP), and VMware vSphere.

In passthrough mode, the Cloud Credential Operator (CCO) passes the provided cloud credential to the components that request cloud credentials. The credential must have permissions to perform the installation and complete the operations that are required by components in the cluster, but does not need to be able to create new credentials. The CCO does not attempt to create additional limited-scoped credentials in passthrough mode.

Note

Manual mode is the only supported CCO configuration for Microsoft Azure Stack Hub.

19.3.1. Passthrough mode permissions requirements

When using the CCO in passthrough mode, ensure that the credential you provide meets the requirements of the cloud on which you are running or installing OpenShift Container Platform. If the provided credentials the CCO passes to a component that creates a CredentialsRequest CR are not sufficient, that component will report an error when it tries to call an API that it does not have permissions for.

19.3.1.1. Amazon Web Services (AWS) permissions

The credential you provide for passthrough mode in AWS must have all the requested permissions for all CredentialsRequest CRs that are required by the version of OpenShift Container Platform you are running or installing.

To locate the CredentialsRequest CRs that are required, see Manually creating long-term credentials for AWS.

19.3.1.2. Microsoft Azure permissions

The credential you provide for passthrough mode in Azure must have all the requested permissions for all CredentialsRequest CRs that are required by the version of OpenShift Container Platform you are running or installing.

To locate the CredentialsRequest CRs that are required, see Manually creating long-term credentials for Azure.

19.3.1.3. Google Cloud Platform (GCP) permissions

The credential you provide for passthrough mode in GCP must have all the requested permissions for all CredentialsRequest CRs that are required by the version of OpenShift Container Platform you are running or installing.

To locate the CredentialsRequest CRs that are required, see Manually creating long-term credentials for GCP.

19.3.1.4. Red Hat OpenStack Platform (RHOSP) permissions

To install an OpenShift Container Platform cluster on RHOSP, the CCO requires a credential with the permissions of a member user role.

19.3.1.5. VMware vSphere permissions

To install an OpenShift Container Platform cluster on VMware vSphere, the CCO requires a credential with the following vSphere privileges:

Table 19.2. Required vSphere privileges
CategoryPrivileges

Datastore

Allocate space

Folder

Create folder, Delete folder

vSphere Tagging

All privileges

Network

Assign network

Resource

Assign virtual machine to resource pool

Profile-driven storage

All privileges

vApp

All privileges

Virtual machine

All privileges

19.3.2. Admin credentials root secret format

Each cloud provider uses a credentials root secret in the kube-system namespace by convention, which is then used to satisfy all credentials requests and create their respective secrets. This is done either by minting new credentials with mint mode, or by copying the credentials root secret with passthrough mode.

The format for the secret varies by cloud, and is also used for each CredentialsRequest secret.

Amazon Web Services (AWS) secret format

apiVersion: v1
kind: Secret
metadata:
  namespace: kube-system
  name: aws-creds
stringData:
  aws_access_key_id: <base64-encoded_access_key_id>
  aws_secret_access_key: <base64-encoded_secret_access_key>

Microsoft Azure secret format

apiVersion: v1
kind: Secret
metadata:
  namespace: kube-system
  name: azure-credentials
stringData:
  azure_subscription_id: <base64-encoded_subscription_id>
  azure_client_id: <base64-encoded_client_id>
  azure_client_secret: <base64-encoded_client_secret>
  azure_tenant_id: <base64-encoded_tenant_id>
  azure_resource_prefix: <base64-encoded_resource_prefix>
  azure_resourcegroup: <base64-encoded_resource_group>
  azure_region: <base64-encoded_region>

On Microsoft Azure, the credentials secret format includes two properties that must contain the cluster’s infrastructure ID, generated randomly for each cluster installation. This value can be found after running create manifests: 

$ cat .openshift_install_state.json | jq '."*installconfig.ClusterID".InfraID' -r

Example output

mycluster-2mpcn

This value would be used in the secret data as follows: 

azure_resource_prefix: mycluster-2mpcn
azure_resourcegroup: mycluster-2mpcn-rg

Google Cloud Platform (GCP) secret format

apiVersion: v1
kind: Secret
metadata:
  namespace: kube-system
  name: gcp-credentials
stringData:
  service_account.json: <base64-encoded_service_account>

Red Hat OpenStack Platform (RHOSP) secret format

apiVersion: v1
kind: Secret
metadata:
  namespace: kube-system
  name: openstack-credentials
data:
  clouds.yaml: <base64-encoded_cloud_creds>
  clouds.conf: <base64-encoded_cloud_creds_init>

VMware vSphere secret format

apiVersion: v1
kind: Secret
metadata:
  namespace: kube-system
  name: vsphere-creds
data:
 vsphere.openshift.example.com.username: <base64-encoded_username>
 vsphere.openshift.example.com.password: <base64-encoded_password>

19.3.3. Passthrough mode credential maintenance

If CredentialsRequest CRs change over time as the cluster is upgraded, you must manually update the passthrough mode credential to meet the requirements. To avoid credentials issues during an upgrade, check the CredentialsRequest CRs in the release image for the new version of OpenShift Container Platform before upgrading. To locate the CredentialsRequest CRs that are required for your cloud provider, see Manually creating long-term credentials for AWS, Azure, or GCP.

19.3.3.1. Maintaining cloud provider credentials

If your cloud provider credentials are changed for any reason, you must manually update the secret that the Cloud Credential Operator (CCO) uses to manage cloud provider credentials.

The process for rotating cloud credentials depends on the mode that the CCO is configured to use. After you rotate credentials for a cluster that is using mint mode, you must manually remove the component credentials that were created by the removed credential.

Prerequisites

  • Your cluster is installed on a platform that supports rotating cloud credentials manually with the CCO mode that you are using:

    • For passthrough mode, Amazon Web Services (AWS), Microsoft Azure, Google Cloud Platform (GCP), Red Hat OpenStack Platform (RHOSP), and VMware vSphere are supported.
  • You have changed the credentials that are used to interface with your cloud provider.
  • The new credentials have sufficient permissions for the mode CCO is configured to use in your cluster.

Procedure

  1. In the Administrator perspective of the web console, navigate to Workloads Secrets.

  2. In the table on the Secrets page, find the root secret for your cloud provider.

    PlatformSecret name

    AWS

    aws-creds

    Azure

    azure-credentials

    GCP

    gcp-credentials

    RHOSP

    openstack-credentials

    VMware vSphere

    vsphere-creds

  3. Click the Options menu kebab in the same row as the secret and select Edit Secret.
  4. Record the contents of the Value field or fields. You can use this information to verify that the value is different after updating the credentials.
  5. Update the text in the Value field or fields with the new authentication information for your cloud provider, and then click Save.

  6. If you are updating the credentials for a vSphere cluster that does not have the vSphere CSI Driver Operator enabled, you must force a rollout of the Kubernetes controller manager to apply the updated credentials.

    Note

    If the vSphere CSI Driver Operator is enabled, this step is not required.

    To apply the updated vSphere credentials, log in to the OpenShift Container Platform CLI as a user with the cluster-admin role and run the following command: 

    $ oc patch kubecontrollermanager cluster \
  -p='{"spec": {"forceRedeploymentReason": "recovery-'"$( date )"'"}}' \
  --type=merge

    While the credentials are rolling out, the status of the Kubernetes Controller Manager Operator reports Progressing=true. To view the status, run the following command: 

    $ oc get co kube-controller-manager

Verification

To verify that the credentials have changed:

  1. In the Administrator perspective of the web console, navigate to Workloads Secrets.
  2. Verify that the contents of the Value field or fields have changed.

Additional resources

19.3.4. Reducing permissions after installation

When using passthrough mode, each component has the same permissions used by all other components. If you do not reduce the permissions after installing, all components have the broad permissions that are required to run the installer.

After installation, you can reduce the permissions on your credential to only those that are required to run the cluster, as defined by the CredentialsRequest CRs in the release image for the version of OpenShift Container Platform that you are using.

To locate the CredentialsRequest CRs that are required for AWS, Azure, or GCP and learn how to change the permissions the CCO uses, see Manually creating long-term credentials for AWS, Azure, or GCP.

19.3.5. Additional resources

19.4. Manual mode with long-term credentials for components

Manual mode is supported for Alibaba Cloud, Amazon Web Services (AWS), global Microsoft Azure, Microsoft Azure Stack Hub, Google Cloud Platform (GCP), IBM Cloud®, and Nutanix.

19.4.1. User-managed credentials

In manual mode, a user manages cloud credentials instead of the Cloud Credential Operator (CCO). To use this mode, you must examine the CredentialsRequest CRs in the release image for the version of OpenShift Container Platform that you are running or installing, create corresponding credentials in the underlying cloud provider, and create Kubernetes Secrets in the correct namespaces to satisfy all CredentialsRequest CRs for the cluster’s cloud provider. Some platforms use the CCO utility (ccoctl) to facilitate this process during installation and updates.

Using manual mode with long-term credentials allows each cluster component to have only the permissions it requires, without storing an administrator-level credential in the cluster. This mode also does not require connectivity to services such as the AWS public IAM endpoint. However, you must manually reconcile permissions with new release images for every upgrade.

For information about configuring your cloud provider to use manual mode, see the manual credentials management options for your cloud provider.

Note

An AWS, global Azure, or GCP cluster that uses manual mode might be configured to use short-term credentials for different components. For more information, see Manual mode with short-term credentials for components.

19.4.2. Additional resources

19.5. Manual mode with short-term credentials for components

During installation, you can configure the Cloud Credential Operator (CCO) to operate in manual mode and use the CCO utility (ccoctl) to implement short-term security credentials for individual components that are created and managed outside the OpenShift Container Platform cluster.

Note

This credentials strategy is supported for Amazon Web Services (AWS), Google Cloud Platform (GCP), and global Microsoft Azure only. The strategy must be configured during installation of a new OpenShift Container Platform cluster. You cannot configure an existing cluster that uses a different credentials strategy to use this feature.

Cloud providers use different terms for their implementation of this authentication method.

Table 19.3. Short-term credentials provider terminology
Cloud providerProvider nomenclature

Amazon Web Services (AWS)

AWS Security Token Service (STS)

Google Cloud Platform (GCP)

GCP Workload Identity

Global Microsoft Azure

Microsoft Entra Workload ID

19.5.1. AWS Security Token Service

In manual mode with STS, the individual OpenShift Container Platform cluster components use the AWS Security Token Service (STS) to assign components IAM roles that provide short-term, limited-privilege security credentials. These credentials are associated with IAM roles that are specific to each component that makes AWS API calls.

Additional resources

19.5.1.1. AWS Security Token Service authentication process

The AWS Security Token Service (STS) and the AssumeRole API action allow pods to retrieve access keys that are defined by an IAM role policy.

The OpenShift Container Platform cluster includes a Kubernetes service account signing service. This service uses a private key to sign service account JSON web tokens (JWT). A pod that requires a service account token requests one through the pod specification. When the pod is created and assigned to a node, the node retrieves a signed service account from the service account signing service and mounts it onto the pod.

Clusters that use STS contain an IAM role ID in their Kubernetes configuration secrets. Workloads assume the identity of this IAM role ID. The signed service account token issued to the workload aligns with the configuration in AWS, which allows AWS STS to grant access keys for the specified IAM role to the workload.

AWS STS grants access keys only for requests that include service account tokens that meet the following conditions:

  • The token name and namespace match the service account name and namespace.
  • The token is signed by a key that matches the public key.

The public key pair for the service account signing key used by the cluster is stored in an AWS S3 bucket. AWS STS federation validates that the service account token signature aligns with the public key stored in the S3 bucket.

19.5.1.1.1. Authentication flow for AWS STS

The following diagram illustrates the authentication flow between AWS and the OpenShift Container Platform cluster when using AWS STS.

  • Token signing is the Kubernetes service account signing service on the OpenShift Container Platform cluster.
  • The Kubernetes service account in the pod is the signed service account token.

Figure 19.2. AWS Security Token Service authentication flow

Detailed authentication flow between AWS and the cluster when using AWS STS

Requests for new and refreshed credentials are automated by using an appropriately configured AWS IAM OpenID Connect (OIDC) identity provider combined with AWS IAM roles. Service account tokens that are trusted by AWS IAM are signed by OpenShift Container Platform and can be projected into a pod and used for authentication.

19.5.1.1.2. Token refreshing for AWS STS

The signed service account token that a pod uses expires after a period of time. For clusters that use AWS STS, this time period is 3600 seconds, or one hour.

The kubelet on the node that the pod is assigned to ensures that the token is refreshed. The kubelet attempts to rotate a token when it is older than 80 percent of its time to live.

19.5.1.1.3. OpenID Connect requirements for AWS STS

You can store the public portion of the encryption keys for your OIDC configuration in a public or private S3 bucket.

The OIDC spec requires the use of HTTPS. AWS services require a public endpoint to expose the OIDC documents in the form of JSON web key set (JWKS) public keys. This allows AWS services to validate the bound tokens signed by Kubernetes and determine whether to trust certificates. As a result, both S3 bucket options require a public HTTPS endpoint and private endpoints are not supported.

To use AWS STS, the public AWS backbone for the AWS STS service must be able to communicate with a public S3 bucket or a private S3 bucket with a public CloudFront endpoint. You can choose which type of bucket to use when you process CredentialsRequest objects during installation:

  • By default, the CCO utility (ccoctl) stores the OIDC configuration files in a public S3 bucket and uses the S3 URL as the public OIDC endpoint.
  • As an alternative, you can have the ccoctl utility store the OIDC configuration in a private S3 bucket that is accessed by the IAM identity provider through a public CloudFront distribution URL.

19.5.1.2. AWS component secret formats

Using manual mode with the AWS Security Token Service (STS) changes the content of the AWS credentials that are provided to individual OpenShift Container Platform components. Compare the following secret formats:

AWS secret format using long-term credentials

apiVersion: v1
kind: Secret
metadata:
  namespace: <target_namespace> 1
  name: <target_secret_name> 2
data:
  aws_access_key_id: <base64_encoded_access_key_id>
  aws_secret_access_key: <base64_encoded_secret_access_key>

1
The namespace for the component.
2
The name of the component secret.

AWS secret format using AWS STS

apiVersion: v1
kind: Secret
metadata:
  namespace: <target_namespace> 1
  name: <target_secret_name> 2
stringData:
  credentials: |-
    [default]
    sts_regional_endpoints = regional
    role_name: <operator_role_name> 3
    web_identity_token_file: <path_to_token> 4

1
The namespace for the component.
2
The name of the component secret.
3
The IAM role for the component.
4
The path to the service account token inside the pod. By convention, this is /var/run/secrets/openshift/serviceaccount/token for OpenShift Container Platform components.

19.5.1.3. AWS component secret permissions requirements

OpenShift Container Platform components require the following permissions. These values are in the CredentialsRequest custom resource (CR) for each component.

Note

These permissions apply to all resources. Unless specified, there are no request conditions on these permissions.

ComponentCustom resourceRequired permissions for services

Cluster CAPI Operator

openshift-cluster-api-aws

EC2

  • ec2:CreateTags
  • ec2:DescribeAvailabilityZones
  • ec2:DescribeDhcpOptions
  • ec2:DescribeImages
  • ec2:DescribeInstances
  • ec2:DescribeInternetGateways
  • ec2:DescribeSecurityGroups
  • ec2:DescribeSubnets
  • ec2:DescribeVpcs
  • ec2:DescribeNetworkInterfaces
  • ec2:DescribeNetworkInterfaceAttribute
  • ec2:ModifyNetworkInterfaceAttribute
  • ec2:RunInstances
  • ec2:TerminateInstances

Elastic load balancing

  • elasticloadbalancing:DescribeLoadBalancers
  • elasticloadbalancing:DescribeTargetGroups
  • elasticloadbalancing:DescribeTargetHealth
  • elasticloadbalancing:RegisterInstancesWithLoadBalancer
  • elasticloadbalancing:RegisterTargets
  • elasticloadbalancing:DeregisterTargets

Identity and Access Management (IAM)

  • iam:PassRole
  • iam:CreateServiceLinkedRole

Key Management Service (KMS)

  • kms:Decrypt
  • kms:Encrypt
  • kms:GenerateDataKey
  • kms:GenerateDataKeyWithoutPlainText
  • kms:DescribeKey
  • kms:RevokeGrant[1]
  • kms:CreateGrant [1]
  • kms:ListGrants [1]

Machine API Operator

openshift-machine-api-aws

EC2

  • ec2:CreateTags
  • ec2:DescribeAvailabilityZones
  • ec2:DescribeDhcpOptions
  • ec2:DescribeImages
  • ec2:DescribeInstances
  • ec2:DescribeInstanceTypes
  • ec2:DescribeInternetGateways
  • ec2:DescribeSecurityGroups
  • ec2:DescribeRegions
  • ec2:DescribeSubnets
  • ec2:DescribeVpcs
  • ec2:RunInstances
  • ec2:TerminateInstances

Elastic load balancing

  • elasticloadbalancing:DescribeLoadBalancers
  • elasticloadbalancing:DescribeTargetGroups
  • elasticloadbalancing:DescribeTargetHealth
  • elasticloadbalancing:RegisterInstancesWithLoadBalancer
  • elasticloadbalancing:RegisterTargets
  • elasticloadbalancing:DeregisterTargets

Identity and Access Management (IAM)

  • iam:PassRole
  • iam:CreateServiceLinkedRole

Key Management Service (KMS)

  • kms:Decrypt
  • kms:Encrypt
  • kms:GenerateDataKey
  • kms:GenerateDataKeyWithoutPlainText
  • kms:DescribeKey
  • kms:RevokeGrant[1]
  • kms:CreateGrant [1]
  • kms:ListGrants [1]

Cloud Credential Operator

cloud-credential-operator-iam-ro

Identity and Access Management (IAM)

  • iam:GetUser
  • iam:GetUserPolicy
  • iam:ListAccessKeys

Cluster Image Registry Operator

openshift-image-registry

S3

  • s3:CreateBucket
  • s3:DeleteBucket
  • s3:PutBucketTagging
  • s3:GetBucketTagging
  • s3:PutBucketPublicAccessBlock
  • s3:GetBucketPublicAccessBlock
  • s3:PutEncryptionConfiguration
  • s3:GetEncryptionConfiguration
  • s3:PutLifecycleConfiguration
  • s3:GetLifecycleConfiguration
  • s3:GetBucketLocation
  • s3:ListBucket
  • s3:GetObject
  • s3:PutObject
  • s3:DeleteObject
  • s3:ListBucketMultipartUploads
  • s3:AbortMultipartUpload
  • s3:ListMultipartUploadParts

Ingress Operator

openshift-ingress

Elastic load balancing

  • elasticloadbalancing:DescribeLoadBalancers

Route 53

  • route53:ListHostedZones
  • route53:ListTagsForResources
  • route53:ChangeResourceRecordSets

Tag

  • tag:GetResources

Security Token Service (STS)

  • sts:AssumeRole

Cluster Network Operator

openshift-cloud-network-config-controller-aws

EC2

  • ec2:DescribeInstances
  • ec2:DescribeInstanceStatus
  • ec2:DescribeInstanceTypes
  • ec2:UnassignPrivateIpAddresses
  • ec2:AssignPrivateIpAddresses
  • ec2:UnassignIpv6Addresses
  • ec2:AssignIpv6Addresses
  • ec2:DescribeSubnets
  • ec2:DescribeNetworkInterfaces

AWS Elastic Block Store CSI Driver Operator

aws-ebs-csi-driver-operator

EC2

  • ec2:AttachVolume
  • ec2:CreateSnapshot
  • ec2:CreateTags
  • ec2:CreateVolume
  • ec2:DeleteSnapshot
  • ec2:DeleteTags
  • ec2:DeleteVolume
  • ec2:DescribeInstances
  • ec2:DescribeSnapshots
  • ec2:DescribeTags
  • ec2:DescribeVolumes
  • ec2:DescribeVolumesModifications
  • ec2:DetachVolume
  • ec2:ModifyVolume
  • ec2:DescribeAvailabilityZones
  • ec2:EnableFastSnapshotRestores

Key Management Service (KMS)

  • kms:ReEncrypt*
  • kms:Decrypt
  • kms:Encrypt
  • kms:GenerateDataKey
  • kms:GenerateDataKeyWithoutPlainText
  • kms:DescribeKey
  • kms:RevokeGrant[1]
  • kms:CreateGrant [1]
  • kms:ListGrants [1]
  1. Request condition: kms:GrantIsForAWSResource: true

19.5.1.4. OLM-managed Operator support for authentication with AWS STS

In addition to OpenShift Container Platform cluster components, some Operators managed by the Operator Lifecycle Manager (OLM) on AWS clusters can use manual mode with STS. These Operators authenticate with limited-privilege, short-term credentials that are managed outside the cluster. To determine if an Operator supports authentication with AWS STS, see the Operator description in OperatorHub.

Additional resources

19.5.2. GCP Workload Identity

In manual mode with GCP Workload Identity, the individual OpenShift Container Platform cluster components use the GCP workload identity provider to allow components to impersonate GCP service accounts using short-term, limited-privilege credentials.

Additional resources

19.5.2.1. GCP Workload Identity authentication process

Requests for new and refreshed credentials are automated by using an appropriately configured OpenID Connect (OIDC) identity provider combined with IAM service accounts. Service account tokens that are trusted by GCP are signed by OpenShift Container Platform and can be projected into a pod and used for authentication. Tokens are refreshed after one hour.

The following diagram details the authentication flow between GCP and the OpenShift Container Platform cluster when using GCP Workload Identity.

Figure 19.3. GCP Workload Identity authentication flow

Detailed authentication flow between GCP and the cluster when using GCP Workload Identity

19.5.2.2. GCP component secret formats

Using manual mode with GCP Workload Identity changes the content of the GCP credentials that are provided to individual OpenShift Container Platform components. Compare the following secret content:

GCP secret format

apiVersion: v1
kind: Secret
metadata:
  namespace: <target_namespace> 1
  name: <target_secret_name> 2
data:
  service_account.json: <service_account> 3

1
The namespace for the component.
2
The name of the component secret.
3
The Base64 encoded service account.

Content of the Base64 encoded service_account.json file using long-term credentials

{
   "type": "service_account", 1
   "project_id": "<project_id>",
   "private_key_id": "<private_key_id>",
   "private_key": "<private_key>", 2
   "client_email": "<client_email_address>",
   "client_id": "<client_id>",
   "auth_uri": "https://accounts.google.com/o/oauth2/auth",
   "token_uri": "https://oauth2.googleapis.com/token",
   "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
   "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/<client_email_address>"
}

1
The credential type is service_account.
2
The private RSA key that is used to authenticate to GCP. This key must be kept secure and is not rotated.

Content of the Base64 encoded service_account.json file using GCP Workload Identity

{
   "type": "external_account", 1
   "audience": "//iam.googleapis.com/projects/123456789/locations/global/workloadIdentityPools/test-pool/providers/test-provider", 2
   "subject_token_type": "urn:ietf:params:oauth:token-type:jwt",
   "token_url": "https://sts.googleapis.com/v1/token",
   "service_account_impersonation_url": "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/<client_email_address>:generateAccessToken", 3
   "credential_source": {
      "file": "<path_to_token>", 4
      "format": {
         "type": "text"
      }
   }
}

1
The credential type is external_account.
2
The target audience is the GCP Workload Identity provider.
3
The resource URL of the service account that can be impersonated with these credentials.
4
The path to the service account token inside the pod. By convention, this is /var/run/secrets/openshift/serviceaccount/token for OpenShift Container Platform components.

19.5.3. Microsoft Entra Workload ID

In manual mode with Microsoft Entra Workload ID, the individual OpenShift Container Platform cluster components use the Workload ID provider to assign components short-term security credentials.

Additional resources

19.5.3.1. Microsoft Entra Workload ID authentication process

The following diagram details the authentication flow between Azure and the OpenShift Container Platform cluster when using Microsoft Entra Workload ID.

Figure 19.4. Workload ID authentication flow

Detailed authentication flow between Azure and the cluster when using Workload ID

19.5.3.2. Azure component secret formats

Using manual mode with Microsoft Entra Workload ID changes the content of the Azure credentials that are provided to individual OpenShift Container Platform components. Compare the following secret formats:

Azure secret format using long-term credentials

apiVersion: v1
kind: Secret
metadata:
  namespace: <target_namespace> 1
  name: <target_secret_name> 2
data:
  azure_client_id: <client_id> 3
  azure_client_secret: <client_secret> 4
  azure_region: <region>
  azure_resource_prefix: <resource_group_prefix> 5
  azure_resourcegroup: <resource_group_prefix>-rg 6
  azure_subscription_id: <subscription_id>
  azure_tenant_id: <tenant_id>
type: Opaque

1
The namespace for the component.
2
The name of the component secret.
3
The client ID of the Microsoft Entra ID identity that the component uses to authenticate.
4
The component secret that is used to authenticate with Microsoft Entra ID for the <client_id> identity.
5
The resource group prefix.
6
The resource group. This value is formed by the <resource_group_prefix> and the suffix -rg.

Azure secret format using Microsoft Entra Workload ID

apiVersion: v1
kind: Secret
metadata:
  namespace: <target_namespace> 1
  name: <target_secret_name> 2
data:
  azure_client_id: <client_id> 3
  azure_federated_token_file: <path_to_token_file> 4
  azure_region: <region>
  azure_subscription_id: <subscription_id>
  azure_tenant_id: <tenant_id>
type: Opaque

1
The namespace for the component.
2
The name of the component secret.
3
The client ID of the user-assigned managed identity that the component uses to authenticate.
4
The path to the mounted service account token file.

19.5.3.3. Azure component secret permissions requirements

OpenShift Container Platform components require the following permissions. These values are in the CredentialsRequest custom resource (CR) for each component.

ComponentCustom resourceRequired permissions for services

Cloud Controller Manager Operator

openshift-azure-cloud-controller-manager

  • Microsoft.Compute/virtualMachines/read
  • Microsoft.Network/loadBalancers/read
  • Microsoft.Network/loadBalancers/write
  • Microsoft.Network/networkInterfaces/read
  • Microsoft.Network/networkSecurityGroups/read
  • Microsoft.Network/networkSecurityGroups/write
  • Microsoft.Network/publicIPAddresses/join/action
  • Microsoft.Network/publicIPAddresses/read
  • Microsoft.Network/publicIPAddresses/write

Cluster CAPI Operator

openshift-cluster-api-azure

role: Contributor [1]

Machine API Operator

openshift-machine-api-azure

  • Microsoft.Compute/availabilitySets/delete
  • Microsoft.Compute/availabilitySets/read
  • Microsoft.Compute/availabilitySets/write
  • Microsoft.Compute/diskEncryptionSets/read
  • Microsoft.Compute/disks/delete
  • Microsoft.Compute/galleries/images/versions/read
  • Microsoft.Compute/skus/read
  • Microsoft.Compute/virtualMachines/delete
  • Microsoft.Compute/virtualMachines/extensions/delete
  • Microsoft.Compute/virtualMachines/extensions/read
  • Microsoft.Compute/virtualMachines/extensions/write
  • Microsoft.Compute/virtualMachines/read
  • Microsoft.Compute/virtualMachines/write
  • Microsoft.ManagedIdentity/userAssignedIdentities/assign/action
  • Microsoft.Network/applicationSecurityGroups/read
  • Microsoft.Network/loadBalancers/backendAddressPools/join/action
  • Microsoft.Network/loadBalancers/read
  • Microsoft.Network/loadBalancers/write
  • Microsoft.Network/networkInterfaces/delete
  • Microsoft.Network/networkInterfaces/join/action
  • Microsoft.Network/networkInterfaces/loadBalancers/read
  • Microsoft.Network/networkInterfaces/read
  • Microsoft.Network/networkInterfaces/write
  • Microsoft.Network/networkSecurityGroups/read
  • Microsoft.Network/networkSecurityGroups/write
  • Microsoft.Network/publicIPAddresses/delete
  • Microsoft.Network/publicIPAddresses/join/action
  • Microsoft.Network/publicIPAddresses/read
  • Microsoft.Network/publicIPAddresses/write
  • Microsoft.Network/routeTables/read
  • Microsoft.Network/virtualNetworks/delete
  • Microsoft.Network/virtualNetworks/read
  • Microsoft.Network/virtualNetworks/subnets/join/action
  • Microsoft.Network/virtualNetworks/subnets/read
  • Microsoft.Resources/subscriptions/resourceGroups/read

Cluster Image Registry Operator

openshift-image-registry-azure

Data permissions

  • Microsoft.Storage/storageAccounts/blobServices/containers/blobs/delete
  • Microsoft.Storage/storageAccounts/blobServices/containers/blobs/write
  • Microsoft.Storage/storageAccounts/blobServices/containers/blobs/read
  • Microsoft.Storage/storageAccounts/blobServices/containers/blobs/add/action
  • Microsoft.Storage/storageAccounts/blobServices/containers/blobs/move/action

General permissions

  • Microsoft.Storage/storageAccounts/blobServices/read
  • Microsoft.Storage/storageAccounts/blobServices/containers/read
  • Microsoft.Storage/storageAccounts/blobServices/containers/write
  • Microsoft.Storage/storageAccounts/blobServices/generateUserDelegationKey/action
  • Microsoft.Storage/storageAccounts/read
  • Microsoft.Storage/storageAccounts/write
  • Microsoft.Storage/storageAccounts/delete
  • Microsoft.Storage/storageAccounts/listKeys/action
  • Microsoft.Resources/tags/write

Ingress Operator

openshift-ingress-azure

  • Microsoft.Network/dnsZones/A/delete
  • Microsoft.Network/dnsZones/A/write
  • Microsoft.Network/privateDnsZones/A/delete
  • Microsoft.Network/privateDnsZones/A/write

Cluster Network Operator

openshift-cloud-network-config-controller-azure

  • Microsoft.Network/networkInterfaces/read
  • Microsoft.Network/networkInterfaces/write
  • Microsoft.Compute/virtualMachines/read
  • Microsoft.Network/virtualNetworks/read
  • Microsoft.Network/virtualNetworks/subnets/join/action
  • Microsoft.Network/loadBalancers/backendAddressPools/join/action

Azure File CSI Driver Operator

azure-file-csi-driver-operator

  • Microsoft.Network/networkSecurityGroups/join/action
  • Microsoft.Network/virtualNetworks/subnets/read
  • Microsoft.Network/virtualNetworks/subnets/write
  • Microsoft.Storage/storageAccounts/delete
  • Microsoft.Storage/storageAccounts/fileServices/read
  • Microsoft.Storage/storageAccounts/fileServices/shares/delete
  • Microsoft.Storage/storageAccounts/fileServices/shares/read
  • Microsoft.Storage/storageAccounts/fileServices/shares/write
  • Microsoft.Storage/storageAccounts/listKeys/action
  • Microsoft.Storage/storageAccounts/read
  • Microsoft.Storage/storageAccounts/write

Azure Disk CSI Driver Operator

azure-disk-csi-driver-operator

  • Microsoft.Compute/disks/*
  • Microsoft.Compute/snapshots/*
  • Microsoft.Compute/virtualMachineScaleSets/*/read
  • Microsoft.Compute/virtualMachineScaleSets/read
  • Microsoft.Compute/virtualMachineScaleSets/virtualMachines/write
  • Microsoft.Compute/virtualMachines/*/read
  • Microsoft.Compute/virtualMachines/write
  • Microsoft.Resources/subscriptions/resourceGroups/read
  1. This component requires a role rather than a set of permissions.

19.5.3.4. OLM-managed Operator support for authentication with Microsoft Entra Workload ID

In addition to OpenShift Container Platform cluster components, some Operators managed by the Operator Lifecycle Manager (OLM) on Azure clusters can use manual mode with Microsoft Entra Workload ID. These Operators authenticate with short-term credentials that are managed outside the cluster. To determine if an Operator supports authentication with Workload ID, see the Operator description in OperatorHub.

Additional resources

19.5.4. Additional resources

Red Hat logoGithubRedditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

© 2024 Red Hat, Inc.