Chapter 10. Changing the cloud provider credentials configuration

Procedure

1. In the Administrator perspective of the web console, navigate to Workloads Secrets.

2. In the table on the Secrets page, find the root secret for your cloud provider.

   Expand

   Platform	Secret name
   AWS	aws-creds
   Azure	azure-credentials
   Google Cloud	gcp-credentials
   RHOSP	openstack-credentials
   VMware vSphere	vsphere-creds

   Show more
3. Click the Options menu in the same row as the secret and select Edit Secret.
4. Record the contents of the Value field or fields. You can use this information to verify that the value is different after updating the credentials.
5. Update the text in the Value field or fields with the new authentication information for your cloud provider, and then click Save.

6. If you are updating the credentials for a vSphere cluster that does not have the vSphere CSI Driver Operator enabled, you must force a rollout of the Kubernetes controller manager to apply the updated credentials.

   Note

   If the vSphere CSI Driver Operator is enabled, this step is not required.

   To apply the updated vSphere credentials, log in to the OpenShift Container Platform CLI as a user with the cluster-admin role and run the following command:

   $ oc patch kubecontrollermanager cluster \
     -p='{"spec": {"forceRedeploymentReason": "recovery-'"$( date )"'"}}' \
     --type=merge

   While the credentials are rolling out, the status of the Kubernetes Controller Manager Operator reports Progressing=true. To view the status, run the following command:

   $ oc get co kube-controller-manager

7. If the CCO for your cluster is configured to use mint mode, delete each component secret that is referenced by the individual CredentialsRequest objects.

   1. Log in to the OpenShift Container Platform CLI as a user with the cluster-admin role.

   2. Get the names and namespaces of all referenced component secrets:

      $ oc -n openshift-cloud-credential-operator get CredentialsRequest \
        -o json | jq -r '.items[] | select (.spec.providerSpec.kind=="<provider_spec>") | .spec.secretRef'

      where <provider_spec> is the corresponding value for your cloud provider:

      • AWS: AWSProviderSpec
      • Google Cloud: GCPProviderSpec

      Partial example output for AWS

      {
        "name": "ebs-cloud-credentials",
        "namespace": "openshift-cluster-csi-drivers"
      }
      {
        "name": "cloud-credential-operator-iam-ro-creds",
        "namespace": "openshift-cloud-credential-operator"
      }

   3. Delete each of the referenced component secrets:

      $ oc delete secret <secret_name> \

      1

      
        -n <secret_namespace> 

      2

      1

      Specify the name of a secret.

      2

      Specify the namespace that contains the secret.

      Example deletion of an AWS secret

      $ oc delete secret ebs-cloud-credentials -n openshift-cluster-csi-drivers

      You do not need to manually delete the credentials from your provider console. Deleting the referenced component secrets will cause the CCO to delete the existing credentials from the platform and create new ones.

1. In the Administrator perspective of the web console, navigate to Workloads Secrets.
2. Verify that the contents of the Value field or fields have changed.

Procedure

1. In the Administrator perspective of the web console, navigate to Workloads Secrets.

2. In the table on the Secrets page, find the root secret for your cloud provider.

   Expand

   Platform	Secret name
   AWS	aws-creds
   Google Cloud	gcp-credentials

   Show more
3. Click the Options menu in the same row as the secret and select Delete Secret.

Procedure

1. Set a variable for the OpenShift Container Platform release image by running the following command:

   $ RELEASE_IMAGE=$(oc get clusterversion -o jsonpath={..desired.image})

2. Obtain the CCO container image from the OpenShift Container Platform release image by running the following command:

   $ CCO_IMAGE=$(oc adm release info --image-for='cloud-credential-operator' $RELEASE_IMAGE -a ~/.pull-secret)

   Note

   Ensure that the architecture of the $RELEASE_IMAGE matches the architecture of the environment in which you will use the ccoctl tool.

3. Extract the ccoctl binary from the CCO container image within the OpenShift Container Platform release image by running the following command:

   $ oc image extract $CCO_IMAGE \
     --file="/usr/bin/ccoctl.<rhel_version>" \

   1

   
     -a ~/.pull-secret

   1

   For <rhel_version>, specify the value that corresponds to the version of Red Hat Enterprise Linux (RHEL) that the host uses. If no value is specified, ccoctl.rhel8 is used by default. The following values are valid:

   • rhel8: Specify this value for hosts that use RHEL 8.
   • rhel9: Specify this value for hosts that use RHEL 9.

   Note

   The ccoctl binary is created in the directory from where you executed the command and not in /usr/bin/. You must rename the directory or move the ccoctl.<rhel_version> binary to ccoctl.

4. Change the permissions to make ccoctl executable by running the following command:

   $ chmod 775 ccoctl

Procedure

1. Create an output directory for the manifests that the ccoctl utility generates. This procedure uses ./output_dir as an example.

2. Extract the service account public signing key for the cluster to the output directory by running the following command:

   $ oc get secret/next-bound-service-account-signing-key \
     -n openshift-kube-apiserver-operator \
     -ojsonpath='{ .data.service-account\.pub }' | base64 -d \
     > output_dir/serviceaccount-signer.public 

   1

   1

   This procedure uses a file named serviceaccount-signer.public as an example.

3. Use the extracted service account public signing key to create an OpenID Connect (OIDC) issuer and Azure blob storage container with OIDC configuration files by running the following command:

   $ ./ccoctl azure create-oidc-issuer \
     --name <azure_infra_name> \

   1

   
     --output-dir ./output_dir \
     --region <azure_region> \

   2

   
     --subscription-id <azure_subscription_id> \

   3

   
     --tenant-id <azure_tenant_id> \
     --public-key-file ./output_dir/serviceaccount-signer.public 

   4

   1

   The value of the name parameter is used to create an Azure resource group. To use an existing Azure resource group instead of creating a new one, specify the --oidc-resource-group-name argument with the existing group name as its value.

   2

   Specify the region of the existing cluster.

   3

   Specify the subscription ID of the existing cluster.

   4

   Specify the file that contains the service account public signing key for the cluster.

4. Verify that the configuration file for the Azure pod identity webhook was created by running the following command:

   $ ll ./output_dir/manifests

   Example output

   total 8
   -rw-------. 1 cloud-user cloud-user 193 May 22 02:29 azure-ad-pod-identity-webhook-config.yaml 

   1

   
   -rw-------. 1 cloud-user cloud-user 165 May 22 02:29 cluster-authentication-02-config.yaml

   1

   The file azure-ad-pod-identity-webhook-config.yaml contains the Azure pod identity webhook configuration.

5. Set an OIDC_ISSUER_URL variable with the OIDC issuer URL from the generated manifests in the output directory by running the following command:

   $ OIDC_ISSUER_URL=`awk '/serviceAccountIssuer/ { print $2 }' ./output_dir/manifests/cluster-authentication-02-config.yaml`

6. Update the spec.serviceAccountIssuer parameter of the cluster authentication configuration by running the following command:

   $ oc patch authentication cluster \
     --type=merge \
     -p "{\"spec\":{\"serviceAccountIssuer\":\"${OIDC_ISSUER_URL}\"}}"

7. Monitor the configuration update progress by running the following command:

   $ oc adm wait-for-stable-cluster

   This process might take 15 minutes or longer. The following output indicates that the process is complete:

   All clusteroperators are stable

8. Restart all of the pods in the cluster by running the following command:

   $ oc adm reboot-machine-config-pool mcp/worker mcp/master

   Restarting a pod updates the serviceAccountIssuer field and refreshes the service account public signing key.

9. Monitor the restart and update process by running the following command:

   $ oc adm wait-for-node-reboot nodes --all

   This process might take 15 minutes or longer. The following output indicates that the process is complete:

   All nodes rebooted

10. Update the Cloud Credential Operator spec.credentialsMode parameter to Manual by running the following command:

    $ oc patch cloudcredential cluster \
      --type=merge \
      --patch '{"spec":{"credentialsMode":"Manual"}}'

11. Extract the list of CredentialsRequest objects from the OpenShift Container Platform release image by running the following command:

    $ oc adm release extract \
      --credentials-requests \
      --included \
      --to <path_to_directory_for_credentials_requests> \
      --registry-config ~/.pull-secret

    Note

    This command might take a few moments to run.

12. Set an AZURE_INSTALL_RG variable with the Azure resource group name by running the following command:

    $ AZURE_INSTALL_RG=`oc get infrastructure cluster -o jsonpath --template '{ .status.platformStatus.azure.resourceGroupName }'`

13. Use the ccoctl utility to create managed identities for all CredentialsRequest objects by running the following command:

    Note

    The following command does not show all available options. For a complete list of options, including those that might be necessary for your specific use case, run $ ccoctl azure create-managed-identities --help.

    $ ccoctl azure create-managed-identities \
      --name <azure_infra_name> \
      --output-dir ./output_dir \
      --region <azure_region> \
      --subscription-id <azure_subscription_id> \
      --credentials-requests-dir <path_to_directory_for_credentials_requests> \
      --issuer-url "${OIDC_ISSUER_URL}" \
      --dnszone-resource-group-name <azure_dns_zone_resourcegroup_name> \

    1

    
      --installation-resource-group-name "${AZURE_INSTALL_RG}" \
      --network-resource-group-name <azure_resource_group> 

    2

    1

    Specify the name of the resource group that contains the DNS zone.

    2

    Optional: Specify the virtual network resource group if it is different from the cluster resource group.

14. Apply the Azure pod identity webhook configuration for Workload ID by running the following command:

    $ oc apply -f ./output_dir/manifests/azure-ad-pod-identity-webhook-config.yaml

15. Apply the secrets generated by the ccoctl utility by running the following command:

    $ find ./output_dir/manifests -iname "openshift*yaml" -print0 | xargs -I {} -0 -t oc replace -f {}

    This process might take several minutes.

16. Restart all of the pods in the cluster by running the following command:

    $ oc adm reboot-machine-config-pool mcp/worker mcp/master

    Restarting a pod updates the serviceAccountIssuer field and refreshes the service account public signing key.

17. Monitor the restart and update process by running the following command:

    $ oc adm wait-for-node-reboot nodes --all

    This process might take 15 minutes or longer. The following output indicates that the process is complete:

    All nodes rebooted

18. Monitor the configuration update progress by running the following command:

    $ oc adm wait-for-stable-cluster

    This process might take 15 minutes or longer. The following output indicates that the process is complete:

    All clusteroperators are stable

19. Optional: Remove the Azure root credentials secret by running the following command:

    $ oc delete secret -n kube-system azure-credentials

Procedure

1. Create an output directory for ccoctl generated manifests by running the following command:

   $ mkdir ./output_dir

2. Create the AWS Identity and Access Management (IAM) OpenID Connect (OIDC) provider.

   1. Extract the service account public signing key for the cluster by running the following command:

      $ oc get secret/next-bound-service-account-signing-key \
        -n openshift-kube-apiserver-operator \
        -ojsonpath='{ .data.service-account\.pub }' | base64 -d \
        > output_dir/serviceaccount-signer.public 

      1

      1

      This procedure uses a file named serviceaccount-signer.public as an example.

   2. Create the AWS IAM identity provider and S3 bucket by running the following command:

      $ ./ccoctl aws create-identity-provider \
        --output-dir output_dir \ 

      1

      
        --name <name_you_choose> \ 

      2

      
        --region us-east-2 \ 

      3

      
        --public-key-file output_dir/serviceaccount-signer.public 

      4

      1

      Specify the output directory you created earlier.

      2

      Specify a globally unique name. This name functions as a prefix for AWS resources created by this command.

      3

      Specify the AWS region of the cluster.

      4

      Specify the relative path to the serviceaccount-signer.public file you created earlier.

   3. Save or note the Amazon Resource Name (ARN) for the IAM identity provider. You can find this information in the final line of the output of the previous command.

3. Update the cluster authentication configuration.

   1. Extract the OIDC issuer URL and update the authentication configuration of the cluster by running the following commands:

      $ OIDC_ISSUER_URL=`awk '/serviceAccountIssuer/ { print $2 }' output_dir/manifests/cluster-authentication-02-config.yaml`
      $ oc patch authentication cluster --type=merge -p "{\"spec\":{\"serviceAccountIssuer\":\"${OIDC_ISSUER_URL}\"}}"

   2. Monitor the configuration update progress by running the following command:

      $ oc adm wait-for-stable-cluster

      This process might take 15 minutes or longer. The following output indicates that the process is complete:

      All clusteroperators are stable

4. Restart pods to apply the issuer update.

   1. Restart all of the pods in the cluster by running the following command:

      $ oc adm reboot-machine-config-pool mcp/worker mcp/master

      Restarting a pod updates the serviceAccountIssuer field and refreshes the service account public signing key.

   2. Monitor the restart and update process by running the following command:

      $ oc adm wait-for-node-reboot nodes --all

      This process might take 15 minutes or longer. The following output indicates that the process is complete:

      All nodes rebooted

5. Update the Cloud Credential Operator spec.credentialsMode parameter to Manual by running the following command:

   $ oc patch cloudcredential cluster \
     --type=merge \
     --patch '{"spec":{"credentialsMode":"Manual"}}'

6. Extract CredentialsRequests objects.

   1. Create a CLUSTER_VERSION environment variable by running the following command:

      $ CLUSTER_VERSION=$(oc get clusterversion version -o json | jq -r '.status.desired.version')

   2. Create a CLUSTER_IMAGE environment variable by running the following command:

      $ CLUSTER_IMAGE=$(oc get clusterversion version -o json | jq -r ".status.history[] | select(.version == \"${CLUSTER_VERSION}\") | .image")

   3. Extract CredentialsRequests objects from the release image by running the following command:

      $ oc adm release extract \
        --credentials-requests \
        --cloud=aws \
        --from ${CLUSTER_IMAGE} \
        --to output_dir/cred-reqs

7. Create AWS IAM roles and apply secrets.

   1. Create an IAM role for each CredentialsRequests object by running the following command:

      $ ./ccoctl aws create-iam-roles \
        --output-dir ./output_dir/ \ 

      1

      
        --name <name_you_choose> \ 

      2

      
        --identity-provider-arn <identity_provider_arn> \ 

      3

      
        --region us-east-2 \ 

      4

      
        --credentials-requests-dir ./output_dir/cred-reqs/ 

      5

      1

      Specify the output directory you created earlier.

      2

      Specify a globally unique name. This name functions as a prefix for AWS resources created by this command.

      3

      Specify the ARN for the IAM identity provider.

      4

      Specify the AWS region of the cluster.

      5

      Specify the relative path to the folder where you extracted the CredentialsRequest files with the oc adm release extract command.

   2. Apply the generated secrets by running the following command:

      $ find ./output_dir/manifests -iname "openshift*yaml" -print0 | xargs -I {} -0 -t oc replace -f {}

8. Finish the configuration process by restarting the cluster.

   1. Restart all of the pods in the cluster by running the following command:

      $ oc adm reboot-machine-config-pool mcp/worker mcp/master

   2. Monitor the restart and update process by running the following command:

      $ oc adm wait-for-node-reboot nodes --all

      This process might take 15 minutes or longer. The following output indicates that the process is complete:

      All nodes rebooted

   3. Monitor the configuration update progress by running the following command:

      $ oc adm wait-for-stable-cluster

      This process might take 15 minutes or longer. The following output indicates that the process is complete:

      All clusteroperators are stable

9. Optional: Remove the AWS root credentials secret by running the following command:

   $ oc delete secret -n kube-system aws-creds