10장.
10.1. 링크 복사링크가 클립보드에 복사되었습니다!
10.1.1. 링크 복사링크가 클립보드에 복사되었습니다!
$ oc adm wait-for-stable-cluster --minimum-stable-period=5s
INFRA_ID=$(oc get infrastructures cluster -o jsonpath='{.status.infrastructureName}') CLUSTER_NAME=${INFRA_ID%-*}- 참고
AWS_BUCKET=$(oc get authentication cluster -o jsonpath={'.spec.serviceAccountIssuer'} | awk -F'://' '{print$2}' |awk -F'.' '{print$1}')$ basename $(oc get authentication cluster -o jsonpath={'.spec.serviceAccountIssuer'} )<subdomain>.cloudfront.net$ aws cloudfront list-distributions --query "DistributionList.Items[].{DomainName: DomainName, OriginDomainName: Origins.Items[0].DomainName}[?contains(DomainName, '<subdomain>.cloudfront.net')]"[ { "DomainName": "<subdomain>.cloudfront.net", "OriginDomainName": "<s3_bucket>.s3.us-east-2.amazonaws.com" } ]AWS_BUCKET=$<s3_bucket>
$ TEMPDIR=$(mktemp -d)- 중요
$ oc delete secrets/next-bound-service-account-signing-key \ -n openshift-kube-apiserver-operator $ oc get secret/next-bound-service-account-signing-key \ -n openshift-kube-apiserver-operator \ -ojsonpath='{ .data.service-account\.pub }' | base64 \ -d > ${TEMPDIR}/serviceaccount-signer.public$ ccoctl aws create-identity-provider \ --dry-run \ --output-dir ${TEMPDIR} \ --public-key-file=${TEMPDIR}/serviceaccount-signer.public \ --name fake \ --region us-east-1$ cp ${TEMPDIR}/<number>-keys.json ${TEMPDIR}/jwks.new.json$ aws s3api get-object \ --bucket ${AWS_BUCKET} \ --key keys.json ${TEMPDIR}/jwks.current.json$ jq -s '{ keys: map(.keys[])}' ${TEMPDIR}/jwks.current.json ${TEMPDIR}/jwks.new.json > ${TEMPDIR}/jwks.combined.json$ aws s3api put-object \ --bucket ${AWS_BUCKET} \ --tagging "openshift.io/cloud-credential-operator/${CLUSTER_NAME}=owned" \ --key keys.json \ --body ${TEMPDIR}/jwks.combined.json$ oc adm wait-for-stable-clusterAll clusteroperators are stable- 중요
$ oc adm reboot-machine-config-pool mcp/worker mcp/master $ oc adm wait-for-node-reboot nodes --allAll nodes rebooted$ oc adm wait-for-stable-clusterAll clusteroperators are stable$ aws s3api put-object \ --bucket ${AWS_BUCKET} \ --tagging "openshift.io/cloud-credential-operator/${CLUSTER_NAME}=owned" \ --key keys.json \ --body ${TEMPDIR}/jwks.new.json
10.1.2. 링크 복사링크가 클립보드에 복사되었습니다!
$ oc adm wait-for-stable-cluster --minimum-stable-period=5s
CURRENT_ISSUER=$(oc get authentication cluster -o jsonpath='{.spec.serviceAccountIssuer}') GCP_BUCKET=$(echo ${CURRENT_ISSUER} | cut -d "/" -f4) CLUSTER_NAME=${GCP_BUCKET%-*}참고$ TEMPDIR=$(mktemp -d)- 중요
$ oc delete secrets/next-bound-service-account-signing-key \ -n openshift-kube-apiserver-operator $ oc get secret/next-bound-service-account-signing-key \ -n openshift-kube-apiserver-operator \ -ojsonpath='{ .data.service-account\.pub }' | base64 \ -d > ${TEMPDIR}/serviceaccount-signer.public$ ccoctl gcp create-workload-identity-provider \ --dry-run \ --output-dir=${TEMPDIR} \ --public-key-file=${TEMPDIR}/serviceaccount-signer.public \ --name fake \ --project fake \ --workload-identity-pool fake$ cp ${TEMPDIR}/<number>-keys.json ${TEMPDIR}/jwks.new.json$ gcloud storage cp gs://${GCP_BUCKET}/keys.json ${TEMPDIR}/jwks.current.json$ gcloud iam workload-identity-pools providers describe \ --format json \ --location global \ --workload-identity-pool ${CLUSTER_NAME} ${CLUSTER_NAME} \ | jq -r ".oidc.jwksJson" > ${TEMPDIR}/jwks.current.json
$ jq -s '{ keys: map(.keys[])}' ${TEMPDIR}/jwks.current.json ${TEMPDIR}/jwks.new.json > ${TEMPDIR}/jwks.combined.json$ gcloud storage cp ${TEMPDIR}/jwks.combined.json gs://${GCP_BUCKET}/keys.json$ gcloud iam workload-identity-pools providers update-oidc ${CLUSTER_NAME} \ --location=global \ --workload-identity-pool=${CLUSTER_NAME} \ --jwk-json-path=${TEMPDIR}/jwks.combined.json
$ oc adm wait-for-stable-clusterAll clusteroperators are stable- 중요
$ oc adm reboot-machine-config-pool mcp/worker mcp/master $ oc adm wait-for-node-reboot nodes --allAll nodes rebooted$ oc adm wait-for-stable-clusterAll clusteroperators are stable$ gcloud storage cp ${TEMPDIR}/jwks.new.json gs://${GCP_BUCKET}/keys.json$ gcloud iam workload-identity-pools providers update-oidc ${CLUSTER_NAME} \ --location=global \ --workload-identity-pool=${CLUSTER_NAME} \ --jwk-json-path=${TEMPDIR}/jwks.new.json
10.1.3. 링크 복사링크가 클립보드에 복사되었습니다!
$ oc adm wait-for-stable-cluster --minimum-stable-period=5s
CURRENT_ISSUER=$(oc get authentication cluster -o jsonpath='{.spec.serviceAccountIssuer}') AZURE_STORAGE_ACCOUNT=$(echo ${CURRENT_ISSUER} | cut -d "/" -f3 | cut -d "." -f1) AZURE_STORAGE_CONTAINER=$(echo ${CURRENT_ISSUER} | cut -d "/" -f4)참고$ TEMPDIR=$(mktemp -d)- 중요
$ oc delete secrets/next-bound-service-account-signing-key \ -n openshift-kube-apiserver-operator $ oc get secret/next-bound-service-account-signing-key \ -n openshift-kube-apiserver-operator \ -ojsonpath='{ .data.service-account\.pub }' | base64 \ -d > ${TEMPDIR}/serviceaccount-signer.public$ ccoctl aws create-identity-provider \ --dry-run \ --output-dir ${TEMPDIR} \ --public-key-file=${TEMPDIR}/serviceaccount-signer.public \ --name fake \ --region us-east-1$ cp ${TEMPDIR}/<number>-keys.json ${TEMPDIR}/jwks.new.json$ az storage blob download \ --container-name ${AZURE_STORAGE_CONTAINER} \ --account-name ${AZURE_STORAGE_ACCOUNT} \ --name 'openid/v1/jwks' \ -f ${TEMPDIR}/jwks.current.json$ jq -s '{ keys: map(.keys[])}' ${TEMPDIR}/jwks.current.json ${TEMPDIR}/jwks.new.json > ${TEMPDIR}/jwks.combined.json$ az storage blob upload \ --overwrite \ --account-name ${AZURE_STORAGE_ACCOUNT} \ --container-name ${AZURE_STORAGE_CONTAINER} \ --name 'openid/v1/jwks' \ -f ${TEMPDIR}/jwks.combined.json$ oc adm wait-for-stable-clusterAll clusteroperators are stable- 중요
$ oc adm reboot-machine-config-pool mcp/worker mcp/master $ oc adm wait-for-node-reboot nodes --allAll nodes rebooted$ oc adm wait-for-stable-clusterAll clusteroperators are stable$ az storage blob upload \ --overwrite \ --account-name ${AZURE_STORAGE_ACCOUNT} \ --container-name ${AZURE_STORAGE_CONTAINER} \ --name 'openid/v1/jwks' \ -f ${TEMPDIR}/jwks.new.json
10.1.4. 링크 복사링크가 클립보드에 복사되었습니다!
$ ccoctl <provider_name> refresh-keys \ --kubeconfig <openshift_kubeconfig_file> \ --credentials-requests-dir <path_to_credential_requests_directory> \ --name <name>- 참고
10.2. 링크 복사링크가 클립보드에 복사되었습니다!
10.2.1. 링크 복사링크가 클립보드에 복사되었습니다!
- 참고
$ oc patch kubecontrollermanager cluster \ -p='{"spec": {"forceRedeploymentReason": "recovery-'"$( date )"'"}}' \ --type=merge$ oc get co kube-controller-manager $ oc -n openshift-cloud-credential-operator get CredentialsRequest \ -o json | jq -r '.items[] | select (.spec.providerSpec.kind=="<provider_spec>") | .spec.secretRef'{ "name": "ebs-cloud-credentials", "namespace": "openshift-cluster-csi-drivers" } { "name": "cloud-credential-operator-iam-ro-creds", "namespace": "openshift-cloud-credential-operator" }$ oc delete secret <secret_name> \1 -n <secret_namespace>2 $ oc delete secret ebs-cloud-credentials -n openshift-cluster-csi-drivers
10.3. 링크 복사링크가 클립보드에 복사되었습니다!
10.3.1. 링크 복사링크가 클립보드에 복사되었습니다!
10.4. 링크 복사링크가 클립보드에 복사되었습니다!
10.4.1. 링크 복사링크가 클립보드에 복사되었습니다!
$ RELEASE_IMAGE=$(oc get clusterversion -o jsonpath={..desired.image})$ CCO_IMAGE=$(oc adm release info --image-for='cloud-credential-operator' $RELEASE_IMAGE -a ~/.pull-secret)참고$ oc image extract $CCO_IMAGE \ --file="/usr/bin/ccoctl.<rhel_version>" \ -a ~/.pull-secret참고$ chmod 775 ccoctl
$ ./ccoctlOpenShift credentials provisioning tool Usage: ccoctl [command] Available Commands: aws Manage credentials objects for AWS cloud azure Manage credentials objects for Azure gcp Manage credentials objects for Google cloud help Help about any command ibmcloud Manage credentials objects for IBM Cloud nutanix Manage credentials objects for Nutanix Flags: -h, --help help for ccoctl Use "ccoctl [command] --help" for more information about a command.
10.4.2. 링크 복사링크가 클립보드에 복사되었습니다!
$ oc get secret/next-bound-service-account-signing-key \ -n openshift-kube-apiserver-operator \ -ojsonpath='{ .data.service-account\.pub }' | base64 -d \ > output_dir/serviceaccount-signer.public$ ./ccoctl azure create-oidc-issuer \ --name <azure_infra_name> \ --output-dir ./output_dir \ --region <azure_region> \ --subscription-id <azure_subscription_id> \ --tenant-id <azure_tenant_id> \ --public-key-file ./output_dir/serviceaccount-signer.public$ ll ./output_dir/manifeststotal 8 -rw-------. 1 cloud-user cloud-user 193 May 22 02:29 azure-ad-pod-identity-webhook-config.yaml -rw-------. 1 cloud-user cloud-user 165 May 22 02:29 cluster-authentication-02-config.yaml$ OIDC_ISSUER_URL=`awk '/serviceAccountIssuer/ { print $2 }' ./output_dir/manifests/cluster-authentication-02-config.yaml`$ oc patch authentication cluster \ --type=merge \ -p "{\"spec\":{\"serviceAccountIssuer\":\"${OIDC_ISSUER_URL}\"}}"$ oc adm wait-for-stable-clusterAll clusteroperators are stable$ oc adm reboot-machine-config-pool mcp/worker mcp/master$ oc adm wait-for-node-reboot nodes --allAll nodes rebooted$ oc patch cloudcredential cluster \ --type=merge \ --patch '{"spec":{"credentialsMode":"Manual"}}'$ oc adm release extract \ --credentials-requests \ --included \ --to <path_to_directory_for_credentials_requests> \ --registry-config ~/.pull-secret참고$ AZURE_INSTALL_RG=`oc get infrastructure cluster -o jsonpath --template '{ .status.platformStatus.azure.resourceGroupName }'`- 참고
$ ccoctl azure create-managed-identities \ --name <azure_infra_name> \ --output-dir ./output_dir \ --region <azure_region> \ --subscription-id <azure_subscription_id> \ --credentials-requests-dir <path_to_directory_for_credentials_requests> \ --issuer-url "${OIDC_ISSUER_URL}" \ --dnszone-resource-group-name <azure_dns_zone_resourcegroup_name> \ --installation-resource-group-name "${AZURE_INSTALL_RG}" \ --network-resource-group-name <azure_resource_group> \ --preserve-existing-roles $ oc apply -f ./output_dir/manifests/azure-ad-pod-identity-webhook-config.yaml$ find ./output_dir/manifests -iname "openshift*yaml" -print0 | xargs -I {} -0 -t oc replace -f {}$ oc adm reboot-machine-config-pool mcp/worker mcp/master$ oc adm wait-for-node-reboot nodes --allAll nodes rebooted$ oc adm wait-for-stable-clusterAll clusteroperators are stable$ oc delete secret -n kube-system azure-credentials
10.4.3. 링크 복사링크가 클립보드에 복사되었습니다!
$ mkdir ./output_dir$ oc get secret/next-bound-service-account-signing-key \ -n openshift-kube-apiserver-operator \ -ojsonpath='{ .data.service-account\.pub }' | base64 -d \ > output_dir/serviceaccount-signer.public$ ./ccoctl aws create-identity-provider \ --output-dir output_dir \ --name <name_you_choose> \ --region us-east-2 \ --public-key-file output_dir/serviceaccount-signer.public
$ OIDC_ISSUER_URL=`awk '/serviceAccountIssuer/ { print $2 }' output_dir/manifests/cluster-authentication-02-config.yaml` $ oc patch authentication cluster --type=merge -p "{\"spec\":{\"serviceAccountIssuer\":\"${OIDC_ISSUER_URL}\"}}"$ oc adm wait-for-stable-clusterAll clusteroperators are stable
$ oc adm reboot-machine-config-pool mcp/worker mcp/master$ oc adm wait-for-node-reboot nodes --allAll nodes rebooted
$ oc patch cloudcredential cluster \ --type=merge \ --patch '{"spec":{"credentialsMode":"Manual"}}'$ CLUSTER_VERSION=$(oc get clusterversion version -o json | jq -r '.status.desired.version')$ CLUSTER_IMAGE=$(oc get clusterversion version -o json | jq -r ".status.history[] | select(.version == \"${CLUSTER_VERSION}\") | .image")$ oc adm release extract \ --credentials-requests \ --cloud=aws \ --from ${CLUSTER_IMAGE} \ --to output_dir/cred-reqs
$ ./ccoctl aws create-iam-roles \ --output-dir ./output_dir/ \ --name <name_you_choose> \ --identity-provider-arn <identity_provider_arn> \ --region us-east-2 \ --credentials-requests-dir ./output_dir/cred-reqs/ \ --permissions-boundary-arn=<policy_arn>$ find ./output_dir/manifests -iname "openshift*yaml" -print0 | xargs -I {} -0 -t oc replace -f {}
$ oc adm reboot-machine-config-pool mcp/worker mcp/master$ oc adm wait-for-node-reboot nodes --allAll nodes rebooted$ oc adm wait-for-stable-clusterAll clusteroperators are stable
$ oc delete secret -n kube-system aws-creds
10.4.4. 링크 복사링크가 클립보드에 복사되었습니다!
$ oc get cloudcredentials cluster \ -o=jsonpath={.spec.credentialsMode}Manual$ oc get secrets \ -n kube-system <secret_name>Error from server (NotFound): secrets "aws-creds" not found$ oc get authentication cluster \ -o jsonpath \ --template='{ .spec.serviceAccountIssuer }'$ oc get secrets \ -n openshift-image-registry installer-cloud-credentials \ -o jsonpath='{.data}'$ oc get pods \ -n openshift-cloud-credential-operatorNAME READY STATUS RESTARTS AGE cloud-credential-operator-59cf744f78-r8pbq 2/2 Running 2 71m pod-identity-webhook-548f977b4c-859lz 1/1 Running 1 70m