托管 control plane

流程

1. 输入以下命令登录到 etcd pod：

   $ oc rsh -n <hosted_control_plane_namespace> -c etcd <etcd_pod_name>

2. 输入以下命令输出 etcd 集群的健康状况：

   sh-4.4$ etcdctl endpoint health --cluster -w table

   输出示例

   ENDPOINT                                                HEALTH  TOOK        ERROR
   https://etcd-0.etcd-discovery.clusters-hosted.svc:2379  true    9.117698ms

流程

1. 要确认 etcd pod 失败，请输入以下命令：

   $ oc get pods -l app=etcd -n <hosted_control_plane_namespace>

   输出示例

   NAME     READY   STATUS             RESTARTS     AGE
   etcd-0   2/2     Running            0            64m
   etcd-1   2/2     Running            0            45m
   etcd-2   1/2     CrashLoopBackOff   1 (5s ago)   64m

   失败的 etcd pod 可能具有 CrashLoopBackOff 或 Error 状态。

2. 输入以下命令删除失败的 pod 及其 PVC：

   $ oc delete pvc/<etcd_pvc_name> pod/<etcd_pod_name> --wait=false

流程

1. 首先，设置环境变量并缩减 API 服务器：

   1. 输入以下命令为您的托管集群设置环境变量，根据需要替换值：

      $ CLUSTER_NAME=my-cluster

      $ HOSTED_CLUSTER_NAMESPACE=clusters

      $ CONTROL_PLANE_NAMESPACE="${HOSTED_CLUSTER_NAMESPACE}-${CLUSTER_NAME}"

   2. 输入以下命令暂停托管集群的协调，根据需要替换值：

      $ oc patch -n ${HOSTED_CLUSTER_NAMESPACE} hostedclusters/${CLUSTER_NAME} -p '{"spec":{"pausedUntil":"true"}}' --type=merge

   3. 输入以下命令缩减 API 服务器：

      1. 缩减 kube-apiserver ：

         $ oc scale -n ${CONTROL_PLANE_NAMESPACE} deployment/kube-apiserver --replicas=0

      2. 缩减 openshift-apiserver ：

         $ oc scale -n ${CONTROL_PLANE_NAMESPACE} deployment/openshift-apiserver --replicas=0

      3. 缩减 openshift-oauth-apiserver ：

         $ oc scale -n ${CONTROL_PLANE_NAMESPACE} deployment/openshift-oauth-apiserver --replicas=0

2. 接下来，使用以下方法之一生成 etcd 快照：

   1. 使用之前备份的 etcd 快照。

   2. 如果您有可用的 etcd pod，通过完成以下步骤从活跃 etcd pod 创建快照：

      1. 输入以下命令列出 etcd pod：

         $ oc get -n ${CONTROL_PLANE_NAMESPACE} pods -l app=etcd

      2. 输入以下命令为 pod 数据库生成快照并将其保存在您的机器中：

         $ ETCD_POD=etcd-0

         $ oc exec -n ${CONTROL_PLANE_NAMESPACE} -c etcd -t ${ETCD_POD} -- env ETCDCTL_API=3 /usr/bin/etcdctl \
         --cacert /etc/etcd/tls/etcd-ca/ca.crt \
         --cert /etc/etcd/tls/client/etcd-client.crt \
         --key /etc/etcd/tls/client/etcd-client.key \
         --endpoints=https://localhost:2379 \
         snapshot save /var/lib/snapshot.db

      3. 输入以下命令验证快照是否成功：

         $ oc exec -n ${CONTROL_PLANE_NAMESPACE} -c etcd -t ${ETCD_POD} -- env ETCDCTL_API=3 /usr/bin/etcdctl -w table snapshot status /var/lib/snapshot.db

   3. 输入以下命令制作快照的本地副本：

      $ oc cp -c etcd ${CONTROL_PLANE_NAMESPACE}/${ETCD_POD}:/var/lib/snapshot.db /tmp/etcd.snapshot.db

      1. 从 etcd 持久性存储生成快照数据库副本：

         1. 输入以下命令列出 etcd pod：

            $ oc get -n ${CONTROL_PLANE_NAMESPACE} pods -l app=etcd

         2. 输入以下命令查找正在运行的 pod，并将其名称设置为 ETCD_POD: ETCD_POD=etcd-0，然后复制其快照数据库：

            $ oc cp -c etcd ${CONTROL_PLANE_NAMESPACE}/${ETCD_POD}:/var/lib/data/member/snap/db /tmp/etcd.snapshot.db

3. 接下来，输入以下命令缩减 etcd statefulset：

   $ oc scale -n ${CONTROL_PLANE_NAMESPACE} statefulset/etcd --replicas=0

   1. 输入以下命令删除第二个和第三个成员的卷：

      $ oc delete -n ${CONTROL_PLANE_NAMESPACE} pvc/data-etcd-1 pvc/data-etcd-2

   2. 创建 pod 以访问第一个 etcd 成员的数据：

      1. 输入以下命令来获取 etcd 镜像：

         $ ETCD_IMAGE=$(oc get -n ${CONTROL_PLANE_NAMESPACE} statefulset/etcd -o jsonpath='{ .spec.template.spec.containers[0].image }')

      2. 创建允许访问 etcd 数据的 pod：

         $ cat << EOF | oc apply -n ${CONTROL_PLANE_NAMESPACE} -f -
         apiVersion: apps/v1
         kind: Deployment
         metadata:
           name: etcd-data
         spec:
           replicas: 1
           selector:
             matchLabels:
               app: etcd-data
           template:
             metadata:
               labels:
                 app: etcd-data
             spec:
               containers:
               - name: access
                 image: $ETCD_IMAGE
                 volumeMounts:
                 - name: data
                   mountPath: /var/lib
                 command:
                 - /usr/bin/bash
                 args:
                 - -c
                 - |-
                   while true; do
                     sleep 1000
                   done
               volumes:
               - name: data
                 persistentVolumeClaim:
                   claimName: data-etcd-0
         EOF

      3. 输入以下命令检查 etcd-data pod 的状态并等待它正在运行：

         $ oc get -n ${CONTROL_PLANE_NAMESPACE} pods -l app=etcd-data

      4. 输入以下命令来获取 etcd-data pod 的名称：

         $ DATA_POD=$(oc get -n ${CONTROL_PLANE_NAMESPACE} pods --no-headers -l app=etcd-data -o name | cut -d/ -f2)

   3. 输入以下命令将 etcd 快照复制到 pod 中：

      $ oc cp /tmp/etcd.snapshot.db ${CONTROL_PLANE_NAMESPACE}/${DATA_POD}:/var/lib/restored.snap.db

   4. 输入以下命令从 etcd-data pod 中删除旧数据：

      $ oc exec -n ${CONTROL_PLANE_NAMESPACE} ${DATA_POD} -- rm -rf /var/lib/data

      $ oc exec -n ${CONTROL_PLANE_NAMESPACE} ${DATA_POD} -- mkdir -p /var/lib/data

   5. 输入以下命令恢复 etcd 快照：

      $ oc exec -n ${CONTROL_PLANE_NAMESPACE} ${DATA_POD} -- etcdutl snapshot restore /var/lib/restored.snap.db \
           --data-dir=/var/lib/data --skip-hash-check \
           --name etcd-0 \
           --initial-cluster-token=etcd-cluster \
           --initial-cluster etcd-0=https://etcd-0.etcd-discovery.${CONTROL_PLANE_NAMESPACE}.svc:2380,etcd-1=https://etcd-1.etcd-discovery.${CONTROL_PLANE_NAMESPACE}.svc:2380,etcd-2=https://etcd-2.etcd-discovery.${CONTROL_PLANE_NAMESPACE}.svc:2380 \
           --initial-advertise-peer-urls https://etcd-0.etcd-discovery.${CONTROL_PLANE_NAMESPACE}.svc:2380

   6. 输入以下命令从 pod 中删除临时 etcd 快照：

      $ oc exec -n ${CONTROL_PLANE_NAMESPACE} ${DATA_POD} -- rm /var/lib/restored.snap.db

   7. 输入以下命令删除数据访问部署：

      $ oc delete -n ${CONTROL_PLANE_NAMESPACE} deployment/etcd-data

   8. 输入以下命令扩展 etcd 集群：

      $ oc scale -n ${CONTROL_PLANE_NAMESPACE} statefulset/etcd --replicas=3

   9. 输入以下命令等待 etcd 成员 pod 返回并报告 available：

      $ oc get -n ${CONTROL_PLANE_NAMESPACE} pods -l app=etcd -w

   10. 输入以下命令扩展所有 etcd-writer 部署：

       $ oc scale deployment -n ${CONTROL_PLANE_NAMESPACE} --replicas=3 kube-apiserver openshift-apiserver openshift-oauth-apiserver

4. 输入以下命令恢复托管集群的协调：

   $ oc patch -n ${CLUSTER_NAMESPACE} hostedclusters/${CLUSTER_NAME} -p '{"spec":{"pausedUntil":""}}' --type=merge

流程

1. 输入以下命令暂停托管集群的协调：

   $ oc patch -n clusters hostedclusters/<hosted_cluster_name> -p '{"spec":{"pausedUntil":"true"}}' --type=merge

2. 输入以下命令停止所有 etcd-writer 部署：

   $ oc scale deployment -n <hosted_cluster_namespace> --replicas=0 kube-apiserver openshift-apiserver openshift-oauth-apiserver

3. 要进行 etcd 快照，请输入以下命令在每个 etcd 容器中使用 exec 命令：

   $ oc exec -it <etcd_pod_name> -n <hosted_cluster_namespace> -- env ETCDCTL_API=3 /usr/bin/etcdctl --cacert /etc/etcd/tls/client/etcd-client-ca.crt --cert /etc/etcd/tls/client/etcd-client.crt --key /etc/etcd/tls/client/etcd-client.key --endpoints=localhost:2379 snapshot save /var/lib/data/snapshot.db

4. 要检查快照状态，请运行以下命令在每个 etcd 容器中使用 exec 命令：

   $ oc exec -it <etcd_pod_name> -n <hosted_cluster_namespace> -- env ETCDCTL_API=3 /usr/bin/etcdctl -w table snapshot status /var/lib/data/snapshot.db

5. 将快照数据复制到稍后检索它的位置，如 S3 存储桶。请参见以下示例。

   注意

   以下示例使用签名版本 2。如果您位于支持签名版本 4 的区域，如 us-east-2 区域，请使用签名版本 4。否则，当将快照复制到 S3 存储桶时，上传会失败。

   Example

   BUCKET_NAME=somebucket
   FILEPATH="/${BUCKET_NAME}/${CLUSTER_NAME}-snapshot.db"
   CONTENT_TYPE="application/x-compressed-tar"
   DATE_VALUE=`date -R`
   SIGNATURE_STRING="PUT\n\n${CONTENT_TYPE}\n${DATE_VALUE}\n${FILEPATH}"
   ACCESS_KEY=accesskey
   SECRET_KEY=secret
   SIGNATURE_HASH=`echo -en ${SIGNATURE_STRING} | openssl sha1 -hmac ${SECRET_KEY} -binary | base64`
   
   oc exec -it etcd-0 -n ${HOSTED_CLUSTER_NAMESPACE} -- curl -X PUT -T "/var/lib/data/snapshot.db" \
     -H "Host: ${BUCKET_NAME}.s3.amazonaws.com" \
     -H "Date: ${DATE_VALUE}" \
     -H "Content-Type: ${CONTENT_TYPE}" \
     -H "Authorization: AWS ${ACCESS_KEY}:${SIGNATURE_HASH}" \
     https://${BUCKET_NAME}.s3.amazonaws.com/${CLUSTER_NAME}-snapshot.db

6. 要稍后在新集群中恢复快照，请保存托管集群引用的加密 secret。

   1. 输入以下命令来获取 secret 加密密钥：

      $ oc get hostedcluster <hosted_cluster_name> -o=jsonpath='{.spec.secretEncryption.aescbc}'
      {"activeKey":{"name":"<hosted_cluster_name>-etcd-encryption-key"}}

   2. 输入以下命令保存 secret 加密密钥：

      $ oc get secret <hosted_cluster_name>-etcd-encryption-key -o=jsonpath='{.data.key}'

      您可以在新集群中恢复快照时解密此密钥。

流程

1. 在 aws 命令行界面 (CLI) 中，创建一个预签名的 URL，以便您可以从 S3 下载 etcd 快照，而无需将凭证传递给 etcd 部署：

   ETCD_SNAPSHOT=${ETCD_SNAPSHOT:-"s3://${BUCKET_NAME}/${CLUSTER_NAME}-snapshot.db"}
   ETCD_SNAPSHOT_URL=$(aws s3 presign ${ETCD_SNAPSHOT})

2. 修改 HostedCluster 规格以引用 URL：

   spec:
     etcd:
       managed:
         storage:
           persistentVolume:
             size: 4Gi
           type: PersistentVolume
           restoreSnapshotURL:
           - "${ETCD_SNAPSHOT_URL}"
       managementType: Managed

3. 确保从 spec.secretEncryption.aescbc 值引用的 secret 包含您在前面的步骤中保存的相同 AES 密钥。

1. 在管理集群 1 中，您可以将其视为源管理集群，control plane 和 worker 使用 ExternalDNS API 进行交互。可以访问外部 DNS API，并且一个负载均衡器位于管理集群之间。

   

2. 您为托管集群执行快照，其中包括 etcd、control plane 和 worker 节点。在此过程中，worker 节点仍然会尝试访问外部 DNS API，即使无法访问它，工作负载正在运行，control plane 存储在本地清单文件中，etcd 会备份到 S3 存储桶。data plane 处于活跃状态，control plane 已暂停。

   

3. 在管理集群 2 中，您可以将其视为目标管理集群，您可以从 S3 存储桶中恢复 etcd，并从本地清单文件恢复 control plane。在此过程中，外部 DNS API 会停止，托管集群 API 变得不可访问，任何使用 API 的 worker 都无法更新其清单文件，但工作负载仍在运行。

   

4. 外部 DNS API 可以再次访问，worker 节点使用它来移至管理集群 2。外部 DNS API 可以访问指向 control plane 的负载均衡器。

   

5. 在管理集群 2 中，control plane 和 worker 节点使用外部 DNS API 进行交互。资源从管理集群 1 中删除，但 etcd 的 S3 备份除外。如果您尝试在 mangagement 集群 1 上再次设置托管集群，它将无法正常工作。

   

流程

1. 输入以下命令创建 configmap 文件来声明源管理集群：

   $ oc create configmap mgmt-parent-cluster -n default --from-literal=from=${MGMT_CLUSTER_NAME}

2. 输入这些命令，在托管集群和节点池中关闭协调：

   PAUSED_UNTIL="true"
   oc patch -n ${HC_CLUSTER_NS} hostedclusters/${HC_CLUSTER_NAME} -p '{"spec":{"pausedUntil":"'${PAUSED_UNTIL}'"}}' --type=merge
   oc scale deployment -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} --replicas=0 kube-apiserver openshift-apiserver openshift-oauth-apiserver control-plane-operator

   PAUSED_UNTIL="true"
   oc patch -n ${HC_CLUSTER_NS} hostedclusters/${HC_CLUSTER_NAME} -p '{"spec":{"pausedUntil":"'${PAUSED_UNTIL}'"}}' --type=merge
   oc patch -n ${HC_CLUSTER_NS} nodepools/${NODEPOOLS} -p '{"spec":{"pausedUntil":"'${PAUSED_UNTIL}'"}}' --type=merge
   oc scale deployment -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} --replicas=0 kube-apiserver openshift-apiserver openshift-oauth-apiserver control-plane-operator

3. 运行此 bash 脚本备份 etcd 并将数据上传到 S3 存储桶：

   提示

   将此脚本嵌套在函数中，并从主功能调用它。

   # ETCD Backup
   ETCD_PODS="etcd-0"
   if [ "${CONTROL_PLANE_AVAILABILITY_POLICY}" = "HighlyAvailable" ]; then
     ETCD_PODS="etcd-0 etcd-1 etcd-2"
   fi
   
   for POD in ${ETCD_PODS}; do
     # Create an etcd snapshot
     oc exec -it ${POD} -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} -- env ETCDCTL_API=3 /usr/bin/etcdctl --cacert /etc/etcd/tls/client/etcd-client-ca.crt --cert /etc/etcd/tls/client/etcd-client.crt --key /etc/etcd/tls/client/etcd-client.key --endpoints=localhost:2379 snapshot save /var/lib/data/snapshot.db
     oc exec -it ${POD} -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} -- env ETCDCTL_API=3 /usr/bin/etcdctl -w table snapshot status /var/lib/data/snapshot.db
   
     FILEPATH="/${BUCKET_NAME}/${HC_CLUSTER_NAME}-${POD}-snapshot.db"
     CONTENT_TYPE="application/x-compressed-tar"
     DATE_VALUE=`date -R`
     SIGNATURE_STRING="PUT\n\n${CONTENT_TYPE}\n${DATE_VALUE}\n${FILEPATH}"
   
     set +x
     ACCESS_KEY=$(grep aws_access_key_id ${AWS_CREDS} | head -n1 | cut -d= -f2 | sed "s/ //g")
     SECRET_KEY=$(grep aws_secret_access_key ${AWS_CREDS} | head -n1 | cut -d= -f2 | sed "s/ //g")
     SIGNATURE_HASH=$(echo -en ${SIGNATURE_STRING} | openssl sha1 -hmac "${SECRET_KEY}" -binary | base64)
     set -x
   
     # FIXME: this is pushing to the OIDC bucket
     oc exec -it etcd-0 -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} -- curl -X PUT -T "/var/lib/data/snapshot.db" \
       -H "Host: ${BUCKET_NAME}.s3.amazonaws.com" \
       -H "Date: ${DATE_VALUE}" \
       -H "Content-Type: ${CONTENT_TYPE}" \
       -H "Authorization: AWS ${ACCESS_KEY}:${SIGNATURE_HASH}" \
       https://${BUCKET_NAME}.s3.amazonaws.com/${HC_CLUSTER_NAME}-${POD}-snapshot.db
   done

   有关备份 etcd 的更多信息，请参阅 "Backing and restore etcd on a hosted cluster"。

4. 输入以下命令备份 Kubernetes 和 OpenShift Container Platform 对象。您需要备份以下对象：

   • 来自 HostedCluster 命名空间的 HostedCluster 和 NodePool 对象
   • 来自 HostedCluster 命名空间中的 HostedCluster secret
   • 来自 Hosted Control Plane 命名空间中的 HostedControlPlane
   • 来自 Hosted Control Plane 命名空间的 Cluster
   • 来自 Hosted Control Plane 命名空间的AWSCluster, AWSMachineTemplate, 和 AWSMachine
   • 来自 Hosted Control Plane 命名空间的 MachineDeployments, MachineSets, 和 Machines。

   • 来自 Hosted Control Plane 命名空间的 ControlPlane

     mkdir -p ${BACKUP_DIR}/namespaces/${HC_CLUSTER_NS} ${BACKUP_DIR}/namespaces/${HC_CLUSTER_NS}-${HC_CLUSTER_NAME}
     chmod 700 ${BACKUP_DIR}/namespaces/
     
     # HostedCluster
     echo "Backing Up HostedCluster Objects:"
     oc get hc ${HC_CLUSTER_NAME} -n ${HC_CLUSTER_NS} -o yaml > ${BACKUP_DIR}/namespaces/${HC_CLUSTER_NS}/hc-${HC_CLUSTER_NAME}.yaml
     echo "--> HostedCluster"
     sed -i '' -e '/^status:$/,$d' ${BACKUP_DIR}/namespaces/${HC_CLUSTER_NS}/hc-${HC_CLUSTER_NAME}.yaml
     
     # NodePool
     oc get np ${NODEPOOLS} -n ${HC_CLUSTER_NS} -o yaml > ${BACKUP_DIR}/namespaces/${HC_CLUSTER_NS}/np-${NODEPOOLS}.yaml
     echo "--> NodePool"
     sed -i '' -e '/^status:$/,$ d' ${BACKUP_DIR}/namespaces/${HC_CLUSTER_NS}/np-${NODEPOOLS}.yaml
     
     # Secrets in the HC Namespace
     echo "--> HostedCluster Secrets:"
     for s in $(oc get secret -n ${HC_CLUSTER_NS} | grep "^${HC_CLUSTER_NAME}" | awk '{print $1}'); do
         oc get secret -n ${HC_CLUSTER_NS} $s -o yaml > ${BACKUP_DIR}/namespaces/${HC_CLUSTER_NS}/secret-${s}.yaml
     done
     
     # Secrets in the HC Control Plane Namespace
     echo "--> HostedCluster ControlPlane Secrets:"
     for s in $(oc get secret -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} | egrep -v "docker|service-account-token|oauth-openshift|NAME|token-${HC_CLUSTER_NAME}" | awk '{print $1}'); do
         oc get secret -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} $s -o yaml > ${BACKUP_DIR}/namespaces/${HC_CLUSTER_NS}-${HC_CLUSTER_NAME}/secret-${s}.yaml
     done
     
     # Hosted Control Plane
     echo "--> HostedControlPlane:"
     oc get hcp ${HC_CLUSTER_NAME} -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} -o yaml > ${BACKUP_DIR}/namespaces/${HC_CLUSTER_NS}-${HC_CLUSTER_NAME}/hcp-${HC_CLUSTER_NAME}.yaml
     
     # Cluster
     echo "--> Cluster:"
     CL_NAME=$(oc get hcp ${HC_CLUSTER_NAME} -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} -o jsonpath={.metadata.labels.\*} | grep ${HC_CLUSTER_NAME})
     oc get cluster ${CL_NAME} -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} -o yaml > ${BACKUP_DIR}/namespaces/${HC_CLUSTER_NS}-${HC_CLUSTER_NAME}/cl-${HC_CLUSTER_NAME}.yaml
     
     # AWS Cluster
     echo "--> AWS Cluster:"
     oc get awscluster ${HC_CLUSTER_NAME} -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} -o yaml > ${BACKUP_DIR}/namespaces/${HC_CLUSTER_NS}-${HC_CLUSTER_NAME}/awscl-${HC_CLUSTER_NAME}.yaml
     
     # AWS MachineTemplate
     echo "--> AWS Machine Template:"
     oc get awsmachinetemplate ${NODEPOOLS} -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} -o yaml > ${BACKUP_DIR}/namespaces/${HC_CLUSTER_NS}-${HC_CLUSTER_NAME}/awsmt-${HC_CLUSTER_NAME}.yaml
     
     # AWS Machines
     echo "--> AWS Machine:"
     CL_NAME=$(oc get hcp ${HC_CLUSTER_NAME} -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} -o jsonpath={.metadata.labels.\*} | grep ${HC_CLUSTER_NAME})
     for s in $(oc get awsmachines -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} --no-headers | grep ${CL_NAME} | cut -f1 -d\ ); do
         oc get -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} awsmachines $s -o yaml > ${BACKUP_DIR}/namespaces/${HC_CLUSTER_NS}-${HC_CLUSTER_NAME}/awsm-${s}.yaml
     done
     
     # MachineDeployments
     echo "--> HostedCluster MachineDeployments:"
     for s in $(oc get machinedeployment -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} -o name); do
         mdp_name=$(echo ${s} | cut -f 2 -d /)
         oc get -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} $s -o yaml > ${BACKUP_DIR}/namespaces/${HC_CLUSTER_NS}-${HC_CLUSTER_NAME}/machinedeployment-${mdp_name}.yaml
     done
     
     # MachineSets
     echo "--> HostedCluster MachineSets:"
     for s in $(oc get machineset -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} -o name); do
         ms_name=$(echo ${s} | cut -f 2 -d /)
         oc get -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} $s -o yaml > ${BACKUP_DIR}/namespaces/${HC_CLUSTER_NS}-${HC_CLUSTER_NAME}/machineset-${ms_name}.yaml
     done
     
     # Machines
     echo "--> HostedCluster Machine:"
     for s in $(oc get machine -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} -o name); do
         m_name=$(echo ${s} | cut -f 2 -d /)
         oc get -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} $s -o yaml > ${BACKUP_DIR}/namespaces/${HC_CLUSTER_NS}-${HC_CLUSTER_NAME}/machine-${m_name}.yaml
     done

5. 输入以下命令清理 ControlPlane 路由：

   $ oc delete routes -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} --all

   输入该命令，您可以启用 ExternalDNS Operator 来删除 Route53 条目。

6. 运行该脚本验证 Route53 条目是否清理：

   function clean_routes() {
   
       if [[ -z "${1}" ]];then
           echo "Give me the NS where to clean the routes"
           exit 1
       fi
   
       # Constants
       if [[ -z "${2}" ]];then
           echo "Give me the Route53 zone ID"
           exit 1
       fi
   
       ZONE_ID=${2}
       ROUTES=10
       timeout=40
       count=0
   
       # This allows us to remove the ownership in the AWS for the API route
       oc delete route -n ${1} --all
   
       while [ ${ROUTES} -gt 2 ]
       do
           echo "Waiting for ExternalDNS Operator to clean the DNS Records in AWS Route53 where the zone id is: ${ZONE_ID}..."
           echo "Try: (${count}/${timeout})"
           sleep 10
           if [[ $count -eq timeout ]];then
               echo "Timeout waiting for cleaning the Route53 DNS records"
               exit 1
           fi
           count=$((count+1))
           ROUTES=$(aws route53 list-resource-record-sets --hosted-zone-id ${ZONE_ID} --max-items 10000 --output json | grep -c ${EXTERNAL_DNS_DOMAIN})
       done
   }
   
   # SAMPLE: clean_routes "<HC ControlPlane Namespace>" "<AWS_ZONE_ID>"
   clean_routes "${HC_CLUSTER_NS}-${HC_CLUSTER_NAME}" "${AWS_ZONE_ID}"

流程

1. 输入以下命令验证新管理集群是否不包含您要恢复的集群中的任何命名空间：

   # Just in case
   export KUBECONFIG=${MGMT2_KUBECONFIG}
   BACKUP_DIR=${HC_CLUSTER_DIR}/backup
   
   # Namespace deletion in the destination Management cluster
   $ oc delete ns ${HC_CLUSTER_NS} || true
   $ oc delete ns ${HC_CLUSTER_NS}-{HC_CLUSTER_NAME} || true

2. 输入以下命令重新创建已删除的命名空间：

   # Namespace creation
   $ oc new-project ${HC_CLUSTER_NS}
   $ oc new-project ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME}

3. 输入以下命令恢复 HC 命名空间中的 secret：

   $ oc apply -f ${BACKUP_DIR}/namespaces/${HC_CLUSTER_NS}/secret-*

4. 输入以下命令恢复 HostedCluster control plane 命名空间中的对象：

   # Secrets
   $ oc apply -f ${BACKUP_DIR}/namespaces/${HC_CLUSTER_NS}-${HC_CLUSTER_NAME}/secret-*
   
   # Cluster
   $ oc apply -f ${BACKUP_DIR}/namespaces/${HC_CLUSTER_NS}-${HC_CLUSTER_NAME}/hcp-*
   $ oc apply -f ${BACKUP_DIR}/namespaces/${HC_CLUSTER_NS}-${HC_CLUSTER_NAME}/cl-*

5. 如果您要恢复节点和节点池来重复利用 AWS 实例，请输入以下命令恢复 HC control plane 命名空间中的对象：

   # AWS
   $ oc apply -f ${BACKUP_DIR}/namespaces/${HC_CLUSTER_NS}-${HC_CLUSTER_NAME}/awscl-*
   $ oc apply -f ${BACKUP_DIR}/namespaces/${HC_CLUSTER_NS}-${HC_CLUSTER_NAME}/awsmt-*
   $ oc apply -f ${BACKUP_DIR}/namespaces/${HC_CLUSTER_NS}-${HC_CLUSTER_NAME}/awsm-*
   
   # Machines
   $ oc apply -f ${BACKUP_DIR}/namespaces/${HC_CLUSTER_NS}-${HC_CLUSTER_NAME}/machinedeployment-*
   $ oc apply -f ${BACKUP_DIR}/namespaces/${HC_CLUSTER_NS}-${HC_CLUSTER_NAME}/machineset-*
   $ oc apply -f ${BACKUP_DIR}/namespaces/${HC_CLUSTER_NS}-${HC_CLUSTER_NAME}/machine-*

6. 运行此 bash 脚本恢复 etcd 数据和托管集群：

   ETCD_PODS="etcd-0"
   if [ "${CONTROL_PLANE_AVAILABILITY_POLICY}" = "HighlyAvailable" ]; then
     ETCD_PODS="etcd-0 etcd-1 etcd-2"
   fi
   
   HC_RESTORE_FILE=${BACKUP_DIR}/namespaces/${HC_CLUSTER_NS}/hc-${HC_CLUSTER_NAME}-restore.yaml
   HC_BACKUP_FILE=${BACKUP_DIR}/namespaces/${HC_CLUSTER_NS}/hc-${HC_CLUSTER_NAME}.yaml
   HC_NEW_FILE=${BACKUP_DIR}/namespaces/${HC_CLUSTER_NS}/hc-${HC_CLUSTER_NAME}-new.yaml
   cat ${HC_BACKUP_FILE} > ${HC_NEW_FILE}
   cat > ${HC_RESTORE_FILE} <<EOF
       restoreSnapshotURL:
   EOF
   
   for POD in ${ETCD_PODS}; do
     # Create a pre-signed URL for the etcd snapshot
     ETCD_SNAPSHOT="s3://${BUCKET_NAME}/${HC_CLUSTER_NAME}-${POD}-snapshot.db"
     ETCD_SNAPSHOT_URL=$(AWS_DEFAULT_REGION=${MGMT2_REGION} aws s3 presign ${ETCD_SNAPSHOT})
   
     # FIXME no CLI support for restoreSnapshotURL yet
     cat >> ${HC_RESTORE_FILE} <<EOF
       - "${ETCD_SNAPSHOT_URL}"
   EOF
   done
   
   cat ${HC_RESTORE_FILE}
   
   if ! grep ${HC_CLUSTER_NAME}-snapshot.db ${HC_NEW_FILE}; then
     sed -i '' -e "/type: PersistentVolume/r ${HC_RESTORE_FILE}" ${HC_NEW_FILE}
     sed -i '' -e '/pausedUntil:/d' ${HC_NEW_FILE}
   fi
   
   HC=$(oc get hc -n ${HC_CLUSTER_NS} ${HC_CLUSTER_NAME} -o name || true)
   if [[ ${HC} == "" ]];then
       echo "Deploying HC Cluster: ${HC_CLUSTER_NAME} in ${HC_CLUSTER_NS} namespace"
       oc apply -f ${HC_NEW_FILE}
   else
       echo "HC Cluster ${HC_CLUSTER_NAME} already exists, avoiding step"
   fi

7. 如果您要恢复节点和节点池来重复利用 AWS 实例，请输入以下命令恢复节点池：

   oc apply -f ${BACKUP_DIR}/namespaces/${HC_CLUSTER_NS}/np-*

流程

1. 输入以下命令来扩展 deployment 和 statefulset 对象：

   重要

   如果其 spec.persistentVolumeClaimRetentionPolicy.whenScaled 字段被设置为 Delete，则不要扩展有状态的集合，因为这可能会导致数据丢失。

   作为临时解决方案，将 spec.persistentVolumeClaimRetentionPolicy.whenScaled 字段的值更新为 Retain。确保不存在协调有状态集的控制器，并将值返回为 Delete，这可能会导致丢失数据。

   # Just in case
   export KUBECONFIG=${MGMT_KUBECONFIG}
   
   # Scale down deployments
   oc scale deployment -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} --replicas=0 --all
   oc scale statefulset.apps -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} --replicas=0 --all
   sleep 15

2. 输入以下命令来删除 NodePool 对象：

   NODEPOOLS=$(oc get nodepools -n ${HC_CLUSTER_NS} -o=jsonpath='{.items[?(@.spec.clusterName=="'${HC_CLUSTER_NAME}'")].metadata.name}')
   if [[ ! -z "${NODEPOOLS}" ]];then
       oc patch -n "${HC_CLUSTER_NS}" nodepool ${NODEPOOLS} --type=json --patch='[ { "op":"remove", "path": "/metadata/finalizers" }]'
       oc delete np -n ${HC_CLUSTER_NS} ${NODEPOOLS}
   fi

3. 输入以下命令删除 machine 和 machineset 对象：

   # Machines
   for m in $(oc get machines -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} -o name); do
       oc patch -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} ${m} --type=json --patch='[ { "op":"remove", "path": "/metadata/finalizers" }]' || true
       oc delete -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} ${m} || true
   done
   
   oc delete machineset -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} --all || true

4. 输入以下命令删除集群对象：

   # Cluster
   C_NAME=$(oc get cluster -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} -o name)
   oc patch -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} ${C_NAME} --type=json --patch='[ { "op":"remove", "path": "/metadata/finalizers" }]'
   oc delete cluster.cluster.x-k8s.io -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} --all

5. 输入这些命令来删除 AWS 机器 (Kubernetes 对象)。不用担心删除实际的 AWS 机器。云实例不会受到影响。

   # AWS Machines
   for m in $(oc get awsmachine.infrastructure.cluster.x-k8s.io -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} -o name)
   do
       oc patch -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} ${m} --type=json --patch='[ { "op":"remove", "path": "/metadata/finalizers" }]' || true
       oc delete -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} ${m} || true
   done

6. 输入以下命令删除 HostedControlPlane 和 ControlPlane HC 命名空间对象：

   # Delete HCP and ControlPlane HC NS
   oc patch -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} hostedcontrolplane.hypershift.openshift.io ${HC_CLUSTER_NAME} --type=json --patch='[ { "op":"remove", "path": "/metadata/finalizers" }]'
   oc delete hostedcontrolplane.hypershift.openshift.io -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} --all
   oc delete ns ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} || true

7. 输入以下命令删除 HostedCluster 和 HC 命名空间对象：

   # Delete HC and HC Namespace
   oc -n ${HC_CLUSTER_NS} patch hostedclusters ${HC_CLUSTER_NAME} -p '{"metadata":{"finalizers":null}}' --type merge || true
   oc delete hc -n ${HC_CLUSTER_NS} ${HC_CLUSTER_NAME}  || true
   oc delete ns ${HC_CLUSTER_NS} || true

1. 使用托管的集群的 kubeconfig 路径加载 KUBECONFIG 环境变量。

2. 输入这个命令：

   $ oc delete pod -n openshift-ovn-kubernetes --all