4.2. AWS 区域托管的集群的灾难恢复


对于托管集群需要灾难恢复 (DR) 的情况,您可以将托管集群恢复到 AWS 中的同一区域。例如,当升级管理集群时,需要 DR,托管的集群处于只读状态。

重要

托管的 control plane 只是一个技术预览功能。技术预览功能不受红帽产品服务等级协议(SLA)支持,且功能可能并不完整。红帽不推荐在生产环境中使用它们。这些技术预览功能可以使用户提早试用新的功能,并有机会在开发阶段提供反馈意见。

有关红帽技术预览功能支持范围的更多信息,请参阅技术预览功能支持范围

DR 进程涉及三个主要步骤:

  1. 在源集群中备份托管集群
  2. 在目标管理集群中恢复托管集群
  3. 从源管理集群中删除托管的集群

您的工作负载在此过程中保持运行。集群 API 可能会在一段时间内不可用,但不会影响 worker 节点上运行的服务。

重要

源管理集群和目标管理集群必须具有 --external-dns 标志才能维护 API 服务器 URL,如下例所示:

示例:外部 DNS 标记

--external-dns-provider=aws \
--external-dns-credentials=<AWS Credentials location> \
--external-dns-domain-filter=<DNS Base Domain>

这样,服务器 URL 以 https://api-sample-hosted.sample-hosted.aws.openshift.com 结尾。

如果没有包含 --external-dns 标志来维护 API 服务器 URL,则无法迁移托管集群。

4.2.1. 环境和上下文示例

假设您有三个集群需要恢复的情况。两个是管理集群,一个是托管的集群。您只能恢复 control plane 或 control plane 和节点。开始前,您需要以下信息:

  • 源 MGMT 命名空间:源管理命名空间
  • Source MGMT ClusterName :源管理集群名称
  • 源 MGMT Kubeconfig :源管理 kubeconfig 文件
  • Destination MGMT Kubeconfig :目标管理 kubeconfig 文件
  • HC Kubeconfig :托管的集群 kubeconfig 文件
  • SSH 密钥文件:SSH 公钥
  • Pull secret:用于访问发行镜像的 pull secret 文件
  • AWS 凭证
  • AWS 区域
  • Base domain:用作外部 DNS 的 DNS 基域
  • S3 bucket name:计划上传 etcd 备份的 AWS 区域中的存储桶

此信息显示在以下示例环境变量中。

环境变量示例

SSH_KEY_FILE=${HOME}/.ssh/id_rsa.pub
BASE_PATH=${HOME}/hypershift
BASE_DOMAIN="aws.sample.com"
PULL_SECRET_FILE="${HOME}/pull_secret.json"
AWS_CREDS="${HOME}/.aws/credentials"
AWS_ZONE_ID="Z02718293M33QHDEQBROL"

CONTROL_PLANE_AVAILABILITY_POLICY=SingleReplica
HYPERSHIFT_PATH=${BASE_PATH}/src/hypershift
HYPERSHIFT_CLI=${HYPERSHIFT_PATH}/bin/hypershift
HYPERSHIFT_IMAGE=${HYPERSHIFT_IMAGE:-"quay.io/${USER}/hypershift:latest"}
NODE_POOL_REPLICAS=${NODE_POOL_REPLICAS:-2}

# MGMT Context
MGMT_REGION=us-west-1
MGMT_CLUSTER_NAME="${USER}-dev"
MGMT_CLUSTER_NS=${USER}
MGMT_CLUSTER_DIR="${BASE_PATH}/hosted_clusters/${MGMT_CLUSTER_NS}-${MGMT_CLUSTER_NAME}"
MGMT_KUBECONFIG="${MGMT_CLUSTER_DIR}/kubeconfig"

# MGMT2 Context
MGMT2_CLUSTER_NAME="${USER}-dest"
MGMT2_CLUSTER_NS=${USER}
MGMT2_CLUSTER_DIR="${BASE_PATH}/hosted_clusters/${MGMT2_CLUSTER_NS}-${MGMT2_CLUSTER_NAME}"
MGMT2_KUBECONFIG="${MGMT2_CLUSTER_DIR}/kubeconfig"

# Hosted Cluster Context
HC_CLUSTER_NS=clusters
HC_REGION=us-west-1
HC_CLUSTER_NAME="${USER}-hosted"
HC_CLUSTER_DIR="${BASE_PATH}/hosted_clusters/${HC_CLUSTER_NS}-${HC_CLUSTER_NAME}"
HC_KUBECONFIG="${HC_CLUSTER_DIR}/kubeconfig"
BACKUP_DIR=${HC_CLUSTER_DIR}/backup

BUCKET_NAME="${USER}-hosted-${MGMT_REGION}"

# DNS
AWS_ZONE_ID="Z07342811SH9AA102K1AC"
EXTERNAL_DNS_DOMAIN="hc.jpdv.aws.kerbeross.com"

4.2.2. 备份和恢复过程概述

备份和恢复过程按如下方式工作:

  1. 在管理集群 1 中,您可以将其视为源管理集群,control plane 和 worker 使用 ExternalDNS API 进行交互。可以访问外部 DNS API,并且一个负载均衡器位于管理集群之间。

    显示 worker 访问外部 DNS API 和通过负载均衡器指向 control plane 的外部 DNS API 图
  2. 您为托管集群执行快照,其中包括 etcd、control plane 和 worker 节点。在此过程中,worker 节点仍然会尝试访问外部 DNS API,即使无法访问它,工作负载正在运行,control plane 存储在本地清单文件中,etcd 会备份到 S3 存储桶。data plane 处于活跃状态,control plane 已暂停。

    298 OpenShift Backup Restore 0123 01
  3. 在管理集群 2 中,您可以将其视为目标管理集群,您可以从 S3 存储桶中恢复 etcd,并从本地清单文件恢复 control plane。在此过程中,外部 DNS API 会停止,托管集群 API 变得不可访问,任何使用 API 的 worker 都无法更新其清单文件,但工作负载仍在运行。

    298 OpenShift Backup Restore 0123 02
  4. 外部 DNS API 可以再次访问,worker 节点使用它来移至管理集群 2。外部 DNS API 可以访问指向 control plane 的负载均衡器。

    298 OpenShift Backup Restore 0123 03
  5. 在管理集群 2 中,control plane 和 worker 节点使用外部 DNS API 进行交互。资源从管理集群 1 中删除,但 etcd 的 S3 备份除外。如果您尝试在 mangagement 集群 1 上再次设置托管集群,它将无法正常工作。

    298 OpenShift Backup Restore 0123 04

您可以手动备份和恢复托管集群,也可以运行脚本来完成这个过程。有关脚本的更多信息,请参阅"运行脚本以备份和恢复托管集群"。

4.2.3. 备份托管集群

要在目标管理集群中恢复托管集群,首先需要备份所有相关数据。

流程

  1. 输入以下命令创建 configmap 文件来声明源管理集群:

    $ oc create configmap mgmt-parent-cluster -n default --from-literal=from=${MGMT_CLUSTER_NAME}
  2. 输入这些命令,在托管集群和节点池中关闭协调:

    $ PAUSED_UNTIL="true"
    $ oc patch -n ${HC_CLUSTER_NS} hostedclusters/${HC_CLUSTER_NAME} -p '{"spec":{"pausedUntil":"'${PAUSED_UNTIL}'"}}' --type=merge
    $ oc scale deployment -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} --replicas=0 kube-apiserver openshift-apiserver openshift-oauth-apiserver control-plane-operator
    $ PAUSED_UNTIL="true"
    $ oc patch -n ${HC_CLUSTER_NS} hostedclusters/${HC_CLUSTER_NAME} -p '{"spec":{"pausedUntil":"'${PAUSED_UNTIL}'"}}' --type=merge
    $ oc patch -n ${HC_CLUSTER_NS} nodepools/${NODEPOOLS} -p '{"spec":{"pausedUntil":"'${PAUSED_UNTIL}'"}}' --type=merge
    $ oc scale deployment -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} --replicas=0 kube-apiserver openshift-apiserver openshift-oauth-apiserver control-plane-operator
  3. 运行此 bash 脚本备份 etcd 并将数据上传到 S3 存储桶:

    提示

    将此脚本嵌套在函数中,并从主功能调用它。

    # ETCD Backup
    ETCD_PODS="etcd-0"
    if [ "${CONTROL_PLANE_AVAILABILITY_POLICY}" = "HighlyAvailable" ]; then
      ETCD_PODS="etcd-0 etcd-1 etcd-2"
    fi
    
    for POD in ${ETCD_PODS}; do
      # Create an etcd snapshot
      oc exec -it ${POD} -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} -- env ETCDCTL_API=3 /usr/bin/etcdctl --cacert /etc/etcd/tls/client/etcd-client-ca.crt --cert /etc/etcd/tls/client/etcd-client.crt --key /etc/etcd/tls/client/etcd-client.key --endpoints=localhost:2379 snapshot save /var/lib/data/snapshot.db
      oc exec -it ${POD} -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} -- env ETCDCTL_API=3 /usr/bin/etcdctl -w table snapshot status /var/lib/data/snapshot.db
    
      FILEPATH="/${BUCKET_NAME}/${HC_CLUSTER_NAME}-${POD}-snapshot.db"
      CONTENT_TYPE="application/x-compressed-tar"
      DATE_VALUE=`date -R`
      SIGNATURE_STRING="PUT\n\n${CONTENT_TYPE}\n${DATE_VALUE}\n${FILEPATH}"
    
      set +x
      ACCESS_KEY=$(grep aws_access_key_id ${AWS_CREDS} | head -n1 | cut -d= -f2 | sed "s/ //g")
      SECRET_KEY=$(grep aws_secret_access_key ${AWS_CREDS} | head -n1 | cut -d= -f2 | sed "s/ //g")
      SIGNATURE_HASH=$(echo -en ${SIGNATURE_STRING} | openssl sha1 -hmac "${SECRET_KEY}" -binary | base64)
      set -x
    
      # FIXME: this is pushing to the OIDC bucket
      oc exec -it etcd-0 -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} -- curl -X PUT -T "/var/lib/data/snapshot.db" \
        -H "Host: ${BUCKET_NAME}.s3.amazonaws.com" \
        -H "Date: ${DATE_VALUE}" \
        -H "Content-Type: ${CONTENT_TYPE}" \
        -H "Authorization: AWS ${ACCESS_KEY}:${SIGNATURE_HASH}" \
        https://${BUCKET_NAME}.s3.amazonaws.com/${HC_CLUSTER_NAME}-${POD}-snapshot.db
    done

    有关备份 etcd 的更多信息,请参阅 "Backing and restore etcd on a hosted cluster"。

  4. 输入以下命令备份 Kubernetes 和 OpenShift Container Platform 对象。您需要备份以下对象:

    • 来自 HostedCluster 命名空间的 HostedClusterNodePool 对象
    • 来自 HostedCluster 命名空间中的 HostedCluster secret
    • 来自 Hosted Control Plane 命名空间中的 HostedControlPlane
    • 来自 Hosted Control Plane 命名空间的 Cluster
    • 来自 Hosted Control Plane 命名空间的AWSCluster, AWSMachineTemplate, 和 AWSMachine
    • 来自 Hosted Control Plane 命名空间的 MachineDeployments, MachineSets, 和 Machines
    • 来自 Hosted Control Plane 命名空间的 ControlPlane

      $ mkdir -p ${BACKUP_DIR}/namespaces/${HC_CLUSTER_NS} ${BACKUP_DIR}/namespaces/${HC_CLUSTER_NS}-${HC_CLUSTER_NAME}
      $ chmod 700 ${BACKUP_DIR}/namespaces/
      
      # HostedCluster
      $ echo "Backing Up HostedCluster Objects:"
      $ oc get hc ${HC_CLUSTER_NAME} -n ${HC_CLUSTER_NS} -o yaml > ${BACKUP_DIR}/namespaces/${HC_CLUSTER_NS}/hc-${HC_CLUSTER_NAME}.yaml
      $ echo "--> HostedCluster"
      $ sed -i '' -e '/^status:$/,$d' ${BACKUP_DIR}/namespaces/${HC_CLUSTER_NS}/hc-${HC_CLUSTER_NAME}.yaml
      
      # NodePool
      $ oc get np ${NODEPOOLS} -n ${HC_CLUSTER_NS} -o yaml > ${BACKUP_DIR}/namespaces/${HC_CLUSTER_NS}/np-${NODEPOOLS}.yaml
      $ echo "--> NodePool"
      $ sed -i '' -e '/^status:$/,$ d' ${BACKUP_DIR}/namespaces/${HC_CLUSTER_NS}/np-${NODEPOOLS}.yaml
      
      # Secrets in the HC Namespace
      $ echo "--> HostedCluster Secrets:"
      for s in $(oc get secret -n ${HC_CLUSTER_NS} | grep "^${HC_CLUSTER_NAME}" | awk '{print $1}'); do
          oc get secret -n ${HC_CLUSTER_NS} $s -o yaml > ${BACKUP_DIR}/namespaces/${HC_CLUSTER_NS}/secret-${s}.yaml
      done
      
      # Secrets in the HC Control Plane Namespace
      $ echo "--> HostedCluster ControlPlane Secrets:"
      for s in $(oc get secret -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} | egrep -v "docker|service-account-token|oauth-openshift|NAME|token-${HC_CLUSTER_NAME}" | awk '{print $1}'); do
          oc get secret -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} $s -o yaml > ${BACKUP_DIR}/namespaces/${HC_CLUSTER_NS}-${HC_CLUSTER_NAME}/secret-${s}.yaml
      done
      
      # Hosted Control Plane
      $ echo "--> HostedControlPlane:"
      $ oc get hcp ${HC_CLUSTER_NAME} -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} -o yaml > ${BACKUP_DIR}/namespaces/${HC_CLUSTER_NS}-${HC_CLUSTER_NAME}/hcp-${HC_CLUSTER_NAME}.yaml
      
      # Cluster
      $ echo "--> Cluster:"
      $ CL_NAME=$(oc get hcp ${HC_CLUSTER_NAME} -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} -o jsonpath={.metadata.labels.\*} | grep ${HC_CLUSTER_NAME})
      $ oc get cluster ${CL_NAME} -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} -o yaml > ${BACKUP_DIR}/namespaces/${HC_CLUSTER_NS}-${HC_CLUSTER_NAME}/cl-${HC_CLUSTER_NAME}.yaml
      
      # AWS Cluster
      $ echo "--> AWS Cluster:"
      $ oc get awscluster ${HC_CLUSTER_NAME} -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} -o yaml > ${BACKUP_DIR}/namespaces/${HC_CLUSTER_NS}-${HC_CLUSTER_NAME}/awscl-${HC_CLUSTER_NAME}.yaml
      
      # AWS MachineTemplate
      $ echo "--> AWS Machine Template:"
      $ oc get awsmachinetemplate ${NODEPOOLS} -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} -o yaml > ${BACKUP_DIR}/namespaces/${HC_CLUSTER_NS}-${HC_CLUSTER_NAME}/awsmt-${HC_CLUSTER_NAME}.yaml
      
      # AWS Machines
      $ echo "--> AWS Machine:"
      $ CL_NAME=$(oc get hcp ${HC_CLUSTER_NAME} -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} -o jsonpath={.metadata.labels.\*} | grep ${HC_CLUSTER_NAME})
      for s in $(oc get awsmachines -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} --no-headers | grep ${CL_NAME} | cut -f1 -d\ ); do
          oc get -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} awsmachines $s -o yaml > ${BACKUP_DIR}/namespaces/${HC_CLUSTER_NS}-${HC_CLUSTER_NAME}/awsm-${s}.yaml
      done
      
      # MachineDeployments
      $ echo "--> HostedCluster MachineDeployments:"
      for s in $(oc get machinedeployment -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} -o name); do
          mdp_name=$(echo ${s} | cut -f 2 -d /)
          oc get -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} $s -o yaml > ${BACKUP_DIR}/namespaces/${HC_CLUSTER_NS}-${HC_CLUSTER_NAME}/machinedeployment-${mdp_name}.yaml
      done
      
      # MachineSets
      $ echo "--> HostedCluster MachineSets:"
      for s in $(oc get machineset -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} -o name); do
          ms_name=$(echo ${s} | cut -f 2 -d /)
          oc get -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} $s -o yaml > ${BACKUP_DIR}/namespaces/${HC_CLUSTER_NS}-${HC_CLUSTER_NAME}/machineset-${ms_name}.yaml
      done
      
      # Machines
      $ echo "--> HostedCluster Machine:"
      for s in $(oc get machine -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} -o name); do
          m_name=$(echo ${s} | cut -f 2 -d /)
          oc get -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} $s -o yaml > ${BACKUP_DIR}/namespaces/${HC_CLUSTER_NS}-${HC_CLUSTER_NAME}/machine-${m_name}.yaml
      done
  5. 输入以下命令清理 ControlPlane 路由:

    $ oc delete routes -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} --all

    输入该命令,您可以启用 ExternalDNS Operator 来删除 Route53 条目。

  6. 运行该脚本验证 Route53 条目是否清理:

    function clean_routes() {
    
        if [[ -z "${1}" ]];then
            echo "Give me the NS where to clean the routes"
            exit 1
        fi
    
        # Constants
        if [[ -z "${2}" ]];then
            echo "Give me the Route53 zone ID"
            exit 1
        fi
    
        ZONE_ID=${2}
        ROUTES=10
        timeout=40
        count=0
    
        # This allows us to remove the ownership in the AWS for the API route
        oc delete route -n ${1} --all
    
        while [ ${ROUTES} -gt 2 ]
        do
            echo "Waiting for ExternalDNS Operator to clean the DNS Records in AWS Route53 where the zone id is: ${ZONE_ID}..."
            echo "Try: (${count}/${timeout})"
            sleep 10
            if [[ $count -eq timeout ]];then
                echo "Timeout waiting for cleaning the Route53 DNS records"
                exit 1
            fi
            count=$((count+1))
            ROUTES=$(aws route53 list-resource-record-sets --hosted-zone-id ${ZONE_ID} --max-items 10000 --output json | grep -c ${EXTERNAL_DNS_DOMAIN})
        done
    }
    
    # SAMPLE: clean_routes "<HC ControlPlane Namespace>" "<AWS_ZONE_ID>"
    clean_routes "${HC_CLUSTER_NS}-${HC_CLUSTER_NAME}" "${AWS_ZONE_ID}"

验证

检查所有 OpenShift Container Platform 对象和 S3 存储桶,以验证所有内容是否如预期。

后续步骤

恢复托管集群。

4.2.4. 恢复托管集群

收集您备份和恢复目标管理集群中的所有对象。

先决条件

您已从源集群备份数据。

提示

确保目标管理集群的 kubeconfig 文件放置在 KUBECONFIG 变量中,或者在 MGMT2_KUBECONFIG 变量中设置。使用 export KUBECONFIG=<Kubeconfig FilePath>,或者使用 export KUBECONFIG=${MGMT2_KUBECONFIG}

流程

  1. 输入以下命令验证新管理集群是否不包含您要恢复的集群中的任何命名空间:

    # Just in case
    $ export KUBECONFIG=${MGMT2_KUBECONFIG}
    $ BACKUP_DIR=${HC_CLUSTER_DIR}/backup
    
    # Namespace deletion in the destination Management cluster
    $ oc delete ns ${HC_CLUSTER_NS} || true
    $ oc delete ns ${HC_CLUSTER_NS}-{HC_CLUSTER_NAME} || true
  2. 输入以下命令重新创建已删除的命名空间:

    # Namespace creation
    $ oc new-project ${HC_CLUSTER_NS}
    $ oc new-project ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME}
  3. 输入以下命令恢复 HC 命名空间中的 secret:

    $ oc apply -f ${BACKUP_DIR}/namespaces/${HC_CLUSTER_NS}/secret-*
  4. 输入以下命令恢复 HostedCluster control plane 命名空间中的对象:

    # Secrets
    $ oc apply -f ${BACKUP_DIR}/namespaces/${HC_CLUSTER_NS}-${HC_CLUSTER_NAME}/secret-*
    
    # Cluster
    $ oc apply -f ${BACKUP_DIR}/namespaces/${HC_CLUSTER_NS}-${HC_CLUSTER_NAME}/hcp-*
    $ oc apply -f ${BACKUP_DIR}/namespaces/${HC_CLUSTER_NS}-${HC_CLUSTER_NAME}/cl-*
  5. 如果您要恢复节点和节点池来重复利用 AWS 实例,请输入以下命令恢复 HC control plane 命名空间中的对象:

    # AWS
    $ oc apply -f ${BACKUP_DIR}/namespaces/${HC_CLUSTER_NS}-${HC_CLUSTER_NAME}/awscl-*
    $ oc apply -f ${BACKUP_DIR}/namespaces/${HC_CLUSTER_NS}-${HC_CLUSTER_NAME}/awsmt-*
    $ oc apply -f ${BACKUP_DIR}/namespaces/${HC_CLUSTER_NS}-${HC_CLUSTER_NAME}/awsm-*
    
    # Machines
    $ oc apply -f ${BACKUP_DIR}/namespaces/${HC_CLUSTER_NS}-${HC_CLUSTER_NAME}/machinedeployment-*
    $ oc apply -f ${BACKUP_DIR}/namespaces/${HC_CLUSTER_NS}-${HC_CLUSTER_NAME}/machineset-*
    $ oc apply -f ${BACKUP_DIR}/namespaces/${HC_CLUSTER_NS}-${HC_CLUSTER_NAME}/machine-*
  6. 运行此 bash 脚本恢复 etcd 数据和托管集群:

    ETCD_PODS="etcd-0"
    if [ "${CONTROL_PLANE_AVAILABILITY_POLICY}" = "HighlyAvailable" ]; then
      ETCD_PODS="etcd-0 etcd-1 etcd-2"
    fi
    
    HC_RESTORE_FILE=${BACKUP_DIR}/namespaces/${HC_CLUSTER_NS}/hc-${HC_CLUSTER_NAME}-restore.yaml
    HC_BACKUP_FILE=${BACKUP_DIR}/namespaces/${HC_CLUSTER_NS}/hc-${HC_CLUSTER_NAME}.yaml
    HC_NEW_FILE=${BACKUP_DIR}/namespaces/${HC_CLUSTER_NS}/hc-${HC_CLUSTER_NAME}-new.yaml
    cat ${HC_BACKUP_FILE} > ${HC_NEW_FILE}
    cat > ${HC_RESTORE_FILE} <<EOF
        restoreSnapshotURL:
    EOF
    
    for POD in ${ETCD_PODS}; do
      # Create a pre-signed URL for the etcd snapshot
      ETCD_SNAPSHOT="s3://${BUCKET_NAME}/${HC_CLUSTER_NAME}-${POD}-snapshot.db"
      ETCD_SNAPSHOT_URL=$(AWS_DEFAULT_REGION=${MGMT2_REGION} aws s3 presign ${ETCD_SNAPSHOT})
    
      # FIXME no CLI support for restoreSnapshotURL yet
      cat >> ${HC_RESTORE_FILE} <<EOF
        - "${ETCD_SNAPSHOT_URL}"
    EOF
    done
    
    cat ${HC_RESTORE_FILE}
    
    if ! grep ${HC_CLUSTER_NAME}-snapshot.db ${HC_NEW_FILE}; then
      sed -i '' -e "/type: PersistentVolume/r ${HC_RESTORE_FILE}" ${HC_NEW_FILE}
      sed -i '' -e '/pausedUntil:/d' ${HC_NEW_FILE}
    fi
    
    HC=$(oc get hc -n ${HC_CLUSTER_NS} ${HC_CLUSTER_NAME} -o name || true)
    if [[ ${HC} == "" ]];then
        echo "Deploying HC Cluster: ${HC_CLUSTER_NAME} in ${HC_CLUSTER_NS} namespace"
        oc apply -f ${HC_NEW_FILE}
    else
        echo "HC Cluster ${HC_CLUSTER_NAME} already exists, avoiding step"
    fi
  7. 如果您要恢复节点和节点池来重复利用 AWS 实例,请输入以下命令恢复节点池:

    $ oc apply -f ${BACKUP_DIR}/namespaces/${HC_CLUSTER_NS}/np-*

验证

  • 要验证节点是否已完全恢复,请使用此功能:

    timeout=40
    count=0
    NODE_STATUS=$(oc get nodes --kubeconfig=${HC_KUBECONFIG} | grep -v NotReady | grep -c "worker") || NODE_STATUS=0
    
    while [ ${NODE_POOL_REPLICAS} != ${NODE_STATUS} ]
    do
        echo "Waiting for Nodes to be Ready in the destination MGMT Cluster: ${MGMT2_CLUSTER_NAME}"
        echo "Try: (${count}/${timeout})"
        sleep 30
        if [[ $count -eq timeout ]];then
            echo "Timeout waiting for Nodes in the destination MGMT Cluster"
            exit 1
        fi
        count=$((count+1))
        NODE_STATUS=$(oc get nodes --kubeconfig=${HC_KUBECONFIG} | grep -v NotReady | grep -c "worker") || NODE_STATUS=0
    done

后续步骤

关闭并删除集群。

4.2.5. 从源集群中删除托管集群

备份托管集群并将其恢复到目标管理集群后,您将关闭并删除源管理集群中的托管集群。

先决条件

您备份了数据并将其恢复到源管理集群。

提示

确保目标管理集群的 kubeconfig 文件放置在 KUBECONFIG 变量中,或者在 MGMT_KUBECONFIG 变量中设置。使用 export KUBECONFIG=<Kubeconfig FilePath> 或使用脚本,请使用 export KUBECONFIG=${MGMT_KUBECONFIG}

流程

  1. 输入以下命令来扩展 deploymentstatefulset 对象:

    # Just in case
    $ export KUBECONFIG=${MGMT_KUBECONFIG}
    
    # Scale down deployments
    $ oc scale deployment -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} --replicas=0 --all
    $ oc scale statefulset.apps -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} --replicas=0 --all
    $ sleep 15
  2. 输入以下命令来删除 NodePool 对象:

    NODEPOOLS=$(oc get nodepools -n ${HC_CLUSTER_NS} -o=jsonpath='{.items[?(@.spec.clusterName=="'${HC_CLUSTER_NAME}'")].metadata.name}')
    if [[ ! -z "${NODEPOOLS}" ]];then
        oc patch -n "${HC_CLUSTER_NS}" nodepool ${NODEPOOLS} --type=json --patch='[ { "op":"remove", "path": "/metadata/finalizers" }]'
        oc delete np -n ${HC_CLUSTER_NS} ${NODEPOOLS}
    fi
  3. 输入以下命令删除 machinemachineset 对象:

    # Machines
    for m in $(oc get machines -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} -o name); do
        oc patch -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} ${m} --type=json --patch='[ { "op":"remove", "path": "/metadata/finalizers" }]' || true
        oc delete -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} ${m} || true
    done
    
    $ oc delete machineset -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} --all || true
  4. 输入以下命令删除集群对象:

    # Cluster
    $ C_NAME=$(oc get cluster -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} -o name)
    $ oc patch -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} ${C_NAME} --type=json --patch='[ { "op":"remove", "path": "/metadata/finalizers" }]'
    $ oc delete cluster.cluster.x-k8s.io -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} --all
  5. 输入这些命令来删除 AWS 机器 (Kubernetes 对象)。不用担心删除实际的 AWS 机器。云实例不会受到影响。

    # AWS Machines
    for m in $(oc get awsmachine.infrastructure.cluster.x-k8s.io -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} -o name)
    do
        oc patch -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} ${m} --type=json --patch='[ { "op":"remove", "path": "/metadata/finalizers" }]' || true
        oc delete -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} ${m} || true
    done
  6. 输入以下命令删除 HostedControlPlaneControlPlane HC 命名空间对象:

    # Delete HCP and ControlPlane HC NS
    $ oc patch -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} hostedcontrolplane.hypershift.openshift.io ${HC_CLUSTER_NAME} --type=json --patch='[ { "op":"remove", "path": "/metadata/finalizers" }]'
    $ oc delete hostedcontrolplane.hypershift.openshift.io -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} --all
    $ oc delete ns ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME} || true
  7. 输入以下命令删除 HostedCluster 和 HC 命名空间对象:

    # Delete HC and HC Namespace
    $ oc -n ${HC_CLUSTER_NS} patch hostedclusters ${HC_CLUSTER_NAME} -p '{"metadata":{"finalizers":null}}' --type merge || true
    $ oc delete hc -n ${HC_CLUSTER_NS} ${HC_CLUSTER_NAME}  || true
    $ oc delete ns ${HC_CLUSTER_NS} || true

验证

  • 要验证所有内容是否正常工作,请输入以下命令:

    # Validations
    $ export KUBECONFIG=${MGMT2_KUBECONFIG}
    
    $ oc get hc -n ${HC_CLUSTER_NS}
    $ oc get np -n ${HC_CLUSTER_NS}
    $ oc get pod -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME}
    $ oc get machines -n ${HC_CLUSTER_NS}-${HC_CLUSTER_NAME}
    
    # Inside the HostedCluster
    $ export KUBECONFIG=${HC_KUBECONFIG}
    $ oc get clusterversion
    $ oc get nodes

后续步骤

删除托管的集群中的 OVN pod,以便您可以连接到新管理集群中运行的新 OVN control plane:

  1. 使用托管的集群的 kubeconfig 路径加载 KUBECONFIG 环境变量。
  2. 输入这个命令:

    $ oc delete pod -n openshift-ovn-kubernetes --all

4.2.6. 运行脚本来备份和恢复托管集群

要加快进程来备份托管集群并在 AWS 上的同一区域恢复它,您可以修改并运行脚本。

流程

  1. 将以下脚本中的变量替换为您的信息:

    # Fill the Common variables to fit your environment, this is just a sample
    SSH_KEY_FILE=${HOME}/.ssh/id_rsa.pub
    BASE_PATH=${HOME}/hypershift
    BASE_DOMAIN="aws.sample.com"
    PULL_SECRET_FILE="${HOME}/pull_secret.json"
    AWS_CREDS="${HOME}/.aws/credentials"
    CONTROL_PLANE_AVAILABILITY_POLICY=SingleReplica
    HYPERSHIFT_PATH=${BASE_PATH}/src/hypershift
    HYPERSHIFT_CLI=${HYPERSHIFT_PATH}/bin/hypershift
    HYPERSHIFT_IMAGE=${HYPERSHIFT_IMAGE:-"quay.io/${USER}/hypershift:latest"}
    NODE_POOL_REPLICAS=${NODE_POOL_REPLICAS:-2}
    
    # MGMT Context
    MGMT_REGION=us-west-1
    MGMT_CLUSTER_NAME="${USER}-dev"
    MGMT_CLUSTER_NS=${USER}
    MGMT_CLUSTER_DIR="${BASE_PATH}/hosted_clusters/${MGMT_CLUSTER_NS}-${MGMT_CLUSTER_NAME}"
    MGMT_KUBECONFIG="${MGMT_CLUSTER_DIR}/kubeconfig"
    
    # MGMT2 Context
    MGMT2_CLUSTER_NAME="${USER}-dest"
    MGMT2_CLUSTER_NS=${USER}
    MGMT2_CLUSTER_DIR="${BASE_PATH}/hosted_clusters/${MGMT2_CLUSTER_NS}-${MGMT2_CLUSTER_NAME}"
    MGMT2_KUBECONFIG="${MGMT2_CLUSTER_DIR}/kubeconfig"
    
    # Hosted Cluster Context
    HC_CLUSTER_NS=clusters
    HC_REGION=us-west-1
    HC_CLUSTER_NAME="${USER}-hosted"
    HC_CLUSTER_DIR="${BASE_PATH}/hosted_clusters/${HC_CLUSTER_NS}-${HC_CLUSTER_NAME}"
    HC_KUBECONFIG="${HC_CLUSTER_DIR}/kubeconfig"
    BACKUP_DIR=${HC_CLUSTER_DIR}/backup
    
    BUCKET_NAME="${USER}-hosted-${MGMT_REGION}"
    
    # DNS
    AWS_ZONE_ID="Z026552815SS3YPH9H6MG"
    EXTERNAL_DNS_DOMAIN="guest.jpdv.aws.kerbeross.com"
  2. 将脚本保存到本地文件系统中。
  3. 输入以下命令运行脚本:

    source <env_file>

    其中: env_file 是保存脚本的文件的名称。

    迁移脚本在以下软件仓库中维护:https://github.com/openshift/hypershift/blob/main/contrib/migration/migrate-hcp.sh

Red Hat logoGithubRedditYoutubeTwitter

学习

尝试、购买和销售

社区

关于红帽文档

通过我们的产品和服务,以及可以信赖的内容,帮助红帽用户创新并实现他们的目标。

让开源更具包容性

红帽致力于替换我们的代码、文档和 Web 属性中存在问题的语言。欲了解更多详情,请参阅红帽博客.

關於紅帽

我们提供强化的解决方案,使企业能够更轻松地跨平台和环境(从核心数据中心到网络边缘)工作。

© 2024 Red Hat, Inc.