Red Hat Summit 红帽全球峰会支持控制台（Console）开发人员开始试用联系人

2.4. Replacing the unhealthy etcd member

Depending on the state of your unhealthy etcd member, use one of the following procedures:

This procedure details the steps to replace an etcd member that is unhealthy either because the machine is not running or because the node is not ready.

Prerequisites

  • You have identified the unhealthy etcd member.
  • You have verified that either the machine is not running or the node is not ready.
  • You have access to the cluster as a user with the cluster-admin role.

  • You have taken an etcd backup.

    重要

    It is important to take an etcd backup before performing this procedure so that your cluster can be restored if you encounter any issues.

Procedure

  1. Remove the unhealthy member.

    1. Choose a pod that is not on the affected node:

      In a terminal that has access to the cluster as a cluster-admin user, run the following command: 

      $ oc get pods -n openshift-etcd | grep etcd
      Copy to Clipboard Toggle word wrap

      Example output

      etcd-ip-10-0-131-183.ec2.internal                3/3     Running     0          123m
etcd-ip-10-0-164-97.ec2.internal                 3/3     Running     0          123m
etcd-ip-10-0-154-204.ec2.internal                3/3     Running     0          124m
      Copy to Clipboard Toggle word wrap

    2. Connect to the running etcd container, passing in the name of a pod that is not on the affected node:

      In a terminal that has access to the cluster as a cluster-admin user, run the following command: 

      $ oc rsh -n openshift-etcd etcd-ip-10-0-154-204.ec2.internal
      Copy to Clipboard Toggle word wrap

    3. View the member list: 

      sh-4.2# etcdctl member list -w table
      Copy to Clipboard Toggle word wrap

      Example output

      +------------------+---------+------------------------------+---------------------------+---------------------------+
|        ID        | STATUS  |             NAME             |        PEER ADDRS         |       CLIENT ADDRS        |
+------------------+---------+------------------------------+---------------------------+---------------------------+
| 6fc1e7c9db35841d | started | ip-10-0-131-183.ec2.internal | https://10.0.131.183:2380 | https://10.0.131.183:2379 |
| 757b6793e2408b6c | started |  ip-10-0-164-97.ec2.internal |  https://10.0.164.97:2380 |  https://10.0.164.97:2379 |
| ca8c2990a0aa29d1 | started | ip-10-0-154-204.ec2.internal | https://10.0.154.204:2380 | https://10.0.154.204:2379 |
+------------------+---------+------------------------------+---------------------------+---------------------------+
      Copy to Clipboard Toggle word wrap

      Take note of the ID and the name of the unhealthy etcd member, because these values are needed later in the procedure.

    4. Remove the unhealthy etcd member by providing the ID to the etcdctl member remove command: 

      sh-4.2# etcdctl member remove 6fc1e7c9db35841d
      Copy to Clipboard Toggle word wrap

      Example output

      Member 6fc1e7c9db35841d removed from cluster baa565c8919b060e
      Copy to Clipboard Toggle word wrap

    5. View the member list again and verify that the member was removed: 

      sh-4.2# etcdctl member list -w table
      Copy to Clipboard Toggle word wrap

      Example output

      +------------------+---------+------------------------------+---------------------------+---------------------------+
|        ID        | STATUS  |             NAME             |        PEER ADDRS         |       CLIENT ADDRS        |
+------------------+---------+------------------------------+---------------------------+---------------------------+
| 757b6793e2408b6c | started |  ip-10-0-164-97.ec2.internal |  https://10.0.164.97:2380 |  https://10.0.164.97:2379 |
| ca8c2990a0aa29d1 | started | ip-10-0-154-204.ec2.internal | https://10.0.154.204:2380 | https://10.0.154.204:2379 |
+------------------+---------+------------------------------+---------------------------+---------------------------+
      Copy to Clipboard Toggle word wrap

      You can now exit the node shell.

  2. Remove the old secrets for the unhealthy etcd member that was removed.

    1. List the secrets for the unhealthy etcd member that was removed. 

      $ oc get secrets -n openshift-etcd | grep ip-10-0-131-183.ec2.internal
      1
      Copy to Clipboard Toggle word wrap
      1
      Pass in the name of the unhealthy etcd member that you took note of earlier in this procedure.

      There is a peer, serving, and metrics secret as shown in the following output:

      Example output

      etcd-peer-ip-10-0-131-183.ec2.internal              kubernetes.io/tls                     2      47m
etcd-serving-ip-10-0-131-183.ec2.internal           kubernetes.io/tls                     2      47m
etcd-serving-metrics-ip-10-0-131-183.ec2.internal   kubernetes.io/tls                     2      47m
      Copy to Clipboard Toggle word wrap

    2. Delete the secrets for the unhealthy etcd member that was removed.

      1. Delete the peer secret: 

        $ oc delete secret -n openshift-etcd etcd-peer-ip-10-0-131-183.ec2.internal
        Copy to Clipboard Toggle word wrap

      2. Delete the serving secret: 

        $ oc delete secret -n openshift-etcd etcd-serving-ip-10-0-131-183.ec2.internal
        Copy to Clipboard Toggle word wrap

      3. Delete the metrics secret: 

        $ oc delete secret -n openshift-etcd etcd-serving-metrics-ip-10-0-131-183.ec2.internal
        Copy to Clipboard Toggle word wrap

  3. Delete and recreate the master machine. After this machine is recreated, a new revision is forced and etcd scales up automatically.

    If you are running installer-provisioned infrastructure, or you used the Machine API to create your machines, follow these steps. Otherwise, you must create the new master using the same method that was used to originally create it.

    1. Obtain the machine for the unhealthy member.

      In a terminal that has access to the cluster as a cluster-admin user, run the following command: 

      $ oc get machines -n openshift-machine-api -o wide
      Copy to Clipboard Toggle word wrap

      Example output

      NAME                                        PHASE     TYPE        REGION      ZONE         AGE     NODE                           PROVIDERID                              STATE
clustername-8qw5l-master-0                  Running   m4.xlarge   us-east-1   us-east-1a   3h37m   ip-10-0-131-183.ec2.internal   aws:///us-east-1a/i-0ec2782f8287dfb7e   stopped
      1 
      
clustername-8qw5l-master-1                  Running   m4.xlarge   us-east-1   us-east-1b   3h37m   ip-10-0-154-204.ec2.internal   aws:///us-east-1b/i-096c349b700a19631   running
clustername-8qw5l-master-2                  Running   m4.xlarge   us-east-1   us-east-1c   3h37m   ip-10-0-164-97.ec2.internal    aws:///us-east-1c/i-02626f1dba9ed5bba   running
clustername-8qw5l-worker-us-east-1a-wbtgd   Running   m4.large    us-east-1   us-east-1a   3h28m   ip-10-0-129-226.ec2.internal   aws:///us-east-1a/i-010ef6279b4662ced   running
clustername-8qw5l-worker-us-east-1b-lrdxb   Running   m4.large    us-east-1   us-east-1b   3h28m   ip-10-0-144-248.ec2.internal   aws:///us-east-1b/i-0cb45ac45a166173b   running
clustername-8qw5l-worker-us-east-1c-pkg26   Running   m4.large    us-east-1   us-east-1c   3h28m   ip-10-0-170-181.ec2.internal   aws:///us-east-1c/i-06861c00007751b0a   running
      Copy to Clipboard Toggle word wrap

      1
      This is the master machine for the unhealthy node, ip-10-0-131-183.ec2.internal.

    2. Save the machine configuration to a file on your file system: 

      $ oc get machine clustername-8qw5l-master-0 \
      1 
      
    -n openshift-machine-api \
    -o yaml \
    > new-master-machine.yaml
      Copy to Clipboard Toggle word wrap
      1
      Specify the name of the master machine for the unhealthy node.

    3. Edit the new-master-machine.yaml file that was created in the previous step to assign a new name and remove unnecessary fields.

      1. Remove the entire status section: 

        status:
  addresses:
  - address: 10.0.131.183
    type: InternalIP
  - address: ip-10-0-131-183.ec2.internal
    type: InternalDNS
  - address: ip-10-0-131-183.ec2.internal
    type: Hostname
  lastUpdated: "2020-04-20T17:44:29Z"
  nodeRef:
    kind: Node
    name: ip-10-0-131-183.ec2.internal
    uid: acca4411-af0d-4387-b73e-52b2484295ad
  phase: Running
  providerStatus:
    apiVersion: awsproviderconfig.openshift.io/v1beta1
    conditions:
    - lastProbeTime: "2020-04-20T16:53:50Z"
      lastTransitionTime: "2020-04-20T16:53:50Z"
      message: machine successfully created
      reason: MachineCreationSucceeded
      status: "True"
      type: MachineCreation
    instanceId: i-0fdb85790d76d0c3f
    instanceState: stopped
    kind: AWSMachineProviderStatus
        Copy to Clipboard Toggle word wrap

      2. Change the metadata.name field to a new name.

        It is recommended to keep the same base name as the old machine and change the ending number to the next available number. In this example, clustername-8qw5l-master-0 is changed to clustername-8qw5l-master-3.

        For example: 

        apiVersion: machine.openshift.io/v1beta1
kind: Machine
metadata:
  ...
  name: clustername-8qw5l-master-3
  ...
        Copy to Clipboard Toggle word wrap

      3. Update the metadata.selfLink field to use the new machine name from the previous step. 

        apiVersion: machine.openshift.io/v1beta1
kind: Machine
metadata:
  ...
  selfLink: /apis/machine.openshift.io/v1beta1/namespaces/openshift-machine-api/machines/clustername-8qw5l-master-3
  ...
        Copy to Clipboard Toggle word wrap

      4. Remove the spec.providerID field: 

          providerID: aws:///us-east-1a/i-0fdb85790d76d0c3f
        Copy to Clipboard Toggle word wrap

      5. Remove the metadata.annotations and metadata.generation fields: 

          annotations:
    machine.openshift.io/instance-state: running
  ...
  generation: 2
        Copy to Clipboard Toggle word wrap

      6. Remove the metadata.resourceVersion and metadata.uid fields: 

          resourceVersion: "13291"
  uid: a282eb70-40a2-4e89-8009-d05dd420d31a
        Copy to Clipboard Toggle word wrap

    4. Delete the machine of the unhealthy member: 

      $ oc delete machine -n openshift-machine-api clustername-8qw5l-master-0
      1
      Copy to Clipboard Toggle word wrap
      1
      Specify the name of the master machine for the unhealthy node.

    5. Verify that the machine was deleted: 

      $ oc get machines -n openshift-machine-api -o wide
      Copy to Clipboard Toggle word wrap

      Example output

      NAME                                        PHASE     TYPE        REGION      ZONE         AGE     NODE                           PROVIDERID                              STATE
clustername-8qw5l-master-1                  Running   m4.xlarge   us-east-1   us-east-1b   3h37m   ip-10-0-154-204.ec2.internal   aws:///us-east-1b/i-096c349b700a19631   running
clustername-8qw5l-master-2                  Running   m4.xlarge   us-east-1   us-east-1c   3h37m   ip-10-0-164-97.ec2.internal    aws:///us-east-1c/i-02626f1dba9ed5bba   running
clustername-8qw5l-worker-us-east-1a-wbtgd   Running   m4.large    us-east-1   us-east-1a   3h28m   ip-10-0-129-226.ec2.internal   aws:///us-east-1a/i-010ef6279b4662ced   running
clustername-8qw5l-worker-us-east-1b-lrdxb   Running   m4.large    us-east-1   us-east-1b   3h28m   ip-10-0-144-248.ec2.internal   aws:///us-east-1b/i-0cb45ac45a166173b   running
clustername-8qw5l-worker-us-east-1c-pkg26   Running   m4.large    us-east-1   us-east-1c   3h28m   ip-10-0-170-181.ec2.internal   aws:///us-east-1c/i-06861c00007751b0a   running
      Copy to Clipboard Toggle word wrap

    6. Create the new machine using the new-master-machine.yaml file: 

      $ oc apply -f new-master-machine.yaml
      Copy to Clipboard Toggle word wrap

    7. Verify that the new machine has been created: 

      $ oc get machines -n openshift-machine-api -o wide
      Copy to Clipboard Toggle word wrap

      Example output

      NAME                                        PHASE          TYPE        REGION      ZONE         AGE     NODE                           PROVIDERID                              STATE
clustername-8qw5l-master-1                  Running        m4.xlarge   us-east-1   us-east-1b   3h37m   ip-10-0-154-204.ec2.internal   aws:///us-east-1b/i-096c349b700a19631   running
clustername-8qw5l-master-2                  Running        m4.xlarge   us-east-1   us-east-1c   3h37m   ip-10-0-164-97.ec2.internal    aws:///us-east-1c/i-02626f1dba9ed5bba   running
clustername-8qw5l-master-3                  Provisioning   m4.xlarge   us-east-1   us-east-1a   85s     ip-10-0-133-53.ec2.internal    aws:///us-east-1a/i-015b0888fe17bc2c8   running
      1 
      
clustername-8qw5l-worker-us-east-1a-wbtgd   Running        m4.large    us-east-1   us-east-1a   3h28m   ip-10-0-129-226.ec2.internal   aws:///us-east-1a/i-010ef6279b4662ced   running
clustername-8qw5l-worker-us-east-1b-lrdxb   Running        m4.large    us-east-1   us-east-1b   3h28m   ip-10-0-144-248.ec2.internal   aws:///us-east-1b/i-0cb45ac45a166173b   running
clustername-8qw5l-worker-us-east-1c-pkg26   Running        m4.large    us-east-1   us-east-1c   3h28m   ip-10-0-170-181.ec2.internal   aws:///us-east-1c/i-06861c00007751b0a   running
      Copy to Clipboard Toggle word wrap

      1
      The new machine, clustername-8qw5l-master-3 is being created and is ready once the phase changes from Provisioning to Running.

      It might take a few minutes for the new machine to be created. The etcd cluster Operator will automatically sync when the machine or node returns to a healthy state.

Verification

  1. Verify that all etcd pods are running properly.

    In a terminal that has access to the cluster as a cluster-admin user, run the following command: 

    $ oc get pods -n openshift-etcd | grep etcd
    Copy to Clipboard Toggle word wrap

    Example output

    etcd-ip-10-0-133-53.ec2.internal                 3/3     Running     0          7m49s
etcd-ip-10-0-164-97.ec2.internal                 3/3     Running     0          123m
etcd-ip-10-0-154-204.ec2.internal                3/3     Running     0          124m
    Copy to Clipboard Toggle word wrap

    If the output from the previous command only lists two pods, you can manually force an etcd redeployment. In a terminal that has access to the cluster as a cluster-admin user, run the following command: 

    $ oc patch etcd cluster -p='{"spec": {"forceRedeploymentReason": "recovery-'"$( date --rfc-3339=ns )"'"}}' --type=merge
    1
    Copy to Clipboard Toggle word wrap
    1
    The forceRedeploymentReason value must be unique, which is why a timestamp is appended.

  2. Verify that there are exactly three etcd members.

    1. Connect to the running etcd container, passing in the name of a pod that was not on the affected node:

      In a terminal that has access to the cluster as a cluster-admin user, run the following command: 

      $ oc rsh -n openshift-etcd etcd-ip-10-0-154-204.ec2.internal
      Copy to Clipboard Toggle word wrap

    2. View the member list: 

      sh-4.2# etcdctl member list -w table
      Copy to Clipboard Toggle word wrap

      Example output

      +------------------+---------+------------------------------+---------------------------+---------------------------+
|        ID        | STATUS  |             NAME             |        PEER ADDRS         |       CLIENT ADDRS        |
+------------------+---------+------------------------------+---------------------------+---------------------------+
| 5eb0d6b8ca24730c | started |  ip-10-0-133-53.ec2.internal |  https://10.0.133.53:2380 |  https://10.0.133.53:2379 |
| 757b6793e2408b6c | started |  ip-10-0-164-97.ec2.internal |  https://10.0.164.97:2380 |  https://10.0.164.97:2379 |
| ca8c2990a0aa29d1 | started | ip-10-0-154-204.ec2.internal | https://10.0.154.204:2380 | https://10.0.154.204:2379 |
+------------------+---------+------------------------------+---------------------------+---------------------------+
      Copy to Clipboard Toggle word wrap

      If the output from the previous command lists more than three etcd members, you must carefully remove the unwanted member.

      警告

      Be sure to remove the correct etcd member; removing a good etcd member might lead to quorum loss.

This procedure details the steps to replace an etcd member that is unhealthy because the etcd pod is crashlooping.

Prerequisites

  • You have identified the unhealthy etcd member.
  • You have verified that the etcd pod is crashlooping.
  • You have access to the cluster as a user with the cluster-admin role.

  • You have taken an etcd backup.

    重要

    It is important to take an etcd backup before performing this procedure so that your cluster can be restored if you encounter any issues.

Procedure

  1. Stop the crashlooping etcd pod.

    1. Debug the node that is crashlooping.

      In a terminal that has access to the cluster as a cluster-admin user, run the following command: 

      $ oc debug node/ip-10-0-131-183.ec2.internal
      1
      Copy to Clipboard Toggle word wrap
      1
      Replace this with the name of the unhealthy node.

    2. Change your root directory to the host: 

      sh-4.2# chroot /host
      Copy to Clipboard Toggle word wrap

    3. Move the existing etcd pod file out of the kubelet manifest directory: 

      sh-4.2# mkdir /var/lib/etcd-backup
      Copy to Clipboard Toggle word wrap 
      sh-4.2# mv /etc/kubernetes/manifests/etcd-pod.yaml /var/lib/etcd-backup/
      Copy to Clipboard Toggle word wrap

    4. Move the etcd data directory to a different location: 

      sh-4.2# mv /var/lib/etcd/ /tmp
      Copy to Clipboard Toggle word wrap

      You can now exit the node shell.

  2. Remove the unhealthy member.

    1. Choose a pod that is not on the affected node.

      In a terminal that has access to the cluster as a cluster-admin user, run the following command: 

      $ oc get pods -n openshift-etcd | grep etcd
      Copy to Clipboard Toggle word wrap

      Example output

      etcd-ip-10-0-131-183.ec2.internal                2/3     Error       7          6h9m
etcd-ip-10-0-164-97.ec2.internal                 3/3     Running     0          6h6m
etcd-ip-10-0-154-204.ec2.internal                3/3     Running     0          6h6m
      Copy to Clipboard Toggle word wrap

    2. Connect to the running etcd container, passing in the name of a pod that is not on the affected node.

      In a terminal that has access to the cluster as a cluster-admin user, run the following command: 

      $ oc rsh -n openshift-etcd etcd-ip-10-0-154-204.ec2.internal
      Copy to Clipboard Toggle word wrap

    3. View the member list: 

      sh-4.2# etcdctl member list -w table
      Copy to Clipboard Toggle word wrap

      Example output

      +------------------+---------+------------------------------+---------------------------+---------------------------+
|        ID        | STATUS  |             NAME             |        PEER ADDRS         |       CLIENT ADDRS        |
+------------------+---------+------------------------------+---------------------------+---------------------------+
| 62bcf33650a7170a | started | ip-10-0-131-183.ec2.internal | https://10.0.131.183:2380 | https://10.0.131.183:2379 |
| b78e2856655bc2eb | started |  ip-10-0-164-97.ec2.internal |  https://10.0.164.97:2380 |  https://10.0.164.97:2379 |
| d022e10b498760d5 | started | ip-10-0-154-204.ec2.internal | https://10.0.154.204:2380 | https://10.0.154.204:2379 |
+------------------+---------+------------------------------+---------------------------+---------------------------+
      Copy to Clipboard Toggle word wrap

      Take note of the ID and the name of the unhealthy etcd member, because these values are needed later in the procedure.

    4. Remove the unhealthy etcd member by providing the ID to the etcdctl member remove command: 

      sh-4.2# etcdctl member remove 62bcf33650a7170a
      Copy to Clipboard Toggle word wrap

      Example output

      Member 62bcf33650a7170a removed from cluster ead669ce1fbfb346
      Copy to Clipboard Toggle word wrap

    5. View the member list again and verify that the member was removed: 

      sh-4.2# etcdctl member list -w table
      Copy to Clipboard Toggle word wrap

      Example output

      +------------------+---------+------------------------------+---------------------------+---------------------------+
|        ID        | STATUS  |             NAME             |        PEER ADDRS         |       CLIENT ADDRS        |
+------------------+---------+------------------------------+---------------------------+---------------------------+
| b78e2856655bc2eb | started |  ip-10-0-164-97.ec2.internal |  https://10.0.164.97:2380 |  https://10.0.164.97:2379 |
| d022e10b498760d5 | started | ip-10-0-154-204.ec2.internal | https://10.0.154.204:2380 | https://10.0.154.204:2379 |
+------------------+---------+------------------------------+---------------------------+---------------------------+
      Copy to Clipboard Toggle word wrap

      You can now exit the node shell.

  3. Remove the old secrets for the unhealthy etcd member that was removed.

    1. List the secrets for the unhealthy etcd member that was removed. 

      $ oc get secrets -n openshift-etcd | grep ip-10-0-131-183.ec2.internal
      1
      Copy to Clipboard Toggle word wrap
      1
      Pass in the name of the unhealthy etcd member that you took note of earlier in this procedure.

      There is a peer, serving, and metrics secret as shown in the following output:

      Example output

      etcd-peer-ip-10-0-131-183.ec2.internal              kubernetes.io/tls                     2      47m
etcd-serving-ip-10-0-131-183.ec2.internal           kubernetes.io/tls                     2      47m
etcd-serving-metrics-ip-10-0-131-183.ec2.internal   kubernetes.io/tls                     2      47m
      Copy to Clipboard Toggle word wrap

    2. Delete the secrets for the unhealthy etcd member that was removed.

      1. Delete the peer secret: 

        $ oc delete secret -n openshift-etcd etcd-peer-ip-10-0-131-183.ec2.internal
        Copy to Clipboard Toggle word wrap

      2. Delete the serving secret: 

        $ oc delete secret -n openshift-etcd etcd-serving-ip-10-0-131-183.ec2.internal
        Copy to Clipboard Toggle word wrap

      3. Delete the metrics secret: 

        $ oc delete secret -n openshift-etcd etcd-serving-metrics-ip-10-0-131-183.ec2.internal
        Copy to Clipboard Toggle word wrap

  4. Force etcd redeployment.

    In a terminal that has access to the cluster as a cluster-admin user, run the following command: 

    $ oc patch etcd cluster -p='{"spec": {"forceRedeploymentReason": "single-master-recovery-'"$( date --rfc-3339=ns )"'"}}' --type=merge
    1
    Copy to Clipboard Toggle word wrap
    1
    The forceRedeploymentReason value must be unique, which is why a timestamp is appended.

    When the etcd cluster Operator performs a redeployment, it ensures that all master nodes have a functioning etcd pod.

Verification

  • Verify that the new member is available and healthy.

    1. Connect to the running etcd container again.

      In a terminal that has access to the cluster as a cluster-admin user, run the following command: 

      $ oc rsh -n openshift-etcd etcd-ip-10-0-154-204.ec2.internal
      Copy to Clipboard Toggle word wrap

    2. Verify that all members are healthy: 

      sh-4.2# etcdctl endpoint health --cluster
      Copy to Clipboard Toggle word wrap

      Example output

      https://10.0.131.183:2379 is healthy: successfully committed proposal: took = 16.671434ms
https://10.0.154.204:2379 is healthy: successfully committed proposal: took = 16.698331ms
https://10.0.164.97:2379 is healthy: successfully committed proposal: took = 16.621645ms
      Copy to Clipboard Toggle word wrap

返回顶部
Red Hat logoGithubredditYoutubeTwitter

学习

尝试、购买和销售

社区

关于红帽文档

通过我们的产品和服务，以及可以信赖的内容，帮助红帽用户创新并实现他们的目标。 了解我们当前的更新.

让开源更具包容性

红帽致力于替换我们的代码、文档和 Web 属性中存在问题的语言。欲了解更多详情，请参阅红帽博客.

關於紅帽

我们提供强化的解决方案，使企业能够更轻松地跨平台和环境（从核心数据中心到网络边缘）工作。

Theme

© 2025 Red Hat