Red Hat Summit Red Hat SummitSupportKonsoleEntwicklerDemo startenKontakt

Dieser Inhalt ist in der von Ihnen ausgewählten Sprache nicht verfügbar.

Chapter 38. Restoring etcd quorum

If you lose etcd quorum, you can restore it.

  • If you run etcd on a separate host, you must back up etcd, take down your etcd cluster, and form a new one. You can use one healthy etcd node to form a new cluster, but you must remove all other healthy nodes.
  • If you run etcd as static pods on your master nodes, you stop the etcd pods, create a temporary cluster, and then restart the etcd pods.
Note

During etcd quorum loss, applications that run on OpenShift Container Platform are unaffected. However, the platform functionality is limited to read-only operations. You cannot take action such as scaling an application up or down, changing deployments, or running or modifying builds.

To confirm the loss of etcd quorum, run one of the following commands and confirm that the cluster is unhealthy:

  • If you use the etcd v2 API, run the following command: 

    # etcd_ctl=2 etcdctl  --cert-file=/etc/origin/master/master.etcd-client.crt  \
          --key-file /etc/origin/master/master.etcd-client.key \
          --ca-file /etc/origin/master/master.etcd-ca.crt \
          --endpoints="https://*master-0.example.com*:2379,\
          https://*master-1.example.com*:2379,\
          https://*master-2.example.com*:2379"\
          cluster-health

member 165201190bf7f217 is unhealthy: got unhealthy result from https://master-0.example.com:2379
member b50b8a0acab2fa71 is unreachable: [https://master-1.example.com:2379] are all unreachable
member d40307cbca7bc2df is unreachable: [https://master-2.example.com:2379] are all unreachable
cluster is unhealthy
    Copy to Clipboard Toggle word wrap

  • If you use the v3 API, run the following command: 

    # ETCDCTL_API=3 etcdctl --cert=/etc/origin/master/master.etcd-client.crt  \
          --key=/etc/origin/master/master.etcd-client.key \
          --cacert=/etc/origin/masterca.crt \
          --endpoints="https://*master-0.example.com*:2379,\
          https://*master-1.example.com*:2379,\
          https://*master-2.example.com*:2379"\
          endpoint health
https://master-0.example.com:2379 is unhealthy: failed to connect: context deadline exceeded
https://master-1.example.com:2379 is unhealthy: failed to connect: context deadline exceeded
https://master-2.example.com:2379 is unhealthy: failed to connect: context deadline exceeded
Error:  unhealthy cluster
    Copy to Clipboard Toggle word wrap

Note the member IDs and host names of the hosts. You use one of the nodes that can be reached to form a new cluster.

38.1. Restoring etcd quorum for separate services
Link kopieren

38.1.1. Backing up etcd
Link kopieren

When you back up etcd, you must back up both the etcd configuration files and the etcd data.

38.1.1.1. Backing up etcd configuration files
Link kopieren

The etcd configuration files to be preserved are all stored in the /etc/etcd directory of the instances where etcd is running. This includes the etcd configuration file (/etc/etcd/etcd.conf) and the required certificates for cluster communication. All those files are generated at installation time by the Ansible installer.

Procedure

For each etcd member of the cluster, back up the etcd configuration. 

$ ssh master-0
1 

# mkdir -p /backup/etcd-config-$(date +%Y%m%d)/
# cp -R /etc/etcd/ /backup/etcd-config-$(date +%Y%m%d)/
Copy to Clipboard Toggle word wrap
1
Replace master-0 with the name of your etcd member.
Note

The certificates and configuration files on each etcd cluster member are unique.

38.1.1.2. Backing up etcd data
Link kopieren

Prerequisites
Note

The OpenShift Container Platform installer creates aliases to avoid typing all the flags named etcdctl2 for etcd v2 tasks and etcdctl3 for etcd v3 tasks.

However, the etcdctl3 alias does not provide the full endpoint list to the etcdctl command, so you must specify the --endpoints option and list all the endpoints.

Before backing up etcd:

  • etcdctl binaries must be available or, in containerized installations, the rhel7/etcd container must be available.
  • Ensure that the OpenShift Container Platform API service is running.
  • Ensure connectivity with the etcd cluster (port 2379/tcp).
  • Ensure the proper certificates to connect to the etcd cluster.
Procedure
Note

While the etcdctl backup command is used to perform the backup, etcd v3 has no concept of a backup. Instead, you either take a snapshot from a live member with the etcdctl snapshot save command or copy the member/snap/db file from an etcd data directory.

The etcdctl backup command rewrites some of the metadata contained in the backup, specifically, the node ID and cluster ID, which means that in the backup, the node loses its former identity. To recreate a cluster from the backup, you create a new, single-node cluster, then add the rest of the nodes to the cluster. The metadata is rewritten to prevent the new node from joining an existing cluster.

Back up the etcd data:

Important

Clusters upgraded from previous versions of OpenShift Container Platform might contain v2 data stores. Back up all etcd data stores.

  1. Obtain the etcd endpoint IP address from the static pod manifest: 

    $ export ETCD_POD_MANIFEST="/etc/origin/node/pods/etcd.yaml"
    Copy to Clipboard Toggle word wrap 
    $ export ETCD_EP=$(grep https ${ETCD_POD_MANIFEST} | cut -d '/' -f3)
    Copy to Clipboard Toggle word wrap

  2. Log in as an administrator: 

    $ oc login -u system:admin
    Copy to Clipboard Toggle word wrap

  3. Obtain the etcd pod name: 

    $ export ETCD_POD=$(oc get pods -n kube-system | grep -o -m 1 '^master-etcd\S*')
    Copy to Clipboard Toggle word wrap

  4. Change to the kube-system project: 

    $ oc project kube-system
    Copy to Clipboard Toggle word wrap

  5. Take a snapshot of the etcd data in the pod and store it locally: 

    $ oc exec ${ETCD_POD} -c etcd -- /bin/bash -c "ETCDCTL_API=3 etcdctl \
    --cert /etc/etcd/peer.crt \
    --key /etc/etcd/peer.key \
    --cacert /etc/etcd/ca.crt \
    --endpoints $ETCD_EP \
    snapshot save /var/lib/etcd/snapshot.db"
    1
    Copy to Clipboard Toggle word wrap
    1
    You must write the snapshot to a directory under /var/lib/etcd/.

38.1.2. Removing an etcd host
Link kopieren

If an etcd host fails beyond restoration, remove it from the cluster. To recover from an etcd quorum loss, you must also remove all healthy etcd nodes but one from your cluster.

Steps to be performed on all masters hosts

Procedure

  1. Remove each other etcd host from the etcd cluster. Run the following command for each etcd node: 

    # etcdctl3 --endpoints=https://<surviving host IP>:2379
  --cacert=/etc/etcd/ca.crt
  --cert=/etc/etcd/peer.crt
  --key=/etc/etcd/peer.key member remove <failed member ID>
    Copy to Clipboard Toggle word wrap

  2. Remove the other etcd hosts from the /etc/origin/master/master-config.yaml +master configuration file on every master: 

    etcdClientInfo:
  ca: master.etcd-ca.crt
  certFile: master.etcd-client.crt
  keyFile: master.etcd-client.key
  urls:
    - https://master-0.example.com:2379
    - https://master-1.example.com:2379
    1 
    
    - https://master-2.example.com:2379
    2
    Copy to Clipboard Toggle word wrap
    1 2
    The host to remove.

  3. Restart the master API service on every master: 

    # master-restart api restart-master controller
    Copy to Clipboard Toggle word wrap

Steps to be performed in the current etcd cluster

Procedure

  1. Remove the failed host from the cluster: 

    # etcdctl2 cluster-health
member 5ee217d19001 is healthy: got healthy result from https://192.168.55.12:2379
member 2a529ba1840722c0 is healthy: got healthy result from https://192.168.55.8:2379
failed to check the health of member 8372784203e11288 on https://192.168.55.21:2379: Get https://192.168.55.21:2379/health: dial tcp 192.168.55.21:2379: getsockopt: connection refused
member 8372784203e11288 is unreachable: [https://192.168.55.21:2379] are all unreachable
member ed4f0efd277d7599 is healthy: got healthy result from https://192.168.55.13:2379
cluster is healthy

# etcdctl2 member remove 8372784203e11288
    1 
    
Removed member 8372784203e11288 from cluster

# etcdctl2 cluster-health
member 5ee217d19001 is healthy: got healthy result from https://192.168.55.12:2379
member 2a529ba1840722c0 is healthy: got healthy result from https://192.168.55.8:2379
member ed4f0efd277d7599 is healthy: got healthy result from https://192.168.55.13:2379
cluster is healthy
    Copy to Clipboard Toggle word wrap
    1
    The remove command requires the etcd ID, not the hostname.

  2. To ensure the etcd configuration does not use the failed host when the etcd service is restarted, modify the /etc/etcd/etcd.conf file on all remaining etcd hosts and remove the failed host in the value for the ETCD_INITIAL_CLUSTER variable: 

    # vi /etc/etcd/etcd.conf
    Copy to Clipboard Toggle word wrap

    For example: 

    ETCD_INITIAL_CLUSTER=master-0.example.com=https://192.168.55.8:2380,master-1.example.com=https://192.168.55.12:2380,master-2.example.com=https://192.168.55.13:2380
    Copy to Clipboard Toggle word wrap

    becomes: 

    ETCD_INITIAL_CLUSTER=master-0.example.com=https://192.168.55.8:2380,master-1.example.com=https://192.168.55.12:2380
    Copy to Clipboard Toggle word wrap
    Note

    Restarting the etcd services is not required, because the failed host is removed using etcdctl.

  3. Modify the Ansible inventory file to reflect the current status of the cluster and to avoid issues when re-running a playbook: 

    [OSEv3:children]
masters
nodes
etcd

... [OUTPUT ABBREVIATED] ...

[etcd]
master-0.example.com
master-1.example.com
    Copy to Clipboard Toggle word wrap

  4. If you are using Flannel, modify the flanneld service configuration located at /etc/sysconfig/flanneld on every host and remove the etcd host: 

    FLANNEL_ETCD_ENDPOINTS=https://master-0.example.com:2379,https://master-1.example.com:2379,https://master-2.example.com:2379
    Copy to Clipboard Toggle word wrap

  5. Restart the flanneld service: 

    # systemctl restart flanneld.service
    Copy to Clipboard Toggle word wrap

38.1.3. Creating a single-node etcd cluster
Link kopieren

To restore the full functionality of your OpenShift Container Platform instance, make a remaining etcd node a standalone etcd cluster.

Procedure

  1. On the etcd node that you did not remove from the cluster, stop all etcd services by removing the etcd pod definition: 

    # mkdir -p /etc/origin/node/pods-stopped
# mv /etc/origin/node/pods/etcd.yaml /etc/origin/node/pods-stopped/
# systemctl stop atomic-openshift-node
# mv /etc/origin/node/pods-stopped/etcd.yaml /etc/origin/node/pods/
    Copy to Clipboard Toggle word wrap

  2. Run the etcd service on the host, forcing a new cluster.

    These commands create a custom file for the etcd service, which adds the --force-new-cluster option to the etcd start command: 

    # mkdir -p /etc/systemd/system/etcd.service.d/
# echo "[Service]" > /etc/systemd/system/etcd.service.d/temp.conf
# echo "ExecStart=" >> /etc/systemd/system/etcd.service.d/temp.conf
# sed -n '/ExecStart/s/"$/ --force-new-cluster"/p' \
    /usr/lib/systemd/system/etcd.service \
    >> /etc/systemd/system/etcd.service.d/temp.conf

# systemctl daemon-reload
# master-restart etcd
    Copy to Clipboard Toggle word wrap

  3. List the etcd member and confirm that the member list contains only your single etcd host: 

    # etcdctl member list
165201190bf7f217: name=192.168.34.20 peerURLs=http://localhost:2380 clientURLs=https://master-0.example.com:2379 isLeader=true
    Copy to Clipboard Toggle word wrap

  4. After restoring the data and creating a new cluster, you must update the peerURLs parameter value to use the IP address where etcd listens for peer communication: 

    # etcdctl member update 165201190bf7f217 https://192.168.34.20:2380
    1
    Copy to Clipboard Toggle word wrap
    1
    165201190bf7f217 is the member ID shown in the output of the previous command, and https://192.168.34.20:2380 is its IP address.

  5. To verify, check that the IP is in the member list: 

    $ etcdctl2 member list
5ee217d17301: name=master-0.example.com peerURLs=https://*192.168.55.8*:2380 clientURLs=https://192.168.55.8:2379 isLeader=true
    Copy to Clipboard Toggle word wrap

38.1.4. Adding etcd nodes after restoring
Link kopieren

After the first instance is running, you can add multiple etcd servers to your cluster.

Procedure

  1. Get the etcd name for the instance in the ETCD_NAME variable: 

    # grep ETCD_NAME /etc/etcd/etcd.conf
    Copy to Clipboard Toggle word wrap

  2. Get the IP address where etcd listens for peer communication: 

    # grep ETCD_INITIAL_ADVERTISE_PEER_URLS /etc/etcd/etcd.conf
    Copy to Clipboard Toggle word wrap

  3. If the node was previously part of a etcd cluster, delete the previous etcd data: 

    # rm -Rf /var/lib/etcd/*
    Copy to Clipboard Toggle word wrap

  4. On the etcd host where etcd is properly running, add the new member: 

    # etcdctl3 member add *<name>* \
  --peer-urls="*<advertise_peer_urls>*"
    Copy to Clipboard Toggle word wrap

    The command outputs some variables. For example: 

    ETCD_NAME="master2"
ETCD_INITIAL_CLUSTER="master-0.example.com=https://192.168.55.8:2380"
ETCD_INITIAL_CLUSTER_STATE="existing"
    Copy to Clipboard Toggle word wrap

  5. Add the values from the previous command to the /etc/etcd/etcd.conf file of the new host: 

    # vi /etc/etcd/etcd.conf
    Copy to Clipboard Toggle word wrap

  6. Start the etcd service in the node joining the cluster: 

    # systemctl start etcd.service
    Copy to Clipboard Toggle word wrap

  7. Check for error messages: 

    # master-logs etcd etcd
    Copy to Clipboard Toggle word wrap

  8. Once you add all the nodes, verify the cluster status and cluster health: 

    # etcdctl3 endpoint health --endpoints="https://<etcd_host1>:2379,https://<etcd_host2>:2379,https://<etcd_host3>:2379"
https://master-0.example.com:2379 is healthy: successfully committed proposal: took = 1.423459ms
https://master-1.example.com:2379 is healthy: successfully committed proposal: took = 1.767481ms
https://master-2.example.com:2379 is healthy: successfully committed proposal: took = 1.599694ms

# etcdctl3 endpoint status --endpoints="https://<etcd_host1>:2379,https://<etcd_host2>:2379,https://<etcd_host3>:2379"
https://master-0.example.com:2379, 40bef1f6c79b3163, 3.2.5, 28 MB, true, 9, 2878
https://master-1.example.com:2379, 1ea57201a3ff620a, 3.2.5, 28 MB, false, 9, 2878
https://master-2.example.com:2379, 59229711e4bc65c8, 3.2.5, 28 MB, false, 9, 2878
    Copy to Clipboard Toggle word wrap
  9. Add the remaining peers back into the cluster.

38.2. Restoring etcd quorum for static pods
Link kopieren

If you lose etcd quorum on a cluster that uses static pods for etcd, take the following steps:

Procedure

  1. Stop the etcd pod: 

    mv /etc/origin/node/pods/etcd.yaml .
    Copy to Clipboard Toggle word wrap

  2. Temporarily force a new cluster on the etcd host: 

    $ cp /etc/etcd/etcd.conf etcd.conf.bak
$ echo "ETCD_FORCE_NEW_CLUSTER=true" >> /etc/etcd/etcd.conf
    Copy to Clipboard Toggle word wrap

  3. Restart the etcd pod: 

    $ mv etcd.yaml /etc/origin/node/pods/.
    Copy to Clipboard Toggle word wrap

  4. Stop the etcd pod and remove the FORCE_NEW_CLUSTER command: 

    $ mv /etc/origin/node/pods/etcd.yaml .
$ rm /etc/etcd/etcd.conf
$ mv etcd.conf.bak /etc/etcd/etcd.conf
    Copy to Clipboard Toggle word wrap

  5. Restart the etcd pod: 

    $ mv etcd.yaml /etc/origin/node/pods/.
    Copy to Clipboard Toggle word wrap
Nach oben
Red Hat logoGithubredditYoutubeTwitter

Lernen

Testen, kaufen und verkaufen

Communitys

Über Red Hat Dokumentation

Wir helfen Red Hat Benutzern, mit unseren Produkten und Diensten innovativ zu sein und ihre Ziele zu erreichen – mit Inhalten, denen sie vertrauen können. Entdecken Sie unsere neuesten Updates.

Mehr Inklusion in Open Source

Red Hat hat sich verpflichtet, problematische Sprache in unserem Code, unserer Dokumentation und unseren Web-Eigenschaften zu ersetzen. Weitere Einzelheiten finden Sie in Red Hat Blog.

Über Red Hat

Wir liefern gehärtete Lösungen, die es Unternehmen leichter machen, plattform- und umgebungsübergreifend zu arbeiten, vom zentralen Rechenzentrum bis zum Netzwerkrand.

Theme

© 2025 Red Hat