Chapter 1. Backing up etcd
etcd is the key-value store for OpenShift Container Platform, which persists the state of all resource objects.
Back up your cluster’s etcd data regularly and store in a secure location ideally outside the OpenShift Container Platform environment. Do not take an etcd backup before the first certificate rotation completes, which occurs 24 hours after installation, otherwise the backup will contain expired certificates. It is also recommended to take etcd backups during non-peak usage hours, as it is a blocking action.
Be sure to take an etcd backup after you upgrade your cluster. This is important because when you restore your cluster, you must use an etcd backup that was taken from the same z-stream release. For example, an OpenShift Container Platform 4.4.2 cluster must use an etcd backup that was taken from 4.4.2.
Back up your cluster’s etcd data by performing a single invocation of the backup script on a master host. Do not take a backup for each master host.
After you have an etcd backup, you can restore to a previous cluster state.
You can perform the etcd data backup process on any master host that has a running etcd instance.
1.1. Backing up etcd data
Follow these steps to back up etcd data by creating an etcd snapshot and backing up the resources for the static pods. This backup can be saved and used at a later time if you need to restore etcd.
Only save a backup from a single master host. Do not take a backup from each master host in the cluster.
Prerequisites
-
You have access to the cluster as a user with the
cluster-adminrole. You have checked whether the cluster-wide proxy is enabled.
TipYou can check whether the proxy is enabled by reviewing the output of
oc get proxy cluster -o yaml. The proxy is enabled if thehttpProxy,httpsProxy, andnoProxyfields have values set.
Procedure
Start a debug session for a master node:
$ oc debug node/<node_name>
Change your root directory to the host:
sh-4.2# chroot /host
-
If the cluster-wide proxy is enabled, be sure that you have exported the
NO_PROXY,HTTP_PROXY, andHTTPS_PROXYenvironment variables. Run the
cluster-backup.shscript and pass in the location to save the backup to.sh-4.4# /usr/local/bin/cluster-backup.sh /home/core/assets/backup 1bf371f1b5a483927cd01bb593b0e12cff406eb8d7d0acf4ab079c36a0abd3f7 etcdctl version: 3.3.18 API version: 3.3 found latest kube-apiserver-pod: /etc/kubernetes/static-pod-resources/kube-apiserver-pod-7 found latest kube-controller-manager-pod: /etc/kubernetes/static-pod-resources/kube-controller-manager-pod-8 found latest kube-scheduler-pod: /etc/kubernetes/static-pod-resources/kube-scheduler-pod-6 found latest etcd-pod: /etc/kubernetes/static-pod-resources/etcd-pod-2 Snapshot saved at /home/core/assets/backup/snapshot_2020-03-18_220218.db snapshot db and kube resources are successfully saved to /home/core/assets/backup
In this example, two files are created in the
/home/core/assets/backup/directory on the master host:-
snapshot_<datetimestamp>.db: This file is the etcd snapshot. static_kuberesources_<datetimestamp>.tar.gz: This file contains the resources for the static pods. If etcd encryption is enabled, it also contains the encryption keys for the etcd snapshot.NoteIf etcd encryption is enabled, it is recommended to store this second file separately from the etcd snapshot for security reasons. However, this file is required in order to restore from the etcd snapshot.
Keep in mind that etcd encryption only encrypts values, not keys. This means that resource types, namespaces, and object names are unencrypted.
-
