Chapter 3. Restoring OpenShift Container Platform components

Stop all etcd services by removing the etcd pod definition and rebooting the host:

# mkdir -p /etc/origin/node/pods-stopped
# mv /etc/origin/node/pods/* /etc/origin/node/pods-stopped/
# reboot

To ensure the proper backup is restored, delete the etcd directories:

Restore a healthy backup data file to each of the etcd nodes. Perform this step on all etcd hosts, including master hosts collocated with etcd.

# cp -R /backup/etcd-xxx/* /var/lib/etcd/
# mv /var/lib/etcd/db /var/lib/etcd/member/snap/db
# chcon -R --reference /backup/etcd-xxx/* /var/lib/etcd/

Run the etcd service on each host, forcing a new cluster.

This creates a custom file for the etcd service, which overwrites the execution command adding the --force-new-cluster option:

# mkdir -p /etc/systemd/system/etcd.service.d/
# echo "[Service]" > /etc/systemd/system/etcd.service.d/temp.conf
# echo "ExecStart=" >> /etc/systemd/system/etcd.service.d/temp.conf
# sed -n '/ExecStart/s/"$/ --force-new-cluster"/p' \
    /usr/lib/systemd/system/etcd.service \
    >> /etc/systemd/system/etcd.service.d/temp.conf

# systemctl daemon-reload
# master-restart etcd

Check for error messages:

$ master-logs etcd etcd

Check for health status:

# etcdctl2 cluster-health
member 5ee217d17301 is healthy: got healthy result from https://192.168.55.8:2379
cluster is healthy

Restart the etcd service in cluster mode:

# rm -f /etc/systemd/system/etcd.service.d/temp.conf
# systemctl daemon-reload
# master-restart etcd

Check for health status and member list:

# etcdctl2 cluster-health
member 5ee217d17301 is healthy: got healthy result from https://192.168.55.8:2379
cluster is healthy

# etcdctl2 member list
5ee217d17301: name=master-0.example.com peerURLs=http://localhost:2380 clientURLs=https://192.168.55.8:2379 isLeader=true

Stop all etcd services by removing the etcd pod definition and rebooting the host:

# mkdir -p /etc/origin/node/pods-stopped
# mv /etc/origin/node/pods/* /etc/origin/node/pods-stopped/
# reboot

Clear all old data, because etcdctl recreates it in the node where the restore procedure is going to be performed:

# rm -Rf /var/lib/etcd

Run the snapshot restore command, substituting the values from the /etc/etcd/etcd.conf file:

# etcdctl3 snapshot restore /backup/etcd-xxxxxx/backup.db \
  --data-dir /var/lib/etcd \
  --name master-0.example.com \
  --initial-cluster "master-0.example.com=https://192.168.55.8:2380" \
  --initial-cluster-token "etcd-cluster-1" \
  --initial-advertise-peer-urls https://192.168.55.8:2380 \
  --skip-hash-check=true

2017-10-03 08:55:32.440779 I | mvcc: restore compact to 1041269
2017-10-03 08:55:32.468244 I | etcdserver/membership: added member 40bef1f6c79b3163 [https://192.168.55.8:2380] to cluster 26841ebcf610583c

Restore permissions and selinux context to the restored files:

# restorecon -Rv /var/lib/etcd

Start the etcd service:

# systemctl start etcd

Check for any error messages:

# master-logs etcd etcd

If the pod is running, stop the etcd pod by moving the pod manifest YAML file to another directory:

$ mv /etc/origin/node/pods/etcd.yaml .

Clear all old data:

$ rm -rf /var/lib/etcd

You use the etcdctl to recreate the data in the node where you restore the pod.

Restore the etcd snapshot to the mount path for the etcd pod:

$ export ETCDCTL_API=3
$ etcdctl snapshot restore /etc/etcd/backup/etcd/snapshot.db
	 --data-dir /var/lib/etcd/
	 --name ip-172-18-3-48.ec2.internal
	 --initial-cluster "ip-172-18-3-48.ec2.internal=https://172.18.3.48:2380"
	 --initial-cluster-token "etcd-cluster-1"
	 --initial-advertise-peer-urls https://172.18.3.48:2380
	 --skip-hash-check=true

Obtain the values for your cluster from the $/backup_files/etcd.conf file.

Set required permissions and selinux context on the data directory:

$ restorecon -Rv /var/lib/etcd/

Restart the etcd pod by moving the pod manifest YAML file to the required directory:

$ mv etcd.yaml /etc/origin/node/pods/.

In the Ansible inventory file, create a new group named [new_etcd] and add the new host. Then, add the new_etcd group as a child of the [OSEv3] group:

[OSEv3:children]
masters
nodes
etcd
new_etcd 1

... [OUTPUT ABBREVIATED] ...

[etcd]
master-0.example.com
master-1.example.com
master-2.example.com

[new_etcd] 2
etcd0.example.com 3

From the host that installed OpenShift Container Platform and hosts the Ansible inventory file, run the etcd scaleup playbook:

$ ansible-playbook  /usr/share/ansible/openshift-ansible/playbooks/openshift-etcd/scaleup.yml

After the playbook runs, modify the inventory file to reflect the current status by moving the new etcd host from the [new_etcd] group to the [etcd] group:

[OSEv3:children]
masters
nodes
etcd
new_etcd

... [OUTPUT ABBREVIATED] ...

[etcd]
master-0.example.com
master-1.example.com
master-2.example.com
etcd0.example.com

If you use Flannel, modify the flanneld service configuration on every OpenShift Container Platform host, located at /etc/sysconfig/flanneld, to include the new etcd host:

FLANNEL_ETCD_ENDPOINTS=https://master-0.example.com:2379,https://master-1.example.com:2379,https://master-2.example.com:2379,https://etcd0.example.com:2379

Restart the flanneld service:

# systemctl restart flanneld.service

Create some environment variables:

export NEW_ETCD_HOSTNAME="*etcd0.example.com*"
export NEW_ETCD_IP="192.168.55.21"

export CN=$NEW_ETCD_HOSTNAME
export SAN="IP:${NEW_ETCD_IP}, DNS:${NEW_ETCD_HOSTNAME}"
export PREFIX="/etc/etcd/generated_certs/etcd-$CN/"
export OPENSSLCFG="/etc/etcd/ca/openssl.cnf"

Create the directory to store the configuration and certificates:

# mkdir -p ${PREFIX}

Create the server certificate request and sign it: (server.csr and server.crt)

# openssl req -new -config ${OPENSSLCFG} \
    -keyout ${PREFIX}server.key  \
    -out ${PREFIX}server.csr \
    -reqexts etcd_v3_req -batch -nodes \
    -subj /CN=$CN

# openssl ca -name etcd_ca -config ${OPENSSLCFG} \
    -out ${PREFIX}server.crt \
    -in ${PREFIX}server.csr \
    -extensions etcd_v3_ca_server -batch

Create the peer certificate request and sign it: (peer.csr and peer.crt)

# openssl req -new -config ${OPENSSLCFG} \
    -keyout ${PREFIX}peer.key \
    -out ${PREFIX}peer.csr \
    -reqexts etcd_v3_req -batch -nodes \
    -subj /CN=$CN

# openssl ca -name etcd_ca -config ${OPENSSLCFG} \
  -out ${PREFIX}peer.crt \
  -in ${PREFIX}peer.csr \
  -extensions etcd_v3_ca_peer -batch

Copy the current etcd configuration and ca.crt files from the current node as examples to modify later:

# cp /etc/etcd/etcd.conf ${PREFIX}
# cp /etc/etcd/ca.crt ${PREFIX}

While still on the surviving etcd host, add the new host to the cluster. To add additional etcd members to the cluster, you must first adjust the default localhost peer in the peerURLs value for the first member:

Add the new host to the etcd cluster. Note that the new host is not yet configured, so the status stays as unstarted until the you configure the new host.

# etcdctl -C https://${CURRENT_ETCD_HOST}:2379 \
  --ca-file=/etc/etcd/ca.crt     \
  --cert-file=/etc/etcd/peer.crt     \
  --key-file=/etc/etcd/peer.key member add ${NEW_ETCD_HOSTNAME} https://${NEW_ETCD_IP}:2380 1

Added member named 10.3.9.222 with ID 4e1db163a21d7651 to cluster

ETCD_NAME="<NEW_ETCD_HOSTNAME>"
ETCD_INITIAL_CLUSTER="<NEW_ETCD_HOSTNAME>=https://<NEW_HOST_IP>:2380,<CLUSTERMEMBER1_NAME>=https:/<CLUSTERMEMBER2_IP>:2380,<CLUSTERMEMBER2_NAME>=https:/<CLUSTERMEMBER2_IP>:2380,<CLUSTERMEMBER3_NAME>=https:/<CLUSTERMEMBER3_IP>:2380"
ETCD_INITIAL_CLUSTER_STATE="existing"

Update the sample ${PREFIX}/etcd.conf file.

Create a tgz file that contains the certificates, the sample configuration file, and the ca and copy it to the new host:

# tar -czvf /etc/etcd/generated_certs/${CN}.tgz -C ${PREFIX} .
# scp /etc/etcd/generated_certs/${CN}.tgz ${CN}:/tmp/

Install iptables-services to provide iptables utilities to open the required ports for etcd:

# yum install -y iptables-services

Create the OS_FIREWALL_ALLOW firewall rules to allow etcd to communicate:

Install etcd:

# yum install -y etcd

Ensure version etcd-2.3.7-4.el7.x86_64 or greater is installed,

Ensure the etcd service is not running by removing the etcd pod definition:

# mkdir -p /etc/origin/node/pods-stopped
# mv /etc/origin/node/pods/* /etc/origin/node/pods-stopped/

Remove any etcd configuration and data:

# rm -Rf /etc/etcd/*
# rm -Rf /var/lib/etcd/*

Extract the certificates and configuration files:

# tar xzvf /tmp/etcd0.example.com.tgz -C /etc/etcd/

Start etcd on the new host:

# systemctl enable etcd --now

Verify that the host is part of the cluster and the current cluster health:

Modify the master configuration in the etcClientInfo section of the /etc/origin/master/master-config.yaml file on every master. Add the new etcd host to the list of the etcd servers OpenShift Container Platform uses to store the data, and remove any failed etcd hosts:

etcdClientInfo:
  ca: master.etcd-ca.crt
  certFile: master.etcd-client.crt
  keyFile: master.etcd-client.key
  urls:
    - https://master-0.example.com:2379
    - https://master-1.example.com:2379
    - https://master-2.example.com:2379
    - https://etcd0.example.com:2379

Restart the master API service:

If you use Flannel, modify the flanneld service configuration located at /etc/sysconfig/flanneld on every OpenShift Container Platform host to include the new etcd host:

FLANNEL_ETCD_ENDPOINTS=https://master-0.example.com:2379,https://master-1.example.com:2379,https://master-2.example.com:2379,https://etcd0.example.com:2379

Restart the flanneld service:

# systemctl restart flanneld.service

Delete the file:

$ oc rsh demo-2-fxx6d
sh-4.2$ ls */opt/app-root/src/uploaded/*
lost+found  ocp_sop.txt
sh-4.2$ *rm -rf /opt/app-root/src/uploaded/ocp_sop.txt*
sh-4.2$ *ls /opt/app-root/src/uploaded/*
lost+found

Replace the file from the server that contains the rsync backup of the files that were in the pvc:

$ oc rsync uploaded demo-2-fxx6d:/opt/app-root/src/

Validate that the file is back on the pod by using oc rsh to connect to the pod and view the contents of the directory:

$ oc rsh demo-2-fxx6d
sh-4.2$ *ls /opt/app-root/src/uploaded/*
lost+found  ocp_sop.txt

Overwrite the currently defined claim-name:

$ oc volume dc/demo --add --name=persistent-volume \
		--type=persistentVolumeClaim --claim-name=filestore \ --mount-path=/opt/app-root/src/uploaded --overwrite

Validate that the pod is using the new PVC:

$ oc describe dc/demo
Name:		demo
Namespace:	test
Created:	3 hours ago
Labels:		app=demo
Annotations:	openshift.io/generated-by=OpenShiftNewApp
Latest Version:	3
Selector:	app=demo,deploymentconfig=demo
Replicas:	1
Triggers:	Config, Image(demo@latest, auto=true)
Strategy:	Rolling
Template:
  Labels:	app=demo
		deploymentconfig=demo
  Annotations:	openshift.io/container.demo.image.entrypoint=["container-entrypoint","/bin/sh","-c","$STI_SCRIPTS_PATH/usage"]
		openshift.io/generated-by=OpenShiftNewApp
  Containers:
   demo:
    Image:	docker-registry.default.svc:5000/test/demo@sha256:0a9f2487a0d95d51511e49d20dc9ff6f350436f935968b0c83fcb98a7a8c381a
    Port:	8080/TCP
    Volume Mounts:
      /opt/app-root/src/uploaded from persistent-volume (rw)
    Environment Variables:	<none>
  Volumes:
   persistent-volume:
    Type:	PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    *ClaimName:	filestore*
    ReadOnly:	false
...omitted...

Now that the deployement configuration uses the new pvc, run oc rsync to place the files onto the new pvc:

$ oc rsync uploaded demo-3-2b8gs:/opt/app-root/src/
sending incremental file list
uploaded/
uploaded/ocp_sop.txt
uploaded/lost+found/

sent 181 bytes  received 39 bytes  146.67 bytes/sec
total size is 32  speedup is 0.15

Validate that the file is back on the pod by using oc rsh to connect to the pod and view the contents of the directory:

$ oc rsh demo-3-2b8gs
sh-4.2$ ls /opt/app-root/src/uploaded/
lost+found  ocp_sop.txt