Chapter 32. Backup and Restore

1. Because the restore procedure involves a complete reinstallation, save all the files used in the initial installation. This may include:

   • ~/.config/openshift/installer.cfg.yml (from the Quick Installation method)
   • Ansible playbooks and inventory files (from the Advanced Installation method)
   • /etc/yum.repos.d/ose.repo (from the Disconnected Installation method)
2. Backup the procedures for post-installation steps. Some installations may involve steps that are not included in the installer. This may include changes to the services outside of the control of OpenShift Container Platform or the installation of extra services like monitoring agents. Additional configuration that is not supported yet by the advanced installer might also be affected, for example when using multiple authentication providers.

3. Install packages that provide various utility commands:

   # yum install etcd

4. If using a container-based installation, pull the etcd image instead:

   # docker pull rhel7/etcd

1. Reinstall OpenShift Container Platform.

   This should be done in the same way that OpenShift Container Platform was previously installed.

2. Run all necessary post-installation steps.

3. Restore the certificates and keys, on each master:

   # cd /etc/origin/master
   # tar xvf /tmp/certs-and-keys-$(hostname).tar

4. Restore from the etcd backup:

   # mv $ETCD_DATA_DIR $ETCD_DATA_DIR.orig
   # cp -Rp $ETCD_DATA_DIR.bak $ETCD_DATA_DIR
   # chcon -R --reference $ETCD_DATA_DIR.orig $ETCD_DATA_DIR
   # chown -R etcd:etcd $ETCD_DATA_DIR

5. Create the new single node cluster using etcd’s --force-new-cluster option. You can do this using the values from /etc/etcd/etcd.conf, or you can temporarily modify the systemd unit file and start the service normally.

   To do so, edit the /usr/lib/systemd/system/etcd.service file, and add --force-new-cluster:

   # sed -i '/ExecStart/s/"$/  --force-new-cluster"/' /usr/lib/systemd/system/etcd.service
   # systemctl show etcd.service --property ExecStart --no-pager
   
   ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/bin/etcd --force-new-cluster"

   Then, restart the etcd service:

   # systemctl daemon-reload
   # systemctl start etcd

6. Verify the etcd service started correctly, then re-edit the /usr/lib/systemd/system/etcd.service file and remove the --force-new-cluster option:

   # sed -i '/ExecStart/s/ --force-new-cluster//' /usr/lib/systemd/system/etcd.service
   # systemctl show etcd.service --property ExecStart --no-pager
   
   ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/bin/etcd"

7. Restart the etcd service, then verify the etcd cluster is running correctly and displays OpenShift Container Platform’s configuration:

   # systemctl daemon-reload
   # systemctl restart etcd

1. If the new etcd members will also be OpenShift Container Platform nodes, see Add the desired number of hosts to the cluster. The rest of this procedure presumes you have added just one host, but if adding multiple, perform all steps on each host.

2. Upgrade etcd and iptables on the surviving nodes:

   # yum update etcd iptables-services

   Ensure version etcd-2.3.7-4.el7.x86_64 or greater is installed, and that the same version is installed on each host.

3. Install etcd and iptables on the new host

   # yum install etcd iptables-services

   Ensure version etcd-2.3.7-4.el7.x86_64 or greater is installed, and that the same version is installed on the new host.

4. Backup the etcd data store on surviving hosts before making any cluster configuration changes.

5. If replacing a failed etcd member, remove the failed member before adding the new member.

   # etcdctl -C https://<surviving host IP>:2379 \
     --ca-file=/etc/etcd/ca.crt     \
     --cert-file=/etc/etcd/peer.crt     \
     --key-file=/etc/etcd/peer.key cluster-health
   
   # etcdctl -C https://<surviving host IP>:2379 \
     --ca-file=/etc/etcd/ca.crt     \
     --cert-file=/etc/etcd/peer.crt     \
     --key-file=/etc/etcd/peer.key member remove <failed member identifier>

   Stop the etcd service on the failed etcd member:

   # systemctl stop etcd

6. On the new host, add the appropriate iptables rules:

   # systemctl enable iptables.service --now
   # iptables -N OS_FIREWALL_ALLOW
   # iptables -t filter -I INPUT -j OS_FIREWALL_ALLOW
   # iptables -A OS_FIREWALL_ALLOW -p tcp -m state \
     --state NEW -m tcp --dport 2379 -j ACCEPT
   # iptables -A OS_FIREWALL_ALLOW -p tcp -m state \
     --state NEW -m tcp --dport 2380 -j ACCEPT
   # iptables-save > /etc/sysconfig/iptables

7. Generate the required certificates for the new host. On a surviving etcd host:

   1. Make a backup of the /etc/etcd/ca/ directory. Ensure that the directory contains the etcd certificates.

   2. Set the variables and working directory for the certificates, ensuring to create the PREFIX directory if one has not been created:

      # cd /etc/etcd
      # export NEW_ETCD="<NEW_HOST_NAME>"
      
      # export CN=$NEW_ETCD
      # export SAN="IP:<NEW_HOST_IP>"
      # export PREFIX="./generated_certs/etcd-$CN/"

   3. Create the $PREFIX directory:

      $ mkdir -p $PREFIX

   4. Create the server.csr and server.crt certificates:

      # openssl req -new -keyout ${PREFIX}server.key \
        -config ca/openssl.cnf \
        -out ${PREFIX}server.csr \
        -reqexts etcd_v3_req -batch -nodes \
        -subj /CN=$CN
      
      # openssl ca -name etcd_ca -config ca/openssl.cnf \
        -out ${PREFIX}server.crt \
        -in ${PREFIX}server.csr \
        -extensions etcd_v3_ca_server -batch

   5. Create the peer.csr and peer.crt certificates:

      # openssl req -new -keyout ${PREFIX}peer.key \
        -config ca/openssl.cnf \
        -out ${PREFIX}peer.csr \
        -reqexts etcd_v3_req -batch -nodes \
        -subj /CN=$CN
      
      # openssl ca -name etcd_ca -config ca/openssl.cnf \
        -out ${PREFIX}peer.crt \
        -in ${PREFIX}peer.csr \
        -extensions etcd_v3_ca_peer -batch

   6. Copy the etcd.conf and ca.crt files, and archive the contents of the directory:

      # cp etcd.conf ${PREFIX}
      # cp ca.crt ${PREFIX}
      # tar -czvf ${PREFIX}${CN}.tgz -C ${PREFIX} .

   7. Transfer the files to the new etcd hosts:

      # scp ${PREFIX}${CN}.tgz  $CN:/etc/etcd/

8. While still on the surviving etcd host, add the new host to the cluster:

   1. Add the new host to the cluster:

      # export ETCD_CA_HOST="<SURVIVING_ETCD_HOSTNAME>"
      # export NEW_ETCD="<NEW_ETCD_HOSTNAME>"
      # export NEW_ETCD_IP="<NEW_HOST_IP>"
      
      # etcdctl -C https://${ETCD_CA_HOST}:2379 \
        --ca-file=/etc/etcd/ca.crt     \
        --cert-file=/etc/etcd/peer.crt     \
        --key-file=/etc/etcd/peer.key member add ${NEW_ETCD} https://${NEW_ETCD_IP}:2380
      
      ETCD_NAME="<NEW_ETCD_HOSTNAME>"
      ETCD_INITIAL_CLUSTER="<NEW_ETCD_HOSTNAME>=https://<NEW_HOST_IP>:2380,<SURVIVING_ETCD_HOST>=https:/<SURVIVING_HOST_IP>:2380
      ETCD_INITIAL_CLUSTER_STATE="existing"

      Copy the three environment variables in the etcdctl member add output. They will be used later.

   2. On the new host, extract the copied configuration data and set the permissions:

      # tar -xf /etc/etcd/<NEW_ETCD_HOSTNAME>.tgz -C /etc/etcd/ --overwrite
      # chown -R etcd:etcd /etc/etcd/*

   3. On the new host, remove any etcd data:

      # rm -rf /var/lib/etcd/member
      # chown -R etcd:etcd /var/lib/etcd

9. On the new etcd host, update the etcd.conf file:

   1. Replace the following with the values generated in the previous step:

      • ETCD_NAME
      • ETCD_INITIAL_CLUSTER
      • ETCD_INITIAL_CLUSTER_STATE

   2. Replace the IP address with the "NEW_ETCD" value for:

      • ETCD_LISTEN_PEER_URLS
      • ETCD_LISTEN_CLIENT_URLS
      • ETCD_INITIAL_ADVERTISE_PEER_URLS
      • ETCD_ADVERTISE_CLIENT_URLS
   3. For replacing failed members, replace the failed hosts with the new hosts.
10. To ensure the etcd configuration does not use the failed host when the etcd service is restarted, modify the etcd.conf file on all remaining etcd hosts and remove the failed host in the value for the ETCD_INITIAL_CLUSTER variable.
11. On the node that hosts the installation files, update the [etcd] hosts group in the /etc/ansible/hosts inventory file. Remove the old etcd hosts and add the new ones.

12. Start etcd on the new host:

    # systemctl enable etcd --now

13. To verify that the new member has been added successfully:

    etcdctl -C https://${ETCD_CA_HOST}:2379 --ca-file=/etc/etcd/ca.crt \
      --cert-file=/etc/etcd/peer.crt     \
      --key-file=/etc/etcd/peer.key cluster-health

14. Update the master configuration on all masters to point to the new etcd host

    1. On every master in the cluster, edit /etc/origin/master/master-config.yaml
    2. Find the etcdClientInfo section.
    3. Add the new etcd host to the urls list.
    4. If a failed etcd host was replaced, remove it from the list.

    5. Restart the master API service.

       On a single master cluster installation:

       # systemctl restart atomic-openshift-master

       On a multi-master cluster installation, on each master:

       # systemctl restart atomic-openshift-master-api

Example of Backing up a Jenkins Deployment’s Application Data

1. Get the application data mountPath from the deploymentconfig:

   $ oc get dc/jenkins -o jsonpath='{ .spec.template.spec.containers[?(@.name=="jenkins")].volumeMounts[?(@.name=="jenkins-data")].mountPath }'
   /var/lib/jenkins

2. Get the name of the pod that is currently running:

   $ oc get pod --selector=deploymentconfig=jenkins -o jsonpath='{ .metadata.name }'
   jenkins-1-37nux

3. Use the oc rsync command to copy application data:

   $ oc rsync jenkins-1-37nux:/var/lib/jenkins /tmp/

Example of Restoring a Jenkins Deployment’s Application Data

1. Verify the backup:

   $ ls -la /tmp/jenkins-backup/
   total 8
   drwxrwxr-x.  3 user     user   20 Sep  6 11:14 .
   drwxrwxrwt. 17 root     root 4096 Sep  6 11:16 ..
   drwxrwsrwx. 12 user     user 4096 Sep  6 11:14 jenkins

2. Use the oc rsync tool to copy the data into the running pod:

   $ oc rsync /tmp/jenkins-backup/jenkins jenkins-1-37nux:/var/lib

   Note

   Depending on the application, you may be required to restart the application.

3. Restart the application with new data (optional):

   $ oc delete pod jenkins-1-37nux

   Alternatively, you can scale down the deployment to 0, and then up again:

   $ oc scale --replicas=0 dc/jenkins
   $ oc scale --replicas=1 dc/jenkins