Chapter 30. Backup and Restore

1. Because the restore procedure involves a complete reinstallation, save all the files used in the initial installation. This may include:

   • ~/.config/openshift/installer.cfg.yml (from the Quick Installation method)
   • Ansible playbooks and inventory files (from the Advanced Installation method)
   • /etc/yum.repos.d/ose.repo (from the Disconnected Installation method)
2. Backup the procedures for post-installation steps. Some installations may involve steps that are not included in the installer. This may include changes to the services outside of the control of OpenShift Container Platform or the installation of extra services like monitoring agents. Additional configuration that is not supported yet by the advanced installer might also be affected, for example when using multiple authentication providers.

3. Install packages that provide various utility commands:

   # yum install etcd

   Copy to Clipboard Toggle word wrap

4. If using a container-based installation, pull the etcd image instead:

   # docker pull rhel7/etcd

   Copy to Clipboard Toggle word wrap

1. Reinstall OpenShift Container Platform.

   This should be done in the same way that OpenShift Container Platform was previously installed.

2. Run all necessary post-installation steps.

3. Restore the certificates and keys, on each master:

   # cd /etc/origin/master
   # tar xvf /tmp/certs-and-keys-$(hostname).tar

   Copy to Clipboard Toggle word wrap

4. Restore from the etcd backup:

   # mv $ETCD_DATA_DIR $ETCD_DATA_DIR.orig
   # cp -Rp $ETCD_DATA_DIR.bak $ETCD_DATA_DIR
   # chcon -R --reference $ETCD_DATA_DIR.orig $ETCD_DATA_DIR
   # chown -R etcd:etcd $ETCD_DATA_DIR

   Copy to Clipboard Toggle word wrap

5. Create the new single node cluster using etcd’s --force-new-cluster option. You can do this using the values from /etc/etcd/etcd.conf, or you can temporarily modify the systemd unit file and start the service normally.

   To do so, edit the /usr/lib/systemd/system/etcd.service file, and add --force-new-cluster:

   # sed -i '/ExecStart/s/"$/  --force-new-cluster"/' /usr/lib/systemd/system/etcd.service
   # systemctl show etcd.service --property ExecStart --no-pager
   
   ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/bin/etcd --force-new-cluster"

   Copy to Clipboard Toggle word wrap

   Then, restart the etcd service:

   # systemctl daemon-reload
   # systemctl start etcd

   Copy to Clipboard Toggle word wrap

6. Verify the etcd service started correctly, then re-edit the /usr/lib/systemd/system/etcd.service file and remove the --force-new-cluster option:

   # sed -i '/ExecStart/s/ --force-new-cluster//' /usr/lib/systemd/system/etcd.service
   # systemctl show etcd.service --property ExecStart --no-pager
   
   ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/bin/etcd"

   Copy to Clipboard Toggle word wrap

7. Restart the etcd service, then verify the etcd cluster is running correctly and displays OpenShift Container Platform’s configuration:

   # systemctl daemon-reload
   # systemctl restart etcd

   Copy to Clipboard Toggle word wrap

1. If the new etcd members will also be OpenShift Container Platform nodes, see Add the desired number of hosts to the cluster. The rest of this procedure presumes you have added just one host, but if adding multiple, perform all steps on each host.

2. Upgrade etcd and iptables on the surviving nodes:

   # yum update etcd iptables-services

   Copy to Clipboard Toggle word wrap

   Ensure version etcd-2.3.7-4.el7.x86_64 or greater is installed, and that the same version is installed on each host.

3. Install etcd and iptables on the new host

   # yum install etcd iptables-services

   Copy to Clipboard Toggle word wrap

   Ensure version etcd-2.3.7-4.el7.x86_64 or greater is installed, and that the same version is installed on the new host.

4. Backup the etcd data store on surviving hosts before making any cluster configuration changes.

5. If replacing a failed etcd member, remove the failed member before adding the new member.

   # etcdctl -C https://<surviving host IP>:2379 \
     --ca-file=/etc/etcd/ca.crt     \
     --cert-file=/etc/etcd/peer.crt     \
     --key-file=/etc/etcd/peer.key cluster-health
   
   # etcdctl -C https://<surviving host IP>:2379 \
     --ca-file=/etc/etcd/ca.crt     \
     --cert-file=/etc/etcd/peer.crt     \
     --key-file=/etc/etcd/peer.key member remove <failed member identifier>

   Copy to Clipboard Toggle word wrap

   Stop the etcd service on the failed etcd member:

   # systemctl stop etcd

   Copy to Clipboard Toggle word wrap

6. On the new host, add the appropriate iptables rules:

   # systemctl enable iptables.service --now
   # iptables -N OS_FIREWALL_ALLOW
   # iptables -t filter -I INPUT -j OS_FIREWALL_ALLOW
   # iptables -A OS_FIREWALL_ALLOW -p tcp -m state \
     --state NEW -m tcp --dport 2379 -j ACCEPT
   # iptables -A OS_FIREWALL_ALLOW -p tcp -m state \
     --state NEW -m tcp --dport 2380 -j ACCEPT
   # iptables-save > /etc/sysconfig/iptables

   Copy to Clipboard Toggle word wrap

7. Generate the required certificates for the new host. On a surviving etcd host:

   1. Make a backup of the /etc/etcd/ca/ directory.

   2. Set the variables and working directory for the certificates, ensuring to create the PREFIX directory if one has not been created:

      # cd /etc/etcd
      # export NEW_ETCD="<NEW_HOST_NAME>"
      
      # export CN=$NEW_ETCD
      # export SAN="IP:<NEW_HOST_IP>"
      # export PREFIX="./generated_certs/etcd-$CN/"

      Copy to Clipboard Toggle word wrap

   3. Create the $PREFIX directory:

      $ mkdir -p $PREFIX

      Copy to Clipboard Toggle word wrap

   4. Create the server.csr and server.crt certificates:

      # openssl req -new -keyout ${PREFIX}server.key \
        -config ca/openssl.cnf \
        -out ${PREFIX}server.csr \
        -reqexts etcd_v3_req -batch -nodes \
        -subj /CN=$CN
      
      # openssl ca -name etcd_ca -config ca/openssl.cnf \
        -out ${PREFIX}server.crt \
        -in ${PREFIX}server.csr \
        -extensions etcd_v3_ca_server -batch

      Copy to Clipboard Toggle word wrap

   5. Create the peer.csr and peer.crt certificates:

      # openssl req -new -keyout ${PREFIX}peer.key \
        -config ca/openssl.cnf \
        -out ${PREFIX}peer.csr \
        -reqexts etcd_v3_req -batch -nodes \
        -subj /CN=$CN
      
      # openssl ca -name etcd_ca -config ca/openssl.cnf \
        -out ${PREFIX}peer.crt \
        -in ${PREFIX}peer.csr \
        -extensions etcd_v3_ca_peer -batch

      Copy to Clipboard Toggle word wrap

   6. Copy the etcd.conf and ca.crt files, and archive the contents of the directory:

      # cp etcd.conf ${PREFIX}
      # cp ca.crt ${PREFIX}
      # tar -czvf ${PREFIX}${CN}.tgz -C ${PREFIX} .

      Copy to Clipboard Toggle word wrap

   7. Transfer the files to the new etcd hosts:

      # scp ${PREFIX}${CN}.tgz  $CN:/etc/etcd/

      Copy to Clipboard Toggle word wrap

8. While still on the surviving etcd host, add the new host to the cluster:

   1. Add the new host to the cluster:

      # export ETCD_CA_HOST="<SURVIVING_ETCD_HOSTNAME>"
      # export NEW_ETCD="<NEW_ETCD_HOSTNAME>"
      # export NEW_ETCD_IP="<NEW_HOST_IP>"
      
      # etcdctl -C https://${ETCD_CA_HOST}:2379 \
        --ca-file=/etc/etcd/ca.crt     \
        --cert-file=/etc/etcd/peer.crt     \
        --key-file=/etc/etcd/peer.key member add ${NEW_ETCD} https://${NEW_ETCD_IP}:2380
      
      ETCD_NAME="<NEW_ETCD_HOSTNAME>"
      ETCD_INITIAL_CLUSTER="<NEW_ETCD_HOSTNAME>=https://<NEW_HOST_IP>:2380,<SURVIVING_ETCD_HOST>=https:/<SURVIVING_HOST_IP>:2380
      ETCD_INITIAL_CLUSTER_STATE="existing"

      Copy to Clipboard Toggle word wrap

      Copy the three environment variables in the etcdctl member add output. They will be used later.

   2. On the new host, extract the copied configuration data and set the permissions:

      # tar -xf /etc/etcd/<NEW_ETCD_HOSTNAME>.tgz -C /etc/etcd/ --overwrite
      # chown -R etcd:etcd /etc/etcd/*

      Copy to Clipboard Toggle word wrap

   3. On the new host, remove any etcd data:

      # rm -rf /var/lib/etcd/member
      # chown -R etcd:etcd /var/lib/etcd

      Copy to Clipboard Toggle word wrap

9. On the new etcd host, update the etcd.conf file:

   1. Replace the following with the values generated in the previous step:

      • ETCD_NAME
      • ETCD_INITIAL_CLUSTER
      • ETCD_INITIAL_CLUSTER_STATE

   2. Replace the IP address with the "NEW_ETCD" value for:

      • ETCD_LISTEN_PEER_URLS
      • ETCD_LISTEN_CLIENT_URLS
      • ETCD_INITIAL_ADVERTISE_PEER_URLS
      • ETCD_ADVERTISE_CLIENT_URLS
   3. For replacing failed members, replace the failed hosts with the new hosts.
10. To ensure the etcd configuration does not use the failed host when the etcd service is restarted, modify the etcd.conf file on all remaining etcd hosts and remove the failed host in the value for the ETCD_INITIAL_CLUSTER variable.
11. On the node that hosts the installation files, update the [etcd] hosts group in the /etc/ansible/hosts inventory file. Remove the old etcd hosts and add the new ones.

12. Start etcd on the new host:

    # systemctl enable etcd --now

    Copy to Clipboard Toggle word wrap

13. To verify that the new member has been added successfully:

    etcdctl -C https://${ETCD_CA_HOST}:2379 --ca-file=/etc/etcd/ca.crt \
      --cert-file=/etc/etcd/peer.crt     \
      --key-file=/etc/etcd/peer.key cluster-health

    Copy to Clipboard Toggle word wrap

14. Update the master configuration on all masters to point to the new etcd host

    1. On every master in the cluster, edit /etc/origin/master/master-config.yaml
    2. Find the etcdClientInfo section.
    3. Add the new etcd host to the urls list.
    4. If a failed etcd host was replaced, remove it from the list.

    5. Restart the master API service.

       On a single master cluster installation:

       # systemctl restart atomic-openshift-master

       Copy to Clipboard Toggle word wrap

       On a multi-master cluster installation, on each master:

       # systemctl restart atomic-openshift-master-api

       Copy to Clipboard Toggle word wrap

Example of Backing up a Jenkins Deployment’s Application Data

1. Get the application data mountPath from the deploymentconfig:

   $ oc get dc/jenkins -o jsonpath='{ .spec.template.spec.containers[?(@.name=="jenkins")].volumeMounts[?(@.name=="jenkins-data")].mountPath }'
   /var/lib/jenkins

   Copy to Clipboard Toggle word wrap

2. Get the name of the pod that is currently running:

   $ oc get pod --selector=deploymentconfig=jenkins -o jsonpath='{ .metadata.name }'
   jenkins-1-37nux

   Copy to Clipboard Toggle word wrap

3. Use the oc rsync command to copy application data:

   $ oc rsync jenkins-1-37nux:/var/lib/jenkins /tmp/

   Copy to Clipboard Toggle word wrap

Example of Restoring a Jenkins Deployment’s Application Data

1. Verify the backup:

   $ ls -la /tmp/jenkins-backup/
   total 8
   drwxrwxr-x.  3 user     user   20 Sep  6 11:14 .
   drwxrwxrwt. 17 root     root 4096 Sep  6 11:16 ..
   drwxrwsrwx. 12 user     user 4096 Sep  6 11:14 jenkins

   Copy to Clipboard Toggle word wrap

2. Use the oc rsync tool to copy the data into the running pod:

   $ oc rsync /tmp/jenkins-backup/jenkins jenkins-1-37nux:/var/lib

   Copy to Clipboard Toggle word wrap
   Note

   Depending on the application, you may be required to restart the application.

3. Restart the application with new data (optional):

   $ oc delete pod jenkins-1-37nux

   Copy to Clipboard Toggle word wrap

   Alternatively, you can scale down the deployment to 0, and then up again:

   $ oc scale --replicas=0 dc/jenkins
   $ oc scale --replicas=1 dc/jenkins

   Copy to Clipboard Toggle word wrap