Chapter 22. Backup and Restore

1. Because the restore procedure involves a complete reinstallation, save all the files used in the initial installation. This may include:

   • ~/.config/openshift/installer.cfg.yml (from the Quick Installation method)
   • Ansible playbooks and inventory files (from the Advanced Installation method)
   • /etc/yum.repos.d/ose.repo (from the Disconnected Installation method)
2. Backup the procedures for post-installation steps. Some installations may involve steps that are not included in the installer. This may include changes to the services outside of the control of OpenShift Enterprise or the installation of extra services like monitoring agents. Additional configuration that is not supported yet by the advanced installer might also be affected, for example when using multiple authentication providers.

3. Install packages that provide various utility commands:

   # yum install etcd

   Copy to Clipboard Toggle word wrap

4. If using a container-based installation, pull the etcd image instead:

   # docker pull rhel7/etcd

   Copy to Clipboard Toggle word wrap

1. Save all the certificates and keys, on each master:

   # cd /etc/origin/master
   # tar cf /tmp/certs-and-keys-$(hostname).tar *.key *.crt

   Copy to Clipboard Toggle word wrap

2. If etcd is running on more than one host, stop it on each host:

   # sudo systemctl stop etcd

   Copy to Clipboard Toggle word wrap

   Although this step is not strictly necessary, doing so ensures that the etcd data is fully synchronized.

3. Create an etcd backup:

   # etcdctl backup \
       --data-dir $ETCD_DATA_DIR \
       --backup-dir $ETCD_DATA_DIR.bak

   Copy to Clipboard Toggle word wrap
   Note

   If etcd is running on more than one host, the various instances regularly synchronize their data, so creating a backup for one of them is sufficient.

   Note

   For a container-based installation, you must use docker exec to run etcdctl inside the container.

4. Copy the db file over to the backup you created:

   # cp "$ETCD_DATA_DIR"/member/snap/db "$ETCD_DATA_DIR.bak"/member/snap/db

   Copy to Clipboard Toggle word wrap

1. Reinstall OpenShift Enterprise.

   This should be done in the same way that OpenShift Enterprise was previously installed.

2. Run all necessary post-installation steps.

3. Restore the certificates and keys, on each master:

   # cd /etc/origin/master
   # tar xvf /tmp/certs-and-keys-$(hostname).tar

   Copy to Clipboard Toggle word wrap

4. Restore from the etcd backup:

   # mv $ETCD_DATA_DIR $ETCD_DATA_DIR.orig
   # cp -Rp $ETCD_DATA_DIR.bak $ETCD_DATA_DIR
   # chcon -R --reference $ETCD_DATA_DIR.orig $ETCD_DATA_DIR
   # chown -R etcd:etcd $ETCD_DATA_DIR

   Copy to Clipboard Toggle word wrap

5. Create the new single node cluster using etcd’s --force-new-cluster option. You can do this using the values from /etc/etcd/etcd.conf, or you can temporarily modify the systemd unit file and start the service normally.

   To do so, edit the /usr/lib/systemd/system/etcd.service file, and add --force-new-cluster:

   # sed -i '/ExecStart/s/"$/  --force-new-cluster"/' /usr/lib/systemd/system/etcd.service
   # systemctl show etcd.service --property ExecStart --no-pager
   
   ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/bin/etcd --force-new-cluster"

   Copy to Clipboard Toggle word wrap

   Then, restart the etcd service:

   # systemctl daemon-reload
   # systemctl start etcd

   Copy to Clipboard Toggle word wrap

6. Verify the etcd service started correctly, then re-edit the /usr/lib/systemd/system/etcd.service file and remove the --force-new-cluster option:

   # sed -i '/ExecStart/s/ --force-new-cluster//' /usr/lib/systemd/system/etcd.service
   # systemctl show etcd.service --property ExecStart --no-pager
   
   ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/bin/etcd"

   Copy to Clipboard Toggle word wrap

7. Restart the etcd service, then verify the etcd cluster is running correctly and displays OpenShift Enterprise’s configuration:

   # systemctl daemon-reload
   # systemctl restart etcd

   Copy to Clipboard Toggle word wrap

Example of Backing up a Jenkins Deployment’s Application Data

1. Get the application data mountPath from the deploymentconfig:

   $ oc get dc/jenkins -o jsonpath='{ .spec.template.spec.containers[?(@.name=="jenkins")].volumeMounts[?(@.name=="jenkins-data")].mountPath }'
   /var/lib/jenkins

   Copy to Clipboard Toggle word wrap

2. Get the name of the pod that is currently running:

   $ oc get pod --selector=deploymentconfig=jenkins -o jsonpath='{ .metadata.name }'
   jenkins-1-37nux

   Copy to Clipboard Toggle word wrap

3. Use the oc rsync command to copy application data:

   $ oc rsync jenkins-1-37nux:/var/lib/jenkins /tmp/

   Copy to Clipboard Toggle word wrap

Example of Restoring a Jenkins Deployment’s Application Data

1. Verify the backup:

   $ ls -la /tmp/jenkins-backup/
   total 8
   drwxrwxr-x.  3 user     user   20 Sep  6 11:14 .
   drwxrwxrwt. 17 root     root 4096 Sep  6 11:16 ..
   drwxrwsrwx. 12 user     user 4096 Sep  6 11:14 jenkins

   Copy to Clipboard Toggle word wrap

2. Use the oc rsync tool to copy the data into the running pod:

   $ oc rsync /tmp/jenkins-backup/jenkins jenkins-1-37nux:/var/lib

   Copy to Clipboard Toggle word wrap
   Note

   Depending on the application, you may be required to restart the application.

3. Restart the application with new data (optional):

   $ oc delete pod jenkins-1-37nux

   Copy to Clipboard Toggle word wrap

   Alternatively, you can scale down the deployment to 0, and then up again:

   $ oc scale --replicas=0 dc/jenkins
   $ oc scale --replicas=1 dc/jenkins

   Copy to Clipboard Toggle word wrap