Undercloud and Control Plane Back Up and Restore

1. Install the NFS server on the backup node.

   [root@backup ~]# yum install -y nfs-utils

   Copy to Clipboard Toggle word wrap

2. Add the NFS service to the firewall to ensure ports 111 and 2049 are open. For example:

   [root@backup ~]# firewall-cmd --add-service=nfs
   [root@backup ~]# firewall-cmd --add-service=nfs --permanent

   Copy to Clipboard Toggle word wrap

3. Enable the NFS server and start it.

   [root@backup ~]# systemctl enable nfs-server
   [root@backup ~]# systemctl restart nfs-server

   Copy to Clipboard Toggle word wrap

1. Create the backup directory.

   [root@backup ~]# mkdir /ctl_plane_backups

   Copy to Clipboard Toggle word wrap

2. Export the directory.

   [root@backup ~]# cat > /etc/exports << EOF
   /ctl_plane_backups <ip-addr-x>/24(rw,sync,no_root_squash,no_subtree_check)
   EOF

   Copy to Clipboard Toggle word wrap

   Replace <ip-addr-x> with the IP address of the network. For example, replace <ip-addr-x>/24 with 10.0.0.1/24, where 10.0.0.1/24 is the IP address and subnet mask in CIDR notation. If the undercloud and the overcloud Control Plane nodes use different networks or subnets, repeat this step for each network or subnet.

1. Install the required packages on the undercloud and on each Control Plane node. For example:

   [root@controller-x ~]# yum install rear genisoimage nfs-utils -y

   Copy to Clipboard Toggle word wrap

2. Create a backup directory on the undercloud and on each Control Plane node. For example:

   [root@controller-x ~]# mkdir -p /ctl_plane_backups

   Copy to Clipboard Toggle word wrap

3. Mount the ctl_plane_backups NFS directory from the backup node running NFS on the undercloud and on each Control Plane node. For example:

   [root@controller-x ~]# mount -t nfs  <ip-addr>:/ctl_plane_backups /ctl_plane_backups

   Copy to Clipboard Toggle word wrap

   Replace <ip-addr> with the IP address of the backup node running the NFS server.

1. Create the ReaR configuration file.

   [root@controller-x ~]# mkdir -p /etc/rear
   [root@controller-x ~]# tee -a "/etc/rear/local.conf" > /dev/null <<'EOF'
   OUTPUT=ISO
   OUTPUT_URL=nfs://{{nfs_server_ip}}/ctl_plane_backups
   ISO_PREFIX={{SERVER_NAME-X}}
   BACKUP=NETFS
   BACKUP_PROG_COMPRESS_OPTIONS=( --gzip )
   BACKUP_PROG_COMPRESS_SUFFIX=".gz"
   BACKUP_PROG_EXCLUDE=( '/tmp/*' '/data/*' )
   BACKUP_URL=nfs://{{nfs_server_ip}}/ctl_plane_backups
   BACKUP_PROG_EXCLUDE=("${BACKUP_PROG_EXCLUDE[@]}" '/media' '/var/tmp' '/var/crash')
   BACKUP_PROG_OPTIONS+=( --anchored --xattrs-include='*.*' --xattrs )
   EOF

   Copy to Clipboard Toggle word wrap

   Replace {{SERVER_NAME-X}} with the hostname of the node. For example, if the node hostname is controller-0, replace {{SERVER_NAME-X}} with controller-0. Replace {{nfs_server_ip}} with the IP address of the backup node running the NFS server configured in Chapter 2, Prepare the Backup Node.

   Important

   If the undercloud or Control Plane nodes use UEFI as their boot mode, you MUST add USING_UEFI_BOOTLOADER=1 to the configuration file too.

2. Create the rescue.conf file.

   [root@controller-x ~]# tee  -a "/etc/rear/rescue.conf" > /dev/null <<'EOF'
   BACKUP_PROG_OPTIONS+=( --anchored --xattrs-include='*.*' --xattrs )
   EOF

   Copy to Clipboard Toggle word wrap

1. Locate the database password:

   # /bin/hiera -c /etc/puppet/hiera.yaml mysql::server::root_password

   Copy to Clipboard Toggle word wrap

2. Back up the databases:

   [heat-admin@overcloud-controller-x ~]$ mysql -uroot -p{{ mysql_root_password }} -s -N -e "select distinct table_schema from information_schema.tables where engine='innodb' and table_schema != 'mysql';" | xargs mysqldump -uroot -p{{ mysql_root_password }} --single-transaction --databases > openstack-backup-mysql.sql

   Copy to Clipboard Toggle word wrap

   [heat-admin@overcloud-controller-x ~]$ mysql -uroot -p{{ mysql_root_password }} -s -N -e "SELECT CONCAT('\"SHOW GRANTS FOR ''',user,'''@''',host,''';\"') FROM mysql.user where (length(user) > 0 and user NOT LIKE 'root')" | xargs -n1 mysql -uroot -p{{ mysql_root_password }} -s -N -e | sed 's/$/;/' > openstack-backup-mysql-grants.sql

   Copy to Clipboard Toggle word wrap

3. Stop the pacemaker cluster and the OpenStack services:

   Important

   This procedure results in the temporary interruption of control plane services to Compute nodes. You will not be able to spin up new instances, migrate instances, authenticate requests or monitor the health of the cluster until you restore the pacemaker cluster and OpenStack services in the final step of this procedure.

   On one of the control plane nodes, enter the following command:

   # pcs cluster stop --all

   Copy to Clipboard Toggle word wrap

   On each control plane node, stop the OpenStack services:

   # systemctl stop openstack-*
   # systemctl stop neutron-*
   # systemctl stop haproxy
   # systemctl stop httpd

   Copy to Clipboard Toggle word wrap

4. To back up the control plane, execute the following as root in the command line interface of each control plane node:

   # rear -d -v mkbackup

   Copy to Clipboard Toggle word wrap

   The procedure creates an ISO image on the backup node under the /ctl_plane_backups directory.

5. When the backup procedure generates ISO images for each of the control plane nodes, restart the control plane nodes:

   # reboot

   Copy to Clipboard Toggle word wrap

   After you restart the control plane nodes, check the status of pacemaker to ensure that the services are running correctly:

   # pcs status

   Copy to Clipboard Toggle word wrap

Procedure

1. Shutdown the undercloud node. Ensure that the undercloud node is shutdown completely before proceeding.
2. Restore the undercloud node by booting it with the ISO image created during the backup process. The ISO image is located under the /ctl_plane_backups directory of the Backup node.

3. During the boot phase a boot menu will appear.

   

   Select Recover <Undercloud Node> where <Undercloud Node> is the name of the Undercloud node.

   Note

   DO NOT select Automatic Recover <Undercloud Node>. It could lead to an SELinux context mismatch.

   You should see the following:

   Welcome to Relax-and-Recover. Run "rear recover" to restore your system!
   RESCUE <Undercloud Node>:~ # rear recover

   Copy to Clipboard Toggle word wrap

   The image restore should progress quickly. Once complete, the console should echo the following:

   Finished recovering your system
   Exiting rear recover
   Running exit tasks

   Copy to Clipboard Toggle word wrap

   Once the command line interface is available, the image is restored. Switch the node off.

   RESCUE <Undercloud Node>:~ #  poweroff

   Copy to Clipboard Toggle word wrap

   On boot up, the node will resume with its previous state.

1. Shutdown each Control Plane node. Ensure that the Control Plane nodes are shutdown completely before proceeding.
2. Restore the Control Plane nodes by booting them with the ISO image created during the backup process. The ISO images are located under the /ctl_plane_backups directory of the Backup node.

3. During the boot phase a boot menu will appear.

   

   Select Recover <Control Plane Node> where <Control Plane Node> is the name of the Control Plane node.

   Note

   DO NOT select Automatic Recover <Control Plane Node>. It could lead to an SELinux context mismatch.

   You should see the following:

   Welcome to Relax-and-Recover. Run "rear recover" to restore your system!
   RESCUE <Control Plane Node>:~ # rear recover

   Copy to Clipboard Toggle word wrap

   The image restore should progress quickly. Once complete, the console should echo the following:

   Finished recovering your system
   Exiting rear recover
   Running exit tasks

   Copy to Clipboard Toggle word wrap

   Once the command line interface is available, the image is restored. Switch the node off.

   RESCUE <Control Plane Node>:~ #  poweroff

   Copy to Clipboard Toggle word wrap

   On boot up, the node will resume with its previous state.

4. Check the status of pacemaker to ensure the services are running properly.

   # pcs status

   Copy to Clipboard Toggle word wrap