Undercloud and Control Plane Back Up and Restore

1. Install the NFS server on the backup node.

   [root@backup ~]# dnf install -y nfs-utils

   Copy to Clipboard Toggle word wrap

2. Add the NFS service to the firewall to ensure ports 111 and 2049 are open. For example:

   [root@backup ~]# firewall-cmd --add-service=nfs
   [root@backup ~]# firewall-cmd --add-service=nfs --permanent

   Copy to Clipboard Toggle word wrap

3. Enable the NFS server and start it.

   [root@backup ~]# systemctl enable nfs-server
   [root@backup ~]# systemctl restart nfs-server

   Copy to Clipboard Toggle word wrap

Procedure

1. Create the backup directory:

   [root@backup ~]# mkdir /ctl_plane_backups

   Copy to Clipboard Toggle word wrap

2. Export the directory. Replace <ip-addr>/24 with the IP address and subnet mask of the network:

   [root@backup ~]# cat >> /etc/exports << EOF
   /ctl_plane_backups <ip-addr>/24(rw,sync,no_root_squash,no_subtree_check)
   EOF

   Copy to Clipboard Toggle word wrap

   The entries in the /etc/exports file are in a space-delimited list. If the undercloud and the overcloud control plane nodes use different networks or subnets, repeat this step for each network or subnet, as shown in this example:

   cat >> /etc/exports << EOF
   /ctl_plane_backups 192.168.24.0/24(rw,sync,no_root_squash,no_subtree_check)
   /ctl_plane_backups 10.0.0.0/24(rw,sync,no_root_squash,no_subtree_check)
   /ctl_plane_backups 172.16.0.0/24(rw,sync,no_root_squash,no_subtree_check)
   EOF

   Copy to Clipboard Toggle word wrap

3. Restart the NFS server:

   [root@backup ~]# systemctl restart nfs-server

   Copy to Clipboard Toggle word wrap

4. Verify that the entries are correctly configured in the NFS server:

   [root@backup ~]# showmount -e `hostname`

   Copy to Clipboard Toggle word wrap

Procedure

1. Install the required packages on the undercloud and on each control plane node. For example:

   [root@controller-x ~]# dnf install rear genisoimage nfs-utils -y

   Copy to Clipboard Toggle word wrap

2. Create a backup directory on the undercloud and on each control plane node. For example:

   [root@controller-x ~]# mkdir -p /ctl_plane_backups

   Copy to Clipboard Toggle word wrap

3. Mount the ctl_plane_backups NFS directory from the backup node running NFS on the undercloud and on each control plane node. For example:

   [root@controller-x ~]# mount -t nfs  <ip-addr>:/ctl_plane_backups /ctl_plane_backups

   Copy to Clipboard Toggle word wrap

   Replace <ip-addr> with the IP address of the backup node running the NFS server.

1. Create the ReaR configuration file:

   [root@controller-x ~]# mkdir -p /etc/rear
   [root@controller-x ~]# tee -a "/etc/rear/local.conf" > /dev/null <<'EOF'
   OUTPUT=ISO
   OUTPUT_URL=nfs://<ip-addr>/ctl_plane_backups
   ISO_PREFIX=<SERVER_NAME-X>
   BACKUP=NETFS
   BACKUP_PROG_COMPRESS_OPTIONS=( --gzip )
   BACKUP_PROG_COMPRESS_SUFFIX=".gz"
   BACKUP_PROG_EXCLUDE=( '/tmp/*' '/data/*' )
   BACKUP_URL=nfs://<ip-addr>/ctl_plane_backups
   BACKUP_PROG_EXCLUDE=("${BACKUP_PROG_EXCLUDE[@]}" '/media' '/var/tmp' '/var/crash')
   BACKUP_PROG_OPTIONS+=( --anchored --xattrs-include='*.*' --xattrs )
   EOF

   Copy to Clipboard Toggle word wrap

   Replace <SERVER_NAME-X> with the hostname of the node. For example, if the node hostname is controller-0, replace <SERVER_NAME-X> with controller-0. Replace <ip-addr> with the IP address of the backup node running the NFS server configured in Chapter 2, Preparing the backup node.

   Important

   If the undercloud or control plane nodes use UEFI as their boot mode, you must add USING_UEFI_BOOTLOADER=1 to the configuration file too.

2. Create the rescue.conf file:

   [root@controller-x ~]# tee  -a "/etc/rear/rescue.conf" > /dev/null <<'EOF'
   BACKUP_PROG_OPTIONS+=( --anchored --xattrs-include='*.*' --xattrs )
   EOF

   Copy to Clipboard Toggle word wrap

Procedure

1. List failures for all available stacks:

   (undercloud) [stack@undercloud-0 ~]$ source stackrc && for i in `openstack stack list -c 'Stack Name' -f value`;  do openstack stack failures list $i; done

   Copy to Clipboard Toggle word wrap

2. Verify that there are no ongoing tasks in the cloud:

   (undercloud) [stack@undercloud-0 ~]$ openstack stack list --nested | grep -v "_COMPLETE"

   Copy to Clipboard Toggle word wrap

   If the command returns no results, there are no ongoing tasks.

3. Stop all OpenStack services in the cloud:

   # systemctl stop tripleo_*

   Copy to Clipboard Toggle word wrap

4. Start the tripleo_mysql service:

   # systemctl start tripleo_mysql

   Copy to Clipboard Toggle word wrap

5. Verify that the tripleo_mysql service is running:

   # systemctl status tripleo_mysql

   Copy to Clipboard Toggle word wrap

Procedure

1. Locate the database password.

   [root@undercloud stack]# PASSWORD=$(/bin/hiera -c /etc/puppet/hiera.yaml mysql::server::root_password)

   Copy to Clipboard Toggle word wrap

2. Back up the databases:

   # podman exec mysql bash -c "mysql -uroot -p$PASSWORD -s -N -e \"SELECT CONCAT('\\\"SHOW GRANTS FOR ''',user,'''@''',host,''';\\\"') FROM mysql.user where (length(user) > 0 and user NOT LIKE 'root')\" | xargs -n1 mysql -uroot -p$PASSWORD -s -N -e | sed 's/$/;/' " > openstack-backup-mysql-grants.sql

   Copy to Clipboard Toggle word wrap

   # podman exec mysql bash -c "mysql -uroot -p$PASSWORD -s -N -e \"select distinct table_schema from information_schema.tables where engine='innodb' and table_schema != 'mysql';\" | xargs mysqldump -uroot -p$PASSWORD --single-transaction --databases" > openstack-backup-mysql.sql

   Copy to Clipboard Toggle word wrap

3. Stop the mariadb database service:

   [root@undercloud stack]# systemctl stop tripleo_mysql

   Copy to Clipboard Toggle word wrap

4. Create the backup:

   [root@undercloud stack]# rear -d -v mkbackup

   Copy to Clipboard Toggle word wrap

   You can find the backup ISO file that you create with ReaR on the backup node in the /ctl_plane_backups directory.

Procedure

1. Locate the database password:

   [heat-admin@overcloud-controller-x ~]# PASSWORD=$(/bin/hiera -c /etc/puppet/hiera.yaml mysql::server::root_password)

   Copy to Clipboard Toggle word wrap

2. Back up the databases:

   [heat-admin@overcloud-controller-x ~]# podman exec galera-bundle-podman-X bash -c "mysql -uroot -p$PASSWORD -s -N -e \"SELECT CONCAT('\\\"SHOW GRANTS FOR ''',user,'''@''',host,''';\\\"') FROM mysql.user where (length(user) > 0 and user NOT LIKE 'root')\" | xargs -n1 mysql -uroot -p$PASSWORD -s -N -e | sed 's/$/;/' " > openstack-backup-mysql-grants.sql

   Copy to Clipboard Toggle word wrap

   [heat-admin@overcloud-controller-x ~]# podman exec galera-bundle-podman-X bash -c "mysql -uroot -p$PASSWORD -s -N -e \"select distinct table_schema from information_schema.tables where engine='innodb' and table_schema != 'mysql';\" | xargs mysqldump -uroot -p$PASSWORD --single-transaction --databases" > openstack-backup-mysql.sql

   Copy to Clipboard Toggle word wrap

3. On one of control plane nodes, stop the pacemaker cluster:

   Important

   Do not operate the stack. When you stop the pacemaker cluster and the containers, this results in the temporary interruption of control plane services to Compute nodes. There is also disruption to network connectivity, Ceph, and the NFS data plane service. You cannot create instances, migrate instances, authenticate requests, or monitor the health of the cluster until the pacemaker cluster and the containers return to service following the final step of this procedure.

   [heat-admin@overcloud-controller-x ~]# pcs cluster stop --all

   Copy to Clipboard Toggle word wrap

4. On each control plane node, stop the containers.

   1. Stop the containers:

      [heat-admin@overcloud-controller-x ~]# systemctl stop tripleo_*

      Copy to Clipboard Toggle word wrap

   2. Stop the ceph-mon@controller.service container:

      [heat-admin@overcloud-controller-x ~]# sudo systemctl stop ceph-mon@$(hostname -s)

      Copy to Clipboard Toggle word wrap

   3. Stop the ceph-mgr@controller.service container:

      [heat-admin@overcloud-controller-x ~]# sudo systemctl stop ceph-mgr@$(hostname -s)

      Copy to Clipboard Toggle word wrap

5. To back up the control plane, run the following command as root in the command line interface of each control plane node:

   [heat-admin@overcloud-controller-x ~]# rear -d -v mkbackup

   Copy to Clipboard Toggle word wrap

   You can find the backup ISO file that you create with ReaR on the backup node in the /ctl_plane_backups directory.

   Note

   When you execute the backup command, you might see warning messages regarding the tar command and sockets that are ignored during the tar process, similar to the following:

   WARNING: tar ended with return code 1 and below output:
   ---snip---
   tar: /var/spool/postfix/public/qmgr: socket ignored
   ...
   ...
   This message indicates that files have been modified during the archiving process and the backup might be inconsistent. Relax-and-Recover continues to operate, however, it is important that you verify the backup to ensure that you can use this backup to recover your system.

   Copy to Clipboard Toggle word wrap

6. When the backup procedure generates ISO images for each of the control plane nodes, restart the pacemaker cluster and the containers:

   On one of the control plane nodes, enter the following command:

   [heat-admin@overcloud-controller-x ~]# pcs cluster start --all

   Copy to Clipboard Toggle word wrap

7. On each control plane node, start the containers.

   1. Start the ceph-mon@controller.service container:

      [heat-admin@overcloud-controller-x ~]# systemctl start ceph-mon@$(hostname -s)

      Copy to Clipboard Toggle word wrap

   2. Start the ceph-mgr@controller.service container:

      [heat-admin@overcloud-controller-x ~]# systemctl start ceph-mgr@$(hostname -s)

      Copy to Clipboard Toggle word wrap

Procedure

1. Shutdown the undercloud node. Ensure that the undercloud node is shutdown completely before you proceed.
2. Restore the undercloud node by booting it with the ISO image created during the backup process. The ISO image is located under the /ctl_plane_backups directory of the Backup node.
3. When the Relax-and-Recover boot menu appears, select Recover <Undercloud Node> where <Undercloud Node> is the name of the undercloud node.

4. Log in as user root.

   The following message displays:

   Welcome to Relax-and-Recover. Run "rear recover" to restore your system!
   RESCUE <Undercloud Node>:~ # rear recover

   Copy to Clipboard Toggle word wrap

   The image restore progresses quickly. When it is complete, the console echoes the following message:

   Finished recovering your system
   Exiting rear recover
   Running exit tasks

   Copy to Clipboard Toggle word wrap

5. When the command line interface is available, the image is restored. Switch the node off.

   RESCUE <Undercloud Node>:~ #  poweroff

   Copy to Clipboard Toggle word wrap

   On boot up, the node resumes with its previous state.

1. Shutdown each control plane node. Ensure that the control plane nodes are shutdown completely before you proceed.
2. Restore the control plane nodes by booting them with the ISO image that you created during the backup process. The ISO images are located under the /ctl_plane_backups directory of the backup node.

3. When the Relax-and-Recover boot menu appears, select Recover <Control Plane Node> where <Control Plane Node> is the name of the control plane node.

   The following message displays:

   Welcome to Relax-and-Recover. Run "rear recover" to restore your system!
   RESCUE <Control Plane Node>:~ # rear recover

   Copy to Clipboard Toggle word wrap

   The image restore progresses quickly. When the restore completes, the console echoes the following message:

   Finished recovering your system
   Exiting rear recover
   Running exit tasks

   Copy to Clipboard Toggle word wrap

   When the command line interface is available, the image is restored. Switch the node off.

   RESCUE <Control Plane Node>:~ #  poweroff

   Copy to Clipboard Toggle word wrap

   Set the boot sequence to the normal boot device. On boot up, the node resumes with its previous state.

4. To ensure that the services are running correctly, check the status of pacemaker. Log in to a controller as root user and run the following command:

   # pcs status

   Copy to Clipboard Toggle word wrap
5. To view the status of the overcloud, use Tempest. For more information about Tempest, see Chapter 4 of the OpenStack Integration Test Suite Guide.