Chapter 4. Executing the backup procedure

Procedure

1. List failures for all available stacks:

   (undercloud) [stack@undercloud-0 ~]$ source stackrc && for i in `openstack stack list -c 'Stack Name' -f value`;  do openstack stack failures list $i; done

   Copy to Clipboard

2. Verify that there are no ongoing tasks in the cloud:

   (undercloud) [stack@undercloud-0 ~]$ openstack stack list --nested | grep -v "_COMPLETE"

   Copy to Clipboard

   If the command returns no results, there are no ongoing tasks.

3. Stop all OpenStack services in the cloud:

   # systemctl stop tripleo_*

   Copy to Clipboard

4. Start the tripleo_mysql service:

   # systemctl start tripleo_mysql

   Copy to Clipboard

5. Verify that the tripleo_mysql service is running:

   # systemctl status tripleo_mysql

   Copy to Clipboard

Procedure

1. Locate the database password.

   [root@undercloud stack]# PASSWORD=$(/bin/hiera -c /etc/puppet/hiera.yaml mysql::server::root_password)

   Copy to Clipboard

2. Back up the databases:

   # podman exec mysql bash -c "mysql -uroot -p$PASSWORD -s -N -e \"SELECT CONCAT('\\\"SHOW GRANTS FOR ''',user,'''@''',host,''';\\\"') FROM mysql.user where (length(user) > 0 and user NOT LIKE 'root')\" | xargs -n1 mysql -uroot -p$PASSWORD -s -N -e | sed 's/$/;/' " > openstack-backup-mysql-grants.sql

   Copy to Clipboard

   # podman exec mysql bash -c "mysql -uroot -p$PASSWORD -s -N -e \"select distinct table_schema from information_schema.tables where engine='innodb' and table_schema != 'mysql';\" | xargs mysqldump -uroot -p$PASSWORD --single-transaction --databases" > openstack-backup-mysql.sql

   Copy to Clipboard

3. Stop the mariadb database service:

   [root@undercloud stack]# systemctl stop tripleo_mysql

   Copy to Clipboard

4. Create the backup:

   [root@undercloud stack]# rear -d -v mkbackup

   Copy to Clipboard

   You can find the backup ISO file that you create with ReaR on the backup node in the /ctl_plane_backups directory.

Procedure

1. Locate the database password:

   [heat-admin@overcloud-controller-x ~]# PASSWORD=$(/bin/hiera -c /etc/puppet/hiera.yaml mysql::server::root_password)

   Copy to Clipboard

2. Back up the databases:

   [heat-admin@overcloud-controller-x ~]# podman exec galera-bundle-podman-X bash -c "mysql -uroot -p$PASSWORD -s -N -e \"SELECT CONCAT('\\\"SHOW GRANTS FOR ''',user,'''@''',host,''';\\\"') FROM mysql.user where (length(user) > 0 and user NOT LIKE 'root')\" | xargs -n1 mysql -uroot -p$PASSWORD -s -N -e | sed 's/$/;/' " > openstack-backup-mysql-grants.sql

   Copy to Clipboard

   [heat-admin@overcloud-controller-x ~]# podman exec galera-bundle-podman-X bash -c "mysql -uroot -p$PASSWORD -s -N -e \"select distinct table_schema from information_schema.tables where engine='innodb' and table_schema != 'mysql';\" | xargs mysqldump -uroot -p$PASSWORD --single-transaction --databases" > openstack-backup-mysql.sql

   Copy to Clipboard

3. On one of control plane nodes, stop the pacemaker cluster:

   Important

   Do not operate the stack. When you stop the pacemaker cluster and the containers, this results in the temporary interruption of control plane services to Compute nodes. There is also disruption to network connectivity, Ceph, and the NFS data plane service. You cannot create instances, migrate instances, authenticate requests, or monitor the health of the cluster until the pacemaker cluster and the containers return to service following the final step of this procedure.

   [heat-admin@overcloud-controller-x ~]# pcs cluster stop --all

   Copy to Clipboard

4. On each control plane node, stop the containers.

   1. Stop the containers:

      [heat-admin@overcloud-controller-x ~]# systemctl stop tripleo_*

      Copy to Clipboard

   2. Stop the ceph-mon@controller.service container:

      [heat-admin@overcloud-controller-x ~]# sudo systemctl stop ceph-mon@$(hostname -s)

      Copy to Clipboard

   3. Stop the ceph-mgr@controller.service container:

      [heat-admin@overcloud-controller-x ~]# sudo systemctl stop ceph-mgr@$(hostname -s)

      Copy to Clipboard

5. To back up the control plane, run the following command as root in the command line interface of each control plane node:

   [heat-admin@overcloud-controller-x ~]# rear -d -v mkbackup

   Copy to Clipboard

   You can find the backup ISO file that you create with ReaR on the backup node in the /ctl_plane_backups directory.

   Note

   When you execute the backup command, you might see warning messages regarding the tar command and sockets that are ignored during the tar process, similar to the following:

   WARNING: tar ended with return code 1 and below output:
   ---snip---
   tar: /var/spool/postfix/public/qmgr: socket ignored
   ...
   ...
   This message indicates that files have been modified during the archiving process and the backup might be inconsistent. Relax-and-Recover continues to operate, however, it is important that you verify the backup to ensure that you can use this backup to recover your system.

   Copy to Clipboard

6. When the backup procedure generates ISO images for each of the control plane nodes, restart the pacemaker cluster and the containers:

   On one of the control plane nodes, enter the following command:

   [heat-admin@overcloud-controller-x ~]# pcs cluster start --all

   Copy to Clipboard

7. On each control plane node, start the containers.

   1. Start the ceph-mon@controller.service container:

      [heat-admin@overcloud-controller-x ~]# systemctl start ceph-mon@$(hostname -s)

      Copy to Clipboard

   2. Start the ceph-mgr@controller.service container:

      [heat-admin@overcloud-controller-x ~]# systemctl start ceph-mgr@$(hostname -s)

      Copy to Clipboard