Chapter 4. Executing the back up procedure

Procedure

1. Confirm that there are no failures on the stack. Replace <STACKNAME> with the name of the stack. Use the command for every stack that is deployed and available:

   (undercloud) [stack@undercloud-0 ~]$ openstack stack failures list <STACKNAME>

   Copy to Clipboard Toggle word wrap

2. Verify that there are no ongoing tasks on the undercloud:

   (undercloud) [stack@undercloud-0 ~]$ openstack stack list --nested | grep -v "_COMPLETE"

   Copy to Clipboard Toggle word wrap

   If the command returns no results, there are no ongoing tasks.

3. Stop all OpenStack services on the undercloud:

   # systemctl stop openstack-*
   # systemctl stop neutron-*
   # systemctl stop ironic*
   # systemctl stop haproxy
   # systemctl stop httpd

   Copy to Clipboard Toggle word wrap

4. Verify that mariadb is running:

   # sudo systemctl status mariadb

   Copy to Clipboard Toggle word wrap

Procedure

1. Locate the database password:

   [root@undercloud stack]# PASSWORD=$(sudo /bin/hiera -c /etc/puppet/hiera.yaml mysql::server::root_password)

   Copy to Clipboard Toggle word wrap

2. Back up the databases:

   [root@undercloud stack]# mysql -uroot -p$PASSWORD -s -N -e "select distinct table_schema from information_schema.tables where engine='innodb' and table_schema != 'mysql';" | xargs mysqldump -uroot -p$PASSWORD --single-transaction --databases > openstack-backup-mysql.sql

   Copy to Clipboard Toggle word wrap

   [root@undercloud stack]# mysql -uroot -p$PASSWORD -s -N -e "SELECT CONCAT('\"SHOW GRANTS FOR ''',user,'''@''',host,''';\"') FROM mysql.user where (length(user) > 0 and user NOT LIKE 'root')" | xargs -n1 mysql -uroot -p$PASSWORD -s -N -e | sed 's/$/;/' > openstack-backup-mysql-grants.sql

   Copy to Clipboard Toggle word wrap

3. Stop the mariadb database service:

   [root@undercloud stack]# systemctl stop mariadb

   Copy to Clipboard Toggle word wrap

4. Create the backup:

   [root@undercloud stack]# rear -d -v mkbackup

   Copy to Clipboard Toggle word wrap

   You can find the backup ISO file that you create with ReaR on the backup node under the /ctl_plane_backups directory.

5. Restart the undercloud:

   1. Log in to the undercloud as the stack user.

   2. Restart the undercloud:

      [stack@undercloud]$ sudo reboot

      Copy to Clipboard Toggle word wrap

Procedure

1. Locate the database password:

   [heat-admin@overcloud-controller-x ~]# PASSWORD=$(sudo /bin/hiera -c /etc/puppet/hiera.yaml mysql::server::root_password)

   Copy to Clipboard Toggle word wrap

2. Back up the databases:

   [heat-admin@overcloud-controller-x ~]# mysql -uroot -p$PASSWORD -s -N -e "select distinct table_schema from information_schema.tables where engine='innodb' and table_schema != 'mysql';" | xargs mysqldump -uroot -p$PASSWORD --single-transaction --databases > openstack-backup-mysql.sql

   Copy to Clipboard Toggle word wrap

   [heat-admin@overcloud-controller-x ~]# mysql -uroot -p$PASSWORD -s -N -e "SELECT CONCAT('\"SHOW GRANTS FOR ''',user,'''@''',host,''';\"') FROM mysql.user where (length(user) > 0 and user NOT LIKE 'root')" | xargs -n1 mysql -uroot -p$PASSWORD -s -N -e | sed 's/$/;/' > openstack-backup-mysql-grants.sql

   Copy to Clipboard Toggle word wrap
   Note

   Backing up the databases is a precautionary measure. This step ensures that you can manually restore the Galera cluster if it does not restore automatically as part of the restoration procedure. For more information about restoring the Galera cluster, see Troubleshooting the Galera cluster.

3. On one of the control plane nodes, stop the pacemaker cluster:

   Important

   Do not operate the stack. When you stop the pacemaker cluster and the containers, this results in the temporary interruption of control plane services to Compute nodes. There is also disruption to network connectivity, Ceph, and the NFS data plane service. You cannot create instances, migrate instances, authenticate requests, or monitor the health of the cluster until the pacemaker cluster and the containers return to service following the final step of this procedure.

   [heat-admin@overcloud-controller-x ~]# sudo pcs cluster stop --all

   Copy to Clipboard Toggle word wrap

4. On each control plane node, stop the containers:

   1. Stop the containers:

      [heat-admin@overcloud-controller-x ~]# sudo docker stop $(sudo docker ps -a -q)

      Copy to Clipboard Toggle word wrap

   2. Stop the ceph-mon@controller.service container:

      [heat-admin@overcloud-controller-x ~]# sudo systemctl stop ceph-mon@$(hostname -s)

      Copy to Clipboard Toggle word wrap

   3. Stop the ceph-mgr@controller.service container:

      [heat-admin@overcloud-controller-x ~]# sudo systemctl stop ceph-mgr@$(hostname -s)

      Copy to Clipboard Toggle word wrap

5. Optional: If you use ganesha-nfs, disable the file server on one controller:

   [heat-admin@overcloud-controller-x ~]# sudo pcs resource disable ceph-nfs

   Copy to Clipboard Toggle word wrap

6. Optional: If you use the ceph services ceph-mds and ceph-rgw, stop these services:

   [heat-admin@overcloud-controller-x ~]# sudo systemctl stop ceph-mds@$(hostname -s)
   
   [heat-admin@overcloud-controller-x ~]# sudo systemctl stop ceph-rgw@$(hostname -s)

   Copy to Clipboard Toggle word wrap

7. To back up the control plane, run the control plane backup on each control plane node:

   [heat-admin@overcloud-controller-x ~]# sudo rear -d -v mkbackup

   Copy to Clipboard Toggle word wrap

   You can find the backup ISO file that you create with ReaR on the backup node under the /ctl_plane_backups directory.

   Note

   When you execute the backup command, you might see warning messages regarding the tar command and sockets that are ignored during the tar process, similar to the following warning:

   WARNING: tar ended with return code 1 and below output:
   ---snip---
   tar: /var/spool/postfix/public/qmgr: socket ignored
   ...
   ...
   This message indicates that files have been modified during the archiving process and the backup might be inconsistent. Relax-and-Recover continues to operate, however, it is important that you verify the backup to ensure that you can use this backup to recover your system.

   Copy to Clipboard Toggle word wrap

8. When the backup procedure generates ISO images for each of the control plane nodes, restart the pacemaker cluster. On one of the control plane nodes, enter the following command:

   [heat-admin@overcloud-controller-x ~]# sudo pcs cluster start --all

   Copy to Clipboard Toggle word wrap

9. On each control plane node, start the containers:

   1. Start the containers:

      [heat-admin@overcloud-controller-x ~]# sudo systemctl restart docker

      Copy to Clipboard Toggle word wrap

   2. Start the ceph-mon@controller.service container:

      [heat-admin@overcloud-controller-x ~]# sudo systemctl start ceph-mon@$(hostname -s)

      Copy to Clipboard Toggle word wrap

   3. Start the ceph-mgr@controller.service container:

      [heat-admin@overcloud-controller-x ~]# sudo systemctl start ceph-mgr@$(hostname -s)

      Copy to Clipboard Toggle word wrap

10. Optional: If you use ceph-mds and ceph-rgw, start these services:

    [heat-admin@overcloud-controller-x ~]# sudo systemctl start ceph-rgw@$(hostname -s)
    [heat-admin@overcloud-controller-x ~]# sudo systemctl start ceph-mds@$(hostname -s)

    Copy to Clipboard Toggle word wrap

11. Optional: If you use ganesha-nfs, enable the file server on one controller:

    [heat-admin@overcloud-controller-x ~]# sudo pcs resource enable ceph-nfs

    Copy to Clipboard Toggle word wrap