Chapter 2. Backing up the undercloud and the control plane nodes by using the Relax-and-Recover tool

Procedure

1. Log in to the undercloud as the stack user.

2. Source the stackrc file:

   $ source ~/stackrc

3. Create the bar-vars.yaml file:

   touch /home/stack/bar-vars.yaml

4. In the bar-vars.yaml file, configure the backup storage location:

   1. If you use an NFS server, add the following parameters and set the values of the IP address of your NFS server and backup storage folder:

      tripleo_backup_and_restore_server: <ip_address>
      tripleo_backup_and_restore_shared_storage_folder: <backup_dir>

      • Replace <ip_address> and <backup_dir> with the values that apply to your environment. By default, the tripleo_backup_and_restore_server parameter value is 192.168.24.1.

   2. If you use an SFTP server, add the tripleo_backup_and_restore_output_url parameter and set the values of the URL and credentials of the SFTP server:

      tripleo_backup_and_restore_output_url: sftp://<user>:<password>@<backup_node>/
      tripleo_backup_and_restore_backup_url: iso:///backup/

      Replace <user>, <password>, and <backup_node> with the backup node URL and credentials.

Procedure

1. On the undercloud node, source the undercloud credentials:

   [stack@undercloud-0 ~]$ source stackrc
   (undercloud) [stack@undercloud ~]$

2. On the undercloud node, create an inventory file for the backup node:

   (undercloud) [stack@undercloud ~]$ cat <<'EOF'> ~/nfs-inventory.yaml
   [BackupNode]
   <backup_node> ansible_host=<ip_address> ansible_user=<user>
   EOF

   Replace <backup_node>, <ip_address>, and <user> with the values that apply to your environment.

3. Copy the public SSH key from the undercloud node to the backup node.

   (undercloud) [stack@undercloud ~]$ ssh-copy-id -i ~/.ssh/id_rsa.pub <backup_node>

   Replace <backup_node> with the path and name of the backup node.

4. Configure the NFS server on the backup node:

   (undercloud) [stack@undercloud ~]$ openstack undercloud backup --setup-nfs --extra-vars /home/stack/bar-vars.yaml --inventory /home/stack/nfs-inventory.yaml

Procedure

1. On the undercloud node, source the undercloud credentials:

   [stack@undercloud-0 ~]$ source stackrc

2. If you have not done so before, extract the static ansible inventory file from the location in which it was saved during installation:

   (undercloud) [stack@undercloud ~]$ cp ~/overcloud-deploy/<stack>/tripleo-ansible-inventory.yaml ~/tripleo-inventory.yaml

   • Replace <stack> with the name of your stack. By default, the name of the stack is overcloud.

3. Install ReaR on the undercloud node:

   (undercloud) [stack@undercloud ~]$ openstack undercloud backup --setup-rear --extra-vars /home/stack/bar-vars.yaml --inventory /home/stack/tripleo-inventory.yaml

4. If your system uses the UEFI boot loader, perform the following steps on the undercloud node:

   1. Install the following tools:

      $ sudo dnf install dosfstools efibootmgr

   2. Enable UEFI backup in the ReaR configuration file located in /etc/rear/local.conf by replacing the USING_UEFI_BOOTLOADER parameter value 0 with the value 1 and adding UEFI_BOOTLOADER=/boot/efi/EFI/redhat/shimx64.efi.

Procedure

1. Log in to the undercloud as the stack user.

2. Retrieve the MySQL root password:

   [stack@undercloud ~]$ PASSWORD=$(sudo /bin/hiera -c /etc/puppet/hiera.yaml mysql::server::root_password)

3. Create a database backup of the undercloud node:

   [stack@undercloud ~]$ sudo podman exec mysql bash -c "mysqldump -uroot -p$PASSWORD --opt --all-databases" | sudo tee /root/undercloud-all-databases.sql

4. On the undercloud node, source the undercloud credentials:

   [stack@undercloud-0 ~]$ source stackrc

5. Create a backup of the undercloud node:

   (undercloud) [stack@undercloud ~]$ openstack undercloud backup
   --inventory /home/stack/tripleo-inventory.yaml

Procedure

1. To schedule a backup of your control plane nodes, run the following command. The default schedule is Sundays at midnight:

   openstack undercloud backup --cron

2. Optional: Customize the scheduled backup according to your deployment:

   • To change the default backup schedule, pass a different cron schedule on the tripleo_backup_and_restore_cron parameter:

     openstack undercloud backup --cron --extra-vars
     '{"tripleo_backup_and_restore_cron": "0 0 * * 0"}'

   • To define additional parameters that are added to the backup command when cron runs the scheduled backup, pass the tripleo_backup_and_restore_cron_extra parameter to the backup command, as shown in the following example:

     openstack undercloud backup --cron --extra-vars '{"tripleo_backup_and_restore_cron_extra":"--extra-vars bar-vars.yaml --inventory /home/stack/tripleo-inventory.yaml"}'

   • To change the default user that executes the backup, pass the tripleo_backup_and_restore_cron_user parameter to the backup command, as shown in the following example:

     openstack undercloud backup --cron --extra-vars '{"tripleo_backup_and_restore_cron_user": "root"}

Procedure

1. On the undercloud node, source the undercloud credentials:

   [stack@undercloud-0 ~]$ source stackrc
   (undercloud) [stack@undercloud ~]$

2. On the undercloud node, create an inventory file for the backup node:

   (undercloud) [stack@undercloud ~]$ cat <<'EOF'> ~/nfs-inventory.yaml
   [BackupNode]
   <backup_node> ansible_host=<ip_address> ansible_user=<user>
   EOF

   Replace <backup_node>, <ip_address>, and <user> with the values that apply to your environment.

3. Copy the public SSH key from the undercloud node to the backup node.

   (undercloud) [stack@undercloud ~]$ ssh-copy-id -i ~/.ssh/id_rsa.pub <backup_node>

   Replace <backup_node> with the path and name of the backup node.

4. Configure the NFS server on the backup node:

   (undercloud) [stack@undercloud ~]$ openstack undercloud backup --setup-nfs --extra-vars /home/stack/bar-vars.yaml --inventory /home/stack/nfs-inventory.yaml

Procedure

1. On the undercloud node, source the undercloud credentials:

   [stack@undercloud-0 ~]$ source stackrc

2. If you have not done so before, extract the static ansible inventory file from the location in which it was saved during installation:

   (undercloud) [stack@undercloud ~]$ cp ~/overcloud-deploy/<stack>/tripleo-ansible-inventory.yaml ~/tripleo-inventory.yaml

   • Replace <stack> with the name of your stack. By default, the name of the stack is overcloud.

3. In the bar-vars.yaml file, configure the backup storage location:

   • If you installed and configured your own NFS server, add the tripleo_backup_and_restore_server parameter and set the value to the IP address of your NFS server:

     tripleo_backup_and_restore_server: <ip_address>
     tripleo_backup_and_restore_shared_storage_folder: <backup_dir>

     • Replace <ip_address> and <backup_dir> with the values that apply to your environment. By default, the tripleo_backup_and_restore_server parameter value is 192.168.24.1.*

   • If you use an SFTP server, add the tripleo_backup_and_restore_output_url parameter and set the values of the URL and credentials of the SFTP server:

     tripleo_backup_and_restore_output_url: sftp://<user>:<password>@<backup_node>/
     tripleo_backup_and_restore_backup_url: iso:///backup/

     Replace <user>, <password>, and <backup_node> with the backup node URL and credentials.

4. Install ReaR on the control plane nodes:

   (undercloud) [stack@undercloud ~]$ openstack overcloud backup --setup-rear --extra-vars /home/stack/bar-vars.yaml --inventory /home/stack/tripleo-inventory.yaml

5. If your system uses the UEFI boot loader, perform the following steps on the control plane nodes:

   1. Install the following tools:

      $ sudo dnf install dosfstools efibootmgr

   2. Enable UEFI backup in the ReaR configuration file located in /etc/rear/local.conf by replacing the USING_UEFI_BOOTLOADER parameter value 0 with the value 1 and adding UEFI_BOOTLOADER=/boot/efi/EFI/redhat/shimx64.efi.

Procedure

1. Locate the config-drive partition on each control plane node:

   [stack@undercloud-0 ~]$ blkid -t LABEL="config-2" -odevice

2. On each control plane node, back up the config-drive partition of each node as the root user:

   [root@controller-x ~]# dd if=<config_drive_partition> of=/mnt/config-drive

   Replace <config_drive_partition> with the name of the config-drive partition that you located in step 1.

3. On the undercloud node, source the undercloud credentials:

   [stack@undercloud-0 ~]$ source stackrc

4. Create a backup of the control plane nodes:

   (undercloud) [stack@undercloud ~]$ openstack overcloud backup --inventory /home/stack/tripleo-inventory.yaml

   The backup process runs sequentially on each control plane node without disrupting the service to your environment.

Procedure

1. To schedule a backup of your control plane nodes, run the following command. The default schedule is Sundays at midnight:

   openstack overcloud backup --cron

2. Optional: Customize the scheduled backup according to your deployment:

   • To change the default backup schedule, pass a different cron schedule on the tripleo_backup_and_restore_cron parameter:

     openstack overcloud backup --cron --extra-vars
     '{"tripleo_backup_and_restore_cron": "0 0 * * 0"}'

   • To define additional parameters that are added to the backup command when cron runs the scheduled backup, pass the tripleo_backup_and_restore_cron_extra parameter to the backup command, as shown in the following example:

     openstack overcloud backup --cron --extra-vars '{"tripleo_backup_and_restore_cron_extra":"--extra-vars bar-vars.yaml --inventory /home/stack/tripleo-inventory.yaml"}'

   • To change the default user that executes the backup, pass the tripleo_backup_and_restore_cron_user parameter to the backup command, as shown in the following example:

     openstack overcloud backup --cron --extra-vars '{"tripleo_backup_and_restore_cron_user": "root"}

Procedure

1. Power off the undercloud node. Ensure that the undercloud node is powered off completely before you proceed.
2. Boot the undercloud node with the backup ISO image.

3. When the Relax-and-Recover boot menu displays, select Recover <undercloud_node>. Replace <undercloud_node> with the name of your undercloud node.

   Note

   If your system uses UEFI, select the Relax-and-Recover (no Secure Boot) option.

4. Log in as the root user and restore the node:

   The following message displays:

   Welcome to Relax-and-Recover. Run "rear recover" to restore your system!
   RESCUE <undercloud_node>:~ # rear recover

   When the undercloud node restoration process completes, the console displays the following message:

   Finished recovering your system
   Exiting rear recover
   Running exit tasks

5. Power off the node:

   RESCUE <undercloud_node>:~ #  poweroff

   On boot up, the node resumes its previous state.

Procedure

1. Power off each control plane node. Ensure that the control plane nodes are powered off completely before you proceed.
2. Boot each control plane node with the corresponding backup ISO image.

3. When the Relax-and-Recover boot menu displays, on each control plane node, select Recover <control_plane_node>. Replace <control_plane_node> with the name of the corresponding control plane node.

   Note

   If your system uses UEFI, select the Relax-and-Recover (no Secure Boot) option.

4. On each control plane node, log in as the root user and restore the node:

   The following message displays:

   Welcome to Relax-and-Recover. Run "rear recover" to restore your system!
   RESCUE <control_plane_node>:~ # rear recover

   When the control plane node restoration process completes, the console displays the following message:

   Finished recovering your system
   Exiting rear recover
   Running exit tasks

5. When the command line console is available, restore the config-drive partition of each control plane node:

   # once completed, restore the config-drive partition (which is ISO9660)
   RESCUE <control_plane_node>:~ $ dd if=/mnt/local/mnt/config-drive of=<config_drive_partition>

6. Power off the node:

   RESCUE <control_plane_node>:~ #  poweroff

7. Set the boot sequence to the normal boot device. On boot up, the node resumes its previous state.

8. To ensure that the services are running correctly, check the status of pacemaker. Log in to a Controller node as the root user and enter the following command:

   # pcs status

9. To view the status of the overcloud, use the OpenStack Integration Test Suite (tempest). For more information, see Validating your OpenStack cloud with the Integration Test Suite (tempest).

Procedure

1. On Controller-0, retrieve the Galera cluster virtual IP:

   $ sudo hiera -c /etc/puppet/hiera.yaml mysql_vip

2. Disable the database connections through the virtual IP on all Controller nodes:

   $ sudo iptables -I INPUT -p tcp --destination-port 3306 -d $MYSQL_VIP=<galera_cluster_vip> -j DROP

   • Replace <galera_cluster_vip> with the IP address you retrieved in step 1.

3. On Controller-0, retrieve the MySQL root password:

   $ sudo hiera -c /etc/puppet/hiera.yaml mysql::server::root_password

4. On Controller-0, set the Galera resource to unmanaged mode:

   $ sudo pcs resource unmanage galera-bundle

5. Stop the MySQL containers on all Controller nodes:

   $ sudo podman container stop $(sudo podman container ls --all --format "{{.Names}}" --filter=name=galera-bundle)

6. Move the current directory on all Controller nodes:

   $ sudo mv /var/lib/mysql /var/lib/mysql-save

7. Create the new directory /var/lib/mysq on all Controller nodes:

   $ sudo mkdir /var/lib/mysql
   $ sudo chown 42434:42434 /var/lib/mysql
   $ sudo chcon -t container_file_t /var/lib/mysql
   $ sudo chmod 0755 /var/lib/mysql
   $ sudo chcon -r object_r /var/lib/mysql
   $ sudo chcon -u system_u /var/lib/mysql

8. Start the MySQL containers on all Controller nodes:

   $ sudo podman container start $(sudo podman container ls --all --format "{{ .Names }}" --filter=name=galera-bundle)

9. Create the MySQL database on all Controller nodes:

   $ sudo podman exec -i $(sudo podman container ls --all --format "{{ .Names }}" \
         --filter=name=galera-bundle) bash -c "mysql_install_db --datadir=/var/lib/mysql --user=mysql --log_error=/var/log/mysql/mysql_init.log"

10. Start the database on all Controller nodes:

    $ sudo podman exec $(sudo podman container ls --all --format "{{ .Names }}" \
          --filter=name=galera-bundle) bash -c "mysqld_safe --skip-networking --wsrep-on=OFF --log-error=/var/log/mysql/mysql_safe.log" &

11. Move the .my.cnf Galera configuration file on all Controller nodes:

    $ sudo podman exec $(sudo podman container ls --all --format "{{ .Names }}" \
          --filter=name=galera-bundle) bash -c "mv /root/.my.cnf /root/.my.cnf.bck"

12. Reset the Galera root password on all Controller nodes:

    $ sudo podman exec $(sudo podman container ls --all --format "{{ .Names }}"  \
          --filter=name=galera-bundle) bash -c "mysql -uroot -e'use mysql;set password for root@localhost = password(\"$ROOTPASSWORD\");flush privileges;'"

13. Restore the .my.cnf Galera configuration file inside the Galera container on all Controller nodes:

    $ sudo podman exec $(sudo podman container ls --all --format "{{ .Names }}"   \
          --filter=name=galera-bundle) bash -c "mv /root/.my.cnf.bck /root/.my.cnf"

14. On Controller-0, copy the backup database files to /var/lib/MySQL:

    $ sudo cp openstack-backup-mysql.sql /var/lib/mysql
    $ sudo cp openstack-backup-mysql-grants.sql /var/lib/mysql

    Note

    The path to these files is /home/tripleo-admin/.

15. On Controller-0, restore the MySQL database:

    $ sudo podman exec $(sudo podman container ls --all --format "{{ .Names }}"    \
    --filter=name=galera-bundle) bash -c "mysql -u root -p$ROOT_PASSWORD < \"/var/lib/mysql/$BACKUP_FILE\"  "
    
    $ sudo podman exec $(sudo podman container ls --all --format "{{ .Names }}"    \
    --filter=name=galera-bundle) bash -c "mysql -u root -p$ROOT_PASSWORD < \"/var/lib/mysql/$BACKUP_GRANT_FILE\"  "

16. Shut down the databases on all Controller nodes:

    $ sudo podman exec $(sudo podman container ls --all --format "{{ .Names }}"    \
          --filter=name=galera-bundle) bash -c "mysqladmin shutdown"

17. On Controller-0, start the bootstrap node:

    $ sudo podman exec $(sudo podman container ls --all --format "{{ .Names }}"  --filter=name=galera-bundle) \
            /usr/bin/mysqld_safe --pid-file=/var/run/mysql/mysqld.pid --socket=/var/lib/mysql/mysql.sock --datadir=/var/lib/mysql \
            --log-error=/var/log/mysql/mysql_cluster.log  --user=mysql --open-files-limit=16384 \
            --wsrep-cluster-address=gcomm:// &

18. Verification: On Controller-0, check the status of the cluster:

    $ sudo podman exec $(sudo podman container ls --all --format "{{ .Names }}" \
             --filter=name=galera-bundle) bash -c "clustercheck"

    Ensure that the following message is displayed: “Galera cluster node is synced”, otherwise you must recreate the node.

19. On Controller-0, retrieve the cluster address from the configuration:

    $ sudo podman exec $(sudo podman container ls --all --format "{{ .Names }}" \
    --filter=name=galera-bundle) bash -c "grep wsrep_cluster_address /etc/my.cnf.d/galera.cnf" | awk '{print $3}'

20. On each of the remaining Controller nodes, start the database and validate the cluster:

    1. Start the database:

       $ sudo podman exec $(sudo podman container ls --all --format "{{ .Names }}" \
             --filter=name=galera-bundle) /usr/bin/mysqld_safe --pid-file=/var/run/mysql/mysqld.pid --socket=/var/lib/mysql/mysql.sock \
             --datadir=/var/lib/mysql --log-error=/var/log/mysql/mysql_cluster.log  --user=mysql --open-files-limit=16384 \
             --wsrep-cluster-address=$CLUSTER_ADDRESS &

    2. Check the status of the MYSQL cluster:

       $ sudo podman exec $(sudo podman container ls --all --format "{{ .Names }}" \
                --filter=name=galera-bundle) bash -c "clustercheck"

       Ensure that the following message is displayed: “Galera cluster node is synced”, otherwise you must recreate the node.

21. Stop the MySQL container on all Controller nodes:

    $ sudo podman exec $(sudo podman container ls --all --format "{{ .Names }}" --filter=name=galera-bundle) \
            /usr/bin/mysqladmin -u root shutdown

22. On all Controller nodes, remove the following firewall rule to allow database connections through the virtual IP address:

    $ sudo iptables -D INPUT -p tcp --destination-port 3306 -d <galera_cluster_vip> -j DROP

    • Replace <galera_cluster_vip> with the IP address you retrieved in step 1.

23. Restart the MySQL container on all Controller nodes:

    $ sudo podman container restart $(sudo podman container ls --all --format  "{{ .Names }}" --filter=name=galera-bundle)

24. Restart the clustercheck container on all Controller nodes:

    $ sudo podman container restart $(sudo podman container ls --all --format  "{{ .Names }}" --filter=name=clustercheck)

25. On Controller-0, set the Galera resource to managed mode:

    $ sudo pcs resource manage galera-bundle

Verification

1. To ensure that services are running correctly, check the status of pacemaker:

   $ sudo pcs status

2. To view the status of the overcloud, use the OpenStack Integration Test Suite (tempest). For more information, see Validating your OpenStack cloud with the Integration Test Suite (tempest).

3. If you suspect an issue with a particular node, check the state of the cluster with clustercheck:

   $ sudo podman exec clustercheck /usr/bin/clustercheck

Procedure

1. Log in to the director undercloud node as the root user.

2. Stop all tripleo services:

   [root@director ~]# systemctl  stop  tripleo_*

3. Ensure that no containers are running on the server by entering the following command:

   [root@director ~]# podman ps

   If any containers are running, enter the following command to stop the containers:

   [root@director ~]# podman stop <container_name>

4. Create a backup of the current /var/lib/mysql directory and then delete the directory:

   [root@director ~]# cp -a /var/lib/mysql /var/lib/mysql_bck
   [root@director ~]# rm -rf /var/lib/mysql

5. Recreate the database directory and set the SELinux attributes for the new directory:

   [root@director ~]# mkdir /var/lib/mysql
   [root@director ~]# chown 42434:42434 /var/lib/mysql
   [root@director ~]# chmod 0755 /var/lib/mysql
   [root@director ~]# chcon -t container_file_t /var/lib/mysql
   [root@director ~]# chcon -r object_r /var/lib/mysql
   [root@director ~]# chcon -u system_u /var/lib/mysql

6. Create a local tag for the mariadb image. Replace <image_id> and <undercloud.ctlplane.example.com> with the values applicable in your environment:

   [root@director ~]# podman images | grep mariadb
   <undercloud.ctlplane.example.com>:8787/rh-osbs/rhosp16-openstack-mariadb                 	16.2_20210322.1   <image_id>   3 weeks ago   718 MB

   [root@director ~]# podman tag <image_id> mariadb

   [root@director ~]# podman images | grep maria
   localhost/mariadb                                                                         	latest        	<image_id>   3 weeks ago   718 MB
   <undercloud.ctlplane.example.com>:8787/rh-osbs/rhosp16-openstack-mariadb                 	16.2_20210322.1   <image_id>   3 weeks ago   718 MB

7. Initialize the /var/lib/mysql directory with the container:

   [root@director ~]# podman run --net=host -v /var/lib/mysql:/var/lib/mysql localhost/mariadb mysql_install_db --datadir=/var/lib/mysql --user=mysql

   Note

   You can ignore the following error messages, which are generated because the directories do not exist in the container image used in RHOSP.

   chown: cannot access '/usr/lib64/mariadb/plugin/auth_pam_tool_dir/auth_pam_tool': No such file or directory
   Couldn't set an owner to '/usr/lib64/mariadb/plugin/auth_pam_tool_dir/auth_pam_tool'.
   It must be root, the PAM authentication plugin doesn't work otherwise.

8. Copy the database backup file that you want to import to the database:

   [root@director ~]# cp /root/undercloud-all-databases.sql /var/lib/mysql

9. Start the database service to import the data:

   [root@director ~]# podman run --net=host -dt -v /var/lib/mysql:/var/lib/mysql  localhost/mariadb  /usr/libexec/mysqld

10. Import the data and configure the max_allowed_packet parameter:

    1. Log in to the container and configure it:

       [root@director ~]# podman exec --user 0 -it <container_id> /bin/bash
           ()[mysql@5a4e429c6f40 /]$ mysql -u root -e "set global max_allowed_packet = 1073741824;"
           ()[mysql@5a4e429c6f40 /]$ mysql -u root < /var/lib/mysql/undercloud-all-databases.sql
           ()[mysql@5a4e429c6f40 /]$ mysql -u root -e 'flush privileges'
           ()[mysql@5a4e429c6f40 /]$ exit
           exit

    2. Stop the container:

       [root@director ~]# podman stop <container_id>

    3. Check that no containers are running:

       [root@director ~]# podman ps
       CONTAINER ID  IMAGE  COMMAND  CREATED  STATUS  PORTS  NAMES
       [root@director ~]#

11. Restart all tripleo services:

    [root@director ~]# systemctl start multi-user.target