Chapter 3. Backing up control plane nodes that use composable roles

Procedure

1. On the undercloud node, source the undercloud credentials:

   [stack@undercloud ~]$ source stackrc
   (undercloud) [stack@undercloud ~]$

2. On the undercloud node, create an inventory file for the backup node and replace the <ip_address> and <user> with the values that apply to your environment:

   (undercloud) [stack@undercloud ~]$ cat <<'EOF'> ~/nfs-inventory.yaml
   [BackupNode]
   <backup_node> ansible_host=<ip_address> ansible_user=<user>
   EOF

3. On the undercloud node, create the following Ansible playbook and replace <backup_node> with the host name of the backup node:

   (undercloud) [stack@undercloud ~]$ cat <<'EOF' > ~/bar_nfs_setup.yaml
   # Playbook
   # Substitute <backup_node> with the host name of your backup node.
   - become: true
     hosts: <backup_node>
     name: Setup NFS server for ReaR
     roles:
     - role: backup-and-restore
   EOF

4. Copy the public SSH key from the undercloud node to the backup node.

   (undercloud) [stack@undercloud ~]$ ssh-copy-id -i ~/.ssh/id_rsa.pub <backup_node>

   Replace <backup_node> with the path and name of the backup node.

5. On the undercloud node, enter the following ansible-playbook commands to configure the backup node:

   (undercloud) [stack@undercloud ~]$ ansible-playbook \
       -v -i ~/nfs-inventory.yaml \
       --extra="ansible_ssh_common_args='-o StrictHostKeyChecking=no'" \
       --become \
       --become-user root \
       --tags bar_setup_nfs_server \
       ~/bar_nfs_setup.yaml

Procedure

1. On the undercloud node, create the following Ansible playbook:

   (undercloud) [stack@undercloud ~]$ cat <<'EOF' > ~/bar_rear_setup-controller.yaml
   # Playbook
   # Install and configuring ReaR on the control plane nodes
   - become: true
     hosts: Controller
     name: Install ReaR
     roles:
     - role: backup-and-restore
   EOF

   Note

   If you deployed the control plane nodes with composable roles, replace the host type Controller with the types of nodes in your control plane. For example, if you deployed the database, messaging, and networking on separate nodes, enter ControllerOpenstack,Database,Messaging,Networker.

2. Choose one of the following options:

   1. If you use NFS and the IP address of the NFS server is the default value 192.168.24.1, on the undercloud node, enter the following Ansible command to install ReaR on the control plane nodes:

      (undercloud) [stack@undercloud ~]$ ansible-playbook \
          -v -i ~/tripleo-inventory.yaml \
          --extra="ansible_ssh_common_args='-o StrictHostKeyChecking=no'" \
          --become \
          --become-user root \
          --tags bar_setup_rear \
          ~/bar_rear_setup-controller.yaml

   2. If you use SFTP and the IP address of the NFS server is not the default value 192.168.24.1, enter the following Ansible command to install ReaR on the control plane nodes:

      (undercloud) [stack@undercloud ~]$ ansible-playbook \
          -v -i ~/tripleo-inventory.yaml \
          --extra="ansible_ssh_common_args='-o StrictHostKeyChecking=no'" \
          --become \
          --become-user root \
          -e tripleo_backup_and_restore_server=<nfs_ip> \
          --tags bar_setup_rear \
          ~/bar_rear_setup-controller.yaml

      Replace <nfs_ip> with the IP address of your NFS server.

   3. If you use SFTP, enter the following Ansible command do install ReaR on the control plane nodes:

      (undercloud) [stack@undercloud ~]$ ansible-playbook \
          -v -i ~/tripleo-inventory.yaml \
          --extra="ansible_ssh_common_args='-o StrictHostKeyChecking=no'" \
          --become \
          --become-user root \
          -e tripleo_backup_and_restore_output_url=sftp://<user>:<password>@<backup_node_ip>/ \
          -e tripleo_backup_and_restore_backup_url=iso:///backup/ \
          --tags bar_setup_rear \
          ~/bar_rear_setup-undercloud.yaml

3. If your system uses the UEFI boot loader, perform the following steps on the control plane nodes:

   1. Install the following tools:

      $ sudo dnf install dosfstools efibootmgr

   2. Enable UEFI backup in the ReaR configuration file located in /etc/rear/local.conf by replacing the USING_UEFI_BOOTLOADER parameter value 0 with the value 1.

Procedure

1. On each Controller node, back up the config-drive partition of each node:

   [heat-admin@controller-x ~]$ mkdir /mnt/config-drive
   [heat-admin@controller-x ~]$ dd if=<config_drive_partition> of=/mnt/config-drive

   Note

   You only need to perform this step on the Controller nodes.

2. On the undercloud node, create the following Ansible playbook:

   (undercloud) [stack@undercloud ~]$ cat <<'EOF' > ~/bar_rear_create_restore_images-controller.yaml
   # Playbook
   # Using ReaR on the Contorl-Plane - Composable Roles
   
   - become: true
     hosts: ControllerOpenstack,Database,Messaging,Networker
     name: Stop service management
     tasks:
       - include_role:
           name: backup-and-restore
           tasks_from: ../backup/tasks/service_manager_pause
         when:
          - tripleo_backup_and_restore_service_manager
   
   - become: true
     hosts: Database
     name: Database Backup
     tasks:
       - include_role:
           name: backup-and-restore
           tasks_from: ../backup/tasks/db_backup
   
   - become: true
     hosts: pacemaker
     name: Backup pacemaker configuration
     tasks:
       - include_role:
           name: backup-and-restore
           tasks_from: pacemaker_backup
   
   - become: true
     hosts: ControllerOpenstack,Database,Messaging,Networker
     name: Create recovery images with ReaR
     tasks:
       - include_role:
           name: backup-and-restore
           tasks_from: ../backup/tasks/main
   
   - become: true
     hosts: pacemaker
     name: Enabled pacemaker
     tasks:
       - name: Enable pacemaker
         command: pcs cluster start --all
         when: enabled_galera
         run_once: true
         tags:
          - bar_create_recover_image
   
   - become: true
     hosts: Database
     name: Restart galera
     tasks:
       - name: unPause database container
         command: "{{ tripleo_container_cli }} unpause {{ tripleo_backup_and_restore_mysql_container }}"
         when:
          - tripleo_container_cli is defined
           - not enabled_galera
           - tripleo_backup_and_restore_mysql_container is defined
         tags:
          - bar_create_recover_image
   
   - become: true
     hosts: ControllerOpenstack,Database,Messaging,Networker
     name: Unpause everything
     tasks:
       - name: Gather Container Service Name
         shell: |
          set -o pipefail
           /usr/bin/{{ tripleo_container_cli }} ps -a --filter='status=paused' --format '{{ '{{' }}.Names {{ '}}' }} '
         register: container_services
         changed_when: container_services.stdout is defined
         tags:
          - bar_create_recover_image
   
       - name: Unpause containers for database backup.
         command: "{{ tripleo_container_cli }} unpause {{ item }}"
         with_items: "{{ container_services.stdout_lines }}"
         when: tripleo_container_cli is defined
         tags:
          - bar_create_recover_image

3. On the undercloud node, enter the following ansible-playbook command to create a backup of the control plane nodes:

   Important

   Do not operate the stack. When you stop the pacemaker cluster and the containers, this results in the temporary interruption of control plane services to Compute nodes. There is also disruption to network connectivity, Ceph, and the NFS or SFTP data plane service. You cannot create instances, migrate instances, authenticate requests, or monitor the health of the cluster until the pacemaker cluster and the containers return to service following the final step of this procedure.

   (undercloud) [stack@undercloud ~]$ ansible-playbook \
       -v -i ~/tripleo-inventory.yaml \
       --extra="ansible_ssh_common_args='-o StrictHostKeyChecking=no'" \
       --become \
       --become-user root \
       --tags bar_create_recover_image \
       ~/bar_rear_create_restore_images-controller.yaml

Procedure

1. On the undercloud node, enter the following command to create the backup script:

   [stack@undercloud ~]$ cat <<'EOF' > /home/stack/execute-rear-cron.sh
   
   #!/bin/bash
   
   OWNER="stack"
   TODAY=`date +%Y%m%d`
   FILE="/var/log/rear-cron.${TODAY}"
   sudo touch ${FILE}
   sudo chown ${OWNER}:${OWNER} ${FILE}
   
   CURRENTTIME=`date`
   echo "[$CURRENTTIME] rear start" >> ${FILE}
   source /home/stack/stackrc && /usr/bin/openstack overcloud backup 2>&1 >> ${FILE}
   CURRENTTIME=`date`
   echo "[$CURRENTTIME] rear end" >> ${FILE}
   EOF

2. Set executable privileges for the /home/stack/execute-rear-cron.sh script:

   [stack@undercloud ~]$ chmod 755 /home/stack/execute-rear-cron.sh

3. Edit the crontab file with the crontab -e command and use an editor of your choice to add the following cron job. Ensure you save the changes to the file:

   [stack@undercloud ~]# $ crontab -e
   #adding the following line
   0 0 * * * /home/stack/execute-rear-cron.sh

   The /home/stack/execute-rear-cron.sh script is scheduled to be executed by the stack user at midnight.

4. To verify that the cron job is scheduled, enter the following command:

   [stack@undercloud ~]$ crontab -l

   The command output displays the scheduled cron jobs:

   0 0 * * * /home/stack/execute-rear-cron.sh