Chapter 2. Backing up the control plane nodes
To back up the control plane nodes, you configure the backup node, install the Relax-and-Recover tool on the control plane nodes, and create the backup image. You can create backups as a part of your regular environment maintenance.
In addition, you must back up the control plane nodes before performing updates or upgrades. You can use the backups to restore the control plane nodes to their previous state if an error occurs during an update or upgrade.
2.1. Supported backup formats and protocols Copy linkLink copied to clipboard!
The undercloud and backup and restore process uses the open-source tool Relax-and-Recover (ReaR) to create and restore bootable backup images. ReaR is written in Bash and supports multiple image formats and multiple transport protocols.
The following list shows the backup formats and protocols that Red Hat OpenStack Platform supports when you use ReaR to back up and restore the undercloud and control plane.
- Bootable media formats
- ISO
- File transport protocols
- SFTP
- NFS
2.2. Installing and configuring an NFS server on the backup node Copy linkLink copied to clipboard!
You can install and configure a new NFS server to store the backup file. To install and configure an NFS server on the backup node, create an inventory file, create an SSH key, and run the openstack undercloud backup command with the NFS server options.
- If you previously installed and configured an NFS or SFTP server, you do not need to complete this procedure. You enter the server information when you set up ReaR on the node that you want to back up.
-
By default, the Relax and Recover (ReaR) IP address parameter for the NFS server is
192.168.24.1. You must add the parametertripleo_backup_and_restore_serverto set the IP address value that matches your environment.
Procedure
On the undercloud node, source the undercloud credentials:
source stackrc
[stack@undercloud ~]$ source stackrc (undercloud) [stack@undercloud ~]$Copy to Clipboard Copied! Toggle word wrap Toggle overflow On the undercloud node, create an inventory file for the backup node:
(undercloud) [stack@undercloud ~]$ cat <<'EOF'> ~/nfs-inventory.yaml [BackupNode] <backup_node> ansible_host=<ip_address> ansible_user=<user> EOF
(undercloud) [stack@undercloud ~]$ cat <<'EOF'> ~/nfs-inventory.yaml [BackupNode] <backup_node> ansible_host=<ip_address> ansible_user=<user> EOFCopy to Clipboard Copied! Toggle word wrap Toggle overflow Replace
<ip_address>and<user>with the values that apply to your environment.Copy the public SSH key from the undercloud node to the backup node.
ssh-copy-id -i ~/.ssh/id_rsa.pub <backup_node>
(undercloud) [stack@undercloud ~]$ ssh-copy-id -i ~/.ssh/id_rsa.pub <backup_node>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Replace
<backup_node>with the path and name of the backup node.Configure the NFS server on the backup node:
openstack undercloud backup --setup-nfs --extra-vars /home/stack/bar-vars.yaml --inventory /home/stack/nfs-inventory.yaml
(undercloud) [stack@undercloud ~]$ openstack undercloud backup --setup-nfs --extra-vars /home/stack/bar-vars.yaml --inventory /home/stack/nfs-inventory.yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow
2.3. Installing ReaR on the control plane nodes Copy linkLink copied to clipboard!
Before you create a backup of the overcloud control plane, install and configure Relax and Recover (ReaR) on each of the control plane nodes.
Due to a known issue, the ReaR backup of overcloud nodes continues even if a Controller node is down. Ensure that all your Controller nodes are running before you run the ReaR backup. A fix is planned for a later Red Hat OpenStack Platform (RHOSP) release. For more information, see BZ#2077335 - Back up of the overcloud ctlplane keeps going even if one controller is unreachable.
Prerequisites
- You have an NFS or SFTP server installed and configured on the backup node. For more information about creating a new NFS server, see Section 2.2, “Installing and configuring an NFS server on the backup node”.
Procedure
On the undercloud node, source the undercloud credentials:
source stackrc
[stack@undercloud ~]$ source stackrcCopy to Clipboard Copied! Toggle word wrap Toggle overflow If you have not done so before, create an inventory file and use the
tripleo-ansible-inventorycommand to generate a static inventory file that contains hosts and variables for all the overcloud nodes:tripleo-ansible-inventory \ --ansible_ssh_user heat-admin \ --static-yaml-inventory /home/stack/tripleo-inventory.yaml
(undercloud) [stack@undercloud ~]$ tripleo-ansible-inventory \ --ansible_ssh_user heat-admin \ --static-yaml-inventory /home/stack/tripleo-inventory.yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow In the
bar-vars.yamlfile, configure the backup storage location. Follow the appropriate steps for your NFS server or SFTP server.If you use an NFS server, add the following parameters to the
bar-vars.yamlfile:tripleo_backup_and_restore_server: <ip_address> tripleo_backup_and_restore_shared_storage_folder: <backup_server_dir_path> tripleo_backup_and_restore_output_url: "nfs://{{ tripleo_backup_and_restore_server }}{{ tripleo_backup_and_restore_shared_storage_folder }}" tripleo_backup_and_restore_backup_url: "nfs://{{ tripleo_backup_and_restore_server }}{{ tripleo_backup_and_restore_shared_storage_folder }}"tripleo_backup_and_restore_server: <ip_address> tripleo_backup_and_restore_shared_storage_folder: <backup_server_dir_path> tripleo_backup_and_restore_output_url: "nfs://{{ tripleo_backup_and_restore_server }}{{ tripleo_backup_and_restore_shared_storage_folder }}" tripleo_backup_and_restore_backup_url: "nfs://{{ tripleo_backup_and_restore_server }}{{ tripleo_backup_and_restore_shared_storage_folder }}"Copy to Clipboard Copied! Toggle word wrap Toggle overflow Replace
<ip_address>`and `<backup_server_dir_path>. The default value fortripleo_backup_and_restore_serverparameter value is192.168.24.1.If you use an SFTP server, add the
tripleo_backup_and_restore_output_urlparameter and set the values of the URL and credentials of the SFTP server:tripleo_backup_and_restore_output_url: sftp://<user>:<password>@<backup_node>/ tripleo_backup_and_restore_backup_url: iso:///backup/
tripleo_backup_and_restore_output_url: sftp://<user>:<password>@<backup_node>/ tripleo_backup_and_restore_backup_url: iso:///backup/Copy to Clipboard Copied! Toggle word wrap Toggle overflow Replace
<user>,<password>, and<backup_node>with the backup node URL and credentials.
Install ReaR on the control plane nodes:
openstack overcloud backup --setup-rear --extra-vars /home/stack/bar-vars.yaml --inventory /home/stack/tripleo-inventory.yaml
(undercloud) [stack@undercloud ~]$ openstack overcloud backup --setup-rear --extra-vars /home/stack/bar-vars.yaml --inventory /home/stack/tripleo-inventory.yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow If your system uses the UEFI boot loader, perform the following steps on the control plane nodes:
Install the following tools:
sudo dnf install dosfstools efibootmgr
$ sudo dnf install dosfstools efibootmgrCopy to Clipboard Copied! Toggle word wrap Toggle overflow -
Enable UEFI backup in the ReaR configuration file located in
/etc/rear/local.confby replacing theUSING_UEFI_BOOTLOADERparameter value0with the value1.
2.4. Configuring Open vSwitch (OVS) interfaces for backup Copy linkLink copied to clipboard!
If you use an Open vSwitch (OVS) bridge in your environment, you must manually configure the OVS interfaces before you create a backup of the undercloud or control plane nodes. The restoration process uses this information to restore the network interfaces.
Procedure
In the
/etc/rear/local.conffile, add theNETWORKING_PREPARATION_COMMANDSparameter in the following format:NETWORKING_PREPARATION_COMMANDS=('<command_1>' '<command_2>' ...')NETWORKING_PREPARATION_COMMANDS=('<command_1>' '<command_2>' ...')Copy to Clipboard Copied! Toggle word wrap Toggle overflow Replace
<command_1>and<command_2>with commands that configure the network interfaces. For example, you can add theip link add br-ctlplane type bridgecommand to create the control plane bridge or add theip link set eth0 upcommand to change the state ofeth0to up. You can add more commands to the parameter based on your network configuration.For example, if your undercloud has the following configuration:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow The
NETWORKING_PREPARATION_COMMANDSparameter is formatted as follows:NETWORKING_PREPARATION_COMMANDS=('ip link add br-ctlplane type bridge' 'ip link set br-ctlplane up' 'ip link set eth0 up' 'ip link set eth0 master br-ctlplane' 'ip addr add 172.16.9.1/24 dev br-ctlplane')NETWORKING_PREPARATION_COMMANDS=('ip link add br-ctlplane type bridge' 'ip link set br-ctlplane up' 'ip link set eth0 up' 'ip link set eth0 master br-ctlplane' 'ip addr add 172.16.9.1/24 dev br-ctlplane')Copy to Clipboard Copied! Toggle word wrap Toggle overflow
2.5. Creating a backup of the control plane nodes Copy linkLink copied to clipboard!
To create a backup of the control plane nodes, use the openstack overcloud backup command. You can then use the backup to restore the control plane nodes to their previous state in case the nodes become corrupted or inaccessible. The backup of the control plane nodes includes the backup of the database that runs on the control plane nodes.
Before you back up your control plane nodes, stop each control plane node and check that the file system is free of corruption. For example, disk corruption on the control plane nodes prevents you from performing a backup of the nodes.
Prerequisites
- You have an NFS or SFTP server installed and configured on the backup node. For more information about creating a new NFS server, see Section 2.2, “Installing and configuring an NFS server on the backup node”.
- You have installed ReaR on the control plane nodes. For more information, see Section 2.3, “Installing ReaR on the control plane nodes”.
- If you use an OVS bridge for your network interfaces, you have configured the OVS interfaces. For more information, see Section 2.4, “Configuring Open vSwitch (OVS) interfaces for backup”.
Procedure
Locate the
config-drivepartition on each control plane node:Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- The
config-drivepartition is the 1M partition that is not mounted.
On each control plane node, back up the
config-drivepartition of each node as therootuser:dd if=<config_drive_partition> of=/mnt/config-drive
[root@controller-x ~]# dd if=<config_drive_partition> of=/mnt/config-driveCopy to Clipboard Copied! Toggle word wrap Toggle overflow Replace
<config_drive_partition>with the name of theconfig-drivepartition that you located in step 1.On the undercloud node, source the undercloud credentials:
source stackrc
[stack@undercloud ~]$ source stackrcCopy to Clipboard Copied! Toggle word wrap Toggle overflow If you have not done so before, use the
tripleo-ansible-inventorycommand to generate a static inventory file that contains hosts and variables for all the overcloud nodes:tripleo-ansible-inventory \ --ansible_ssh_user heat-admin \ --static-yaml-inventory /home/stack/tripleo-inventory.yaml
(undercloud) [stack@undercloud ~]$ tripleo-ansible-inventory \ --ansible_ssh_user heat-admin \ --static-yaml-inventory /home/stack/tripleo-inventory.yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow Create a backup of the control plane nodes:
openstack overcloud backup --inventory /home/stack/tripleo-inventory.yaml
(undercloud) [stack@undercloud ~]$ openstack overcloud backup --inventory /home/stack/tripleo-inventory.yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow The backup process runs sequentially on each control plane node without disrupting the service to your environment.
2.6. Scheduling control plane node backups with cron Copy linkLink copied to clipboard!
You can schedule backups of the control plane nodes with ReaR by using the Ansible backup-and-restore role. You can view the logs in the /var/log/rear-cron directory.
Prerequisites
- You have an NFS or SFTP server installed and configured on the backup node. For more information about creating a new NFS server, see Section 1.3, “Installing and configuring an NFS server on the backup node”.
- You have installed ReaR on the undercloud and control plane nodes. For more information, see Section 2.3, “Installing ReaR on the control plane nodes”.
- You have sufficient available disk space at your backup location to store the backup.
Procedure
To schedule a backup of your control plane nodes, run the following command. The default schedule is Sundays at midnight:
openstack overcloud backup --cron
openstack overcloud backup --cronCopy to Clipboard Copied! Toggle word wrap Toggle overflow Optional: Customize the scheduled backup according to your deployment:
To change the default backup schedule, pass a different cron schedule on the
tripleo_backup_and_restore_cronparameter:openstack overcloud backup --cron --extra-vars '{"tripleo_backup_and_restore_cron": "0 0 * * 0"}'openstack overcloud backup --cron --extra-vars '{"tripleo_backup_and_restore_cron": "0 0 * * 0"}'Copy to Clipboard Copied! Toggle word wrap Toggle overflow To define additional parameters that are added to the backup command when cron runs the scheduled backup, pass the
tripleo_backup_and_restore_cron_extraparameter to the backup command, as shown in the following example:openstack overcloud backup --cron --extra-vars '{"tripleo_backup_and_restore_cron_extra":"--extra-vars bar-vars.yaml --inventory /home/stack/tripleo-inventory.yaml"}'openstack overcloud backup --cron --extra-vars '{"tripleo_backup_and_restore_cron_extra":"--extra-vars bar-vars.yaml --inventory /home/stack/tripleo-inventory.yaml"}'Copy to Clipboard Copied! Toggle word wrap Toggle overflow To change the default user that executes the backup, pass the
tripleo_backup_and_restore_cron_userparameter to the backup command, as shown in the following example:openstack overcloud backup --cron --extra-vars '{"tripleo_backup_and_restore_cron_user": "root"}openstack overcloud backup --cron --extra-vars '{"tripleo_backup_and_restore_cron_user": "root"}Copy to Clipboard Copied! Toggle word wrap Toggle overflow