Chapter 8. Backing Up and Restoring Satellite Server and Capsule Server
This chapter describes the minimum backup and restore procedures required to ensure continuity of your Red Hat Satellite deployment and associated data in the event of a disaster. If your deployment uses custom configurations you should take these into account when planning your backup and disaster recovery policy.
8.1. Backing up Satellite Server or Capsule Server
This section describes creating a backup of your Satellite Server or Capsule Server and all associated data using the satellite-backup
script. Backing up to a separate location is recommended. Backing up to a separate storage device on a separate system is highly recommended. Satellite services are unavailable during the backup. The backup can be scheduled for a quiet time using cron
, see the Example 8.1, “A Weekly Full Backup Followed by Daily Incremental Backups”.
Prerequisites
- When planning a scheduled backup, ensure that no other tasks are scheduled by other administrators for the same time. This is particularly important when administrators are working in different locations and time zones.
- Ensure that you either encrypt the backups or move them to a secure location to minimize the risk of damage or unauthorized access to hosts.
You can also use conventional backup methods such as that described in the System Backup and Recovery section of the Red Hat Enterprise Linux 7 System Administrator’s Guide. When creating a snapshot or conventional backup, stop all services (Do not do this if using the satellite-backup
script):
# katello-service stop
Start the services after creating a snapshot or conventional backup:
# katello-service start
8.1.1. Estimating the size of a Backup
The full backup creates uncompressed archives of MongoDB, PostgreSQL, and Pulp database files and Satellite configuration files. Compression occures after the archives are created to decrease the time when Satellite services are unavailable. Consequently, a full backup requires space to store the following data:
- Uncompressed Satellite databases and configuration files.
- Compressed Satellite databases and configuration files.
- An extra 20% of the total estimated space to ensure a reliable backup.
To Estimate the Size of a Backup
Enter the
du
command to estimate the space required to store uncompressed directories containing Satellite databases and configuration files, for example:# du -csh /var/lib/mongodb /var/lib/pgsql/data /var/lib/pulp \ /etc /root/ssl-build /var/www/html/pub /opt/puppetlabs 480G /var/lib/mongodb 100G /var/lib/pgsql/data 100G /var/lib/pulp 680G total 37M /etc 900K /root/ssl-build 100K /var/www/html/pub 2M /opt/puppetlabs 680GB total
In this example, the uncompressed backup data occupies 680 GB in total.
NoteThe
/opt/puppetlabs
directory is used for Puppet 4. Use/var/lib/puppet
for Puppet 3.Calculate how much space is required to store the compressed data.
Table 8.1, “Backup Data Compression Ratio” demonstrates compression ratio of all data items used in the backup.
Table 8.1. Backup Data Compression Ratio Data type Directory Ratio Example results MongoDB database files
/var/lib/mongodb
10 - 15%
480 GB
420 GB PostgreSQL database files
/var/lib/pgsql/data
15 - 20%
100 GB
80 GB Pulp RPM files
/var/lib/pulp
-
(not compressed)
Configuration files
/etc
/root-ssl/build
/var/www/html/pub
/opt/puppetlabs
5 - 10%
50 MB
45 MB In this example, the compressed backup data occupies 500 GB in total.
Calculate how much space is required to store a backup. Sum up the estimated values of compressed and uncompressed backup data and add extra 20% to ensure a reliable backup.
This example requires 680 GB plus 500 GB for the uncompressed and compressed backup data, 1180 GB in total. With 240 GB of extra space, 1420 GB must be allocated for the backup location.
8.1.2. Performing a Full Backup of Satellite Server or Capsule Server
Red Hat Satellite 6.3 uses the satellite-backup
script to make and restore backups. To see the usage statement, enter a command as follows:
# satellite-backup --help
From Satellite 6.2.8, the satellite-backup
creates a time-stamped subdirectory in the backup directory you specify. The satellite-backup
script does not overwrite backups and the correct directory or subdirectory has to be selected when restoring from a backup or an incremental backup. The script stops and restarts services as required.
To Perform a Full Offline Backup of Satellite Server or Capsule Server:
This procedure performs a full offline backup. Satellite services are unavailable during the backup process.
Request other users of Satellite Server or Capsule Server to save any changes and warn them that Satellite services are unavailable for the duration of the backup. Ensure no other tasks are scheduled for the same time as the backup.
- Ensure your backup location has enough disk space to store the backup. For more information, see Section 8.1.1, “Estimating the size of a Backup”.
Run the backup script:
# satellite-backup backup_directory
The
satellite-backup
script stops all services which could impact the backup, performs the backup, then restarts the required services. The script creates the target directory when trying to create a backup file if the target directory does not exist.This process can take a long time to complete, due to the amount of data to copy.
8.1.3. Performing a Backup without Pulp Content
To Perform a Backup without Pulp Content:
This procedure performs an off-line backup but excludes the contents of the Pulp directory. This backup is useful for debugging purposes and is only intended to provide access to configuration files without spending time backing up the Pulp database. You cannot restore from a directory that does not contain Pulp content.
Request other users of Satellite Server or Capsule Server to save any changes and warn them that Satellite services are unavailable for the duration of the backup. Ensure no other tasks are scheduled for the same time as the backup.
- Ensure your backup location has enough disk space to store the backup. For more information, see Section 8.1.1, “Estimating the size of a Backup”.
Run the backup script:
# satellite-backup --skip-pulp-content backup_directory
The
satellite-backup
script stops all services which could impact the backup, performs the backup, then restarts the required services. The script creates the target directory when trying to create a backup file if the target directory does not exist.
8.1.4. Performing an Incremental Backup
To Perform an Incremental Backup:
This procedure performs an off-line backup of any changes since a previous backup. Use a full backup as a reference to make the first incremental backup of a sequence. Keep at least the last known good full backup and a complete sequence of incremental backups to restore from.
Request other users of Satellite Server or Capsule Server to save any changes and warn them that Satellite services are unavailable for the duration of the backup. Ensure no other tasks are scheduled for the same time as the backup.
- Ensure your backup location has enough disk space to store the backup. For more information, see Section 8.1.1, “Estimating the size of a Backup”.
Run the backup script:
With Pulp content:
# satellite-backup backup_directory --incremental backup_directory/previous_time-stamped_subdirectory
Without Pulp content:
# satellite-backup backup_directory --skip-pulp-content --incremental backup_directory/previous_time-stamped_subdirectory
The
satellite-backup
script stops all services which could impact the backup, performs the backup, then restarts the required services. If the target directory does not exist when trying to create a backup file the script creates it. It is possible to make incremental backups using a backup older than the previous backup as a starting point, but with a corresponding increase in time to make the backup.
8.1.5. Example - A Weekly Full Backup Followed by Daily Incremental Backups
Example 8.1. A Weekly Full Backup Followed by Daily Incremental Backups
An example script that makes a full backup on a Sunday and incremental backups on all other days of the week. Backups are stored in subdirectories named $YEAR-$WEEK
. Each subdirectory contains the full backup from the previous week’s Sunday and incremental backups from the subsequent days. This script requires a daily cron job.
#!/bin/bash -e export PATH=/sbin:/bin:/usr/sbin:/usr/bin DESTINATION=/var/backup YEAR=$(date +%Y) WEEK=$(date +%-V) if [[ $(date +%w) == 0 ]]; then satellite-backup $DESTINATION/$YEAR-$((WEEK + 1)) --assumeyes else LAST=$(ls -td -- $DESTINATION/$YEAR-$WEEK/*/ | head -n 1) satellite-backup $DESTINATION/$YEAR-$WEEK --incremental "$LAST" --assumeyes fi exit 0
Note that the satellite-backup
script requires /sbin
and /usr/sbin
directories to be in PATH
.
8.1.6. Performing an Online Backup
To Perform an Online Backup:
This procedure performs a full backup while Satellite Server or Capsule Server is running. If there are procedures affecting the Pulp database, the Pulp part of the backup procedure repeats until it is no longer being altered. Since the backup of the Pulp database is the most time consuming part of backing up a Satellite, it is highly recommended to not alter the Pulp database during this time. This prolongs the backup procedure as the Pulp part of the backup is restarted.
Satellite 6 uses two database systems, PostgreSQL and MongoDB. There are records that exist in both PostgreSQL and MongoDB that need to remain synchronized.
The --online-backup
option keeps all services running which means there is a possibility that data can be modified while the backup is being made. There is a basic check to see if the databases were modified during the backup. If this occurs, the script starts the database portion of the backup again. This check is rudimentary and cannot ensure with 100% certainty that there were no modifications to the databases while the backup script was running. This check can also result in repeated loops if there is continuous modification occurring to the databases.
It is recommended to use a snapshot backup method in production, as described in Section 8.1.7, “Performing a Snapshot Backup”. If you still want to use the online backup method in production, ensure that no modifications occur during the backup.
Request other users of Satellite Server or Capsule Server to save any changes and warn them that Satellite services are unavailable for the duration of the backup. Ensure no other tasks are scheduled for the same time as the backup.
- Ensure your backup location has enough disk space to store the backup. For more information, see Section 8.1.1, “Estimating the size of a Backup”.
Run the backup script:
# satellite-backup --online-backup /tmp/backup_directory
8.1.7. Performing a Snapshot Backup
The snapshot backup method uses Logical Volume Manager (LVM) snapshots of the Pulp, MongoDB, and PostgreSQL directories. The actual backup is then created from the LVM snapshots and not from the running Satellite as with online backup, which mitigates the risk of creating an inconsistent backup. The snapshot backup method is faster than a full off-line backup, which reduces Satellite downtime. As such, it suits well for backing up highly populated Satellite servers with long backup times.
To Perform a Snapshot Backup:
This procedure performs a snapshot backup. It can be combined with other satellite-backup
sub-commands, except an online-backup.
Prerequisites
-
The system uses LVM for the snapshotted directories (
/var/lib/pulp/
,/var/lib/mongodb/
, and/var/lib/pgsql/
). - The free disk space in the relevant volume group (VG) is three times the size of the snapshot. More precisely, the VG has to have enough space unreserved by the member logical volumes (LVs) to accommodate new snapshots. In addition, one of the LVs has to have enough free space for the backup directory.
- The target backup directory is on a different LV than the snapshotted directories.
Request other Satellite Server or Capsule Server users to save any changes and warn them that Satellite services are unavailable for the duration of the backup. Ensure no other tasks are scheduled for the same time as the backup.
Run the backup script:
# satellite-backup --snapshot backup_directory
The satellite-backup
script stops all services which could impact the backup. After the successful backup, all services are restarted and LVM snapshots are removed.
8.2. Restoring Satellite Server or Capsule Server from a Backup
This section describes how to restore a Red Hat Satellite Server or Red Hat Capsule Server from the backup data created as a result of following the steps in Section 8.1, “Backing up Satellite Server or Capsule Server”. This process is intended for restoring the backup on the same server that generated the backup, and all data covered by the backup is deleted on the target system. If the original system is unavailable, provision a system with the same configuration settings (in particular, the host name must be the same).
Prerequisites
- Ensure that you are restoring to the correct instance. The Red Hat Satellite instance must have the same host name, configuration, and be the same major version as the original system.
-
Ensure that you run the
satellite-restore
script asroot
. Ensure that all SELinux contexts are correct. Enter the following command to restore the correct SELinux contexts:
# restorecon -Rnv /
To Restore Satellite Server or Capsule Server from a Full Backup:
- Install Satellite 6 using the procedures in Installing Satellite Server in the Installation Guide.
Copy the backup data to the Satellite’s local file system. Use
/tmp/
or/var/tmp/
. Ensure you have enough space to store this data on the base system of Satellite Server or Capsule Server as well as enough space after the restoration to contain all the data in the/etc/
and/var/
directories contained within the backup.You can use the
du -sh directory_name
command to check the space used by a directory and thedf -h directory_name
command to check for free space. Add the--total
option to sum the results from more than one directory.Run the restoration script:
# satellite-restore backup_directory
Where backup_directory is the time-stamped directory or subdirectory containing the backed-up data. The target directory is read from the configuration files contained within the archive. If the target directory does not exist when trying to recover, it gives an error and asks for the correct directory. The restore process can take a long time to complete, due to the amount of data to copy. Where incremental backups exist, see To Restore Satellite Server or Capsule Server from an Incremental Backup:.
When this process completes, all services should be running and Satellite Server or Capsule Server should be available for use.
To Restore Satellite Server or Capsule Server from an Incremental Backup:
- Install Satellite 6 using the procedures in Installing Satellite Server in the Installation Guide.
- Restore the last full backup as described in To Restore Satellite Server or Capsule Server from a Full Backup:.
-
Copy the backup data to the Satellite’s local file system, for example,
/var/tmp/satellite-backup/
. Ensure you have enough space to store this data on the base system of Satellite Server or Capsule Server as well as enough space after the restoration to contain all the data in the/etc/
and/var/
directories contained within the backup. Run the restoration script:
# satellite-restore backup_directory_X
Where backup_directory_X is a time-stamped directory or subdirectory containing an incremental backup. Restore the incremental backups in the same sequence that they were made. For example: backup_directory_1, backup_directory_2. The target directory is read from the configuration files contained within the archive. If the target directory does not exist when trying to recover, it gives an error and asks for the correct directory.
When this process completes, all services should be running and Satellite Server or Capsule Server should be available for use.
8.3. Backing up and Restoring Capsule Server Using a Virtual Machine Snapshot
There are three methods of backing up Capsule Server:
-
Using the
satellite-backup
script as described in Section 8.1, “Backing up Satellite Server or Capsule Server”. Thesatellite-backup
script is convenient if your Capsule Server is a physical machine. You can also use the script if your Capsule Server is a virtual machine but it creates only a backup of the data and not the machine itself. - Using the conventional backup methods as described in System Backup and Recovery in the Red Hat Enterprise Linux 7 System Administrator’s Guide.
- Using the snapshot of a virtual machine with your Capsule Server on it as described below. Note that this method is different from the snapshot backup method described in Section 8.1.7, “Performing a Snapshot Backup”.
If your Capsule Server is a virtual machine, you can restore it from a snapshot. Creating weekly snapshots to restore from is recommended. In the event of failure, you can reinstall, or configure a new Capsule Server, and then synchronize the database content from the Satellite Server.
When creating a snapshot or conventional backup, stop all services (Do not do this if using the satellite-backup
script):
# katello-service stop
Start the services after creating a snapshot or conventional backup:
# katello-service start
If you have a snapshot or conventional backup, restore from that and then synchronize from the Satellite Server as described below.
If required, deploy a new Capsule Server, ensuring the host name is the same as before, and then install the Capsule certificates. You might still have them on the Satellite Server, the package name ends in -certs.tar
, alternately create new ones. Follow the procedures in Installing Capsule Server in the Installation Guide until you see in the web UI that the Capsule Server is connected to the Satellite Server. Then synchronize from the Satellite Server as described below.
Synchronizing an External Capsule
- To synchronize an external Capsule, select the relevant organization and location in the web UI, or choose Any Organization and Any Location.
- Navigate to Infrastructure > Capsules and click the name of the Capsule to synchronize.
- On the Overview tab, select Synchronize.
8.4. Renaming a Satellite Server or Capsule Server
Renaming a Satellite Server or Capsule Server requires use of the satellite-change-hostname
script. Red Hat Satellite contains references to the host’s name and these changes are made using the script. Renaming a Satellite Server affects itself, all Capsule Servers and all hosts registered to it. Renaming a Capsule Server affects itself and all hosts registered to it.
The renaming process shuts down all Satellite Server services on the host being renamed. When the renaming is complete, all services are restarted.
8.4.1. Renaming a Satellite Server
The host name of a Satellite Server is used by Satellite Server components, all Capsule Servers, and hosts registered to it for communication. Renaming a Satellite Server requires that these references be updated.
If you use external authentication, you must reconfigure Satellite Server for external authentication after you run the satellite-change-hostname
script. The satellite-change-hostname
script breaks external authentication for Satellite Server. For more information about configuring external authentication, see Chapter 11, Configuring External Authentication
Prerequisites
- (Optional) If the Satellite Server has a custom X.509 certificate installed, a new certificate must be obtained in the host’s new name. When all hosts are re-registered to the Satellite Server, the new certificate is installed. For more information on obtaining a custom X.509 certificate, see Configuring Satellite Server with a Custom Server Certificate in the Red Hat Satellite Installation Guide.
-
Backup the Satellite Server. The
satellite-change-hostname
script makes irreversible changes to the Satellite Server. If the renaming process is not successful, you must restore it from backup. For more information, see Section 11.2, “Using Identity Management”.
Rename a Satellite Server
On the Satellite Server, run the
satellite-change-hostname
script, providing the host’s new name, and Satellite credentials.# satellite-change-hostname new_satellite --username admin \ --password password
The message
***** Hostname change complete! *****
confirms that the rename completed successfully.- (Optional) If you have obtained a new X.509 certificate for the Satellite Server’s new host name, run the Satellite installation script to install the certificate. For more information on installing a custom X.509 certificate, see Configuring Satellite Server with a Custom Server Certificate in the Red Hat Satellite Installation Guide.
On all Capsule Servers and hosts registered to the Satellite Server, reinstall the bootstrap RPM and re-register them to the Satellite Server. Substitute the example organization and environment values with those matching your environment.
# yum remove -y katello-ca-consumer*
# rpm -Uvh http://new-satellite.example.com/pub/katello-ca-consumer-latest.noarch.rpm
# subscription-manager register --org="Default_Organization" \ --environment="Library" \ --force
Use of the Red Hat Satellite remote execution feature is recommended for this step. For more information, see Configuring and Running Remote Commands in Managing Hosts.
Reattach subscriptions to all Capsule Servers and hosts registered to the Satellite Server, then refresh the subscription.
# subscription-manager refresh
# yum repolist
Use of the Red Hat Satellite remote execution feature is recommended for this step. For more information, see Configuring and Running Remote Commands in Managing Hosts.
On all Capsule Servers, re-run the Satellite installation script to update references to the new host name.
# satellite-installer --foreman-proxy-content-parent-fqdn new-satellite.example.com \ --foreman-proxy-foreman-base-url https://new-satellite.example.com \ --foreman-proxy-trusted-hosts new-satellite.example.com
On the Satellite Server, synchronize content for each Capsule Server.
List all Capsule Servers with their ID numbers:
# hammer capsule list
Enter the following command for each Capsule Server:
# hammer capsule content synchronize --id capsule_id_number
8.4.2. Renaming a Capsule Server
The host name of a Capsule Server is referenced by Satellite Server components, and all hosts registered to it. Renaming a Capsule Server requires that these references be updated.
Prerequisites
- Optional: New X.509 custom certificate files for the Capsule Server. For more information on obtaining a custom X.509 certificate, see Configuring Capsule Server with a Custom Server Certificate in the Red Hat Satellite Installation Guide.
Backup the Capsule Server. The
satellite-change-hostname
script makes irreversible changes to the Capsule Server. If the renaming process is not successful, you must restore it from backup.Red Hat Satellite does not provide a native backup method for a Capsule Server. For more information, see Chapter 8, Backing Up and Restoring Satellite Server and Capsule Server.
Renaming a Capsule Server:
On Satellite Server, create a new certificates archive file.
If you are using the default Satellite Server certificate:
# capsule-certs-generate --capsule-fqdn new-capsule.example.com \ --certs-tar /root/new-capsule.example.com-certs.tar
Ensure that you enter the full path to the
.tar
file.- If you are using a custom X.509 certificate on the Capsule Server, see Create the Capsule Server’s Certificate Archive File in the Red Hat Satellite Installation Guide.
On Satellite Server, copy the certificates archive file to the Capsule Server, providing the
root
user’s password when prompted. In this example the archive file is copied to theroot
user’s home directory, but you may prefer to copy it elsewhere.# scp /root/new-capsule.example.com-certs.tar root@capsule.example.com:
On the Capsule Server, run the
satellite-change-hostname
script, providing the host’s new name, Satellite credentials, and certificates archive filename.# satellite-change-hostname new_capsule --username admin \ --password password \ --certs-tar /root/new-capsule.example.com-certs.tar
Ensure that you enter the full path to the
.tar
file.The message
***** Hostname change complete! *****
confirms that the rename completed successfully.- Optional: If you have obtained a new X.509 certificate in the Capsule Server’s new host name, run the Satellite installation script to install the certificate. For more information on installing a custom X.509 certificate, see Configuring Satellite Server with a Custom Server Certificate in the Red Hat Satellite Installation Guide.
On all hosts registered to the Capsule Server, reinstall the bootstrap RPM and re-register them to the Capsule Server. Substitute the example organization and environment values with those matching your environment.
# yum remove -y katello-ca-consumer*
# rpm -Uvh http://new-capsule.example.com/pub/katello-ca-consumer-latest.noarch.rpm
# subscription-manager register --org="Default_Organization" \ --environment="Library" \ --force
Use of the Red Hat Satellite remote execution feature is recommended for this step. For more information, see Running Jobs on Hosts in Managing Hosts.
Reattach subscriptions to all hosts registered to the Capsule Server, then refresh the subscription.
# subscription-manager refresh
# yum repolist
Edit the Capsule Server’s name.
- In the Satellite web UI, navigate to Infrastructure > Capsules.
- Find the Capsule Server in the list, and click Edit in that row.
- Edit the Name and URL fields to match the Capsule Server’s new host name, then click Submit.
- On your DNS server, add a record for the Capsule Server’s new host name, and delete the record for the previous host name.