Chapter 3. Using the Block Storage backup service
You can use the Block Storage backup service to perform full or incremental backups, and to restore a backup to a volume.
3.1. Full backups Copy linkLink copied to clipboard!
3.1.1. Creating a full volume backup Copy linkLink copied to clipboard!
To back up a volume, use the cinder backup-create command. By default, this command creates a full backup of the volume. If the volume has existing backups, you can choose to create an incremental backup instead. For more information, Section 3.2.2, “Performing incremental backups”
If you are using a Ceph volume backed up to a Ceph cluster, then the second time you do a backup, the backup service automatically performs incremental backups.
You can create backups of the volumes that you have access to. This means that users with administrative privileges can back up any volume, regardless of the owner. For more information, see Section 3.1.2, “Creating a volume backup as an admin”.
Procedure
View the ID or Display Name of the volume you want to back up:
cinder list
# cinder listCopy to Clipboard Copied! Toggle word wrap Toggle overflow Back up the volume:
cinder backup-create <VOLUME>
# cinder backup-create <VOLUME>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Replace <VOLUME> with the
IDorDisplay Nameof the volume you want to back up:Copy to Clipboard Copied! Toggle word wrap Toggle overflow The
volume_idof the resulting backup is identical to the ID of the source volume.Verify that the volume backup creation is complete:
cinder backup-list
# cinder backup-listCopy to Clipboard Copied! Toggle word wrap Toggle overflow
3.1.2. Creating a volume backup as an admin Copy linkLink copied to clipboard!
The volume backup creation is complete when the Status of the backup entry is available.
Users with administrative privileges can back up any volume managed by Red Hat OpenStack Platform. When an admin user backs up a volume that is owned by a non-admin user, the backup is hidden from the volume owner by default.
Procedure
As an admin user, you can use the following command to back up a volume and make the backup available to a specific tenant:
cinder --os-auth-url <KEYSTONEURL> --os-tenant-name <TENANTNAME> --os-username <USERNAME> --os-password <PASSWD> backup-create <VOLUME>
# cinder --os-auth-url <KEYSTONEURL> --os-tenant-name <TENANTNAME> --os-username <USERNAME> --os-password <PASSWD> backup-create <VOLUME>
Replace the following variables according to your environment requirements:
- <TENANTNAME> is the name of the tenant where you want to make the backup available.
- <USERNAME> and <PASSWD> are the username and password credentials of a user within <TENANTNAME>.
- <VOLUME> is the name or ID of the volume you want to back up.
-
<KEYSTONEURL> is the URL endpoint of the Identity service, which is typically http://IP:5000/v2, where
IPis the IP address of the Identity service host. When you perform this operation, the size of the resulting backup counts against the quota ofTENANTNAMErather than the quota of the tenant admin.
3.1.3. Exporting the metadata of a volume backup Copy linkLink copied to clipboard!
You can export and store the metadata of a volume backup so that you can restore the volume backup even if the Block Storage database suffers a catastrophic loss.
Procedure
Run the following command:
cinder backup-export <BACKUPID>
# cinder backup-export <BACKUPID>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Replace <BACKUPID> with the ID or name of the volume backup:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
The volume backup metadata consists of the backup_service and backup_url values.
3.1.4. Backing up an in-use volume Copy linkLink copied to clipboard!
You can create a backup of an in-use volume with the --force option when the Block Storage back end snapshot is supported.
To use the --force option, the Block Storage back end snapshot must be supported. You can verify snapshot support by checking the documentation for the back end that you are using.
By using the --force option, you acknowledge that you are not quiescing the drive before you perform the backup. Using this method creates a crash-consistent, but not application-consistent, backup. This means that the backup does not have an awareness of which applications were running when the backup was performed. However, the data is intact.
Procedure
To create a backup of an in-use volume, run:
openstack volume backup create --incremental --force
$ openstack volume backup create --incremental --forceCopy to Clipboard Copied! Toggle word wrap Toggle overflow
3.1.5. Performing a full snapshot backup Copy linkLink copied to clipboard!
You can create a full backup from a snapshot using the volume ID that is associated with the snapshot.
Procedure
Locate the ID of the snapshot that you want to back up:
cinder snapshot-list --volume-id _VOLUME_ID_
# cinder snapshot-list --volume-id _VOLUME_ID_Copy to Clipboard Copied! Toggle word wrap Toggle overflow If the snapshot is named, you can use the following example to locate the
ID:cinder snapshot-show _SNAPSHOT_NAME_
# cinder snapshot-show _SNAPSHOT_NAME_Copy to Clipboard Copied! Toggle word wrap Toggle overflow Create the backup of the snapshot:
cinder backup-create _VOLUME_ --snapshot-id=_SNAPSHOT_ID_
# cinder backup-create _VOLUME_ --snapshot-id=_SNAPSHOT_ID_Copy to Clipboard Copied! Toggle word wrap Toggle overflow
3.2. Incremental backups Copy linkLink copied to clipboard!
You can use the Block Storage backup service to perform incremental backups.
3.2.1. Performance considerations Copy linkLink copied to clipboard!
Some backup features like incremental and data compression can impact performance. Incremental backups have a performance impact because all of the data in a volume must be read and checksummed for both the full and each incremental backup.
You can use data compression with non-Ceph backends. Enabling data compression requires additional CPU power but uses less network bandwidth and storage space overall.
The multipathing configuration also impacts performance. If you attach multiple volumes without enabling multipathing, you might not be able to connect or have full network capabilities, which impacts performance.
You can use the advanced configuration options to enable or disable compression, define the number of processes, and add additional CPU resources. For more information, see Appendix B, Advanced Block Storage backup configuration options.
3.2.1.1. Impact of backing up from a snapshot Copy linkLink copied to clipboard!
Some back ends support creating a backup from a snapshot. A driver that supports this feature can directly attach a snapshot, which is faster than cloning the snapshot into a volume to be able to attach to it. In general, this feature can affect performance because of the extra step of creating the volume from a snapshot.
3.2.2. Performing incremental backups Copy linkLink copied to clipboard!
By default, the cinder backup-create command creates a full backup of a volume. However, if the volume has existing backups, you can choose to create an incremental backup.
Incremental backups are fully supported on NFS, Object Storage (swift), and Red Hat Ceph Storage backup repositories.
An incremental backup captures any changes to the volume since the last full or incremental backup. Performing numerous, regular, full backups of a volume can become resource intensive because the size of the volume increases over time. With incremental backups, you can capture periodic changes to volumes and minimize resource usage.
Procedure
To create an incremental volume backup, use the
--incrementaloption:cinder backup-create <VOLUME> --incremental
# cinder backup-create <VOLUME> --incrementalCopy to Clipboard Copied! Toggle word wrap Toggle overflow Replace <VOLUME> with the
IDorDisplay Nameof the volume that you want to back up.
You cannot delete a full backup if it already has an incremental backup. In addition, if a full backup has multiple incremental backups, you can only delete the latest backup.
3.3. Canceling a backup Copy linkLink copied to clipboard!
Procedure
To issue a cancellation on a backup, as an admin user, request a force delete on the backup:
openstack volume backup delete --force <backup>
# openstack volume backup delete --force <backup>Copy to Clipboard Copied! Toggle word wrap Toggle overflow
After you complete the cancellation and the backup no longer appears in the backup listings, there can be a slight delay for the backup to be successfully canceled. To verify that the backup is successfully canceled, the backing-up status in the source resource stops.
Before Red Hat OpenStack Platform (RHOSP) 12, the backing-up status was stored in the volume. This meant that if you backed up a snapshot, any delete operation on the snapshot after a cancellation could fail if the snapshot was still mapped. From RHOSP 13, you can cancel the ongoing restoring operation on any of the supported backup drivers.
3.4. Viewing and modifying backup quota of a tenant Copy linkLink copied to clipboard!
Normally, you can use the dashboard to modify tenant storage quotas, for example, the number of volumes, volume storage, snapshots, or other operational limits that a tenant can have. However, the functionality to modify backup quotas with the dashboard is not yet available.
You must use the command-line interface to modify backup quotas.
Procedure
To view the storage quotas of a specific tenant (TENANT_ID), run the following command:
cinder quota-show TENANT_ID
# cinder quota-show TENANT_IDCopy to Clipboard Copied! Toggle word wrap Toggle overflow To update the maximum number of backups (MAXNUM) that can be created in a specific tenant, run the following command:
cinder quota-update --backups MAXNUM TENANT_ID
# cinder quota-update --backups MAXNUM TENANT_IDCopy to Clipboard Copied! Toggle word wrap Toggle overflow To update the maximum total size of all backups (MAXGB) within a specific tenant, run the following command:
cinder quota-update --backup-gigabytes MAXGB TENANT_ID
# cinder quota-update --backup-gigabytes MAXGB TENANT_IDCopy to Clipboard Copied! Toggle word wrap Toggle overflow To view the storage quota usage of a specific tenant, run the following command:
cinder quota-usage TENANT_ID
# cinder quota-usage TENANT_IDCopy to Clipboard Copied! Toggle word wrap Toggle overflow
3.5. Restoring from backups Copy linkLink copied to clipboard!
3.5.1. Restoring a volume from a backup Copy linkLink copied to clipboard!
To create a new volume from a backup, complete the following steps.
Procedure
Find the ID of the volume backup you want to use:
cinder backup-list
# cinder backup-listCopy to Clipboard Copied! Toggle word wrap Toggle overflow Ensure that the Volume ID matches the ID of the volume that you want to restore.
Restore the volume backup:
cinder backup-restore _BACKUP_ID_
# cinder backup-restore _BACKUP_ID_Copy to Clipboard Copied! Toggle word wrap Toggle overflow Replace BACKUP_ID with the ID of the volume backup you want to use.
If you no longer need the backup, delete it:
openstack volume backup delete <backup_ID>
# openstack volume backup delete <backup_ID>Copy to Clipboard Copied! Toggle word wrap Toggle overflow If you need to restore a backed up volume to a volume of a particular type, you can use
--volumeto restore a backup to a specific volume:cinder backup-restore <BACKUP_ID --volume VOLUME_ID>
# cinder backup-restore <BACKUP_ID --volume VOLUME_ID>Copy to Clipboard Copied! Toggle word wrap Toggle overflow NoteIf you restore a volume from an encrypted backup, then the destination volume type must also be encrypted.
3.5.2. Restoring a volume after a Block Storage database loss Copy linkLink copied to clipboard!
When a Block Storage database loss occurs, you cannot restore a volume backup because the database contains metadata that the volume backup service requires. However, after you create the volume backup, you can export and store the metadata, which consists of backup_service and backup_url values, so that when a database loss occurs, you can restore the volume backup. For more information, see Section 3.1.3, “Exporting the metadata of a volume backup”.
For incremental backups, you must import all exported data before you can restore one of the incremental backups
Procedure
As a user with administrative privileges, run:
cinder backup-import <backup_service> <backup_url>
# cinder backup-import <backup_service> <backup_url>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Replace <backup_service> and <backup_url> with the metadata that you exported. For example, to use the exported metadata from Section 3.1.1, “Creating a full volume backup”, run the following command:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow - After the metadata is imported into the Block Storage service database, you can restore the volume. For more information, see Section 3.5.1, “Restoring a volume from a backup”.
3.5.3. Canceling a restoration Copy linkLink copied to clipboard!
To cancel a backup restoration, change the status to any value that is not restoring. For example, you can use either the error or available statuses.
openstack volume backup set --state error <BACKUP_ID>
$ openstack volume backup set --state error <BACKUP_ID>
When the status in the destination volume changes to available, the cancellation has been completed.
This feature is not currently available on RBD backups.
After a restore operation has started, if it is then cancelled, the destination volume is useless, as there is no way to know how much data, if any, was actually restored.
3.6. Troubleshooting Copy linkLink copied to clipboard!
There are two common scenarios that cause many of the issues that occur with the backup service:
-
When the
cinder-backupservice starts, it connects to its configured backend and uses this as a target for backups. Problems with this connection can cause services to fail. - When backups are requested, the backup service connects to the volume service and attaches the requested volume. Problems with this connection are evident only during backup time.
In either case, the logs contain messages that describe the error.
For more information about log files and services, see Log Files for OpenStack Services in the Logging, Monitoring and Troubleshooting Guide.
For more general information about log locations, and troubleshooting suggestions, see the Block Storage (cinder) Log Files in the Logging, Monitoring and Troubleshooting Guide.
3.6.1. Verifying services Copy linkLink copied to clipboard!
You can diagnose many issues by verifying that services are available and by checking log files for error messages. For more information about the key services and their interactions, see Section 1.2, “How backups and restores work”.
After you verify the status of the services, check the cinder-backup.log file. The Block Storage Backup service log is located in /var/log/containers/cinder]/cinder-backup.log.
Procedure
Run the
cinder showcommand on the volume to see if it is stored by the host.openstack volume show <volume>
# openstack volume show <volume>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Run the following command to view the services that are running:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow - Verify that the expected services are available.
3.6.2. Troubleshooting tips Copy linkLink copied to clipboard!
Backups are asynchronous. There are a relatively small number of static checks when there is an API request. These checks include an invalid volume reference (missing) or a volume that is in-use or attached to an instance. The in-use case requires you to use the --force option.
If you use the --force option, I/O is not quiesced and the resulting volume image might be corrupt.
If the API accepts the request, the backup occurs in the background. Usually, the CLI returns immediately even if the backup fails or is approaching failure. You can query the status of a backup by using the cinder backup API. If an error occurs, review the logs to discover the cause.
3.6.3. Pacemaker Copy linkLink copied to clipboard!
By default, Pacemaker deploys the Block Storage backup service. Pacemaker configures virtual IP addresses, containers, services, and other features as resources in a cluster to ensure that the defined set of OpenStack cluster resources are running and available. When a service or an entire node in a cluster fails, Pacemaker can restart the resource, take the node out of the cluster, or reboot the node. Requests to most services are through HAProxy.
For more information about how to use Pacemaker for troubleshooting, see Managing high availability services with Pacemaker in the High Availability Deployment and Usage guide.