Chapter 3. Using the Block Storage backup service
This chapter explains how to use the Block Storage backup service to perform full or incremental backups, and how to restore a backup to a volume. Basic troubleshooting tips are also provided.
3.1. Full backups Copy linkLink copied to clipboard!
3.1.1. Creating a full volume backup Copy linkLink copied to clipboard!
To back up a volume, use the cinder backup-create command. By default, this command creates a full backup of the volume. If the volume has existing backups, you can choose to create an incremental backup instead (see Section 2.4.1.2, “Create an Incremental Volume Backup” for details.)
You can create backups of volumes you have access to. This means that users with administrative privileges can back up any volume, regardless of owner. For more information, see Section 3.1.2, “Creating a volume backup as an admin”.
Procedure
View the ID or Display Name of the volume you wish to back up:
# cinder listBack up the volume:
# cinder backup-create _VOLUME_Replace VOLUME with the
IDorDisplay Nameof the volume you want to back up. For example:+-----------+--------------------------------------+ | Property | Value | +-----------+--------------------------------------+ | id | e9d15fc7-eeae-4ca4-aa72-d52536dc551d | | name | None | | volume_id | 5f75430a-abff-4cc7-b74e-f808234fa6c5 | +-----------+--------------------------------------+The
volume_idof the resulting backup is identical to the ID of the source volume.Verify that the volume backup creation is complete:
# cinder backup-list-
The volume backup creation is complete when the
Statusof the backup entry is available.
3.1.2. Creating a volume backup as an admin Copy linkLink copied to clipboard!
Users with administrative privileges (such as the default admin account) can back up any volume managed by OpenStack. When an admin backs up a volume owned by a non-admin user, the backup is hidden from the volume owner by default.
As an admin, you can still back up a volume and make the backup available to a specific tenant. To do so, run:
# cinder --os-auth-url _KEYSTONEURL_ --os-tenant-name _TENANTNAME_ --os-username _USERNAME_ --os-password _PASSWD_ backup-create _VOLUME_
Where:
- TENANTNAME is the name of the tenant where you want to make the backup available.
- USERNAME and PASSWD are the username/password credentials of a user within TENANTNAME.
- VOLUME is the name or ID of the volume you want to back up.
- KEYSTONEURL is the URL endpoint of the Identity service (typically http://IP:5000/v2, where IP is the IP address of the Identity service host). When performing this operation, the resulting backup’s size counts against the quota of TENANTNAME rather than the admin’s tenant.
3.1.3. Exporting a volume backup’s metadata Copy linkLink copied to clipboard!
You can also export and store the volume backup’s metadata. This allows you to restore the volume backup, even if the Block Storage database suffers a catastrophic loss.
To do so, run:
# cinder backup-export _BACKUPID_
Where BACKUPID is the ID or name of the volume backup. For example,
+----------------+------------------------------------------+
| Property | Value |
+----------------+------------------------------------------+
| backup_service | cinder.backup.drivers.swift |
| backup_url | eyJzdGF0dXMiOiAiYXZhaWxhYmxlIiwgIm9iam...|
| | ...4NS02ZmY4MzBhZWYwNWUiLCAic2l6ZSI6IDF9 |
+----------------+------------------------------------------+
The volume backup metadata consists of the backup_service and backup_url values.
3.1.4. Backing up an in-use volume Copy linkLink copied to clipboard!
You can also use the cinder backup-create command to create a backup of an in-use volume by adding the --force option.
The --force option relies on Block Storage back end snapshot support and should be supported by most drivers. You can verify snapshot support by checking the documentation for the back end you are using.
By using the --force option, you acknowledge that you are not quiescing the drive before performing the backup. Using this method creates a crash-consistent (but not application-consistent) backup. In other words, the backup will not have an awareness of which applications were running when the backup was performed. However, the data will be intact.
Procedure
To create a backup of an in-use volume, run:
# cinder backup-create _VOLUME_ --incremental --force
3.1.5. Backing up a snapshot Copy linkLink copied to clipboard!
When creating a backup from a snapshot, you can do a full backup or an incremental backup (by including the --incremental option). You must identify the volume ID associated with the snapshot.
Procedure
Locate the snapshot ID of the snapshot to backup using
cinder snapshot list.# cinder snapshot-list --volume-id _VOLUME_ID_If the snapshot is named, then you can use this example to locate the
ID:# cinder snapshot-show _SNAPSHOT_NAME_Create the backup of a snapshot:
# cinder backup-create _VOLUME_ --snapshot-id=_SNAPSHOT_ID_
3.2. Incremental backups Copy linkLink copied to clipboard!
The Block Storage backup service provides the option of performing incremental backups.
3.2.1. Performance considerations Copy linkLink copied to clipboard!
Some backup features like incremental and data compression may impact performance. Incremental backups have a performance impact because all of the data in a volume must be read and checksummed for both the full and each incremental backup.
Data compression can be used with non-Ceph backends. Enabling data compression requires additional CPU power but uses less network bandwidth and storage space overall.
Multipathing configuration also impacts performance. If multiple volumes are attached without multipathing enabled, you might not be able to connect and might not be using the full network capabilities, which impacts performance.
You can use the advanced configuration options (see Appendix B, Advanced Block Storage backup configuration options) to enable or disable compression, define the number of processes, and add additional CPU resources.
3.2.1.1. Backup from a snapshot impact Copy linkLink copied to clipboard!
Some back ends support creating a backup from a snapshot. A driver that supports this feature can directly attach a snapshot, which is faster than cloning the snapshot into a volume to be able to attach to it. In general, this feature can affect performance because of the extra step of creating the volume from a snapshot.
3.2.2. Performing incremental backups Copy linkLink copied to clipboard!
By default, the cinder backup-create command creates a full backup of a volume. However, if the volume has existing backups, you can choose to create an incremental backup.
Incremental backups are fully supported on NFS and Object Storage (swift).
Incremental backups are not supported for Ceph Storage volumes in this release.
An incremental backup captures any changes to the volume since the last backup (full or incremental). Performing numerous, regular, full backups of a volume can become resource-intensive as the volume’s size increases over time. In this regard, incremental backups allow you to capture periodic changes to volumes while minimizing resource usage.
To create an incremental volume backup, use the --incremental option. As in:
# cinder backup-create _VOLUME_ --incremental
Replace VOLUME with the ID or Display Name of the volume you want to back up.
You cannot delete a full backup if it already has an incremental backup. In addition, if a full backup has multiple incremental backups, you can only delete the latest one.
3.3. Canceling a backup Copy linkLink copied to clipboard!
To issue a backup cancellation on a backup, an admin must request a force delete on the backup.
# cinder backup-delete --force BACKUP ID
Even if the backup is immediately deleted, and therefore no longer appears in the listings, the cancellation may take a little bit longer. Check the status of the source resource to verify when the status stops being “backing-up”.
Before OpenStack version 12, the “backing-up” status would always be stored in the volume, even when backing up a snapshot, so when backing up a snapshot any delete operation on the snapshot that followed a cancellation could result in an error if the snapshot was still mapped. Since version 13, ongoing restoring operation can be canceled on any of the supported backup drivers.
3.4. Viewing and modifying a tenant’s backup quota Copy linkLink copied to clipboard!
Normally, you can use the dashboard to modify tenant storage quotas, for example, the number of volumes, volume storage, snapshots, or other operational limits that a tenant can have. However, the functionality to modify backup quotas with the dashboard is not yet available.
You must use the command-line interface to modify backup quotas with the cinder quota-update command.
To view the storage quotas of a specific tenant (TENANT_ID), run:
# cinder quota-show TENANT_ID
To update the maximum number of backups (MAXNUM) that can be created in a specific tenant, run:
# cinder quota-update --backups MAXNUM TENANT_ID
To update the maximum total size of all backups (MAXGB) within a specific tenant, run:
# cinder quota-update --backup-gigabytes MAXGB TENANT_ID
To view the storage quota usage of a specific tenant, run:
# cinder quota-usage TENANT_ID
3.5. Restoring from backups Copy linkLink copied to clipboard!
3.5.1. Restoring a volume from a backup Copy linkLink copied to clipboard!
These steps create a new volume from a backup.
Find the ID of the volume backup you want to use:
# cinder backup-list- The Volume ID should match the ID of the volume you want to restore.
Restore the volume backup:
# cinder backup-restore _BACKUP_ID_Where BACKUP_ID is the ID of the volume backup you wish to use.
If you no longer need the backup, delete it:
# cinder backup-delete _BACKUP_ID_
If you need to restore a backed up volume to a volume of a particular type, you can use --volume to restore a backup to a specific volume:
# cinder backup-restore _BACKUP_ID --volume VOLUME_ID_
3.5.2. Restoring a volume after a Block Storage database loss Copy linkLink copied to clipboard!
Typically, a Block Storage database loss prevents you from restoring a volume backup. This is because the Block Storage database contains metadata required by the volume backup service (openstack-cinder-backup). This metadata consists of backup_service and backup_url values, which you can export after creating the volume backup (as shown in Section 3.1.1, “Creating a full volume backup”).
If you exported and stored this metadata, then you can import it to a new Block Storage database (thereby allowing you to restore the volume backup).
For incremental backups, you must import all exported data before you can restore one of the incremental backups
Procedure
As a user with administrative privileges, run:
# cinder backup-import _backup_service_ _backup_url_Where backup_service and backup_url are from the metadata you exported. For example, using the exported metadata from Section 3.1.1, “Creating a full volume backup”:
# cinder backup-import cinder.backup.drivers.swift eyJzdGF0dXMi...c2l6ZSI6IDF9 +----------+--------------------------------------+ | Property | Value | +----------+--------------------------------------+ | id | 77951e2f-4aff-4365-8c64-f833802eaa43 | | name | None | +----------+--------------------------------------+- After the metadata is imported into the Block Storage service database, you can restore the volume as normal (see Section 3.5.1, “Restoring a volume from a backup”).
3.5.3. Canceling a restoration Copy linkLink copied to clipboard!
To issue a backup restoration cancellation, alter its status to anything other than restoring. You can use the “error” state to avoid any confusion on whether the restore was successful or not or the value can be changed to “available.”
$ openstack volume backup set --state error BACKUP_ID
The backup cancelation is an asynchronous action, as the backup driver needs to detect the status change before cancelling the backup. A change to “available” in the destination volume status means that the cancellation has been completed.
This feature is not currently available on RBD backups.
After a restore operation has started, if it is then cancelled, the destination volume is useless, as there is no way of knowing how much data, or if any, was actually restored.
3.6. Troubleshooting Copy linkLink copied to clipboard!
Many issues can be diagnosed by verifying services are available and looking in log files for error messages.
Two scenarios account for many issues with the backup service:
- When the cinder-backup service starts up, it connects to its configured backend and uses this as a target for backups. Problems with this connection can cause service to be down.
- When backups are requested, the backup service connects to the volume service and attaches the requested volume. Connection problems here won’t be seen until backup time.
In either case, messages describing the error should be present in the logs.
For general information, log locations, and troubleshooting suggestions, refer to the Logging, Monitoring and Troubleshooting Guide. Log files and services are listed in the Log Files for OpenStack Services section.
3.6.1. Verifying services Copy linkLink copied to clipboard!
One of the first steps in troubleshooting is to verify that the necessary services are available and to check the logs for additional clues. Section 1.2, “How do backups and restores work?” illustrates the key services and their interaction.
After you verify the status of the services, check the cinder-backup.log file. The Block Storage Backup service log is located in /var/log/containers/cinder]/cinder-backup.log.
Procedure
Use
cinder showon the volume to see if it is stored by the host.# cinder showUse
cinder service-listto view running services.# cinder service-list +------------------+--------------------+------+---------+-------+----------------------------+-----------------+ | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason | +------------------+--------------------+------+---------+-------+----------------------------+-----------------+ | cinder-backup | hostgroup | nova | enabled | up | 2017-05-15T02:42:25.000000 | - | | cinder-scheduler | hostgroup | nova | enabled | up | 2017-05-15T02:42:25.000000 | - | | cinder-volume | hostgroup@sas-pool | nova | enabled | down | 2017-05-14T03:04:01.000000 | - | | cinder-volume | hostgroup@ssd-pool | nova | enabled | down | 2017-05-14T03:04:01.000000 | - | +------------------+--------------------+------+---------+-------+----------------------------+-----------------+- Verify that the expected services are available.
3.6.2. Troubleshooting tips Copy linkLink copied to clipboard!
Backups themselves are asynchronous. A relatively small number of static checks are performed upon receiving an API request. These checks include an invalid volume reference (missing) or a volume that is in-use or attached to an instance. The in-use case requires you to use the --force option.
Using the --force option means that I/O will not be quiesced and the resulting volume image may be corrupt.
If the API accepts the request, the backup will take place in the background. Usually the CLI will return immediately even if the backup has failed or is about to fail. The status of a backup can be queried using cinder’s backup API. If an error does occur, review the logs for the cause.
3.6.3. Pacemaker Copy linkLink copied to clipboard!
The Block Storage backup service is deployed with Pacemaker by default. By configuring virtual IP addresses, containers, services, and other features as resources in a cluster, Pacemaker makes sure that the defined set of OpenStack cluster resources are running and available. When a service or an entire node in a cluster fails, Pacemaker can restart the resource, take the node out of the cluster, or reboot the node. Requests to most of those services are done through HAProxy.
For information on using Pacemaker for troubleshooting, refer to the Using Pacemaker chapter of the Understanding Red Hat OpenStack Platform High Availability guide.