Chapter 3. Using the Block Storage backup service
You can use the Block Storage backup service to perform full or incremental backups, and to restore a backup to a volume.
3.1. Full backups
3.1.1. Creating a full volume backup
To back up a volume, use the cinder backup-create
command. By default, this command creates a full backup of the volume. If the volume has existing backups, you can choose to create an incremental backup instead. For more information, Section 3.2.2, “Performing incremental backups”
If you are using a Ceph volume backed up to a Ceph cluster, then the second time you do a backup, the backup service automatically performs incremental backups.
You can create backups of the volumes that you have access to. This means that users with administrative privileges can back up any volume, regardless of the owner. For more information, see Section 3.1.2, “Creating a volume backup as an admin”.
Procedure
View the ID or Display Name of the volume you want to back up:
cinder list
Copy to clipboardCopied# cinder list
Back up the volume:
cinder backup-create <VOLUME>
Copy to clipboardCopied# cinder backup-create <VOLUME>
Replace <VOLUME> with the
ID
orDisplay Name
of the volume you want to back up:+-----------+--------------------------------------+ | Property | Value | +-----------+--------------------------------------+ | id | e9d15fc7-eeae-4ca4-aa72-d52536dc551d | | name | None | | volume_id | 5f75430a-abff-4cc7-b74e-f808234fa6c5 | +-----------+--------------------------------------+
Copy to clipboardCopied+-----------+--------------------------------------+ | Property | Value | +-----------+--------------------------------------+ | id | e9d15fc7-eeae-4ca4-aa72-d52536dc551d | | name | None | | volume_id | 5f75430a-abff-4cc7-b74e-f808234fa6c5 | +-----------+--------------------------------------+
The
volume_id
of the resulting backup is identical to the ID of the source volume.Verify that the volume backup creation is complete:
cinder backup-list
Copy to clipboardCopied# cinder backup-list
3.1.2. Creating a volume backup as an admin
The volume backup creation is complete when the Status
of the backup entry is available.
Users with administrative privileges can back up any volume managed by Red Hat OpenStack Platform. When an admin user backs up a volume that is owned by a non-admin user, the backup is hidden from the volume owner by default.
Procedure
As an admin user, you can use the following command to back up a volume and make the backup available to a specific tenant:
cinder --os-auth-url <KEYSTONEURL> --os-tenant-name <TENANTNAME> --os-username <USERNAME> --os-password <PASSWD> backup-create <VOLUME>
# cinder --os-auth-url <KEYSTONEURL> --os-tenant-name <TENANTNAME> --os-username <USERNAME> --os-password <PASSWD> backup-create <VOLUME>
Copy to clipboardCopiedReplace the following variables according to your environment requirements:
- <TENANTNAME> is the name of the tenant where you want to make the backup available.
- <USERNAME> and <PASSWD> are the username and password credentials of a user within <TENANTNAME>.
- <VOLUME> is the name or ID of the volume you want to back up.
-
<KEYSTONEURL> is the URL endpoint of the Identity service, which is typically http://IP:5000/v2, where
IP
is the IP address of the Identity service host. When you perform this operation, the size of the resulting backup counts against the quota ofTENANTNAME
rather than the quota of the tenant admin.
3.1.3. Exporting the metadata of a volume backup
You can export and store the metadata of a volume backup so that you can restore the volume backup even if the Block Storage database suffers a catastrophic loss.
Procedure
Run the following command:
cinder backup-export <BACKUPID>
Copy to clipboardCopied# cinder backup-export <BACKUPID>
Replace <BACKUPID> with the ID or name of the volume backup:
+----------------+------------------------------------------+ | Property | Value | +----------------+------------------------------------------+ | backup_service | cinder.backup.drivers.swift | | backup_url | eyJzdGF0dXMiOiAiYXZhaWxhYmxlIiwgIm9iam...| | | ...4NS02ZmY4MzBhZWYwNWUiLCAic2l6ZSI6IDF9 | +----------------+------------------------------------------+
Copy to clipboardCopied+----------------+------------------------------------------+ | Property | Value | +----------------+------------------------------------------+ | backup_service | cinder.backup.drivers.swift | | backup_url | eyJzdGF0dXMiOiAiYXZhaWxhYmxlIiwgIm9iam...| | | ...4NS02ZmY4MzBhZWYwNWUiLCAic2l6ZSI6IDF9 | +----------------+------------------------------------------+
The volume backup metadata consists of the backup_service
and backup_url
values.
3.1.4. Backing up an in-use volume
You can create a backup of an in-use volume with the --force
option when the Block Storage back end snapshot is supported.
To use the --force option, the Block Storage back end snapshot must be supported. You can verify snapshot support by checking the documentation for the back end that you are using.
By using the --force
option, you acknowledge that you are not quiescing the drive before you perform the backup. Using this method creates a crash-consistent, but not application-consistent, backup. This means that the backup does not have an awareness of which applications were running when the backup was performed. However, the data is intact.
Procedure
To create a backup of an in-use volume, run:
openstack volume backup create --incremental --force
Copy to clipboardCopied$ openstack volume backup create --incremental --force
3.1.5. Performing a full snapshot backup
You can create a full backup from a snapshot using the volume ID that is associated with the snapshot.
Procedure
Locate the ID of the snapshot that you want to back up:
cinder snapshot-list --volume-id _VOLUME_ID_
Copy to clipboardCopied# cinder snapshot-list --volume-id _VOLUME_ID_
If the snapshot is named, you can use the following example to locate the
ID
:cinder snapshot-show _SNAPSHOT_NAME_
Copy to clipboardCopied# cinder snapshot-show _SNAPSHOT_NAME_
Create the backup of the snapshot:
cinder backup-create _VOLUME_ --snapshot-id=_SNAPSHOT_ID_
Copy to clipboardCopied# cinder backup-create _VOLUME_ --snapshot-id=_SNAPSHOT_ID_
3.2. Incremental backups
You can use the Block Storage backup service to perform incremental backups.
3.2.1. Performance considerations
Some backup features like incremental and data compression can impact performance. Incremental backups have a performance impact because all of the data in a volume must be read and checksummed for both the full and each incremental backup.
You can use data compression with non-Ceph backends. Enabling data compression requires additional CPU power but uses less network bandwidth and storage space overall.
The multipathing configuration also impacts performance. If you attach multiple volumes without enabling multipathing, you might not be able to connect or have full network capabilities, which impacts performance.
You can use the advanced configuration options to enable or disable compression, define the number of processes, and add additional CPU resources. For more information, see Appendix B, Advanced Block Storage backup configuration options.
3.2.1.1. Impact of backing up from a snapshot
Some back ends support creating a backup from a snapshot. A driver that supports this feature can directly attach a snapshot, which is faster than cloning the snapshot into a volume to be able to attach to it. In general, this feature can affect performance because of the extra step of creating the volume from a snapshot.
3.2.2. Performing incremental backups
By default, the cinder backup-create
command creates a full backup of a volume. However, if the volume has existing backups, you can choose to create an incremental backup.
Incremental backups are fully supported on NFS, Object Storage (swift), and Red Hat Ceph Storage backup repositories.
An incremental backup captures any changes to the volume since the last full or incremental backup. Performing numerous, regular, full backups of a volume can become resource intensive because the size of the volume increases over time. With incremental backups, you can capture periodic changes to volumes and minimize resource usage.
Procedure
To create an incremental volume backup, use the
--incremental
option:cinder backup-create <VOLUME> --incremental
Copy to clipboardCopied# cinder backup-create <VOLUME> --incremental
Replace <VOLUME> with the
ID
orDisplay Name
of the volume that you want to back up.
You cannot delete a full backup if it already has an incremental backup. In addition, if a full backup has multiple incremental backups, you can only delete the latest backup.
3.3. Canceling a backup
Procedure
To issue a cancellation on a backup, as an admin user, request a force delete on the backup:
openstack volume backup delete --force <backup>
Copy to clipboardCopied# openstack volume backup delete --force <backup>
After you complete the cancellation and the backup no longer appears in the backup listings, there can be a slight delay for the backup to be successfully canceled. To verify that the backup is successfully canceled, the backing-up
status in the source resource stops.
Before Red Hat OpenStack Platform (RHOSP) 12, the backing-up
status was stored in the volume. This meant that if you backed up a snapshot, any delete operation on the snapshot after a cancellation could fail if the snapshot was still mapped. From RHOSP 13, you can cancel the ongoing restoring operation on any of the supported backup drivers.
3.4. Viewing and modifying backup quota of a tenant
Normally, you can use the dashboard to modify tenant storage quotas, for example, the number of volumes, volume storage, snapshots, or other operational limits that a tenant can have. However, the functionality to modify backup quotas with the dashboard is not yet available.
You must use the command-line interface to modify backup quotas.
Procedure
To view the storage quotas of a specific tenant (TENANT_ID), run the following command:
cinder quota-show TENANT_ID
Copy to clipboardCopied# cinder quota-show TENANT_ID
To update the maximum number of backups (MAXNUM) that can be created in a specific tenant, run the following command:
cinder quota-update --backups MAXNUM TENANT_ID
Copy to clipboardCopied# cinder quota-update --backups MAXNUM TENANT_ID
To update the maximum total size of all backups (MAXGB) within a specific tenant, run the following command:
cinder quota-update --backup-gigabytes MAXGB TENANT_ID
Copy to clipboardCopied# cinder quota-update --backup-gigabytes MAXGB TENANT_ID
To view the storage quota usage of a specific tenant, run the following command:
cinder quota-usage TENANT_ID
Copy to clipboardCopied# cinder quota-usage TENANT_ID
3.5. Restoring from backups
3.5.1. Restoring a volume from a backup
To create a new volume from a backup, complete the following steps.
Procedure
Find the ID of the volume backup you want to use:
cinder backup-list
Copy to clipboardCopied# cinder backup-list
Ensure that the Volume ID matches the ID of the volume that you want to restore.
Restore the volume backup:
cinder backup-restore _BACKUP_ID_
Copy to clipboardCopied# cinder backup-restore _BACKUP_ID_
Replace BACKUP_ID with the ID of the volume backup you want to use.
If you no longer need the backup, delete it:
openstack volume backup delete <backup_ID>
Copy to clipboardCopied# openstack volume backup delete <backup_ID>
If you need to restore a backed up volume to a volume of a particular type, you can use
--volume
to restore a backup to a specific volume:cinder backup-restore <BACKUP_ID --volume VOLUME_ID>
Copy to clipboardCopied# cinder backup-restore <BACKUP_ID --volume VOLUME_ID>
NoteIf you restore a volume from an encrypted backup, then the destination volume type must also be encrypted.
3.5.2. Restoring a volume after a Block Storage database loss
When a Block Storage database loss occurs, you cannot restore a volume backup because the database contains metadata that the volume backup service requires. However, after you create the volume backup, you can export and store the metadata, which consists of backup_service
and backup_url
values, so that when a database loss occurs, you can restore the volume backup. For more information, see Section 3.1.3, “Exporting the metadata of a volume backup”.
For incremental backups, you must import all exported data before you can restore one of the incremental backups
Procedure
As a user with administrative privileges, run:
cinder backup-import <backup_service> <backup_url>
Copy to clipboardCopied# cinder backup-import <backup_service> <backup_url>
Replace <backup_service> and <backup_url> with the metadata that you exported. For example, to use the exported metadata from Section 3.1.1, “Creating a full volume backup”, run the following command:
cinder backup-import cinder.backup.drivers.swift eyJzdGF0dXMi...c2l6ZSI6IDF9
Copy to clipboardCopied# cinder backup-import cinder.backup.drivers.swift eyJzdGF0dXMi...c2l6ZSI6IDF9 +----------+--------------------------------------+ | Property | Value | +----------+--------------------------------------+ | id | 77951e2f-4aff-4365-8c64-f833802eaa43 | | name | None | +----------+--------------------------------------+
- After the metadata is imported into the Block Storage service database, you can restore the volume. For more information, see Section 3.5.1, “Restoring a volume from a backup”.
3.5.3. Canceling a restoration
To cancel a backup restoration, change the status to any value that is not restoring
. For example, you can use either the error
or available
statuses.
openstack volume backup set --state error <BACKUP_ID>
$ openstack volume backup set --state error <BACKUP_ID>
Copy to clipboardCopied
When the status in the destination volume changes to available
, the cancellation has been completed.
This feature is not currently available on RBD backups.
After a restore operation has started, if it is then cancelled, the destination volume is useless, as there is no way to know how much data, if any, was actually restored.
3.6. Troubleshooting
There are two common scenarios that cause many of the issues that occur with the backup service:
-
When the
cinder-backup
service starts, it connects to its configured backend and uses this as a target for backups. Problems with this connection can cause services to fail. - When backups are requested, the backup service connects to the volume service and attaches the requested volume. Problems with this connection are evident only during backup time.
In either case, the logs contain messages that describe the error.
For more information about log files and services, see Log Files for OpenStack Services in the Logging, Monitoring and Troubleshooting Guide.
For more general information about log locations, and troubleshooting suggestions, see the Block Storage (cinder) Log Files in the Logging, Monitoring and Troubleshooting Guide.
3.6.1. Verifying services
You can diagnose many issues by verifying that services are available and by checking log files for error messages. For more information about the key services and their interactions, see Section 1.2, “How backups and restores work”.
After you verify the status of the services, check the cinder-backup.log
file. The Block Storage Backup service log is located in /var/log/containers/cinder]/cinder-backup.log
.
Procedure
Run the
cinder show
command on the volume to see if it is stored by the host.openstack volume show <volume>
Copy to clipboardCopied# openstack volume show <volume>
Run the following command to view the services that are running:
openstack volume service list
Copy to clipboardCopied# openstack volume service list +------------------+--------------------+------+---------+-------+----------------------------+-----------------+ | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason | +------------------+--------------------+------+---------+-------+----------------------------+-----------------+ | cinder-backup | hostgroup | nova | enabled | up | 2017-05-15T02:42:25.000000 | - | | cinder-scheduler | hostgroup | nova | enabled | up | 2017-05-15T02:42:25.000000 | - | | cinder-volume | hostgroup@sas-pool | nova | enabled | down | 2017-05-14T03:04:01.000000 | - | | cinder-volume | hostgroup@ssd-pool | nova | enabled | down | 2017-05-14T03:04:01.000000 | - | +------------------+--------------------+------+---------+-------+----------------------------+-----------------+
- Verify that the expected services are available.
3.6.2. Troubleshooting tips
Backups are asynchronous. There are a relatively small number of static checks when there is an API request. These checks include an invalid volume reference (missing
) or a volume that is in-use
or attached to an instance. The in-use
case requires you to use the --force
option.
If you use the --force
option, I/O is not quiesced and the resulting volume image might be corrupt.
If the API accepts the request, the backup occurs in the background. Usually, the CLI returns immediately even if the backup fails or is approaching failure. You can query the status of a backup by using the cinder backup API. If an error occurs, review the logs to discover the cause.
3.6.3. Pacemaker
By default, Pacemaker deploys the Block Storage backup service. Pacemaker configures virtual IP addresses, containers, services, and other features as resources in a cluster to ensure that the defined set of OpenStack cluster resources are running and available. When a service or an entire node in a cluster fails, Pacemaker can restart the resource, take the node out of the cluster, or reboot the node. Requests to most services are through HAProxy.
For more information about how to use Pacemaker for troubleshooting, see Managing high availability services with Pacemaker in the High Availability Deployment and Usage guide.