Backing up Block Storage volumes
Deploy and use the Red Hat OpenStack Platform Block Storage (cinder) backup service
Abstract
Making open source more inclusive
Red Hat is committed to replacing problematic language in our code, documentation, and web properties. We are beginning with these four terms: master, slave, blacklist, and whitelist. Because of the enormity of this endeavor, these changes will be implemented gradually over several upcoming releases. For more details, see our CTO Chris Wright’s message.
Providing feedback on Red Hat documentation
We appreciate your input on our documentation. Tell us how we can make it better.
Providing documentation feedback in Jira
Use the Create Issue form to provide feedback on the documentation. The Jira issue will be created in the Red Hat OpenStack Platform Jira project, where you can track the progress of your feedback.
- Ensure that you are logged in to Jira. If you do not have a Jira account, create an account to submit feedback.
- Click the following link to open a the Create Issue page: Create Issue
- Complete the Summary and Description fields. In the Description field, include the documentation URL, chapter or section number, and a detailed description of the issue. Do not modify any other fields in the form.
- Click Create.
Chapter 1. Block Storage backup service overview
The Block Storage service (cinder) of the Red Hat OpenStack Platform (RHOSP) provides an optional backup service that you can deploy on Controller nodes.
You can use the Block Storage backup service to create and restore full or incremental backups of your Block Storage volumes.
A volume backup is a persistent copy of the contents of a Block Storage volume that is saved to a backup repository.
Some features of the Block Storage backup service can impact the performance of the backups. For more information, see Backup performance considerations.
1.1. Backup repository back ends
By default, your backup repository uses the Red Hat OpenStack Platform Object Storage service (swift) back end and the volume backups are created as object stores. However, you can choose to use Red Hat Ceph Storage, NFS, or S3 as your backup repository back end.
The Block Storage backup service can back up volumes on any back end that the Block Storage service (cinder) supports, regardless of which back end you choose to use for your backup repository.
1.2. Block Storage volume backup metadata
When you create a backup of a Block Storage volume, the metadata for this backup is stored in the Block Storage service database. The Block Storage backup service uses this metadata when it restores the volume from the backup.
To ensure that a backup survives a catastrophic loss of the Block Storage service database, you can manually export and store the metadata of this backup. After a catastrophic database loss, you need to create a new Block Storage database and then manually re-import this backup metadata into it. For more information, see Protecting your backups.
Chapter 2. Deploying the Block Storage backup service
The Block Storage (cinder) backup service is optional. You must include it in your Red Hat OpenStack Platform (RHOSP) overcloud deployment to deploy it on your Controller nodes.
2.1. Deploying your active-active Block Storage backup service
Before Red Hat OpenStack Platform (RHOSP) 17.1, the Block Storage backup service was deployed in active-passive mode and was managed by Pacemaker.
In RHOSP 17.1, the Block Storage backup service is deployed in active-active mode and therefore runs on each Controller node and is not managed by Pacemaker.
When you upgrade to RHOSP 17.1, the Block Storage backup service remains in active-passive mode.
If you choose to use the Block Storage backup service, then you must include it in your RHOSP 17.1 overcloud deployment.
Prerequisites
- An available storage source for your backup repository that uses one of the following back ends: Object Storage (swift), Red Hat Ceph Storage, NFS, or S3.
Procedure
-
Log in to the undercloud host as the
stack
user. Source the
stackrc
undercloud credentials file:$ source ~/stackrc
Add this environment file to the stack with your other environment files:
/usr/share/openstack-tripleo-heat-templates/environments/cinder-backup-active-active.yaml
This file deploys the Block Storage backup service in active-active mode and sets all the heat template parameters of this service to their default settings. The default settings configure your backup repository to use the Object Storage (swift) back end and the
zlib
data compression algorithm.If the default configuration meets your deployment requirements, then you do not need to do anything further and you can deploy the overcloud.
If you need to use another back end for your backup repository or to change other default values:
Add these parameters and their new values to the
parameter_defaults
section of either a new or existing environment file. For more information about the parameters that you can change, see Changing the default Block Storage backup service parameter values.For example, a new environment file
/home/stack/templates/custom_backup_environment_file.yaml
specifies an NFS back end and changes the data compression algorithm tozstd
:parameter_defaults: CinderBackupBackend: nfs CinderBackupNfsShare: 192.168.1.1:/var/export/cinder-backup CinderBackupCompressionAlgorithm: zstd
Add the environment file that contains your specific parameter values to the stack with your other environment files, after the
/usr/share/openstack-tripleo-heat-templates/environments/cinder-backup-active-active.yaml
file and deploy the overcloud. In this example:$ openstack overcloud deploy --templates -e [your other environment files] -e /usr/share/openstack-tripleo-heat-templates/environments/cinder-backup-active-active.yaml -e /home/stack/templates/custom_backup_environment_file.yaml
Verification:
- Ensure that the Block Storage services are running correctly on their hosts and then verify that the Block Storage backup service is deployed successfully. For more information, see Verifying the Block Storage backup service deployment.
2.2. Changing the default Block Storage backup service parameter values
When you deploy the Block Storage backup service, it implements default settings for its heat template parameters. For more information, see Deploying your active-active Block Storage backup service.
You can provide your deployment specific values for these parameters.
Procedure
- Select and configure the back end for your backup repository. For more information, see Backup repository back-end configuration.
- Implement the Block Storage backup service configuration supported by your selected back end. For more information, see Block Storage backup service configuration.
2.2.1. Backup repository back-end configuration
Select and configure one of the following back ends for your backup repository.
2.2.1.1. OpenStack Object Storage service (swift) back end
Parameter | Description | Value |
CinderBackupBackend | The back end of your backup repository. Note There are no additional parameters for this default back end. |
The default value. |
2.2.1.2. NFS back end
Parameter | Description | Value |
CinderBackupBackend | The back end of your backup repository. |
|
CinderBackupNfsShare | The remote NFS share that you want to mount to store your backups. Ensure that you specify the server name or IP followed by the export. | |
CinderBackupNfsMountOptions | Optional: A comma-delimited list of options for mounting your NFS share. |
2.2.1.3. Red Hat Ceph Storage back end
Parameter | Description | Value |
CinderBackupBackend | The back end of your backup repository. |
|
CinderBackupRbdPoolName | The RBD pool name of your Ceph cluster that stores your backups. |
|
2.2.1.4. S3 back end
Parameter | Description | Value |
CinderBackupBackend | The back end of your backup repository. |
|
CinderBackupS3Bucket | The S3 bucket that stores your backups. Note Ensure that you create this bucket on the S3 back end and that you have configured the necessary permissions to write to this bucket, before you deploy the Block Storage backup service. |
|
CinderBackupS3AccessKey | The S3 Access key to connect to your S3 bucket. | |
CinderBackupS3SecretKey | The S3 Secret key to connect to your S3 bucket. | |
CinderBackupS3EndpointUrl | The URL of your S3 endpoint. |
2.2.2. Block Storage backup service configuration
You can implement any Block Storage backup service parameter that is supported by your selected back end.
Parameter | Description | Value |
CinderBackupCompressionAlgorithm | If your back end supports it, you can enable the data compression of your backup repository. Data compression requires additional CPU power but uses less network bandwidth and storage space. Note You cannot specify the data compression algorithm for the Red Hat Ceph Storage back end driver. This parameter is ignored for this back end. |
Alternatives:
|
Chapter 3. Using the Block Storage backup service
You can use the Block Storage backup service to perform full or incremental backups, to protect your backups, and to restore a backup to a volume.
3.1. Creating backups
Create a backup of your Block Storage volume to protect your data from being lost if the volume fails. For more information, see Creating a full volume backup. You can also create a backup directly from a snapshot of a volume. For more information, see Creating a full snapshot backup. In addition to the volume data, a backup also stores the volume metadata, such as the name and description.
If you have enabled data compression for your backup repository then your backups will be compressed, which can reduce their performance.
Full backups are easier to manage but they can become resource intensive when the size of the volume increases over time. With incremental backups, you can capture periodic changes to volumes and minimize resource usage. For more information, see Incremental backups.
You can create backups of volumes that you have access to. Project administrators can back up any volume belonging to the project. These backups are hidden from the volume owners unless the administrator provides additional arguments when creating the backup. For more information, see Backup arguments to authenticate volume owners.
Each project (tenant) limits the maximum number of backups and the maximum total size of all backups that can be created for it. As a project administrator, you can view and change these quotas. For more information, see Viewing and modifying project backup quotas.
Normally you can only backup a volume that has an available
status but, if you need to, you can backup a volume with an in-use
status. For more information, see Creating a backup of an in-use volume.
When you create a backup of a Block Storage volume, the metadata for this backup is stored in the Block Storage service database, which is used when restoring this volume. To ensure that a backup survives a catastrophic loss of the Block Storage service database, a project administrator can manually export and store the metadata of this backup. For more information, see Protecting your backups.
3.1.1. Creating a full volume backup
You can create one or more full backups of a volume.
Prerequisites
- Only volume owners and project administrators can backup volumes.
- Your backup repository must have the necessary space.
- The backup quotas specified for your project have not been exceeded. For more information, see Viewing and modifying project backup quotas.
Procedure
-
Log in to the undercloud host as the
stack
user. Source the
stackrc
undercloud credentials file:$ source ~/stackrc
List the volumes to obtain the ID or name of the volume you want to back up:
$ openstack volume list
NoteUsually you can only backup a volume that has an
available
status but, if you need to, you can backup a volume with anin-use
status. For more information, see Creating a backup of an in-use volume.Back up the volume:
NoteIf you are the project administrator and not the volume owner, then to make this backup accessible by the volume owner, you need to provide additional parameters when creating this backup. For more information, see Backup arguments to authenticate volume owners.
$ openstack volume backup create [--name <backup_name>] <volume>
-
Replace
<volume>
with the ID or name of the volume you want to back up. Optional: replace
<backup_name>
with the name of this backup.This command immediately provides the ID of this backup but the volume is backed up asynchronously, in the background. For example:
$ openstack volume backup create --name vol1bu2 vol_1 +-------+--------------------------------------+ | Field | Value | +-------+--------------------------------------+ | id | 83dadc43-2aa9-4c0b-bc05-a12203a8f4cb | | name | vol1bu2 | +-------+--------------------------------------+
-
Replace
Verification
List the backups:
$ openstack volume backup list
The volume backup is created when this backup has an
available
status. For example:+--------------------------------------+---------+-------------+-----------+------+ | ID | Name | Description | Status | Size | +--------------------------------------+---------+-------------+-----------+------+ | 83dadc43-2aa9-4c0b-bc05-a12203a8f4cb | vol1bu2 | None | available | 1 | | b604a932-94a5-468e-bf6b-841ac16f69a8 | None | None | available | 1 | +--------------------------------------+---------+-------------+-----------+------+
Additional resources
3.1.2. Creating a full snapshot backup
You can create a full backup from a snapshot by using the ID of the volume associated with the snapshot.
The backup is created by directly attaching to the snapshot, which is faster than cloning the snapshot into a volume and then backing up this volume. But this feature can affect the backup performance because of the extra step of creating the volume from a snapshot.
Prerequisites:
- Your backup repository must have the necessary space.
- The backup quotas specified for your project have not been exceeded. For more information, see Viewing and modifying project backup quotas.
Procedure
-
Log in to the undercloud host as the
stack
user. Source the
stackrc
undercloud credentials file:$ source ~/stackrc
List the snapshots to obtain the name or ID of the snapshot you want to backup:
$ openstack volume snapshot list
List the details of this snapshot to obtain the ID of the volume associated with this snapshot:
$ openstack volume snapshot show <snapshot>
Replace
<snapshot>
with the name or ID of the snapshot you want to backup.The value of the
volume_id
field is the ID of the volume associated with this snapshot. For example:$ openstack volume snapshot show snap_1 +--------------------------------------------+--------------------------------------+ | Field | Value | +--------------------------------------------+--------------------------------------+ | created_at | 2023-07-18T08:14:16.000000 | | description | None | | id | 2d95b707-6657-49af-ac8f-f9a7417d4650 | | name | snap_1 | | os-extended-snapshot-attributes:progress | 100% | | os-extended-snapshot-attributes:project_id | c2c1da89ed1648fc8b4f35a045f8d34c | | properties | | | size | 1 | | status | available | | updated_at | 2023-07-18T08:14:17.000000 | | volume_id | 6841e3d1-8a1a-4496-bc51-f7c04b787e8f | +--------------------------------------------+--------------------------------------+
Backup the snapshot:
$ openstack volume backup create [--name <backup_name>] --snapshot <snapshot> <volume_id>
-
Replace
<volume_id>
with the ID of the volume associated with this snapshot. Optional: replace
<backup_name>
with the name of this backup.This command immediately provides the ID of this backup but the snapshot is backed up asynchronously, in the background. For example:
$ openstack volume backup create --name snap1bu1 --snapshot snap_1 6841e3d1-8a1a-4496-bc51-f7c04b787e8f +-------+--------------------------------------+ | Field | Value | +-------+--------------------------------------+ | id | 867e6cfb-9be7-47fa-8a79-221b0e80c757 | | name | snap1bu1 | +-------+--------------------------------------+
-
Replace
Verification
List the backups:
$ openstack volume backup list
The snapshot backup is created when this backup has an
available
status. For example:+--------------------------------------+------------+-------------+-----------+------+ | ID | Name | Description | Status | Size | +--------------------------------------+------------+-------------+-----------+------+ | 867e6cfb-9be7-47fa-8a79-221b0e80c757 | snap1bu1 | None | available | 1 | +--------------------------------------+------------+-------------+-----------+------+
3.1.3. Creating a backup of an in-use volume
Usually you can only backup a volume that has an available
status. But you can use the --force
option when creating a backup, to back up a volume that has an in-use
status.
When you use the --force
volume backup option, you create a crash-consistent, but not an application-consistent, backup because the volume is not quiesced before performing the backup. Therefore, the data is intact but the backup does not have an awareness of which applications were running when the backup was performed.
Prerequisites
- Only volume owners and project administrators can backup volumes.
- Your backup repository must have the necessary space.
- The backup quotas specified for your project have not been exceeded. For more information, see Viewing and modifying project backup quotas.
Procedure
-
Log in to the undercloud host as the
stack
user. Source the
stackrc
undercloud credentials file:$ source ~/stackrc
List the volumes to obtain the ID or name of the volume you want to back up:
$ openstack volume list
For example:
+--------------------------------------+----------------+-----------+------+--------------------------------+ | ID | Name | Status | Size | Attached to | +--------------------------------------+----------------+-----------+------+--------------------------------+ | 6841e3d1-8a1a-4496-bc51-f7c04b787e8f | vol_1 | available | 1 | | | 92800cf6-82ae-448a-a2bb-872fa4d98099 | Pansible_vol_2 | in-use | 1 | Attached to inst1 on /dev/vdc | +--------------------------------------+----------------+-----------+------+--------------------------------+
Force the back up, if the volume that you want to backup has an
in-use
status:$ openstack volume backup create [--name <backup_name>] --force <volume>
-
Replace
<volume>
with the ID or name of the volume you want to back up. Optional: replace
<backup_name>
with the name of this backup.This command immediately provides the ID of this backup but the volume is backed up asynchronously, in the background. For example:
$ openstack volume backup create --name panvol2bu1 --force Pansible_vol_2 +-------+--------------------------------------+ | Field | Value | +-------+--------------------------------------+ | id | 8c72bbf3-eb8e-4459-83e9-c7654ebe6343 | | name | panvol2bu1 | +-------+--------------------------------------+
-
Replace
Verification
List the backups:
$ openstack volume backup list
The volume backup is created when this backup has an
available
status. For example:+--------------------------------------+------------+-------------+-----------+------+ | ID | Name | Description | Status | Size | +--------------------------------------+------------+-------------+-----------+------+ | 8c72bbf3-eb8e-4459-83e9-c7654ebe6343 | panvol2bu1 | None | available | 1 | +--------------------------------------+------------+-------------+-----------+------+
Additional resources
3.1.4. Incremental backups
If a volume has at least one full backup, you can use the Block Storage backup service to create an incremental backup. For more information, see Creating an incremental backup.
Full backups are easier to manage but they can become resource intensive when the size of the volume increases over time. With incremental backups, you can capture periodic changes to volumes and minimize your resource usage.
An incremental backup only stores the changes made to the volume since the last full or incremental backup.
Incremental backups increase the administrative overhead required for managing your backups. For instance, you cannot delete a full backup if it already has one or more incremental backups, you can only delete the latest incremental backup.
Incremental backups have a lower performance than full backups: When you create an incremental backup, all of the data in the volume must first be read and compared with the data in both the full backup and each subsequent incremental backup.
3.1.4.1. Creating an incremental backup
You can create an incremental backup to only store the changes made to the volume since the last full or incremental backup.
Prerequisites:
- At least one full backup of the volume. For more information, see Creating a full volume backup.
- Only volume owners and project administrators can backup volumes.
- Your backup repository must have the necessary space.
- The backup quotas specified for your project have not been exceeded. For more information, see Viewing and modifying project backup quotas.
Procedure
-
Log in to the undercloud host as the
stack
user. Source the
stackrc
undercloud credentials file:$ source ~/stackrc
List the volumes to obtain the ID or name of the volume you want to back up:
$ openstack volume list
Back up the volume and use the
--incremental
option:$ openstack volume backup create --incremental [--name <backup_name>] <volume>
-
Replace
<volume>
with the ID or name of the volume you want to back up. Optional: replace
<backup_name>
with the name of this backup.This command immediately provides the ID of this backup but the volume is backed up asynchronously, in the background. For example:
$ openstack volume backup create --name vol3incbu1 --incremental vol_3 +-------+--------------------------------------+ | Field | Value | +-------+--------------------------------------+ | id | f1681313-b5ed-4520-9b63-5b533f7cdc11 | | name | vol3incbu1 | +-------+--------------------------------------+
-
Replace
Verification
List the backups:
$ openstack volume backup list
The volume backup is created when this backup has an
available
status. For example:+--------------------------------------+---------+-------------+-----------+------+ | ID | Name | Description | Status | Size | +--------------------------------------+---------+-------------+-----------+------+ | f1681313-b5ed-4520-9b63-5b533f7cdc11 | vol3incbu1 | None | available | 1 | | f0e9ba67-67e1-4c2c-96ce-221df75bf2c2 | vol3bu1 | None | available | 1 | +--------------------------------------+---------+-------------+-----------+------+
Additional resources
3.1.5. Backup performance considerations
Some features of the Block Storage backup service like incremental backups and data compression can reduce the performance of the backups.
By only capturing the periodic changes to volumes, incremental backups can minimize your resource usage. For more information, see Incremental backups. But incremental backups have a lower performance than full backups: When you create an incremental backup, all of the data in the volume must first be read and compared with the data in both the full backup and each subsequent incremental backup.
You can also create a backup from a snapshot by directly attaching to it, which is faster than cloning the snapshot into a volume. For more information, see Creating a full snapshot backup. But this feature can affect the backup performance because of the extra step of creating the volume from a snapshot.
Enabling data compression of your backup repository requires additional CPU power but uses less network bandwidth and storage space overall. You can configure the Block Storage backup service to enable or disable data compression of your backup repository. For more information, see Block Storage backup service configuration.
You cannot use data compression with the Red Hat Ceph Storage back end.
3.1.6. Backup arguments to authenticate volume owners
Project administrators can back up any volume belonging to the project, but these backups are hidden from the volume owners.
To ensure that the volume owner can also access the volume backup, the project administrator must provide the following additional arguments to authenticate as the volume owner when backing up the volume:
$ openstack --os-auth-url <keystoneurl> --os-project-name <projectname> --os-username <username> --os-password <password> volume backup create [--name <backup_name>] <volume>
-
Replace
<keystoneurl>
with the URL endpoint of the Identity service, which is typically http://IP:5000/v3, whereIP
is the IP address of the Identity service host. -
Replace
<projectname>
with the name of the project (tenant) of the owner of the volume. -
Replace
<username>
and<password>
with the username and password credentials of the user that is the owner of the volume within this project.
[--name <backup_name>] <volume>
are the typical arguments when creating a volume backup.
3.1.7. Viewing and modifying project backup quotas
A project administrator can change or view the maximum number of backups and the maximum total size of all backups, in gigabytes, that can be created for a specific project (tenant) and see the usage of these backup quotas for this project.
Prerequisites
- You must be a project administrator to view or modify the backup quotas of a project.
Procedure
-
Log in to the undercloud host as the
stack
user. Source the
stackrc
undercloud credentials file:$ source ~/stackrc
List the projects to obtain the ID or name of the required project:
$ openstack project list
View the backup quotas for a specific project:
$ openstack quota show <project>
-
Replace
<project>
with the ID or name of the required project.
The value of the
backup-gigabytes
field in the table is the maximum total size of all backups that can be created in this project. The value of thebackups
field in the table is the maximum number of backups that can be created in this project. For example:+
$ openstack quota show c2c1da89ed1648fc8b4f35a045f8d34c +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | backup-gigabytes | 1000 | | backups | 10
-
Replace
Modify the maximum total size of all backups created for a project:
$ openstack quota set --backup-gigabytes <maxgb> <project>
-
Replace
<maxgb>
with the maximum total size, in gigabytes, of the backups that can be created for this project.
-
Replace
Modify the maximum number of backups that can be created for a project:
$ openstack quota set --backups <maxnum> <project>
-
Replace
<maxnum>
with the maximum number of backups that can be created for this project.
-
Replace
View the usage of these backup quotas for a specific project:
$ cinder quota-usage <project_id>
Replace
<project_id>
with the ID of the project.For example:
$ cinder quota-usage c2c1da89ed1648fc8b4f35a045f8d34c +-----------------------+--------+----------+-------+-----------+ | Type | In_use | Reserved | Limit | Allocated | +-----------------------+--------+----------+-------+-----------+ | backup_gigabytes | 7 | 0 | 1000 | | | backups | 7 | 0 | 10 | | | gigabytes | 6 | 0 | 1000 | | | gigabytes_multiattach | 0 | 0 | -1 | | | gigabytes_tripleo | 6 | 0 | -1 | | | groups | 0 | 0 | 10 | | | per_volume_gigabytes | 0 | 0 | -1 | | | snapshots | 1 | 0 | 10 | | | snapshots_multiattach | 0 | 0 | -1 | | | snapshots_tripleo | 1 | 0 | -1 | | | volumes | 5 | 0 | 10 | | | volumes_multiattach | 0 | 0 | -1 | | | volumes_tripleo | 5 | 0 | -1 | | +-----------------------+--------+----------+-------+-----------+
Verification
If you have changed either of these quotas then review these changes:
$ openstack quota show <project>
Ensure that the modified values are specified by the
backup-gigabytes
andbackups
fields in the table. For example:+-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | backup-gigabytes | 500 | | backups | 12
3.1.8. Canceling a backup
You must request a force delete on the backup to cancel it.
You cannot cancel backups if you use the Red Hat Ceph Storage back end for your backup repository.
Procedure
-
Log in to the undercloud host as the
stack
user. Source the
stackrc
undercloud credentials file:$ source ~/stackrc
List the backups to obtain the ID or name of the backup you want to cancel:
$ openstack volume backup list
Cancel the backup:
# openstack volume backup delete --force <backup>
Replace
<backup>
with the ID or name of the volume backup that you want to cancel.There can be a slight delay for the backup to be successfully canceled.
Verification
The backup is canceled when the backup record is not listed by this command:
$ openstack volume backup show <backup>
3.2. Backing up and restoring across edge sites
You can back up and restore Block Storage service (cinder) volumes across distributed compute node (DCN) architectures in edge site and availability zones. The cinder-backup
service runs in the central availability zone (AZ), and backups are stored in the central AZ. The Block Storage service does not store backups at DCN sites.
Prerequisites
- Deploy the optional Block Storage backup service. For more information, see Block Storage backup service deployment in Backing up Block Storage volumes.
- Block Storage (cinder) REST API microversion 3.51 or later.
-
All sites must use a common
openstack
cephx client name. For more information, see Creating a Ceph key for external access in Deploying a Distributed Compute Node (DCN) architecture.
Procedure
Create a backup of a volume in the first DCN site:
$ cinder --os-volume-api-version 3.51 backup-create --name <volume_backup> --availability-zone <az_central> <edge_volume>
-
Replace
<volume_backup>
with a name for the volume backup. -
Replace
<az_central>
with the name of the central availability zone that hosts thecinder-backup
service. Replace
<edge_volume>
with the name of the volume that you want to back up.NoteIf you experience issues with Ceph keyrings, you might need to restart the
cinder-backup
container so that the keyrings copy from the host to the container successfully.
-
Replace
Restore the backup to a new volume in the second DCN site:
$ cinder --os-volume-api-version 3.51 create --availability-zone <az_2> --name <new_volume> --backup-id <volume_backup> <volume_size>
-
Replace
<az_2>
with the name of the availability zone where you want to restore the backup. -
Replace
<new_volume>
with a name for the new volume. -
Replace
<volume_backup>
with the name of the volume backup that you created in the previous step. -
Replace
<volume_size>
with a value in GB equal to or greater than the size of the original volume.
-
Replace
3.3. Protecting your backups
When you create a backup of a Block Storage volume, the metadata for this backup is stored in the Block Storage service database, which is used to restore this volume. To ensure that a backup survives a catastrophic loss of the Block Storage service database, a project administrator can manually export and store the metadata of this backup in a safe location, such as an offsite backup. For more information, see Exporting backup metadata.
When the Block Storage service database experiences a catastrophic loss you cannot restore any of your backups because this database contains the backup metadata used when restoring backups. But if a project administrator manually exported and saved the metadata of a backup, then the project administrator can import this metadata into the new Block Storage database, so that you can use this backup to restore the volume. For more information see Importing backup metadata.
For incremental backups, you must import the metadata of all the preceding backups before you can use it to restore the volume.
3.3.1. Exporting backup metadata
A project administrator can export the metadata of a backup and store it in a file so that you can restore the volume backup even if the Block Storage database suffers a catastrophic loss. For more information, see Protecting your backups.
For an incremental backup, you must export the metadata of all the preceding backups.
Prerequisites
- You must be a project administrator to export the backup metadata.
Procedure
-
Log in to the undercloud host as the
stack
user. Source the
stackrc
undercloud credentials file:$ source ~/stackrc
List the backups to obtain the ID or name of the backup:
$ openstack volume backup list
Export the metadata of the backup and store it in an appropriately named YAML file:
$ openstack volume backup record export -f yaml <backup> > <filename>.yaml
-
Replace
<backup>
with the ID or name of the volume backup. Replace
<filename>
with the name of the YAML file to save the exportedbackup_service
andbackup_url
values for this backup.For example:
$ openstack volume backup record export -f yaml vol1bu2 > vol1bu2.yaml
-
Replace
- Copy the file to a safe location, such as an offsite backup.
Verification
Edit the file to see that the values of the
backup_service
andbackup_url
, match the values provided by this command:$ openstack volume backup record export -f yaml <backup>
For example:
$ openstack volume backup record export -f yaml vol1bu2 backup_service: cinder.backup.drivers.ceph.CephBackupDriver backup_url: eyJkcml2 … YWxzZX0=
3.3.2. Importing backup metadata
If a project administrator has exported and saved the metadata of a volume backup, then after a catastrophic loss of the Block Storage service database, the project administrator can import this metadata, so that you can use this backup.
You can also use this procedure to recreate a backup that was deleted.
For incremental backups, you must also import the metadata of all the preceding backups.
Prerequisites
- You must be a project administrator to import the volume backup metadata to a Block Storage database.
-
You must provide the
backup_service
andbackup_url
metadata values of this backup. For more information see Exporting backup metadata. - A Block Storage database that does not already contain this backup.
Procedure
-
Log in to the undercloud host as the
stack
user. Source the
stackrc
undercloud credentials file:$ source ~/stackrc
-
Locate the file in which you stored the exported
backup_service
andbackup_url
metadata values of this backup. Import the metadata values of this volume backup to the Block Storage database:
$ openstack volume backup record import <backup_service> <backup_url>
-
Replace
<backup_service>
with thebackup_service
metadata value of this volume backup. Replace
<backup_url>
with thebackup_url
metadata value of this volume backup.This command provides the name and the ID of this backup. For example:
$ openstack volume backup record import cinder.backup.drivers.ceph.CephBackupDriver eyJkcml2 … YWxzZX0= +-------+--------------------------------------+ | Field | Value | +-------+--------------------------------------+ | id | 83dadc43-2aa9-4c0b-bc05-a12203a8f4cb | | name | vol1bu2 | +-------+--------------------------------------+
-
Replace
Next steps
3.4. Restoring backups
After you create a Block Storage volume backup, you can restore this backed up data if needed.
You can use one of the following methods to restore your backups:
- Restore the backup to a volume that you specify. For more information, see Restoring a backup to a specific volume.
- Restore the backup to a new volume. For more information, see Restoring a backup to a new volume.
When the Block Storage service database experiences a catastrophic loss you cannot restore any of your backups, unless you have exported and saved their metadata. For more information see Protecting your backups.
Only project administrators can cancel restoring a volume backup. For more information, see Canceling restoring a backup.
3.4.1. Restoring a backup to a specific volume
You can restore a volume backup to an available
volume that you have already created.
If you restore a volume from an encrypted backup, then the destination volume type must also be encrypted.
Procedure
-
Log in to the undercloud host as the
stack
user. Source the
stackrc
undercloud credentials file:$ source ~/stackrc
List the backups to obtain the name or ID of the backup you want to restore:
$ openstack volume backup list
For example:
+--------------------------------------+---------+-------------+-----------+------+ | ID | Name | Description | Status | Size | +--------------------------------------+---------+-------------+-----------+------+ | 83dadc43-2aa9-4c0b-bc05-a12203a8f4cb | vol1bu2 | None | available | 1 |
List the volumes:
$ openstack volume list
Ensure that the status of the required volume is
available
and then obtain the name or ID of this volume. For example:+--------------------------------------+----------------+-----------+------+--------------------------------+ | ID | Name | Status | Size | Attached to | +--------------------------------------+----------------+-----------+------+--------------------------------+ | 654e2be8-bc79-4528-96a7-5f773d31c201 | vol_3 | available | 1 | |
Restore the backup to the volume:
$ openstack volume backup restore <backup> <volume>
-
Replace
<backup>
with the name or ID of the Block Storage volume backup. Replace
<volume>
with the name or ID of theavailable
Block Storage volume.For example:
$ openstack volume backup restore vol1bu2 vol_3 +-------------+--------------------------------------+ | Field | Value | +-------------+--------------------------------------+ | backup_id | 83dadc43-2aa9-4c0b-bc05-a12203a8f4cb | | volume_id | 654e2be8-bc79-4528-96a7-5f773d31c201 | | volume_name | vol_3 | +-------------+--------------------------------------+
-
Replace
-
Verify that the
backup_id
provided by this command corresponds to the ID of the backup that was restored and that thevolume_name
andvolume_id
values correspond to the name and ID of the specified volume. Delete the backup if you no longer need it:
$ openstack volume backup delete <backup>
3.4.2. Restoring a backup to a new volume
You can create a new volume when you restore a backup of a Block Storage volume.
Procedure
-
Log in to the undercloud host as the
stack
user. Source the
stackrc
undercloud credentials file:$ source ~/stackrc
List the backups to obtain the name or ID of the backup you want to restore:
$ openstack volume backup list
For example:
+--------------------------------------+---------+-------------+-----------+------+ | ID | Name | Description | Status | Size | +--------------------------------------+---------+-------------+-----------+------+ | 83dadc43-2aa9-4c0b-bc05-a12203a8f4cb | vol1bu2 | None | available | 1 |
Restore the backup to a new volume:
$ cinder backup-restore <backup>
Replace
<backup>
with the name or ID of the Block Storage volume backup.For example:
$ cinder backup-restore vol1bu2 +-------------+-----------------------------------------------------+ | Property | Value | +-------------+-----------------------------------------------------+ | backup_id | 83dadc43-2aa9-4c0b-bc05-a12203a8f4cb | | volume_id | 296c853c-c749-4eb6-857a-57ec182232a6 | | volume_name | restore_backup_83dadc43-2aa9-4c0b-bc05-a12203a8f4cb | +-------------+-----------------------------------------------------+
Verify that the
backup_id
provided by this command corresponds to the ID of the backup that was restored.The
volume_id
value is the ID of the created volume. But thevolume_name
can be a temporary name that is replaced with the name of the backed up volume.List the volumes to verify that the volume with an ID of
volume_id
has been created and to obtain this volume name:$ openstack volume list
For example:
+--------------------------------------+----------------+-----------+------+--------------------------------+ | ID | Name | Status | Size | Attached to | +--------------------------------------+----------------+-----------+------+--------------------------------+ | 296c853c-c749-4eb6-857a-57ec182232a6 | vol_1 | available | 1 | |
Delete the backup if you no longer need it:
$ openstack volume backup delete <backup>
3.4.3. Canceling restoring a backup
A project administrator can cancel restoring a volume backup by changing the status of the backup to error
. But you cannot cancel restoring a backup when Red Hat Ceph Storage is the back end of the backup repository.
If you cancel restoring a backup after it starts, the destination volume is useless, because there is no way of knowing how much data, if any, was actually restored.
Prerequisites
- You must be a project administrator to cancel restoring a volume backup.
- Ensure that the back end of your backup repository is not Red Hat Ceph Storage.
Procedure
-
Log in to the undercloud host as the
stack
user. Source the
stackrc
undercloud credentials file:$ source ~/stackrc
List the backups to obtain the name or ID of the backup that you want to stop restoring:
$ openstack volume backup list
Change the status of this backup to
error
to cancel its restore operation:$ openstack volume backup set --state error <backup>
-
Replace
<backup>
with the name or ID of the volume backup that you do not want to restore.
Canceling a restore is an asynchronous action, because the back end of the backup repository must detect the change in the backup status before it cancels the restore.
-
Replace
Verification
List the volume backups to verify that the restore is canceled:
$ openstack volume backup list
When the status of the backup changes to
available
, then the restore is canceled.
Chapter 4. Troubleshooting the Block Storage backup service
You can diagnose many issues by verifying that the Block Storage services are running correctly and then by examining the log files for error messages.
4.1. Verifying the Block Storage backup service deployment
After a deployment or when troubleshooting issues, it is important to verify that the necessary Block Storage services are running correctly on their hosts. Ensure that the Block Storage backup service is running on every Controller node, like the Block Storage scheduler service.
After you verify that the necessary Block Storage services are running correctly, then you must verify that the Block Storage backup service is deployed successfully.
Procedure
Run the
openstack volume service list
command:# openstack volume service list +------------------+-------------------------+------+---------+-------+----------------------------+ | Binary | Host | Zone | Status | State | Updated At | +------------------+-------------------------+------+---------+-------+----------------------------+ | cinder-scheduler | controller-0 | nova | enabled | up | 2023-06-21T13:07:42.000000 | | cinder-scheduler | controller-1 | nova | enabled | up | 2023-06-21T13:07:42.000000 | | cinder-scheduler | controller-2 | nova | enabled | up | 2023-06-21T13:07:42.000000 | | cinder-backup | controller-0 | nova | enabled | up | 2023-06-21T13:07:46.000000 | | cinder-backup | controller-1 | nova | enabled | up | 2023-06-21T13:07:46.000000 | | cinder-backup | controller-2 | nova | enabled | up | 2023-06-21T13:07:46.000000 | | cinder-volume | hostgroup@tripleo_iscsi | nova | enabled | up | 2023-06-21T13:07:47.000000 | +------------------+-------------------------+------+---------+-------+----------------------------+
-
Verify that the
State
entry of every service isup
. If not, examine the relevant log files. For more information about the location of these log files, see Block Storage (cinder) Log Files in Managing overcloud observability. - Verify that the Block Storage backup service is deployed successfully, by backing up any Block Storage volume and ensuring that the backup succeeds. For more information, see Troubleshooting backups.
Additional resources
4.2. Troubleshooting backups
The Block Storage backup service performs static checks when receiving a request to back up a Block Storage (cinder) volume. If these checks fail then you will immediately be notified:
-
Check for an invalid volume reference (
missing
). Check if the volume is
in-use
or attached to an instance. Thein-use
case requires you to use the--force
option to perform a backup. For more information, see Creating a backup of an in-use volume.When you use the
--force
volume backup option, you create a crash-consistent, but not an application-consistent, backup because the volume is not quiesced before performing the backup. Therefore, the data is intact but the backup does not have an awareness of which applications were running when the backup was performed.
When these checks succeed: the Block Storage backup service accepts the request to backup this volume, the CLI backup command returns immediately, and the volume is backed up in the background.
Therefore the CLI backup command returns even if the backup fails. You can use the openstack volume backup list
command to verify that the volume backup is successful, when the Status
of the backup entry is available
.
If a backup fails, examine the Block Storage backup service log file for error messages to discover the cause. For more information, see Examining the Block Storage backup service log file.
4.3. Examining the Block Storage backup service log file
When a backup or restore does not succeed, you can examine the Block Storage backup service log file for error messages that can help you to determine the reason.
Procedure
Find the Block Storage backup service log file on the Controller node where the backup service is running.
This log file is located in the following path:
/var/log/containers/cinder/cinder-backup.log
.
4.4. Volume backup workflow
The following diagram and explanation describe the steps that occur when the user requests the cinder API to backup a Block Storage (cinder) volume.
Figure 4.1. Creating a backup of a Block Storage volume
- The user issues a request to the cinder API, which is a REST API, to back up a Block Storage volume.
- The cinder API receives the request from HAProxy and validates the request, the user credentials, and other information.
- The cinder API creates the backup record in the SQL database.
-
The cinder API makes an asynchronous RPC call to the
cinder-backup
service via AMQP to back up the volume. - The cinder API returns the current backup record, with an ID, to the API caller.
- An RPC create message arrives on one of the backup services.
-
The
cinder-backup
service performs a synchronous RPC call toget_backup_device
. -
The
cinder-volume
service ensures that the correct device is returned to the caller. Normally, it is the same volume, but if the volume is in use, the service returns a temporary cloned volume or a temporary snapshot, depending on the configuration. -
The
cinder-backup
service issues another synchronous RPC tocinder-volume
to expose the source device. -
The
cinder-volume
service exports and maps the source device (volume or snapshot) and returns the appropriate connection information. -
The
cinder-backup
service attaches the source device by using the connection information. -
The
cinder-backup
service calls the backup back end driver, with the device already attached, which begins the data transfer to the backup repository. - The source device is detached from the Backup host.
-
The
cinder-backup
service issues a synchronous RPC tocinder-volume
to disconnect the source device. -
The
cinder-volume
service unmaps and removes the export for the device. -
If a temporary volume or temporary snapshot was created,
cinder-backup
callscinder-volume
to remove it. -
The
cinder-volume
service removes the temporary volume. - When the backup is completed, the backup record is updated in the database.
4.5. Volume restore workflow
The following diagram and explanation describe the steps that occur when the user requests the cinder API to restore a Block Storage service (cinder) backup.
Figure 4.2. Restoring a Block Storage backup
- The user issues a request to the cinder API, which is a REST API, to restore a Block Storage backup.
- The cinder API receives the request from HAProxy and validates the request, the user credentials, and other information.
- If the request does not contain an existing volume as the destination, the cinder API makes an asynchronous RPC call to create a new volume and polls the status of the volume until it becomes available.
-
The
cinder-scheduler
selects a volume service and makes the RPC call to create the volume. -
The selected
cinder-volume
service creates the volume. - When the cinder API detects that the volume is available, the backup record is created in the database.
- The cinder API makes an asynchronous RPC call to the backup service via AMQP to restore the backup.
- The cinder API returns the current volume ID, backup ID, and volume name to the API caller.
- An RPC create message arrives on one of the backup services.
-
The
cinder-backup
service performs a synchronous RPC call tocinder-volume
to expose the volume. -
The
cinder-volume
service exports and maps the volume returning the appropriate connection information. -
The
cinder-backup
service attaches the volume by using the connection information. -
The
cinder-backup
service calls the back end driver with the volume already attached, which begins the data restoration to the volume. - The volume is detached from the backup host.
-
The
cinder-backup
service issues a synchronous RPC tocinder-volume
to disconnect the volume. -
The
cinder-volume
service unmaps and removes the export for the volume. - When the volume is restored, the backup record is updated in the database.