Chapter 2. Image service with multiple stores
The Red Hat OpenStack Platform Image service (glance) supports using multiple stores with distributed edge architecture so that you can have an image pool at every edge site. You can copy images between the central site, which is also known as the hub site, and the edge sites.
The image metadata contains the location of each copy. For example, an image present on two edge sites is exposed as a single UUID with three locations: the central site plus the two edge sites. This means you can have copies of image data that share a single UUID on many stores. For more information about locations, see Understanding the location of images.
With an RBD image pool at every edge site, you can boot VMs quickly by using Ceph RBD copy-on-write (COW) and snapshot layering technology. This means that you can boot VMs from volumes and have live migration. For more information about layering with Ceph RBD, see Ceph block device layering in the Block Device Guide.
2.1. Requirements of storage edge architecture
- A copy of each image must exist in the Image service at the central location.
- Prior to creating an instance at an edge site, you must have a local copy of the image at that edge site.
- Images uploaded to an edge site must be copied to the central location before they can be copied to other edge sites.
- Use the Image service RBD driver for all edge sites. Mixed architecture is not supported.
- RBD must be the storage driver for the Image, Compute and Block Storage services.
-
For each site, you must assign the same value to the
NovaComputeAvailabilityZone
andCinderStorageAvailabilityZone
parameters.
2.2. Import an image to multiple stores
Use the interoperable image import workflow to import image data into multiple Ceph Storage clusters. You can import images into the Image service that are available on the local file system or through a web server.
If you import an image from a web server, the image can be imported into multiple stores at once. If the image is not available on a web server, you can import the image from a local file system into the central store and then copy it to additional stores. For more information, see Copy an existing image to multiple stores.
Always store an image copy on the central site, even if there are no instances using the image at the central location. For more information about importing images into the Image service, see the Distributed compute node and storage deployment guide.
2.2.1. Manage image import failures
You can manage failures of the image import operation by using the --allow-failure
parameter:
-
If the value of the
--allow-failure
parameter totrue
, the image status becomesactive
after the first store successfully imports the data. This is the default setting. You can view a list of stores that failed to import the image data by using theos_glance_failed_import
image property. -
If you set the value of the
--allow-failure
parameter tofalse
, the image status only becomesactive
after all specified stores successfully import the data. Failure of any store to import the image data results in an image status offailed
. The image is not imported into any of the specified stores.
2.2.2. Importing image data to multiple stores
Because the default setting of the --allow-failure
parameter is true
, you do not need to include the parameter in the command if it is acceptable for some stores to fail to import the image data.
This procedure does not require all stores to successfully import the image data.
Procedure
Import image data to multiple, specified stores:
$ glance image-create-via-import \ --container-format bare \ --name IMAGE-NAME \ --import-method web-download \ --uri URI \ --stores STORE1,STORE2,STORE3
- Replace IMAGE-NAME with the name of the image you want to import.
- Replace URI with the URI of the image.
- Replace STORE1, STORE2, and STORE3 with the names of the stores to which you want to import the image data.
-
Alternatively, replace
--stores
with--all-stores true
to upload the image to all the stores.
The glance image-create-via-import
command, which automatically converts the QCOW2 image to RAW format, works only with the web-download
method. The glance-direct
method is available, but it works only in deployments with a configured shared file system. For more information, see Storing an image in RAW format.
2.2.3. Importing image data to multiple stores without failure
This procedure requires all stores to successfully import the image data.
Procedure
Import image data to multiple, specified stores:
$ glance image-create-via-import \ --container-format bare \ --name IMAGE-NAME \ --import-method web-download \ --uri URI \ --stores STORE1,STORE2
- Replace IMAGE-NAME with the name of the image you want to import.
- Replace URI with the URI of the image.
- Replace STORE1, STORE2, and STORE3 with the names of stores to which you want to copy the image data.
Alternatively, replace
--stores
with--all-stores true
to upload the image to all the stores.NoteWith the
--allow-failure
parameter set tofalse
, the Image service does not ignore stores that fail to import the image data. You can view the list of failed stores with the image propertyos_glance_failed_import
. For more information see Checking the progress of image import operation.
Verify that the image data was added to specific stores:
$ glance image-show IMAGE-ID | grep stores
Replace IMAGE-ID with the ID of the original existing image.
The output displays a comma-delimited list of stores.
2.2.4. Importing image data to a single store
You can import image data to a single store.
Procedure
Import image data to a single store:
$ glance image-create-via-import \ --container-format bare \ --name IMAGE-NAME \ --import-method web-download \ --uri URI \ --store STORE
- Replace IMAGE-NAME with the name of the image you want to import.
- Replace URI with the URI of the image.
Replace STORE with the name of the store to which you want to copy the image data.
NoteIf you do not include the options of
--stores
,--all-stores
, or--store
in the command, the Image service creates the image in the central store.
Verify that the image data was added to specific store:
$ glance image-show IMAGE-ID | grep stores
Replace IMAGE-ID with the ID of the original existing image.
The output displays a comma-delimited list of stores.
2.2.5. Checking the progress of the image import operation
The interoperable image import workflow sequentially imports image data into stores. The size of the image, the number of stores, and the network speed between the central site and the edge sites impact how long it takes for the image import operation to complete.
You can follow the progress of the image import by looking at two image properties, which appear in notifications sent during the image import operation:
-
The
os_glance_importing_to_stores
property lists the stores that have not imported the image data. At the beginning of the import, all requested stores show up in the list. Each time a store successfully imports the image data, the Image service removes the store from the list. -
The
os_glance_failed_import
property lists the stores that fail to import the image data. This list is empty at the beginning of the image import operation.
In the following procedure, the environment has three Ceph Storage clusters: the central
store and two stores at the edge, dcn0
and dcn1
.
Procedure
Verify that the image data was added to specific stores:
$ glance image-show IMAGE-ID
Replace IMAGE-ID with the ID of the original existing image.
The output displays a comma-delimited list of stores similar to the following example snippet:
| os_glance_failed_import | | os_glance_importing_to_stores | central,dcn0,dcn1 | status | importing
Monitor the status of the image import operation. When you precede a command with
watch
, the command output refreshes every two seconds.$ watch glance image-show IMAGE-ID
Replace IMAGE-ID with the ID of the original existing image.
The status of the operation changes as the image import operation progresses:
| os_glance_failed_import | | os_glance_importing_to_stores | dcn0,dcn1 | status | importing
Output that shows that an image failed to import resembles the following example:
| os_glance_failed_import | dcn0 | os_glance_importing_to_stores | dcn1 | status | importing
After the operation completes, the status changes to active:
| os_glance_failed_import | dcn0 | os_glance_importing_to_stores | | status | active
2.3. Copy an existing image to multiple stores
This feature enables you to copy existing images using Red Hat OpenStack Image service (glance) image data into multiple Ceph Storage stores at the edge by using the interoperable image import workflow.
The image must be present at the central site before you copy it to any edge sites. Only the image owner or administrator can copy existing images to newly added stores.
You can copy existing image data either by setting --all-stores
to true
or by specifying specific stores to receive the image data.
-
The default setting for the
--all-stores
option isfalse
. If--all-stores
isfalse
, you must specify which stores receive the image data by using--stores STORE1,STORE2
. If the image data is already present in any of the specified stores, the request fails. -
If you set
all-stores
totrue
, and the image data already exists in some of the stores, then those stores are excluded from the list.
After you specify which stores receive the image data, the Image service copies data from the central site to a staging area. Then the Image service imports the image data by using the interoperable image import workflow. For more information, see Importing an image to multiple stores.
Red Hat recommends that administrators carefully avoid closely timed image copy requests. Two closely timed copy-image operations for the same image causes race conditions and unexpected results. Existing image data remains as it is, but copying data to new stores fails.
2.3.1. Copying an image to all stores
Use the following procedure to copy image data to all available stores.
Procedure
Copy image data to all available stores:
$ glance image-import IMAGE-ID \ --all-stores true \ --import-method copy-image
Replace IMAGE-ID with the name of the image you want to copy.
Confirm that the image data successfully replicated to all available stores:
$ glance image-list --include-stores
For information about how to check the status of the image import operation, see Checking the progress of the image import operation.
2.3.2. Copying an image to specific stores
Use the following procedure to copy image data to specific stores.
Procedure
Copy image data to specific stores:
$ glance image-import IMAGE-ID \ --stores STORE1,STORE2 \ --import-method copy-image
- Replace IMAGE-ID with the name of the image you want to copy.
- Replace STORE1 and STORE2 with the names of the stores to which you want to copy the image data.
Confirm that the image data successfully replicated to the specified stores:
$ glance image-list --include-stores
For information about how to check the status of the image import operation, see Checking the progress of the image import operation.
2.4. Deleting an image from a specific store
With this feature you can use the OpenStack Image service (glance) to delete an existing image copy on a specific store.
Procedure
Delete an image from a specific store:
$ glance stores-delete --store _STORE_ID_ _IMAGE_ID_
- Replace _STORE_ID with the name of the store on which the image copy should be deleted.
- Replace IMAGE_ID with the ID of the image you want to delete.
Using glance image-delete
will permanently delete the image across all the sites. All image copies will be deleted, as well as the image instance and metadata.
2.5. Understanding the locations of images
Although an image can be present on multiple sites, there is only a single UUID for a given image. The image metadata contains the locations of each copy. For example, an image present on two edge sites is exposed as a single UUID with three locations: the central site plus the two edge sites.
Procedure
Show the sites on which a copy of the image exists:
$ glance image-show ID | grep "stores" | stores | default_backend,dcn1,dcn2
In the example, the image is present on the central site, the
default_backend
, and on the two edge sitesdcn1
anddcn2
.Alternatively, you can run the
glance image-list
command with the--include-stores
option to see the sites where the images exist:$ glance image-list --include-stores | ID | Name | Stores | 2bd882e7-1da0-4078-97fe-f1bb81f61b00 | cirros | default_backend,dcn1,dcn2
List the image locations properties to show the details of each location:
$ openstack image show ID -c properties | properties | (--- cut ---) locations='[{'url': 'rbd://79b70c32-df46-4741-93c0-8118ae2ae284/images/2bd882e7-1da0-4078-97fe-f1bb81f61b00/snap', 'metadata': {'store': 'default_backend'}}, {'url': 'rbd://63df2767-8ddb-4e06-8186-8c155334f487/images/2bd882e7-1da0-4078-97fe-f1bb81f61b00/snap', 'metadata': {'store': 'dcn1'}}, {'url': 'rbd://1b324138-2ef9-4ef9-bd9e-aa7e6d6ead78/images/2bd882e7-1da0-4078-97fe-f1bb81f61b00/snap', 'metadata': {'store': 'dcn2'}}]', (--- cut --)
The image properties show the different Ceph RBD URIs for the location of each image.
In the example, the central image location URI is:
rbd://79b70c32-df46-4741-93c0-8118ae2ae284/images/2bd882e7-1da0-4078-97fe-f1bb81f61b00/snap', 'metadata': {'store': 'default_backend'}}
The URI is composed of the following data:
-
79b70c32-df46-4741-93c0-8118ae2ae284
corresponds to the central Ceph FSID. Each Ceph cluster has a unique FSID. -
The default value for all sites is
images
, which corresponds to the Ceph pool on which the images are stored. -
2bd882e7-1da0-4078-97fe-f1bb81f61b00
corresponds to the image UUID. The UUID is the same for a given image regardless of its location. -
The metadata shows the glance store to which this location maps. In this example, it maps to the
default_backend
, which is the central hub site.
-