Chapter 7. Planning storage and shared file systems

Red Hat Ceph Storage is a distributed data object store designed for performance, reliability, and scalability. Distributed object stores use unstructured data to simultaneously service modern and legacy object interfaces. It provides access to block, file, and object storage.

Red Hat Ceph Storage is deployed as a cluster. A cluster consists of two primary types of daemons:

RHOSO supports Red Hat Ceph Storage 7 in the following deployment scenarios:

For more information about Red Hat Ceph Storage architecture, see the Red Hat Ceph Storage 7 Architecture Guide.

The Block Storage service (cinder) allows users to provision block storage volumes on back ends. Users can attach volumes to instances to augment their ephemeral storage with general-purpose persistent storage. You can detach and re-attach volumes to instances, but you can only access these volumes through the attached instance.

You can also configure instances so that they do not use ephemeral storage. Instead of using ephemeral storage, you can configure the Block Storage service to write images to a volume. You can then use the volume as a bootable root volume for an instance. Volumes also provide inherent redundancy and disaster recovery through backups and snapshots. However, backups are only provided if you deploy the optional Block Storage backup service. In addition, you can encrypt volumes for added security.

The Image service (glance) provides discovery, registration, and delivery services for instance images. It also provides the ability to store snapshots of instances ephemeral disks for cloning or restore purposes. You can use stored images as templates to commission new servers quickly and more consistently than installing a server operating system and individually configuring services.

The Object Storage service (swift) provides a fully-distributed storage solution that you can use to store any kind of static data or binary object; such as media files, large datasets, and disk images. The Object Storage service organizes objects by using object containers, which are similar to directories in a file system, but they cannot be nested. You can use the Object Storage service as a repository for nearly every service in the cloud.

Red Hat Ceph Storage RGW can be used as an alternative to the Object Storage service.

The Shared File Systems service (manila) provides the means to provision remote, shareable file systems. These are known as shares. Shares allow projects in the cloud to share POSIX compliant storage, and they can be consumed by multiple instances simultaneously.

Shares are used for instance consumption, and they can be consumed by multiple instances at the same time with read/write access mode.

Storage best practices recommend using two different networks:

These networks are referred to as storage and storageMgmt. If your deployment diverges from the architecture of two networks, adapt the documented examples as necessary. For example, if the management interface for the storage system is available on the storage network, replace storageMgmt with storage when there is only one network, and remove storageMgmt when the storage network is already present.

The storage services in Red Hat OpenStack Services on OpenShift (RHOSO), with the exception of the Object Storage service (swift), require access to the storage and storageMgmt networks. You can configure the storage and storageMgmt networks in the networkAttachments field of your OpenStackControlPlane CR. The networkAttachments field accepts a list of strings with all the networks the component requires access to. Different components can have different network requirements, for example, the Block Storage service (cinder) API component does not require access to any of the storage networks.

The following example shows the networkAttachments for Block Storage volumes:

apiVersion: core.openstack.org/v1beta1
kind: OpenStackControlPlane
metadata:
  name: openstack
spec:
  cinder:
    template:
      cinderVolumes:
        iscsi:
          networkAttachments:
          - storage
          - storageMgmt

Plan the networking on your cloud to ensure that cloud users can connect their shares to workloads that run on Red Hat OpenStack Services on OpenShift (RHOSO) virtual machines, bare-metal servers, and containers.

Depending on the level of security and isolation required for cloud users, you can set the driver_handles_share_servers parameter (DHSS) to true or false.

The Image service (glance) requires a staging area to manipulate data during an import operation. It is possible to copy image data into multiple stores so some persistence is required for the Image service. Although PVCs represent the main storage model for the Image service, an External model can also be chosen.

...
default:
  storage:
    external: true
...
...
extraMounts:
- extraVol:
  - extraVolType: NFS
    mounts:
    - mountPath: /var/lib/glance/os_glance_staging_store
      name: nfs
    volumes:
    - name: nfs
      nfs:
        path: <nfs_export_path>
        server: <nfs_ip_address>

It should be noted that the configuration sample conflicts with the distributed image import feature. Distributed image import requires RWO storage plugged into a particular instance; it owns the data and receives requests in case its staged data is required for an upload operation. When the External model is adopted, if Red Hat Ceph Storage is used as a backend, and an image conversion operation is run in one of the existing replicas, the glance-operator does not have to make any assumption about the underlying storage that is tied to the staging area, and the conversion operation that uses the os_glance_staging_store directory (within the Pod) interacts with the RWX NFS backend provided via extraMounts. With this scenario, no image-cache PVC can be requested and mounted to a subPath, because it should be the administrator’s responsibility to plan for persistence using extraMounts.

...
default:
  replicas: 3
  storage:
    storageRequest: 10G
...

In this model, if Red Hat Ceph Storage is set as a backend, no dedicated image conversion PVC is created. The administrator must think about the PVC sizing in advance; the size of the PVC should be at least up to the largest converted image size. Concurrent conversions within the same Pod might be problematic in terms of PVC size; a conversion will fail or cannot take place if the PVC is full and there’s not enough space. The upload should be retried after the previous conversion is over and the staging area space is released. However, concurrent conversion operations might happen in different Pods. You should deploy at least 3 replicas for a particular glanceAPI. This helps to handle heavy operations like image conversion.

For a PVC-based layout, the scale out of a glanceAPI in terms of replicas is limited by the available storage provided by the storageClass, and depends on the storageRequest. The storageRequest is a critical parameter, it can be globally defined for all the glanceAPI, or defined with a different value for each API. It will influence the scale out operations for each of them. Other than a local PVC required for the staging area, it is possible to enable image cache, which is translated into an additional PVC bound to each glanceAPI instance. A glance-cache PVC is bound to /var/lib/glance/image-cache. The glance-operator configures the glanceAPI instance accordingly, setting both image_cache_max_size and the image_cache_dir parameters. The number of image cache PVCs follows the same rules described for the local PVC, and the number of requested PVCs is proportional to the number of replicas.

The Object Storage service requires storage devices for data. These devices must be accessible using the same hostname or IP address during their lifetime. The configuration of a StatefulSet with a Headless Service is how this is achieved.

If you want to use storage volumes to provide persistence for your workload, you can use a StatefulSet as part of the solution. Although individual Pods in a StatefulSet are susceptible to failure, the persistent Pod identifiers make it easier to match existing volumes to the new Pods that replace any that have failed.

The Object Storage service requires quite a few services to access these PVs, and all of them are running in a single pod.

Additionally, volumes are not deleted if the StatefulSet is deleted. An unwanted removal of the StatefulSet (or the whole deployment) will not immediately result in a catastrophic data loss, but can be recovered from with administrator interaction.

The Headless Service makes it possible to access the storage pod directly by using a DNS name. For example, if the pod name is swift-storage-0 and the SwiftStorage instance is named swift-storage, it becomes accessible using swift-storage-0.swift-storage. This makes it easily usable within the Object Storage service rings, and IP changes are now transparent and don’t require an update of the rings.

Parallel pod management tells the StatefulSet controller to launch or terminate all Pods in parallel, and to not wait for Pods to become Running and Ready or completely terminated prior to launching or terminating another Pod. This option only affects the behavior for scaling operations. Updates are not affected.

This is required to scale by more than one; including new deployments with more than one replica. It is required to create all pods at the same time, otherwise there will be PVCs that are not bound and the Object Storage service rings cannot be created, eventually blocking the start of these pods.

Storage pods should be distributed to different nodes to avoid single points of failure. A podAntiAffinity rule with preferredDuringSchedulingIgnoredDuringExecution is used to distribute pods to different nodes if possible. Using a separate storageClass and PersistentVolumes that are located on different nodes can be used to enforce further distribution.

Object Storage service backend services must only be accessible by other backend services and the Object Storage service proxy. To limit access, a NetworkPolicy is added to allow only traffic between these pods. The NetworkPolicy itself depends on labels, and these must match to allow traffic. Therefore labels must not be unique; instead all pods must use the same label to allow access. This is also the reason why the swift-operator is not using labels from lib-common.

Object Storage service rings require information about the disks to use, and this includes sizes and hostnames or IPs. Sizes are not known when starting the StatefulSet using PVCs, the size requirement is a lower limit, but the actual PVs might be much bigger.

However, StatefulSets do create PVCs before the ConfigMaps are available and simply wait starting the pods until these become available. The SwiftRing reconciler is watching the SwiftStorage instances and iterates over PVCs to get actual information about the used disks. Once these are bound, the size is known and the swift-ring-rebalance job creates the Swift rings and eventually the ConfigMap. After the ConfigMap becomes available, StatefulSets will start the service pods.

Rings are stored in a ConfigMap mounted by the SwiftProxy and SwiftStorage instances using projected volumes. This makes it possible to mount all required files at the same place, without merging these from other places. Updated ConfigMaps will update these files, and these changes are are detected by the Swift services eventually reloading these.

Some operators are using the customServiceConfig option to customize settings. However, the SwiftRing instance deploys multiple backend services, and each of these requires specific files to be customized. Therefore only defaultConfigOverwrite using specific keys as filenames is supported when using the swift-operator.