Chapter 8. Asynchronous errata updates

Improved rbd_diff_iterate2() API performance

Previously, RBD diff-iterate was not guaranteed to execute locally if exclusive lock was available when diffing against the beginning of time (fromsnapname == NULL) in fast-diff mode (whole_object == true with fast-diff image feature enabled and valid).

Cluster keys and certain configuration directories are removed during RHEL 8 to RHEL 9 upgrade

Due to the RHEL 8 deprecation of the libunwind package, this package is removed when upgrading to RHEL 9. The ceph-common package depends on the libunwind package and therefore is removed as well. Removing the ceph-common package results in the removal of the cluster keys and the certain configurations in the /etc/ceph and /var/log/ceph directories.

The rgw-restore-bucket-index experimental tool restores bucket indices for versioned and un-versioned buckets

With this enhancement, you can restore the bucket indices for versioned buckets with the rgw-restore-bucket-index experimental tool, in addition to its existing ability to work with un-versioned buckets.

Enhanced ordered bucket listing

Previously, in some cases, buckets with larger number of shards and several pseudo-subdirectories would take an unnecessarily long time to complete.

The radosgw-admin bucket stats command prints bucket versioning

With this enhancement, the radosgw-admin bucket stats command prints the versioning status for buckets as one of three values of enabled, off, or suspended since versioning can be enabled or disabled after creation.

The MDS default balancer is now disabled by default

With this release, the MDS default balancer or the automatic dynamic subtree balancer is disabled by default. This prevents accidental subtree migrations, Subtree migrations can be expensive to undo when the operator increases the file system max_mds setting without planning subtree delegations, such as, with pinning.

Each Ceph Manager module has a separate thread to run commands

Previously, there was one thread through which all the ceph-mgr module commands were run. If one of the module’s commands were stuck, all the other module’s commands would hang, waiting on the same thread.

Improved protection against running BlueStore twice

Previously, advisory locking was used to protect against running BlueStore twice. This works well on baremetal deployments. However, when used on containers it would create unrelated inodes that targeted same mknod b block device. As a result, two containers might assume that they can have exclusive access which led to severe errors.

New reports available for sub-events for delayed operations

Previously, slow operations were marked as delayed but without a detailed description.

Some metrics are displayed as null leading to blank spaces in graphs

Some metrics on the Ceph dashboard are shown as null, which leads to blank space in the graphs since you do not initialize a metric until it has some value.

md5 mismatch of replicated objects when testing Ceph Object gateway’s server-side encryption in multi-site

Presently, a md5 mismatch of replicated objects is observed when testing Ceph Object gateway’s server-side encryption in multi-site. The data corruption is specific to S3 multipart uploads with SSE encryption enabled. The corruption only affects the replicated copy. The original object remains intact.

Users can now set crush_device_class in OSD specification

Previously, users would manually set the crush_device_class after the OSDs were made.

Syntax

service_type: osd
service_id: SERVICE_ID_OF_OSD
placement:
hosts:
- HOSTNAME_01
- HOSTNAME_01
crush_device_class: CRUSH_DEVICE_CLASS(SSD/HDD)
spec:
data_devices:
paths:
- DATA_DEVICES
db_devices:
paths:
- DB_DEVICES
wal_devices:
paths:
- WAL_DEVICES

Users can now set retention time in Prometheus specification

Previously, setting the retention time required manually editing the unit.run file, and that would be overwritten whenever the Prometheus daemon was redeployed.

Example

service_type: prometheus
placement:
  count: 1
spec:
  retention_time: "1y"

Client request counters are converted from _u8 type to _u32 type and the limit is set to 256 times

Previously, in multiple active MDSs cases, if a single request failed in the current MDS, the client would forward the request to another MDS. If no MDS could successfully handle the request, it would bounce infinitely between MDSs. The old num_fwd/num_retry counters are _u8 type, which would overflow after bouncing 256 times.

Administrators can now reuse output from rados ls to complete bucket reindexing quickly

Previously, running rados ls command for each bucket was very time-consuming and therefore, slowed down the reindexing of buckets.

Adding or expanding iSCSI gateways in gwcli across the iSCSI daemons works as expected

Previously, due to iSCSI daemons not being reconfigured automatically when a trusted IP list was updated in the specification file, adding or expanding iSCSI gateways in gwcli would fail due to the iscsi-gateway.cfg` not matching across the iSCSI daemons.

ceph orch ps does not display a version for monitoring stack daemons

In cephadm`, due to the version grabbing code currently being incompatible with the downstream monitoring stack containers, version grabbing fails for monitoring stack daemons, such as node-exporter, prometheus, and alertmanager.

cephadm automatically updates the dashboard Grafana password if it is set in the Grafana service spec

Previously, users would have to manually set the Grafana password after applying the specification.

OSDs automatically update their Ceph configuration files with the new mon locations

With this enhancement, whenever a monmap change is detected, cephadm automatically updates the Ceph configuration files for each OSD with the new mon locations.

The Block Device images table is paginated

With this enhancement, the Block Device images table is paginated to use with 10000+ image storage clusters as retrieving information for a block device image is expensive.

Newly added cross_origin_url option allows cross origin resource sharing

Previously, IBM developers faced issues with their storage insights product when they tried to ping the REST API using their front-end because of the tight Cross Origin Resource Sharing (CORS) policies set up in Red Hat’s REST API.

Users can store arbitrary metadata of CephFS subvolume snapshots

With this enhancement, Ceph File System (CephFS) volume users can store arbitrary metadata in the form of key-value pairs for CephFS subvolume snapshots with a set of command-line interface (CLI) commands.

STS max_session_duration for a role can now be updated

With this enhancement, the STS max_session_duration for a role can be updated using the radosgw-admin command-line interface.

ListBucket S3 operation now generates JSON output

With this enhancement, on customers’ request to facilitate integrations, the ListBucket S3 operation generates JSON-formatted output, instead of the default XML, if the request contains an Accept: application/json header.

The option to enable TCP keepalive managed by libcurl is added

With this enhancement, the option to enable TCP keepalive on the HTTP client sockets managed by libcurl is added to make sync and other operations initiated by Ceph Object Gateway more resilient to network instability. This does not apply to connections received by the HTTP frontend, but only to HTTP requests sent by the Ceph Object Gateway, such as Keystone for authentication, sync requests from multi-site, and requests to key management servers for SSE.

Result code 2002 of radosgw-admin commands is explicitly translated to 2

Previously, a change in the S3 error translation of internal NoSuchBucket result inadvertently changed the error code from the radosgw-admin bucket stats command, causing the programs checking the shell result code of those radosgw-admin commands to see a different result code.

You can now use use bucket policies with useful errors

Bucket policies were difficult to use since the error indication was wrong. Additionally, silently dropping principals would cause problems during the upgrade. With this update, useful errors from policy parser and a flag to reject invalid principals with rgw policy reject invalid principals=true parameter is introduced.

The bucket sync run command provides more details

With this enhancement, user-friendly progress reports on the bucket sync run command are added to provide users easier visibility into the progress of the operation. When the user runs the radosgw-admin bucket sync run command with --extra-info flag, users get a message for the start of generation sync and also for each object that is synced.

Multi-site configuration supports dynamic bucket index resharding

Previously, only manual resharding of the buckets for multi-site configurations was supported.

Users can now reshard bucket index dynamically with multi-site archive zones

With this enhancement, multi-site archive zone bucket index can be resharded dynamically when dynamic resharding is enabled for that zone.

Low-level log messages are introduced to warn user about hitting throttle limits

Previously, there was a lack of low-level logging indication that throttle limits were hit, causing these occurrences to incorrectly have the appearance of a networking issue.

Cloned images can now be encrypted with their own encryption format and passphrase

With this enhancement, layered client-side encryption is now supported that enables each cloned image to be encrypted with its own encryption format and passphrase, potentially different from that of the parent image. The efficient copy-on-write semantics intrinsic to unformatted regular cloned images are retained.

Adding or expanding iSCSI gateways in gwcli across the iSCSI daemons works as expected

Previously, due to iSCSI daemons not being reconfigured automatically when a trusted IP list was updated in the specification file, adding or expanding iSCSI gateways in gwcli would fail due to the iscsi-gateway.cfg not matching across the iSCSI daemons.

ceph orch ps does not display a version for monitoring stack daemons

In cephadm, due to the version grabbing code currently being incompatible with the downstream monitoring stack containers, version grabbing fails for monitoring stack daemons, such as node-exporter, prometheus, and alertmanager.

Resharding a bucket having num_shards = 0 results in the bucket’s metadata being lost

Upgrade to Red Hat Ceph Storage 5.3 from older releases with buckets having num_shards = 0 can result in bucket’s metadata loss leading to the bucket’s unavailability while trying to access it. This is a known issue that will be fixed in an upcoming release. The Upgrade guide contains the workaround to disable the dynamic bucket resharding and setting num_shards to a non-zero value before going for the upgrade. For any help on the upgrade or to know more about the issue, contact Red Hat Support.