Chapter 6. Asynchronous errata updates

New clone creation no longer slows down due to parallel clone limit

Previously, upon reaching the limit of parallel clones, the rest of the clones would queue up, slowing down the cloning.

Ceph File System names can now be swapped for enhanced disaster recovery

This enhancement provides the option for two file systems to swap their names, by using the ceph fs swap command. The file system IDs can also optionally be swapped with this command.

quota.max_bytes is now set in more understandable size values

Previously, the quota.max_bytes value was set in bytes, resulting in often very large size values, which was hard to set or changed.

Health warnings for a standby-replay MDS are no longer included

Previously, all inode and stray counters health warnings were displayed during a standby-replay MDS.

S3 requests are no longer cut off in the middle of transmission during shutdown

Previously, a few clients faced issues with the S3 request being cut off in the middle of transmission during shutdown without waiting.

Intel QAT Acceleration for Object Compression & Encryption

Intel QuickAssist Technology (QAT) is implemented to help reduce node CPU usage and improve the performance of Ceph Object Gateway when enabling compression and encryption. It’s a known issue that QAT can only be configured on new setups (Greenfield only). QAT Ceph Object Gateway daemons cannot be configured in the same cluster as non-QAT (regular) Ceph Object Gateway daemons.

Cluster keys and certain configuration directories are removed during RHEL 8 to RHEL 9 upgrade

Due to the RHEL 8 deprecation of the libunwind package, this package is removed when upgrading to RHEL 9. The ceph-common package depends on the libunwind package and therefore is removed as as well. Removing the ceph-common package results in the removal of the cluster keys and the certain configurations in the /etc/ceph and /var/log/ceph directories.

Using ceph orch ls command with the --export flag corrupts the cert/key files format

Previously, long multi-line strings like cert/key files format would be mangled when using ceph orch ls with the --export flag. Specifically, some newlines are stripped. As a result, if users re-apply a specification with a cert/key as they got it from ceph orch ls with --export provided, the cert/key will be unusable by the daemon.

Metrics support for the Replication Start/End Notifications.

With this enhancement, metrics support for the Replication Start/End notifications is provided. These metrics enable monitoring logic for data replication.

New mon_cluster_log_level command option to control the cluster log level verbosity for external entities

Previously, debug verbosity logs were sent to all external logging systems regardless of their level settings. As a result, the /var/ filesystem would rapidly fill up.

Cluster keys and certain configuration directories are removed during RHEL 8 to RHEL 9 upgrade

Due to the RHEL 8 deprecation of the libunwind package, this package is removed when upgrading to RHEL 9. The ceph-common package depends on the libunwind package and therefore is removed as as well. Removing the ceph-common package results in the removal of the cluster keys and the certain configurations in the /etc/ceph and /var/log/ceph directories.

Update rgw configurations using UI and API

Previously, UI and API were dependent on is_updatable_at_runtime flag which returned incorrect values. As a result customers could not update the rgw configuration from the UI or by using the API.

Increased efficiency for ordered bucket listings when namespaced bucket index entries exist

Previously, when ignoring namespaced bucket index entries, the code would still access the ignored entry. As a result, there would be an unnecessary latency in the ignored listings.

Ceph Object Gateway (RGW) users can be created without a key using the radosgw-admin command-line

Previously, there was no provision to create RGW users without a key using the radosgw-admin command-line. This feature is available for the adminops.

quota.max_bytes is now set in more understandable size values

Previously, the quota.max_bytes value was set in bytes, resulting in often very large size values, which were hard to set or change.

Inspection of disk allocator state through the admin socket command

This enhancement provides a middle point between the allocator score which gives a single number, and the allocator dump which lists all free chunks.

The admin socket command now works with default parameters

Previously, listing allocator histogram = admin socket with bluestore allocator fragmentation histogram did not work for bluefs-db and bluefs-wal with default parameters. With this enhancement, the admin socket command works with the default parameters.

Lower CPU usage on s390x for CRC32

Previously, the CPU usage was high due to the CRC32 software implementation.

Cluster keys and certain configuration directories are removed during RHEL 8 to RHEL 9 upgrade

Due to the RHEL 8 deprecation of the libunwind package, this package is removed when upgrading to RHEL 9. The ceph-common package depends on the libunwind package and therefore is removed as as well. Removing the ceph-common package results in the removal of the cluster keys and the certain configurations in the /etc/ceph and /var/log/ceph directories.

Improved core dump handling in cephadm systemd units

Previously, core dumps were not generated or were truncated when services crashed—especially in hard-to-reproduce cases—resulting in the loss of valuable debugging information.

Bucket notifications can now be sent to a multi-node Kafka cluster

Previously, Ceph Object Gateway could only send a message to a single node Kafka cluster.

Sites can now configure RGW error handling for existing bucket creation

Previously, RGW returned a success response when creating a bucket that already existed in the same zone, even if no new bucket was created. This caused confusion in automated workflows. 

PG scrub performance improved by removing unnecessary object ID repair check

Previously, every PG scrub invoked repair_oinfo_oid(), a function meant to fix mismatched object IDs in rare cases linked to a historical filesystem bug. This added unnecessary overhead, as the check applied only under very specific conditions.

New ceph osd rm-pg-up map-primary-all command for OSDMap cleanup

Previously, users had to remove pg_upmap_primary mappings individually using ceph osd rm-pg-upmap-primary PGID, which was time-consuming and error-prone, especially when cleaning up invalid mappings after pool deletion.

Exporter daemons report error state after receiving SIGTERM

Currently, the ceph-exporter and node-exporter daemons return a non-zero return code when they receive a SIGTERM signal. As a result, the system marks these daemons as being in an error state instead of the expected stopped state.

Daemons incorrectly marked as stopped after exiting maintenance mode

When a node is taken out of maintenance mode, cephadm may temporarily display the daemons on that node as being in a stopped state even if they have already started. This occurs because cephadm has not yet refreshed the daemon status on the host. As a result, users may mistakenly see daemons as stopped immediately after maintenance mode ends.