Chapter 6. Bug fixes

Users can upgrade to a local repo image without any issues

Previously, in cephadm, docker.io would be added to the start of the image name by default, if the image name was not a qualified domain name. Due to this, users were unable to upgrade to images on local repositories.

snap-schedules are no longer lost on restarts of Ceph Manager services

Previously, in-memory databases were not written to persistent storage on every change to the schedule. This caused snap-schedules to get lost on restart of Ceph Manager services.

The standby-replay Metadata Server daemon is no longer unexpectedly removed

Previously, the Ceph Monitor would remove a standby-replay Metadata Server (MDS) daemon from the MDS map under certain conditions. This caused the standby-replay MDS daemon to be removed from the Metadata Server cluster, which generated cluster warnings.

Ceph Manager Alert emails are not tagged as spam anymore

Previously, emails sent by the Ceph Manager Alerts module did not have the “Message-Id” and “Date:headers”. This increased the chances of flagging the emails as spam.

The volume list remains empty when no ceph-osd container is found and cephvolumescan actor no longer fails

Previously, if Ceph containers ran collocated with other containers without a ceph-osd container present among them, the process would try to retrieve the volume list from one non-Ceph container which would not work. Due to this, cephvolumescan actor would fail and the upgrade would not complete.

Ceph OSD deployment no longer fails when ceph-volume treats multiple devices.

Previously, ceph-volume computed wrong sizes when there were multiple devices to treat, resulting in failure to deploy OSDs.

Users can now set up Kafka connectivity with SASL in a non-TLS environment

Previously, due to a failure in configuring the TLS certificate for Ceph Object Gateway, it was not possible to configure Kafka topic with SASL (user and password).

Internal handling of tokens is fixed

Previously, internal handling of tokens in the refresh path of Java-based client authentication provider jar for AWS SDK for Java and Hadoop S3A Connector, would not deal correctly with the large tokens, resulting in improper processing of some tokens and preventing the renewal of client tokens.

The object version access is corrected preventing object lock violation

Previously, inadvertent slicing of version information would occur in some call paths, causing any object version protected by object lock to be deleted contrary to policy.

Ceph Object Gateway no longer crashes with malformed URLs

Previously, a refactoring abstraction replaced a bucket value with a pointer to a bucket value that was not always initialized. This caused malformed URLs corresponding to bucket operations on no buckets resulting in Ceph Object Gateway crashing.

The code that parses dates z-amz-date format is changed

Previously, the standard format for x-amz-date was changed which caused issues, since the new software uses the new date format. The new software built with the latest go libraries would not talk to the Ceph Object Gateway.

New logic in processing of lifecycle shards prevents stalling due to deleted buckets

Previously, changes were made to cause lifecycle processing to continuously cycle across days, that is, to not restart from the beginning of the list of eligible buckets each day. However, the changes contained a bug which could stall processing of lifecycle shards that contained deleted buckets, causing the processing of lifecycle shards to stall.

Header processing no longer causes sporadic swift-protocol authentication failures

Previously, a combination of incorrect HTTP header processing and timestamp handling logic would either cause an invalid Keystone admin token to be used for operations, or non-renewal of Keystone’s admin token as required. Due to this, sporadic swift-protocol authentication failures would occur.

Warnings are no longer logged in inappropriate circumstances

Previously, an inverted logic would occasionally report an incorrect warning - unable to find head object, causing the warning to be logged when it was not applicable in a Ceph Object Gateway configuration.

PUT object operation writes to the correct bucket index shards

Previously, due to a race condition, a PUT object operation would rarely write to a former bucket index shard. This caused the former bucket index shard to be recreated, and the object would not appear in the proper bucket index. Therefore, the object would not be listed when the bucket was listed.

Suspending bucket versioning in the primary zone no longer suspends bucket versioning in the archive zone

Previously, if bucket versioning was suspended in the primary zone, bucket versioning in the archive zone would also be suspended.

The radosgw-admin sync status command in multi-site replication works as expected

Previously, in a multisite replication, if one or more participating Ceph Object Gateway nodes are down, you would (5) Input/output error output when running the radosgw-admin sync status command. This status should get resolved after all the Ceph Object Gateway nodes are back online.

Processes trimming retired bucket index entries no longer cause radosgw instance to crash

Previously, under some circumstances, processes trimming retired bucket index entries could access an uninitialized pointer variable resulting in the radosgw instance to crash.

Bucket sync run is given control logic to sync all objects

Previously, to support dynamic bucket resharding on multisite clusters, a singular bucket index log was replaced with multiple bucket index log generations. But, due to how bucket sync run was implemented, only the oldest outstanding generation would be sync run.

Per-bucket replication logical error fix executes policies correctly

Previously, an internal logic error caused failures in per-bucket replication, due to which per-bucket replication policies did not work in some circumstances.

Variable access no longer causes undefined program behavior

Previously, a coverity scan would identify two cases, where variables could be used after a move, potentially causing an undefined program behavior to occur.

Requests with a tenant but no bucket no longer cause a crash

Previously, an upstream refactoring replaced uninitialized bucket data fields with uninitialized pointers. Due to this, any bucket request containing a URL referencing no valid bucket caused crashes.

Performing a DR test with two sites stretch cluster no longer causes Ceph to become unresponsive

Previously, when performing a DR test with two sites stretch-cluster, removing and adding new monitors to the cluster would cause an incorrect rank in ConnectionTracker class. Due to this, the monitor would fail to identify itself in the peer_tracker copy and would never update its correct field, causing a deadlock in the election process which would lead to Ceph becoming unresponsive.

Rank is removed from the live_pinging and dead_pinging set to mitigate the inconsistent connectivity score issue

Previously, when removing two monitors consecutively, if the rank size is equal to Paxos’s size, the monitor would face a condition and would not remove rank from the dead_pinging set. Due to this, the rank remained in the dead_pinging set which would cause problems, such as inconsistent connectivity score when the stretch-cluster mode was enabled.

The Prometheus metrics now reflect the correct Ceph version for all Ceph Monitors whenever requested

Previously, the Prometheus metrics reported mismatched Ceph versions for Ceph Monitors when the monitor was upgraded. As a result, the active Ceph Manager daemon needed to be restarted to resolve this inconsistency.

The ceph daemon heap status command shows the heap status

Previously, due to a failure to get heap information through the ceph daemon command, the ceph daemon heap stats command would return empty output instead of returning current heap usage for a Ceph daemon. This was because ceph::osd_cmds::heap() was confusing the stderr and stdout concept which caused the difference in output.

Ceph Monitors no longer crash when using ceph orch apply mon <num> command

Previously, when the command ceph orch apply mon <num> was used to decrease monitors in a cluster, the monitors were removed before shutting down in ceph-adm causing the monitors to crash.

End-user can now see the scrub or deep-scrub starts message from the Ceph cluster log

Previously, due to the scrub or deep-scrub starts message missing in the Ceph cluster log, the end-user would fail to know if the PG scrubbing had started for a PG from the Ceph cluster log.

No assertion during the Ceph Manager failover

Previously, when activating the Ceph Manager, it would receive several service_map versions sent by the previously active manager. This incorrect check in code would cause assertion failure when the newly activated manager received a map with a higher version sent by the previously active manager.

Users can remove cloned objects after upgrading a cluster

Previously, after upgrading a cluster from Red Hat Ceph Storage 4 to Red Hat Ceph Storage 5 , removing snapshots of objects created in earlier versions would leave clones, which could not be removed. This was because the SnapMapper keys were wrongly converted.

RocksDB error does not occur for small writes

BlueStore employs a strategy of deferring small writes for HDDs and stores data in RocksDB. Cleaning deferred data from RocksDB is a background process which is not synchronized with BlueFS.

Corrupted dups entries of a PG Log can be removed by off-line and on-line trimming

Previously, trimming of PG log dups entries could be prevented during the low-level PG split operation, which is used by the PG autoscaler with far higher frequency than by a human operator. Stalling the trimming of dups resulted in significant memory growth of PG log, leading to OSD crashes as it ran out of memory. Restarting an OSD did not solve the problem as the PG log is stored on disk and reloaded to RAM on startup.

rbd info command no longer fails if executed when the image is being flattened

Previously, due to an implementation defect, rbd info command would fail, although rarely, if run when the image was being flattened. This caused a transient No such file or directory error to occur, although, upon rerun, the command always succeeded.

Removing a pool with pending Block Device tasks no longer causes all the tasks to hang

Previously, due to an implementation defect, removing a pool with pending Block Device tasks caused all Block Device tasks, including other pools, to hang. To resume hung Block Device tasks, the administrator had to restart the ceph-mgr daemon.

The image replayer shuts down as expected

Previously, due to an implementation defect, a request to shut down a particular image replayer would cause the rbd-mirror daemon to hang indefinitely, especially in cases where the daemon was blocklisted on the remote storage cluster.

The rbd mirror pool peer bootstrap create command guarantees correct monitor addresses in the bootstrap token

Previously, a bootstrap token generated with the rbd mirror pool peer bootstrap create command contained monitor addresses as specified by the mon_host option in the ceph.conf file. This was fragile and caused issues to users, such as causing confusion between V1 and V2 endpoints, specifying only one of them, grouping them incorrectly, and the like.

Upgrade from Red Hat Ceph Storage 4.x to 5.x with iSCSI works as expected

Previously, due to version conflict between some of the ceph-iscsi dependent libraries, upgrades from Red Hat Ceph Storage 4.x to 5.x would lead to a persistent HTTP 500 error.

Upgrade workflow with Ceph Object Gateway configuration is fixed

Previously, whenever set_radosgw_address.yml was called from the dashboard playbook execution, the fact is_rgw_instances_defined was expected to be set if rgw_instances was defined in group_vars/host_vars by the user. Otherwise, the next task that sets the fact rgw_instances will be executed under the assumption that it wasn’t user defined. This caused the upgrade workflow to break when deploying the Ceph Object Gateway multisite and Ceph Dashboard.

The fact condition is updated to execute only on the Ceph Object Gateway nodes

Previously, due to set_fact _radosgw_address to radosgw_address_block ipv4 being executed on all nodes, including the ones where no Ceph Object Gateway network range was present, playbooks failed to work.