Chapter 4. Bug fixes

Using the --name NODE flag with the cephadm shell to start a stopped OSD no longer returns the wrong image container

Previously, in some cases, when using the cephadm shell --name NODE command, the command would start the container with the wrong version of the tools. This would occur when a user has a newer ceph container image on the host than the one that their OSDs are using.

Playbooks now remove the RHCS version repositories matching the running RHEL version

Previously, playbooks would try to remove Red Hat Ceph Storage 4 repositories from RHEL 9 even though they do not exist on RHEL 9. This would cause the playbooks to fail.

All memory consumed by the configuration reload process is now released

Previously, reload exports would not release all the memory consumed by the configuration reload process causing the memory footprint to increase.

Users can create volumes with multiple hosts in the Ceph dashboard

With this fix, users can now create volumes with multiple hosts in the Ceph dashboard.

Unset subvolume size is no longer set as 'infinite'

Previously, the unset subvolume size was set to 'infinite', resulting in the failure of the update.

Missing options are added in the kernel mount command

Previously, a few options were missing in the kernel mount command for attaching the filesystem causing the command to not work as intended.

Ceph dashboard now supports both NFS v3 and v4-enabled export management

Previously, the Ceph dashboard only supported the NFSv4-enabled exports management and not the NFSv3-enabled exports. Due to this, any management done for exports via CLI for NFSv3 was corrupted.

Access/secret keys are now not compulsory while creating a zone

Previously, access/secret keys were compulsory when creating a zone in Ceph Object Gateway multi-site. Due to this, users had to first set the non-system user’s keys in the zone and later update with the system user’s keys.

Importing multi-site configuration no longer throws an error on submitting the form

Previously, the multi-site period information did not contain the 'realm' name. Due to this, importing the multi-site configuration threw an error on submitting the form.

The Ceph Object Gateway metrics label names are aligned with the Prometheus label naming format and they are now visible in Prometheus

Previously, the metrics label names were not aligned with the Prometheus label naming format, causing the Ceph Object Gateway metrics to not be visible in Prometheus.

Full names can now include dot in Ceph dashboard

Previously, in the Ceph dashboard, it was not possible to create or modify a full name with a dot in it due to incorrect validation.

MDS metadata with FSMap changes are now added in batches to ensure consistency

Previously, monitors would sometimes lose track of MDS metadata during upgrades and cancelled PAXOS transactions resulting in MDS metadata being no longer available.

The ENOTEMPTY output is detected and the message is displayed correctly

Previously, when running the subvolume group rm command, the ENOTEMPTY output was not detected in the volume’s plugin causing a generalized error message instead of a specific message.

MDS now queues the next client replay request automatically as part of request cleanup

Previously, sometimes, MDS would not queue the next client request for replay in the up:client-replay state causing the MDS to hang.

cephfs-mirroring overall performance is improved

With this fix, the incremental snapshot sync is corrected, which improves the overall performance of cephfs-mirroring.

The loner member is set to true

Previously, for a file lock in the LOCK_EXCL_XSYN state, the non-loner clients would be issued empty caps. However, since the loner of this state is set to false, it could make the locker to issue the Fcb caps to them, which is incorrect. This would cause some client requests to incorrectly revoke some caps and infinitely wait and cause slow requests.

snap-schedule repeat and retention specification for monthly snapshots is changed from m to M

Previously, the snap-schedule repeat specification and retention specification for monthly snapshots was not consistent with other Ceph components.

ceph-mds no longer crashes when some inodes are replicated in multi-mds cluster

Previously, due to incorrect lock assertion in ceph-mds, ceph-mds would crash when some inodes were replicated in a multi-mds cluster.

Missing fields, such as date, client_count, filters are added to the --dump output

With this fix, missing fields, such as date, client_count, filters are added to the --dump output.

MDS no longer fails with the assert function during recovery

Previously, MDS would sometimes report metadata damage incorrectly when recovering a failed rank and thus, fail with an assert function.

The target mon_host details are removed from the peer List and mirror daemon status

Previously, the snapshot mirror peer-list showed more information than just the peer list. This output caused confusion if there should be only one MON IP or all the MON host IP’s should be displayed.

The target mon_host details are removed from the peer List and mirror daemon status

Previously, a regression was introduced by the quiesce protocol code. When killing the client requests, it would just skip choosing the new batch head for the batch operations. This caused the stale batch head requests to stay in the MDS cache forever and then be treated as slow requests.

File system upgrade happens even when no MDS is up

Previously, monitors would not allow an MDS to upgrade a file system when all MDS were down. Due to this, upgrades would fail when the fail_fs setting was set to 'true'.

Auto-generated internal topics are no longer shown in the admin topic list command

Previously, auto-generated internal topics were exposed to the user via the topic list command due to which the users could see a lot more topics than what they had created.

The deprecated bucket name field is no longer shown in the topic list command

Previously, in case of pull mode notifications (pubsub), the notifications were stored in a bucket. However, despite this mode being deprecated, an empty bucket name field is still shown in the topic list command.

Notifications are now sent on lifecycle transition

Previously, logic to dispatch on transition (as distinct from expiration) was missed. Due to this, notifications were not seen on transition.

RGWCopyObjRequest is fixed and rename operations work as expected

Previously, incorrect initialization of RGWCopyObjRequest, after zipper conversion, broke the rename operation. Due to this, many rgw_rename() scenarios failed to copy the source object, and due to a secondary issue, also deleted the source even though the copy had failed.

Ceph Object Gateway can no longer be illegally accessed

Previously, a variable representing a Ceph Object Gateway role was being accessed before it was initialized, resulting in a segfault.

An error message is now shown per wrong CSV object structure

Previously, a CSV file with unclosed double-quotes would cause an assert, followed by a crash.

Users no longer encounter 'user not found' error when querying user-related information in the Ceph dashboard

Previously, in the Ceph dashboard, end users could not retrieve the user-related information from the Ceph Object Gateway due to the presence of a namespace in the full user_id which the dashboard would not identify, resulting in encountering the “user not found” error.

Ceph Object gateway now passes requests with well-formed payloads of the new stream encoding forms

Previously, Ceph Object gateway would not recognize STREAMING-AWS4-HMAC-SHA256-PAYLOAD and STREAMING-UNSIGNED-PAYLOAD-TRAILER encoding forms resulting in request failures.

The check stat calculation for radosgw admin bucket and bucket reshard stat calculation are now correct

Previously, due to a code change, radosgw-admin bucket check stat calculation and bucket reshard stat calculation were incorrect when there were objects that transitioned from unversioned to versioned.

Tail objects are no longer lost during a multipart upload failure

Previously, during a multipart upload, if an upload of a part failed due to scenarios, such as a time-out, and the upload was restarted, the cleaning up of the first attempt would remove tail objects from the subsequent attempt. Due to this, the resulting Ceph Object Gateway multipart object would be damaged as some tail objects would be missing. It would respond to a HEAD request but fail during a GET request.

ETag values in the CompleteMultipartUpload and its notifications are now present

Previously, changes related to notifications caused the object handle corresponding to the completing multipart upload to not contain the resulting ETag. Due to this, ETags were not present for completing multipart uploads as the result of CompleteMultipartUpload and its notifications. (The correct ETag was computed and stored, so subsequent operations contained a correct ETag result.)

Listing a container (bucket) via swift no longer causes a Ceph Object Gateway crash

Previously, a swift-object-storage call path was missing a call to update an object handle with its corresponding bucket (zipper backport issue). Due to this, listing a container (bucket) via swift would cause a Ceph Object Gateway crash when an S3 website was configured for the same bucket.

Processing a lifecycle on a bucket with no lifecycle policy does not crash now

Previously, attempting to manually process a lifecycle on a bucket with no lifecycle policy induced a null pointer reference causing the radosgw-admin program to crash.

Zone details for a datapool can now be modified

The rgw::zone_create() function initializes the default placement target and pool name on zone creation. This function was also previously used for radosgw-admin zone set with exclusive=false. But, zone set does not allow the STANDARD storage class’s data_pool to be modified.

Modulo operation on float numbers now return correct results

Previously, modulo operation on float numbers returned wrong results.

SQL statements correctly return results for case-insensitive boolean expressions

Previously, SQL statements contained a boolean expression with capital letters in parts of the statement resulting in wrong interpretation and wrong results.

SQL engine returns the correct NULL value

Previously, SQL statements contained cast into type from NULL, as a result of which, the wrong result was returned instead of returning NULL.

ETags values are now present in CompleteMultipartUpload and its notifications

Previously, the changes related to notifications caused the object handle, corresponding to the completing multipart upload, to not contain the resulting ETag. As a result, ETags were not present for CompleteMultipartUpload and its notifications. (The correct ETag was computed and stored, so subsequent operations contained a correct ETag result.)

Sending workloads with embedded backslash (/) in object names to cloud-sync no longer causes sync failures

Previously, incorrect URL-escaping of object paths during cloud sync caused sync failures when workloads contained objects with an embedded backslash (/) in the names, that is, when virtual directory paths were used.

SQL statements containing boolean expression return boolean types

Previously, SQL statements containing boolean expression (a projection) would return a string type instead of boolean type.

The work scheduler now takes the next date into account in the should_work function

Previously, the logic used in the should_work function, that decides whether the lifecycle should start running at the current time, would not take the next date notion into account. As a result, any custom work time "XY:TW-AB:CD" would break the lifecycle processing when AB < XY.

merge_and_store_attrs() method no longer causes attribute update operations to fail

Previously, a bug in the merge_and_store_attrs() method, which deals with reconciling changed and the unchanged bucket instance attributes, caused some attribute update operations to fail silently. Due to this, some metadata operations on a subset of buckets would fail. For example, a bucket owner change would fail on a bucket with a rate limit set.

Checksum and malformed trailers can no longer induce a crash

Previously, an exception from AWSv4ComplMulti during java AWS4Test.testMultipartUploadWithPauseAWS4 led to a crash induced by some client input, specifically, by those which use checksum trailers.

Implementation of improved trailing chunk boundary detection

Previously, one valid-form of 0-length trailing chunk boundary formatting was not handled. Due to this, the Ceph Object Gateway failed to correctly recognize the start of the trailing chunk, leading to the 403 error.

Default values for Kafka message and idle timeouts no longer cause hangs

Previously, the default values for Kafka message and idle timeouts caused infrequent hangs while waiting for the Kafka broker.

Delete bucket tagging no longer fails

Previously, an incorrect logic in RADOS SAL merge_and_store_attrs() caused deleted attributes to not materialize. This also affected DeleteLifecycle. As a result, a pure attribute delete did not take effect in some code paths.

Object mtime now advances on S3 PutACL and ACL changes replicate properly

Previously, S3 PutACL operations would not update object mtime. Due to this, the ACL changes once applied would not replicate as the timestamp-based object-change check incorrectly returned false.

All transition cases can now dispatch notifications

Previously, the logic to dispatch notifications on transition was mistakenly scoped to the cloud-transition case due to which notifications on pool transition were not sent.

RetainUntilDate after the year 2106 no longer truncates and works as expected for new PutObjectRetention requests

Previously, PutObjectRetention requests specifying a RetainUntilDate after the year 2106 would truncate, resulting in an earlier date used for object lock enforcement. This did not affect ` PutBucketObjectLockConfiguration` requests, where the duration is specified in days.

Bucket lifecycle processing rules are no longer stalled

Previously, enumeration of per-shard bucket-lifecycle rules contained a logical error related to concurrent removal of lifecycle rules for a bucket. Due to this, a shard could enter a state which would stall processing of that shard, causing some bucket lifecycle rules to not be processed.

Deleting objects in versioned buckets causes statistics mismatch

Due to versioned buckets having a mix of current and non-current objects, deleting objects might cause bucket and user statistics discrepancies on local and remote sites. This does not cause object leaks on either site, just statistics mismatch.

Ceph Object Gateway no longer deadlocks during object deletion

Previously, during object deletion, the Ceph Object Gateway S3 DeleteObjects would run together with a multi-site deployment, causing the Ceph Object Gateway to deadlock and stop accepting new requests. This was caused by the DeleteObjects requests processing several object deletions at a time.

CURL path normalization is now disabled at startup

Previously, due to "path normalization" performed by CURL, by default (part of the Ceph Object Gateway replication stack), object names were illegally reformatted during replication. Due to this, objects whose names contained embedded . and .. were not replicated.

The authentication of the forwarded request on the primary site no longer fails

Previously, an S3 request issued to secondary failed if temporary credentials returned by STS were used to sign the request. The failure occured because the request would be forwarded to the primary and signed using a system user’s credentials which do not match the temporary credentials in the session token of the forwarded request. As a result of unmatched credentials, the authentication of the forwarded request on the primary site fails, which results in the failure of the S3 operation.

Ceph reports a POOL_APP_NOT_ENABLED warning if the pool has zero objects stored in it

Previously, Ceph status failed to report pool application warning if the pool was empty resulting in RGW bucket creation failure if the application tag was enabled for RGW pools.

Checks are added for uneven OSD weights between two sites in a stretch cluster

Previously, there were no checks for equal OSD weights after stretch cluster deployment. Due to this, users could make OSD weights unequal.

Autoscaler no longer runs while the norecover flag is set

Previously, the autoscaler would run while the norecover flag was set leading to creation of new PGs and these PGs requiring to be backfilled. Running of autoscaler while the norecover flag is set allowed in cases where I/O is blocked on missing or degraded objects in order to avoid client I/O hanging indefinitely.

The ceph config dump command output is now consistent

Previously, the ceph config dump command without the pretty print formatted output showed the localized option name and its value. An example of a normalized vs localized option is shown below:

MGR module no longer takes up one CPU core every minute and CPU usage is normal

Previously, expensive calls from the placement group auto-scaler module to get OSDMap from the Monitor resulted in the MGR module taking up one CPU core every minute. Due to this, the CPU usage was high in the MGR daemon.

The correct CRUSH location of the OSDs parent (host) is determined

Previously, when the osd_memory_target_autotune option was enabled, the memory target was applied at the host level. This was done by using a host mask when auto-tuning the memory. But the code that applied to the memory target would not determine the correct CRUSH location of the parent host for the change to be propagated to the OSD(s) of the host. As a result, none of the OSDs hosted by the machine got notified by the config observer and the osd_memory_target remained unchanged for those set of OSDs.

Monitors no longer get stuck in elections during crash/shutdown tests

Previously, the disallowed_leaders attribute of the MonitorMap was conditionally filled only when entering stretch_mode. However, there were instances wherein monitors that got revived would not enter stretch_mode right away because they would be in a probing state. This led to a mismatch in the disallowed_leaders set between the monitors across the cluster. Due to this, monitors would fail to elect a leader, and the election would be stuck, resulting in Ceph being unresponsive.

'Error getting attr on' message no longer occurs

Previously, ceph-objectstore-tool listed pgmeta objects when using --op list resulting in "Error getting attr on" message.

LBA alignment in the allocators are no longer used and the OSD daemon does not assert due to allocation failure

Previously, OSD daemons would assert and fail to restart which could sometimes lead to data unavailability or data loss. This would happen as the OSD daemon would not assert if the allocator got to 4000 requests and configured with a different allocation unit.

A sqlite database using the "libcephsqlite" library no longer may be corrupted due to short reads failing to correctly zero memory pages.

Previously, “libcephsqlite” would not handle short reads correctly which may cause corruption of sqlite databases.

The image status description now shows "orphan (force promoting)" when a peer site is down during force promotion

Previously, upon a force promotion, when a peer site went down, the image status description showed "local image linked to unknown peer", which is not a clear description.

rbd_support module no longer fails to recover from repeated block-listing of its client

Previously, it was observed that the rbd_support module failed to recover from repeated block-listing of its client due to a recursive deadlock in the rbd_support module, a race condition in the rbd_support module’s librbd client, and a bug in the librbd cython bindings that sometimes crashed the ceph-mgr.