Chapter 4. Bug fixes

The haproxy daemon no longer fails deployment when using the haproxy_qat_support setting in the ingress specification

Previously, the haproxy_qat_support was present but not functional in the ingress specification. This was added to allow haproxy to offload encryption operations on machines with QAT hardware, intending to improve performance. The added function did not work as intended, due to an incomplete code update. If the haproxy_qat_support setting was used, then the haproxy daemon failed to deploy.

PROMETHEUS_API_HOST gets set during Cephadm Prometheus deployment

Previously, PROMETHEUS_API_HOST would not always get set when Cephadm initially deployed Prometheus. This issue was seen most commonly when bootstrapping a cluster with --skip-monitoring-stack, then deploying Prometheus at a later time. As a result monitoring information could be unavailable.

Creating a sub-user from the dashboard now works as expected

Previously, a Python coding error prevented the dashboard and API from creating sub-users.

Editing RGW-related configurations from the dashboard is now supported

Previously, the dashboard relied on incorrect flag data to determine whether a configuration was editable, preventing users from modifying RGW-related settings that were editable via the CLI.

Invalid headers during a no longer cause a segmentation fault during journal import

Previously, the cephfs-journal-tool did not check for headers during a journal import operation. This would cause a segmentation fault.

cephfs-data-scan during disaster recovery now completes as expected

Previously, in some cases, the cephfs-data-scan ran during disaster recovery but did not create a missing directory fragment from the backtraces or create duplicate links. As a result, directories were inaccessible or crashed the MDS.

Inode invalidation operations now complete faster

Previously, an extra reference to an inode was taken in some cases that was never released. As a result, operations requiring inode invalidation were delayed until a timeout elapsed, making them very slow.

Space larger than the NFS export disk size can no longer be allocated

Previously, an empty file could be created without storage blocks allocated. These empty files could cause write operations to fail when writing to the desired file region, with a command such as fallocate.

Proxy daemon logs now update immediately

Previously, log messages from the proxy daemon were buffered by the glibc library, causing delays in log file updates. As a result, in the event of a crash, some log entries could be lost, making troubleshooting and debugging more difficult.

Async write deadlock fixed under OSD full conditions

Previously, when asynchronous writes were ongoing and the OSD became full, the client received a notification to cancel the writes. The cancellation method and the callback invoked after the write was canceled both attempted to acquire the same lock. As a result, this led to a deadlock, causing the client to hang indefinitely during an OSD full scenario.

Expanded removexattr support for CephFS virtual extended attributes

Previously, removexattr was not supported on all appropriate Ceph virtual extended attributes, resulting in attempts to remove an extended attribute failing with a "No such attribute" error.

MDS and FS IDs are now verified during health warning checks for fail commands

Previously, the MDS and FS IDs were not checked when executing the ceph mds fail and ceph fs fail commands. As a result, these commands would fail with a "permission denied" error for healthy MDS or FS instances when another instance in the cluster exhibited health warnings.

Error mapping now displays specific error message

Previously, an incorrect mapping of the error code to the user message resulted in a generic message being displayed. As a result, users did not see the specific details of the error encountered.

fscrypt now decrypts long file names

Previously, the alternate name, which holds the raw encrypted version of the file name, was not provided in all decryption cases. As a result, long file names were not being decrypted correctly, and incomplete directory entry data was produced.

Snapshot names are now stored in plain text

Previously, snapshots could be created regardless of whether the fscrypt key was present. When a snapshot was created using the mgr subvolume snapshot create command without the key, the snapshot name was not encrypted during creation. As a result, subsequent attempts to decrypt the plain text name produced unreadable output.

Multipart uploads using AWS CLI no longer cause Ceph Object Gateway to crash

Previously, during a multipart upload using AWS CLI, RGW crashed due to checksum algorithms and reporting behavior introduced in AWS S3 and AWS SDKs, specifically the new CRC64NVME checksum algorithm.

Deleted objects no longer appear in the bucket listing

Previously, a race between CompleteMultipart and AbortMultipart uploads could lead to inconsistent results. As a result, objects could appear in the bucket listing, even when they were no longer present.

Ceph Object Gateway no longer crashes during object deletion

Previously, in some cases, an uninitialized check_objv parameter variable could lead to accessing an invalid memory address in the object delete path. As a result, there was a segmentation fault.

Tail objects no longer wrongly deleted with copy-object

Previously, there was a reference count invariant on tail objects that was not maintained when an object was copied to itself. This caused the existing object was changed, rather than copied. As a result, references to tail objects were being decremented. When the refcount on tail objects dropped to 0, they were deleted during the next garbage collection (GC) cycle.

AssumeRoleWithWebIdentity operations now fails as expected when incorrect thumbprints are added

Previously, due to a boolean flag being incorrectly set in the code, the AssumeRoleWithWebIdentity operation succeeded even when an incorrect thumbprint was registered in the CreateOIDCProvider call. As a result, AssumeRoleWithWebIdentity was able to succeed when it should have failed.

Ceph Object Gateway can now delete objects when RADOS is at maximum pool capacity

Previously, when a RADOS pool was near its maximum quota, the Ceph Object Gateway was not able to delete objects.

User Put Object permissions are now recognized on copied buckets

Previously, bucket policies of a copy source bucket with access permissions for Put Object were not recognized on the copied bucket. As a result, when accessing the copied bucket, an Access Denied error was emitted.

Large queries on Parquet objects no longer emit an Out of memory error

Previously, in some cases, when a query was processed on a Parquet object, that object was read in large chunks. This caused the Ceph Object Gateway to load a larger buffer into the memory, which was too big for low-end machines. The memory would especially be affected when Ceph Object Gateway was co-located with OSD processes, which consume a large amount of memory. With the Out of memory error, the OS killed the Ceph Object Gateway process.

radosgw-admin no longer crashes by non-positive values

Previously, when running a radosgw-admin bucket reshard command, using a non-positive --num-shards value, such as a zero or a negative number, would cause radosgw-admin to crash.

Ceph Object Gateway no longer fails during signature validation

Previously, if the JSON Web Token (JWT) was not signed using the first x5c certification for signature validation, the signature validation fails.

Objects are now removed as per the lifecycle rules set when bucket versioning is suspended

Previously, due to an error in the lifecycle code, the lifecycle process did not remove the objects if the bucket versioning was in the suspended state. As a result, the objects were still seen in the bucket listing.

Multipart uploads can now add object tags

Previously, the Ceph Object Gateway S3 multipart upload object tags were not recognized when sent by the client. As a result, clients were not able to successfully apply object tags during initial object creation during a multipart upload.

STS implementation now supports encryption keys larger than 1024 bytes

Previously, Ceph Object Gateway STS implementation did not support encryption keys larger than 1024 bytes.

Bucket logging configurations no longer allow setting the same source and target buckets

Previously, there was no check in place when setting a bucket logging configuration, verifying that the source and target buckets were different.

Ceph Object Gateway no longer crashes due to mishandled kafka error messages

Previously, error conditions with the kafka message broker were not handled correctly. As a result, in some cases, Ceph Objet Gateway would crash.

ACL bucket operations now work as expected

Previously, a local variable 'uri' shadowed a member variable with the same name. As a result, a subset of bucket ACL operations would fail.

Target buckets now needs a bucket policy for users to write logs to them

Previously, no permission checks were run on the target bucket for bucket logging. As a result, any user could write logs to a target bucket, without needing specific permissions.

S3 requests no longer rejected if local is listed before external for the authentication order

Previously, S3 requests were rejected when the request is not authenticated successfully by the local authentication engine. As a result, S3 requests using OpenStack Keystone EC2 credentials failed to authenticate with Ceph Object Gateway when the authentication order had local before external

Ceph Object Gateway internal HTTP headers are no longer sent while transitioning the object to Cloud

Previously, some Ceph Object Gateway internal HTTP header values were sent to the Cloud endpoint, when transitioning the object to Cloud. As a result, some S3 cloud services did not recognize the headers and failed the transition or failed to restore the operation of the objects.

The radosgw-admin bucket logging flush command now provides works as expected

Previously, using the radosgw-admin bucket logging flush command would return the next lob object name. As a result, the user did not know the name of the log object that was flushed without listing the log bucket.

Upgrading clusters now fetches notification_v2 topics correctly

Previously, upgrading clusters upgraded bucket notifications to notification_v2. As a result, topics in notification_v2 were not retrieved as expected.

olh get now completes as expected

Previously, a 2023 fix for a versioning-related bug caused an internal code path to reference an incorrect attribute name for the object logical head (OLH). As a result, an error would emit when running the radosgw-admin olh get command.

Swift container listings now report object last modified time

Previously, the Ceph Object Gateway Swift container listing implementation was missing the logic to send the last_modified JSON field. As a result, Swift container listings did not report the last modified time of objects.

Ceph Object Gateway now recognizes additional checksums from their checksum-type specific headers and trailers

Previously, the aws-sdk-go-v2 checksum behavior differed from other SDKs, as it did not send either x-amz-checksum-algorithm or x-amz-sdk-checksum and never included x-amz-decoded-content-length, despite AWS documentation requiring it. As a result, additional checksums were not recognized when sent, and some AWS-chunked requests failed an assertion check for decoded content length with an InvalidArgument error.

Shadow users for the AssumeRoleWithWebIdentity call are now created within the oidc namespace

Previously, an incorrect method was used to load the bucket stats, which caused the shadow users for AssumeRoleWithWebIdentity call to not be created within the oidc namespace. As a result, users were not able to differentiate between the shadow users and local rgw users.

Replicating metadata from earlier versions of Red Hat Ceph Storage no longer renders user access keys as “inactive”

Previously, when a secondary zone running Red Hat Ceph Storage 8.0 replicated user metadata from a pre-8.0 metadata master zone, the access keys of those users were erroneously marked as "inactive". Inactive keys cannot be used to authenticate requests, so those users are denied access to the secondary zone.

Invalid URL-encoded text from the client no longer creates errors

Previously, the system improperly handled scenarios where URL decoding resulted in an empty key.name. The empty key.name due to invalid URL-encoded text from the client. As a result, an assertion error during the copy operation would occur, and sometimes led to a crash later.

Network error code is now mapped correctly

Previously, when one or some of the Ceph Object Gateways in a target zone were down, the HTTP client in the Ceph Object Gateway in the source zone did not map the network connection error code correctly internally. As a result, the client kept attempting to connect to a downed Ceph Object Gateway instead of falling back to other active ones.

sync error trim now runs as expected with optional --shard-id input

Previously, the sync error trim command did not mark the --shard-id option as optional.

Objects restored from Cloud/Tape now synchronize correctly to remote locations

Previously, objects restored from Cloud/Tape retained their original mtime, making it insufficient for multisite sync checks. As a result, these restored objects were not synchronized to remote locations.

Sync rate now works as expected

Previously, in some cases, an incorrect internal error return caused sync operations to run slower than expected.

rgw daemons no longer crash due to stack overflows

Previously, large clusters of over 10,000 daemons with a stack-based allocation of variable-length arrays caused a stack overflow. As a result, the Ceph Object Gateway daemons crashed.

Pool removals now remove pg_upmap_primary mappings from the OSDMap

Previously, deleting pools did not remove pg_upmap_primary mappings from the OSDMap. As a result, pg_upmap_primary mappings were seen but could not be removed, since the pool, and pgid no longer existed.

Destroyed OSDs are no longer listed by the ceph node ls command

Previously, destroyed OSDs were listed without any indication of their status, leading to user confusion and causing cephadm to incorrectly report them as stray.

AVX512 support for the ISA-L erasure code plugin is now enabled

Previously, due to an issue in the build scripts, the plugin did not take advantage of the AVX512 instruction set—even on CPUs that supported it—resulting in reduced performance.

Multiple OSDs crashing while replaying bluefs -- ceph_assert(delta.offset == fnode.allocated)

Previously, a fix was implemented to prevent RocksDB’s SST files from expanding, but it contained a bug. As a result, the BlueFS log became corrupted, causing an error that prevented OSD bootup, even though it could be ignored.

BlueFS log no longer gets corrupted due to race conditions

Previously, a rare condition between truncate and unlink operations in BlueFS caused the log to reference deleted files. This corrupted the BlueFS log, triggering an assertion failure during OSD startup and resulting in a crash loop.