9.0 Release Notes

New cephadm certificate lifecycle management for improved Ceph cluster security

cephadm certificate lifecycle management was previously available as limited release. This enhancement provides full availability for new and existing customers in production environments.

Multiple container registries can now be defined in registry credentials

Previously, only a single container registry credential could be configured. However, users may have different registries for different service containers.

Enhanced config parameter to set the maximum number of OSDs to upgrade in parallel

With this enhancement, the config parameter sets the maximum number of OSDs that can be upgraded in parallel. The default value is 16.

New support for managing Ceph Object Gateway accounts

Previously, managing Ceph Object Gateway accounts was only possible through the command-line interface (CLI) using radosgw-admin commands.

New migration from Promtail to Grafana Alloy for centralized logging

Previously, centralized logging relied on Promtail, which is now deprecated and no longer recommended for new deployments.

Case sensitivity and Unicode normalization can now be configured during subvolume group creation

Previously, it was possible to configure Unicode normalization and case sensitivity when creating a subvolume, but not when creating a subvolume group. To apply these settings, users had to run additional commands after the group was created.

Source information of clone subvolumes is now preserved

Previously, after cloning was completed, the source information (subvolume or snapshot) of the clone was removed from the .meta file. As a result, when users ran the subvolume info command for a clone subvolume, they could not view details about its source.

Now supports monitoring subvolume-level metrics

CephFS now provides performance metrics at the subvolume level, including IOPS, throughput, and latency. These metrics help administrators monitor IO allocations for applications and protocol gateways that use CephFS subvolumes. Metrics are available through Prometheus, the Ceph Manager stats module, and the Ceph Dashboard.

Bucket logging support for Ceph Object Gateway with bug fixes and enhancements

Bucket logging was previously available as limited release. This enhancement provides full availability for new and existing customers in production environments.

Restore objects transitioned to remote cloud endpoint back into Ceph Object gateway using the cloud-restore feature

The cloud-restore feature was previously available as limited release. This enhancement provides full availability for new and existing customers in production environments.

New support for updating the restoration period for archived objects

With this enhancement, you can now update the expiry date of a restored object by reissuing the restore-object API request with a new restoration period. The updated period is calculated from the current time, allowing you to retain data longer or expire it sooner without re-downloading from the remote cloud endpoint.

New CLI commands introduced to help monitor and debug restore operations

Previously, administrators had limited visibility into object restore operations, which made monitoring and debugging difficult.

Improved CLI output for topic management

The radosgw-admin topic list command has been enhanced for better usability. The output format is now consistent across v1 and v2 topics and excludes the topics section, reducing complexity for automation and scripting.

Enhanced conditional operations

This enhancement introduces support for conditional PUT and DELETE operations, including bulk and multi-delete requests. These conditional operations improve data consistency for some workloads.

Flushed object name now emitted

Previously, users had no direct way to identify the last object that was flushed. This made it harder to determine the correct starting point when traversing log objects in the log bucket.

Reduced client impact during bucket resharding

With this enhancement, bucket resharding now does most of its processing before it starts to block write operations. This should significantly reduce the client-visible impact of resharding on large buckets.

Committed objects now added to log buckets even without pending records

Previously, when committing an object, it was not added to the log bucket if there were no log records pending. This made it harder for consumers to reliably determine the last committed object when listing log bucket contents.

Clear error propagation for logging failures in journal mode

Previously, when logging failed in journal mode, the customer received generic or misleading error messages. For example, a customer performing a regular S3 operation could see a 403 error if permissions were missing on the log bucket, even though permissions were correct on the target bucket.

Automatic permission setting for D3N cache directory

Previously, configuring the RGW D3N cache directory required manual steps to set permissions, such as running chmod a+rwx rgw_d3n_l1_datacache_persistent_path. This added complexity and increased setup time.

New support for AWS S3 GetAccountSummary

Previously, AWS S3 GetAccountSummary was not supported, which limited certain workloads that require account-level information, such as Terraform-based automation.

New support for AWS STS GetCallerIdentity

Previously, AWS STS GetCallerIdentity was not supported, limiting the ability to validate user identities and enforce access policies before creating or modifying policies. This gap impacted workflows that rely on identity verification, such as Terraform-based automation.

Aligned operation names with AWS for consistent log integration

Previously, operation names in Ceph logs were inconsistent with the operation types used by AWS. This required different approaches for log consumption depending on whether Ceph or AWS logs were being processed.

Improved reliability for multi-site replication data log delivery

Previously, in rare cases, replication data logs could lose updates, which created the appearance of stalled replication even though data consistency was not affected.

Cleanup added for index segments of replicated buckets

Previously, dynamic resharding with multi-site replication had a long-standing limitation: old index segments were not cleaned up due to simultaneous access to old and new index shards during replication. This resulted in persistent space leakage.

Enhanced support for moving stretch mode to normal mode

Previously, Ceph clusters operating in stretch mode could not be reverted to normal mode without manual intervention.

Enhanced detection of network partitions under connectivity election strategy

Previously, monitors operating under the connectivity election strategy did not provide user-facing alerts when network partitions occurred.

New ISA plugin support for erasure coded pools

Previously, erasure coded pools only supported Jerasure plugins.

General enhancements for RADOS and RADOS BlueStore

This version provides several enhancements for RADOS and RADOS BlueStore. These enhancements include the following:

Erasure coding ratio support enhancements

This release introduces new support and qualification for 5+2 and 6+2 erasure coding ratios. These configurations deliver an optimal balance of performance, scalability, and cost efficiency, making them ideal for clusters that require high storage utilization and robust data protection.

Improved tracking of mirror group snapshot states

Previously, rbd-mirror tracked the progress of a mirror group snapshot without distinguishing between a snapshot that was created and one that was fully synced.

Ceph Object Gateway page now loads after a multi-site configuration

Previously, the Ceph Object Gateway page does not load because the dashboard could not find the correct access key and secret key for the new realm during multi-site configuration.

enctag value length is now restricted to 255 characters

Previously, enctags could be stored with values longer than 255 characters. However, operations such as enctag get only supported displaying values up to 255 characters. If an enctag with a longer value was stored, the system returned a general Unexpected error output.

Unsupported encryption algorithms now return an error on CephFS

Previously, the CephFS userspace did not validate encryption algorithms when setting up fscrypt. Only AES-256-XTS and AES-256-CTS were supported, but if a different algorithm was requested, CephFS silently used the default supported algorithm without notifying the user.

Cloud-S3 restore requests no longer lost after Ceph Object Gatway restart

Previously, if the Ceph Object Gateway (RGW) service restarted while a restore request for the cloud-s3 cloud tier was in progress, the request state was lost because it was not stored persistently. As a result, the restore operation was not retried.

Local reads now work as expected

Previously, local reads were sometimes unavailable for recently created or modified RADOS objects due to protocol limitations.

max_objs_per_shard no longer goes below a safe minimum

Previously, in versioned buckets, the max_objs_per_shard value was reduced by a factor of three to account for the additional index entries created by object versioning. In cases, such as debugging, this value could artificially be set low to trigger early resharding. With these two items in combination, these adjustments could result in a max_objs_per_shard value of 0, leading to a division by 0 crash.

RGW STS now supports encryption keys larger than 1024 bytes

Previously, the RGW STS implementation did not support encryption keys larger than 1024 bytes. Users had to manually adjust Keycloak settings by lowering the priority of the rsa-enc-generated provider and reducing the keySize to 1024.

Checksum type and checksum algorithm now display with uncompleted multipart uploads

Previously, when listing parts for uncompleted multipart uploads, the checksum type and checksum algorithm were missing because the logic to extract these fields was not implemented.

radosgw-admin no longer crashes by non-positive values

Previously, when running the radosgw-admin bucket reshard command, using a non-positive --num-shards value, such as a zero or a negative number, would cause radosgw-admin to crash.

Empty string in the HTTP_X_AMZ_COPY_SOURCE header no longer causes crashing

Previously, the HTTP_X_AMZ_COPY_SOURCE header could have an empty string output, rather than NULL. When the empty string was passed to RGWCopyObj::parse_copy_location() the empty name would cause a crash.

Multipart upload completion logs now include object size

Previously, the log record for multipart upload completion did not include the object size.

Operation names in log records now match AWS naming conventions

Previously, the name of the operation in Ceph Object Gateway log records did not match the name used in AWS. As a result, consumers that could parse AWS generator records failed when processing records generated by Ceph Object Gateway.

Bucket logging configuration changes no longer cause data loss

Previously, a race condition occurred when the bucket logging configuration was changed while the system was running. This caused log objects to be garbage collected, and log records were lost after the garbage collector ran.

Copied objects from versioned to non-versioned buckets are now accessible

Previously, when copying an object from a versioned bucket to a non-versioned bucket, some versioning attributes were mistakenly copied to the destination object. This could make the copied object inaccessible.

Data transition to AWS non-default regions is now supported

Previously, the cloud tier module did not handle the location_constraint parameter required by AWS when creating a bucket in a non-default region. As a result, data transition to an AWS cloud endpoint failed if the target bucket was in a non-default region.

Tenant user policy and role-based permissions now work as expected after upgrade

Previously, some policy or role-based permissions involving legacy tenant users behaved differently after upgrading to releases that support IAM accounts. As a result, expected access grants would fail.

Predefined ACLs are now correctly matched

Previously, reversed logic in the comparison functor for predefined ACL matching caused all predefined ACLs to be rejected.

Permission checks for multipart upload initialization are now correct

Previously, the permission check for InitMultipart incorrectly used the bucket Amazon Resource Name (ARN) instead of the object ARN. This could cause PutObject requests to fail unexpectedly.

Improved bulk delete performance in versioned buckets

Previously, the Ceph Object Gateway object deletion logic was inefficient, particularly in how it invoked update_olh. This caused high latency and could eventually lead to system lockups when processing bulk deletes of up to 1,000 object versions per request in versioned buckets.

ACL checks after AssumeRole are now correctly enforced

Previously, incorrect logic failed to verify ACLs after an AssumeRole operation. As a result, checks for explicit ACL grants failed incorrectly.

Log records now correctly indicate ACL-based authorization.

Previously, the aclRequired field in the log record would display as with a hyphen (-), even when the request was authorized by an ACL. This was misleading because it suggested that the operation was authorized by a bucket policy.

Log records now correctly indicate authentication type for unauthenticated requests

Previously, the AuthenticationType field in the log record was incorrectly set to QueryString for unauthenticated requests.

IAM policy now recognizes AbortMultipartUpload Deny requests

Previously, a session policy incorrectly used the AbortMultipartUpload action. As a result, a Deny statement for AbortMultipartUpload in the IAM policy was not respected when PutObject was allowed.

Delete bucket policy now returns correct status code

Previously, the delete bucket policy operation returned HTTP status code 204, instead of the correct 200 code.

api_name field now initializes during zone group rename

Previously, the api_name field in the zone group map was not initialized during a zone group rename because the variable assignment was missing.

Multipart object decryption now works for partNumber requests

Previously, if a multipart object was encrypted using SSE-C or SSE-S3, a get object request with partNumber did not decrypt the part.

Rate limit configurations now synchronize across all Ceph Object Gateway multi-site zones

Previously, user and bucket quota rate limit configurations set on a secondary Ceph Object Gateway site were not synchronized back to the primary site in a multi-site setup. This caused configuration inconsistencies, resulting in different rate-limiting behaviors between zones.

RGW multi-site now automates cleanup of deleted bucket instances and index objects

Previously, deleted bucket instances and related index objects were retained and not trimmed during bilog trimming to allow multi-site sync to finish processing deletions on other zones.

Object lock configuration rule is now synchronized to the secondary zone

Previously, when object lock was enabled on a bucket, the rule failed to replicate to other zones, causing object lock inconsistencies between zones.

RGWBucketFullSyncCR no longer spins indefinitely when the source bucket has been deleted

Previously, the coroutine RGWListRemoteBucketCR reused the bucket_list_result member without clearing its prior state. Stale entries and the is_truncated flag from a previous iteration could persist, causing the loop to continue even after the bucket was deleted.

New OAuth2 SSO

OAuth2 SSO uses the oauth2-proxy service to work with the Ceph Management gateway (mgmt-gateway), providing unified access and improved user experience.

New per-user and per-bucket usage counters in Prometheus

Ceph Object Gateway now exports per-user and per-bucket usage counters via performance counters automatically collected by the ceph-exporter and made available in Prometheus. This provides low-overhead, real-time visibility into the following:

Balanced primary placement groups can now be observed in a cluster

Previously, users could only balance primaries with the offline osdmaptool.

Now supports tracking data availability score of a cluster

This release introduces a feature that tracks the data availability score of a Ceph cluster over time. The score represents how accessible your data is at any given moment, based on factors such as OSD health, placement group states, and redundancy policies.

NFS daemon fails to start for NFSv3

The NFS daemon fails to start when the rpcbind and rpc.statd services are missing or not running. These services are required for NFSv3, and by default, cephadm creates the NFS service for both NFSv3 and NFSv4 protocols. When these services are unavailable, the NFS daemon does not come online and emits the Cannot register NFS V3 on TCP error.

Grafana certificate does not migrate during upgrade

When you upgrade from Red Hat Ceph Storage 8.1 to 9.0, the existing user-signed Grafana certificate is not migrated. Instead, Grafana switches to a cephadm-signed certificate. As a result, duplicate certificate entries may appear, and certificate-related health warnings can persist. Manual reconfiguration is required if you want to use custom TLS certificates.

Management gateway does not open HTTPS port during deployment

When the management gateway (mgmt-gateway) is deployed with default settings and firewalld is active, the default HTTPS port (443) is not opened in firewalld. The gateway listens on port 443 and is reachable locally, but remote access to the dashboard fails until the firewall is manually adjusted.

Cephadm operations may fail when interactive shell aliases are present

In Red Hat Ceph Storage 7, cephadm uses the shell mv command on remote hosts. If the cephadm SSH user has interactive aliases such as mv='mv -i' (and similar for rm or cp), these aliases trigger prompts and block cephadm operations. As a result, commands like ceph orch upgrade, cephadm bootstrap, or adding hosts may hang or fail because mv waits for user confirmation instead of running non-interactively.

Promtail image remains visible after migration to Alloy

During the transition from Promtail to Alloy, cephadm continues to register the Promtail container image to maintain backward compatibility and ensure a smooth migration path. As a result, Promtail still appears in the cephadm list-images output after upgrading, even though Alloy is the new default. The behavior is intentional to prevent breaking log collection on clusters that have not fully migrated.

HAProxy deployment fails when QAT is enabled with ingress

Deploying HAProxy with the QAT feature enabled fails on Red Hat Ceph Storage 9.0 container images when using the ingress feature.

QAT cannot be used for TLS offload or acceleration mode together with SSL set

Enabling QAT on HAProxy with SSL enabled injects legacy OpenSSL engine directives. The legacy OpenSSL engine path breaks the TLS handshake, emitting the tlsv1 alert internal error error. With the TLS handshake broken, the TLS termination fails.

Active alert displays even when Prometheus module is active

In some cases, the Ceph Dashboard shows an active alert for CephMgrPrometheusModuleInactive even though the Prometheus module is enabled. This can happen due to a cluster misconfiguration that causes the Ceph target to go down, falsely triggering the alert.

Dashboard cannot delete non-default zone groups or zones

Users cannot delete non-default zone groups or zones from the Ceph Dashboard. Attempts to delete them fail.

Subvolume operations delayed due to GIL contention during asynchronous cloning

When the asynchronous cloner in the volumes module (mgr/volumes) uses the CephFS Python binding, it invokes the Ceph client library API while holding the Python Global Interpreter Lock (GIL). During asynchronous clone operations, the GIL remains locked for an extended period, which prevents other CephFS subvolume operations such as create and delete from acquiring the GIL in time. As a result, customers may experience delayed responses when performing subvolume operations.

Lifecycle processing stuck in PROCESSING state for a given bucket

If a Ceph Object Gateway server is unexpectedly restarted when the lifecycle processing is in progress for a given bucket, that bucket does not resume processing lifecycle work for at least two scheduling cycles and is stuck in PROCESSING state. This is an expected behavior as it is intended to avoid multiple Ceph Object gateway instances or threads from processing the same bucket simultaneously, especially when the debugging is in progress in production.

Ceph Object Gateway services down after upgrade

After upgrading, Ceph Object Gateway services may fail to start. The service fails to start as the rgw service now enforces the rgw_realm configuration but no realm exists in the Ceph Object Gateway configuration. As a result, the following outputs occur:

Sync failure occurs after renaming a zone or zone group

Renaming a zone or zone group in the Primary zone in the master_zonegroup can cause sync failures. When sync failures occur, the following sync status error is may be emitted and further sync operations are affected:

Secondary site continues to display old zone group name after rename

In some cases, when a zone group is renamed on the Primary site, the Secondary site may still display the old zone group name. This occurs because the old name is not removed from the .rgw.root pool after the rename operation.

Multi-site lifecycle expiration does not clean OLH entries in versioned buckets

Multi-site lifecycle expiration may fail to remove object log header (OLH) entries in versioned buckets. The system leaves stale data in the bucket index. This issue occurs when lifecycle expiration runs on multi-site deployments for versioned buckets. As a result, stale OLH entries remain after object deletion. This causes bucket index bloat and may impact bucket operations for customers.

Kernel client does not support pg-upmap-primary

The kernel client currently does not support the pg-upmap-primary feature. As a result, users may encounter issues when attempting to mount images or filesystems using the kernel client in environments where pg-upmap-primary is configured.

Placement groups are not scaled down in upmap-read and read balancer modes

Currently, pg-upmap-primary entries are not properly removed for placement groups (PGs) that are pending merge. For example, when the bulk flag is removed on a pool, or any case where the number of PGs in a pool decreases. As a result, the PG scale-down process gets stuck and the number of PGs in the affected pool do not decrease as expected.