Release Notes

Add cmount_path option and generate unique user ID

With this enhancement, you can add the optional cmount_path option and generate a unique user ID for each Ceph File System to allow sharing CephFS clients across multiple Ganesha exports thereby reducing the memory usage for a single CephFS client.

TLS is enabled across all monitoring components, enhancing security for Prometheus

With this enhancement, to safeguard data integrity, confidentiality, and alignment with the security best practices, TLS is enabled across the monitoring stack. The enhanced security feature for Prometheus, Alert manager, and Node exporter adds an additional layer of protection by using secure communication across the monitoring stack.

Enhanced security of the monitoring stack

With this enhancement, to safeguard data integrity, confidentiality, and to align with the security best practices, the authentication feature for Prometheus, Alert manager, and the Node exporter is implemented. This enhances the security of the whole monitoring stack by enabling TLS in all the monitoring components and requiring users to provide valid credentials before accessing Prometheus and Alert Manager data. By using this new feature, an additional layer of protection is provided by using secure communication and preventing the unauthorized access to sensitive metrics and monitoring data. With TLS enabled in all the monitoring stack components and authentication in place, users must authenticate before accessing monitoring and metrics data, enhancing overall security and control over data access.

Users can now put a host into maintenance mode

Previously, users could not put a host into maintenance mode as stopping all the daemons on that host would cause data unavailability.

Users can now drain a host of daemons without draining the client conf or keyring files

With this enhancement, users can drain a host of daemons, without also draining the client conf or keyring files deployed on the host by passing the --keep-conf-keyring flag to the ceph orch host drain command. Users can now mark a host to have all daemons drained or not placed there while still having Cephadm manage conf or keyring files on the host.

Cephadm services can now be marked as managed or unmanaged

Previously, the only way to mark the Cephadm services as managed or unmanaged was to edit and re-apply the service specification for the service. This was inconvenient for scenarios, such as temporarily stop making OSDs on devices that match an OSD specification in Cephadm.

Users can now apply client IP restrictions on the NFS deployment using the HAProxy protocol mode

Previously, users could not apply client IP restrictions, while still using HAProxy between the client and NFS. This is because only the HAProxy IP would be recognized by NFS, making proper client IP restriction impossible.

quota.max_bytes is now set in more understandable size values

Previously, the quota.max_bytes value was set in bytes, resulting in often very large size values, which was hard to set or changed.

Laggy clients are now evicted only if there are no laggy OSDs

Previously, monitoring performance dumps from the MDS would sometimes show that the OSDs were laggy, objecter.op_laggy and objecter.osd_laggy, causing laggy clients and the dirty data for cap revokes would not be flushed.

Improved overview dashboard utilization panel

With this enhancement, graphs and graph legends are improved for better usability. In addition, search queries are improved for giving better results.

Upgrade the cluster from the dashboard

Previously, a Red Hat Ceph Storage cluster could only be upgraded through the command-line interface.

The Ceph Dashboard has an Overview page that displays the Ceph Object Gateway information

With this enhancement, an Overview page is added in the Object Gateway section of the Ceph dashboard. From this Overview page, users can now access a dedicated object gateway overview in the Ceph dashboard. This feature enhances the user experience by providing insights into the Object Gateway performance, storage usage, and configuration details. When multi-site is configured in the cluster, the user can also see the multi-site sync status directly on this Overview page.

The Ceph dashboard displays capacity usage information for Block Device images

With this enhancement, a new capacity usage progress bar is visible within the Block > Images table on the Ceph dashboard. The bar provides visible usage information, along with a percentage.

Manage Ceph File System volumes on the dashboard

Previously, Ceph File System (CephFS) volumes could only be managed through the command-line interface.

Manage Ceph File System subvolumes and subvolume groups on the dashboard

Previously, Ceph File System (CephFS) subvolumes and subvolumes groups could only be managed through the command-line interface.

Users can now specify the FQDN for Ceph Object Gateway host through the CLI and the dashboard

Previously, short hostnames were picked to resolve the Ceph Object gateway hostname which would cause issues.

Configure Ceph Object Gateway multi-site on the dashboard

With this enhancement, you can configure multi-site Ceph Object Gateway not only through the command-line interface, but also through the Ceph Dashboard. The dashboard now supports creating, updating, and deleting Object Gateway entities such as realms, zonegroups, and zones. In addition, this feature allows users to configure multi-site between two remote clusters, providing options for data replication and synchronization.

The radosgw-admin bucket command prints bucket versioning

With this enhancement, the ` radosgw-admin bucket stats ` command prints the versioning status for buckets as enabled or off since versioning can be enabled or disabled after creation.

S3 WORM certification with an external entity

With this enhancement, S3 WORM feature is certified with an external entity. This enables data retention in compliance with FSI regulations and a secured object storage deployment that guarantees data retrieval even if the object/buckets in the production zones have been lost or compromised.

Multi-site sync instances now dynamically spread workloads among themselves

Previously, the Ceph Object Gateway multi-site throughput was generally limited to the available bandwidth of a single instance.

Enhanced bucket granular multi-site sync policies

IBM Storage Ceph now supports bucket granular multi-site sync policies.

Reduced object storage query times for data analytics with added JSON, Parquet, and CSV-format object support

Ceph Object Gateway now supports operating on JSON-format, Parquet-format, and CSV-format objects, expanding potential application of S3 select to widely deployed analytics frameworks, for example, Apache Spark and Trino. This added support helps reduce query times.

Data can now be transitioned to the Azure cloud service, using the multi-cloud gateway (MCG)

Previously, transitioning data to the Azure cloud service was a Technology Preview feature. With this enhancement, the feature is ready to be used in a production environment.

Enhanced S3 select feature for more efficient integration with Trino

Object storage now has enhanced integration with Trino for S3 select operations. This improves the query times for semi-structured and structured datasets stored in Ceph Object Gateway.

New performance counters introduced for messenger v2

Previously, there were no dedicated performance counters for accounting encrypted traffic in messenger v2.

New reports available for sub-events for delayed operations

Previously, slow operations were marked as delayed but without a detailed description.

Support HAProxy’s PROXY protocol

With this enhancement, HAProxy uses load balancing servers. This allows load balancing and also enables client restrictions on access.

Python tasks no longer wait for the GIL

Previously, the Ceph manager daemon held the Python global interpreter lock (GIL) during some RPCs with the Ceph MDS, due to which, other Python tasks are starved waiting for the GIL.

cephadm can differentiate between a duplicated hostname and no longer adds the same host to a cluster

Previously, cephadm would consider a host with a shortname and a host with its FQDN as two separate hosts, causing the same host to be added twice to a cluster.

cephadm no longer reports that a non-existing label is removed from the host

Previously, in cephadm, there was no check to verify if a label existed before removing it from a host. Due to this, the ceph orch host label rm command would report that a label was removed from the host, even when the label was non-existent. For example, a misspelled label.

The keepalive daemons communicate and enter the main/primary state

Previously, keepalive configurations were populated with IPs that matched the host IP reported from the ceph orch host ls command. As a result, if the VIP was configured on a different subnet than the host IP listed, the keepalive daemons were not able to communicate, resulting in the keepalive daemons to enter a primary state.

Stopped crash daemons now have the correct state

Previously, when a crash daemon stopped, the return code gave an error state, rather than the expected stopped state, causing systemd to think that the service had failed.

HA proxy now binds to the frontend port on the VIP

Previously, in Cephadm, multiple ingress services could not be deployed on the same host with the same frontend port as the port binding occurred across all host networks.

User-space Ceph File System (CephFS) work as expected post upgrade

Previously, the user-space CephFS client would sometimes crash during a cluster upgrade. This would occur due to stale feature bits on the MDS side that were held on the user-space side.

Blocklist and evict client for large session metadata

Previously, large client metadata buildup in the MDS would sometimes cause the MDS to switch to read-only mode.

Deadlocks no longer occur between the unlink and reintegration requests

Previously, when fixing async dirop bug, a regression was introduced by previous commits, causing deadlocks between the unlink and reintegration request.

Client always sends a caps revocation acknowledgement to the MDS daemon

Previously, whenever an MDS daemon sent a caps revocation request to a client and during this time, if the client released the caps and removed the inode, then the client would drop the request directly, but the MDS daemon would need to wait for a caps revoking acknowledgement from the client. Due to this, even when there was no need for caps revocation, the MDS daemon would continue waiting for an acknowledgement from the client, causing a warning in MDS Daemon health status.

MDS locks are obtained in the correct order

Previously, MDS would acquire metadata tree locks in the wrong order, resulting in a create and getattr RPC request to deadlock.

Sending split_realms information is skipped from CephFS MDS

Previously, the split_realms information would be incorrectly sent from the CephFS MDS which could not be correctly decoded by kclient. Due to this, the clients would not care about the split_realms and treat it as a corrupted snaptrace.

Snapshot data is no longer lost after setting writing flags

Previously, in clients, if the writing flag was set to ‘1’ when the Fb caps were used, it would be skipped in case of any dirty caps and reuse the existing capsnap, which is incorrect. Due to this, two consecutive snapshots would be overwritten and lose data.

Thread renaming no longer fails

Previously, in a few rare cases, during renaming, if another thread tried to lookup the dst dentry, there were chances for it to get inconsistent result, wherein both the src dentry and dst dentry would link to the same inode simultaneously. Due to this,the rename request would fail as two different dentries were being linked to the same inode.

Revocation requests no longer get stuck

Previously, before the revoke request was sent out, which would increase the 'seq', if the clients released the corresponding caps and sent out the cap update request with the old seq, the MDS would miss checking the seq(s) and cap calculation. Due to this, the revocation requests would be stuck infinitely and would throw warnings about the revocation requests not responding from clients.

Errors are handled gracefully in MDLog::_recovery_thread

Previously, a write would fail if the MDS was already blocklisted due to the fs fail issued by the QA tests. For instance, the QA test test_rebuild_moved_file (tasks/data-scan) would fail due to this reason.

Ceph client now verifies the cause of lagging before sending out an alarm

Previously, Ceph would sometimes send out false alerts warning of laggy OSDs. For example, X client(s) laggy due to laggy OSDs. These alerts were sent out without verifying that the lagginess was actually due to the OSD, and not due to some other cause.

Grafana panels for performance of daemons in the Ceph Dashboard now show correct data

Previously, the labels exporter were not compatible with the queries used in the Grafana dashboard. Due to this, the Grafana panels were empty for Ceph daemons performance in the Ceph Dashboard.

Edit layering and deep-flatten features disabled on the Dashboard

Previously, in the Ceph dashboard, it was possible to allow editing the layering & deep-flatten features, which are immutable, resulting in an error - rbd: failed to update image features: (22) Invalid argument.

ceph_daemon label is added to the labeled performance counters in Ceph exporter

Previously, in Ceph exporter, adding the ceph_daemon label to the labeled performance counters was missed.

Protecting snapshot is enabled only if layering for its parent image is enabled

Previously, protecting snapshot was enabled even if layering was disabled for its parent image. This caused errors when trying to protect the snapshot of an image for which layering was disabled.

Newly added host details are now visible on the cluster expansion review page

Previously, users could not see the information about the hosts that were added in the previous step.

Ceph Object Gateway page now loads properly on the Ceph dashboard.

Previously, an incorrect regex matching caused the dashboard to break when trying to load the Ceph Object Gateway page. The Ceph Object Gateway page would not load with specific configurations like rgw_frontends like beast port=80 ssl_port=443.

Ceph Object Gateway daemon no longer crashes where phoneNumbers.addr is NULL

Previously, due to a syntax error, the query for select * from s3object[*].phonenumbers where phoneNumbers.addr is NULL; would cause the Ceph Object Gateway daemon to crash.

Ceph Object Gateway daemon no longer crashes with cast( trim) queries

Previously, due to the trim skip type checking within the query for select cast( trim( leading 132140533849470.72 from _3 ) as float) from s3object;, the Ceph Object Gateway daemon would crash.

Ceph Object Gateway daemon no longer crashes with “where” clause in an s3select JSON query.

Previously, due to a syntax error, an s3select JSON query with a “where” clause would cause the the Ceph Object Gateway daemon to crash.

Ceph Object Gateway daemon no longer crashes with s3 select phonenumbers.type query

Previously, due to a syntax error, the query for select phonenumbers.type from s3object[*].phonenumbers; would cause the Ceph Object Gateway daemon to crash.

Ceph Object Gateway daemon validates arguments and no longer crashes

Previously, due to an operator with missing arguments, the daemon would crash when trying to access the nonexistent arguments.

Ceph Object Gateway daemon no longer crashes with the trim command

Previously, due to the trim skip type checking within the query for select trim(LEADING '1' from '111abcdef111') from s3object;, the Ceph Object Gateway daemon would crash.

Ceph Object Gateway daemon no longer crashes if a big value is entered

Previously, due to too large of a value entry, the query for select DATE_DIFF(SECOND, utcnow(),date_add(year,1111111111111111111, utcnow())) from s3object; would cause the Ceph Object Gateway daemon to crash.

Ceph Object Gateway now parses the CSV objects without processing failures

Previously, Ceph Object Gateway failed to properly parse CSV objects. When the process failed, the requests would stop without a proper error message.

Object version instance IDs beginning with a hyphen are restored

Previously, when restoring the index on a versioned bucket, object versions with an instance ID beginning with a hyphen would not be properly restored into the bucket index.

Multi-delete function notifications work as expected

Previously, due to internal errors, such as a race condition in the code, the Ceph Object Gateway would crash or react unexpectedly when multi-delete functions were performed and the notifications were set for bucket deletions.

RADOS object multipart upload workflows complete properly

Previously, in some cases, a RADOS object that was part of a multipart upload workflow objects that were created on a previous upload would cause certain parts to not complete or stop in the middle of the upload.

Users belonging to a different tenant than the bucket owner can now manage notifications

Previously, a user that belonged to a different tenant than the bucket owner was not able to manage notifications. For example, modify, get, or delete.

Ability to perform NFS setattr on buckets is removed

Previously, changing the attributes stored on a bucket via export as an NFS directory triggered an inconsistency in the Ceph Object gateway bucket information cache. Due to this, subsequent accesses to the bucket via NFS failed.

Testing for reshardable bucket layouts is added to prevent crashes

Previously, with the added bucket layout code to enable dynamic bucket resharding with multi-site, there was no check to verify if the bucket layout supported resharding during dynamic, immediate, or rescheduled resharding. Due to this, the Ceph Object gateway daemon would crash in case of dynamic bucket resharding and the radosgw-admin command would crash in case of immediate or scheduled resharding.

The user modify -placement-id command can now be used with an empty --storage-class argument

Previously, if the --storage-class argument was not used when running the 'user modify --placement-id' command, the command would fail.

Initialization now only unregisters watches that were previously registered

Previously, in some cases, an error in initialization could cause an attempt to unregister a watch that was never registered. This would result in some command line tools crashing unpredictably.

Multi-site replication now maintains consistent states between zones and prevents overwriting deleted objects

Previously, a race condition in multi-site replication would allow objects that should be deleted to be copied back from another site, resulting in an inconsistent state between zones. As a result, the zone which is receiving the workload ends up with some objects which should be deleted still present.

The memory footprint of Ceph Object Gateway has significantly been reduced

Previously, in some cases, a memory leak associated with Lua scripting integration caused excessive RGW memory growth.

Bucket index performance no longer impacted during versioned object operations

Previously, in some cases, space leaks would occur and reduce bucket index performance. This was caused by a race condition related to updates of object logical head (OLH), which relates to versioned bucket current version calculations during updates.

Delete markers are working correctly with the LC rule

Previously, optimization was attempted to reuse a sal object handle. Due to this, delete markers were not being generated as expected.

SQL engine no longer causes Ceph Object Gateway crash with illegal calculations

Previously, in some cases, the SQL engine would throw an exception that was not handled, causing a Ceph Object Gateway crash. This was caused due to an illegal SQL calculation of a date-time operation.

The select trim (LEADING '1' from '111abcdef111') from s3object; query now works when capitals are used in query

Previously, if LEADING or TRAILING were written in all capitals, the string would not properly read, causing a float type to be referred to as a string type, thus leading to a wrong output.

JSON parsing now works for select _1.authors.name from s3object[*] limit 1 query

Previously, an anonymous array given in the select _1.authors.name from s3object[*] limit 1 would give the wrong value output.

Client no longer resets the connection for an incorrect Content-Length header field value

Previously, when returning an error page to the client, for example, a 404 or 403 condition, the </body> and </html> closing tags were missing, although their presence was accounted for in the request’s Content-Length header field value. Due to this, depending on the client, the TCP connection between the client and the Rados Gateway would be closed by an RST packet from the client on account of incorrect Content-Length header field value, instead of a FIN packet under normal circumstances.

Sync notification are sent with the correct object size

Previously, when an object was synced between zones, and sync notifications were configured, the notification was sent with zero as the size of the object.

Multi-site sync properly filters and checks according to allowed zones and filters

Previously, when using the multi-site sync policy, certain commands, such as radosgw-admin sync status, would not filter restricted zones or empty sync group names. The lack of filter caused the output of these commands to be misleading.

The ceph version command no longer returns the empty version list

Previously, if the MDS daemon was not deployed in the cluster then the ceph version command returned an empty version list for MDS daemons that represented version inconsistency. This should not be shown if the daemon is not deployed in the cluster.

ms_osd_compression_algorithm now displays the correct value

Previously, an incorrect value in ms_osd_compression_algorithm displayed a list of algorithms instead of the default value, causing a discrepancy by listing a set of algorithms instead of one.

MGR no longer disconnects from the cluster without retries

Previously, during network issues, clusters would disconnect with MGR without retries and the authentication of monclient would fail.

Increased timeout retry value for client_mount_timeout

Previously, due to the mishandling of the client_mount_timeout configurable, the timeout for authenticating a client to monitors could reach up to 10 retries disregarding its high default value of 5 minutes.

Demoted mirror snapshot is removed following the promotion of the image

Previously, due to an implementation defect, the demoted mirror snapshots would not be removed following the promotion of the image, whether on the secondary image or on the primary image. Due to this, demoted mirror snapshots would pile up and consume storage space.

Non-primary images are now deleted when the primary image is deleted

Previously, a race condition in the rbd-mirror daemon image replayer prevented a non-primary image from being deleted when the primary was deleted. Due to this, the non-primary image would not be deleted and the storage space was used.

The librbd client correctly propagates the block-listing error to the caller

Previously, when the rbd_support module’s RADOS client was block-listed, the module’s mirror_snapshot_schedule handler would not always shut down correctly. The handler’s librbd client would not propagate the block-list error, thereby stalling the handler’s shutdown. This lead to the failures of the mirror_snapshot_schedule handler and the rbd_support module to automatically recover from repeated client block-listing. The rbd_support module stopped scheduling mirror snapshots after its client was repeatedly block-listed.

Newly implemented Crimson-OSD of the core Ceph object storage daemon (OSD) component replaces ceph-osd

With this enhancement, the next generation ceph-osd is implemented for multi-core scalability and to improve performance with fast network and storage devices, employing state-of-the-art technologies that includes DPDK and SPDK. Crimson aims to be compatible with an earlier version of OSD daemon with the class ceph-osd.

Object storage archive zone in Red Hat Ceph Storage

With this enhancement, the archive zone receives all objects from the production zones and keeps every version for every object, providing the user with an object catalogue that contains the full history of the object. This provides a secured object storage deployment that guarantees data retrieval even if the object/buckets in the production zones have been lost or compromised.

Protect object storage data outside of a production cluster using per-bucket enable and disable sync to an archive zone

As an administrator, you can now recover any version of any object that has existed on the primary site from the archive zone. In the case of data loss or a ransomware attack, valid versions of all objects are accessible, if needed.

Balancing Red Hat Ceph Storage cluster using read balancer

With this release, to ensure that each device gets its fair share of primary OSDs so that read requests get distributed across OSDs in the cluster, evenly, read balancer is implemented. Read balancing is cheap and the operation is fast as there is no data movement involved. Read balancing supports replicated pools only. Erasure coded pools are not supported.

Some NFS daemons may not produce logs

Currently, in a cephadm-deployed NFS service with multiple NFS daemons, only one would be serving IO, causing the user to come across NFS daemons that produce no logs. There is no workaround for this as of now.

Prevent Mutex from failing when unlocking.

Previously, when a Mutex that was not locked was attempted to be unlocked, the Mutex crashed.

ceph orch ps command does not display a version for monitoring stack daemons

In cephadm, due to the version grabbing code currently being incompatible with the downstream monitoring stack containers, version grabbing fails for monitoring stack daemons, such as node-exporter, prometheus, and alertmanager.

Ceph Object Gateway page does not load after a multi-site configuration

The Ceph Object Gateway page does not load because the dashboard cannot find the correct access key and secret key for the new realm during multi-site configuration.

JSON select count() from S3Object[]; queries are lagging and cause high CPU usage

When running the select count() from S3Object[]; query, the time lapse and radosgw CPU utilization are very high compared to CSV object queries.

s3select and Trino fail when processing JSON object using s3select

When Trino processes a JSON object using the s3select request, the request fails causing Trino to fail too. This emits the wrong json dataType should use DOCUMENT error message in the Ceph Object Gateway logs.