Release Notes

OS tuning profiles added to manage kernel parameters using cephadm

With this release, to achieve feature parity with ceph-ansible, users can apply tuned profile specifications that cause cephadm to set OS tuning parameters on the hosts matching the specifications

Users can now easily set the Prometheus TSDB retention size and time in the Prometheus specification

Previously, users could not modify the default 15d retention period and disk consumption from Prometheus.

Example

service_type: prometheus
placement:
  count: 1
spec:
  retention_time: "1y"
  retention_size: "1GB"

New Ansible playbook is added to define an insecure registry

Previously, when deploying a Red Hat Ceph Storage cluster with a large number of hosts in a disconnected installation environment, it was tedious to populate the /etc/containers/registries.conf file on each host.

Improved Ceph Dashboard features for rbd-mirroring is now available

Previously, there was no Ceph Block Device Snapshot mirroring support from the user interface.

A new logging functionality is added to the Ceph dashboard

With this release, a centralized logging functionality for a single cluster, named Daemon Logs, is implemented on the dashboard under the Cluster → Logs section. This makes it easier for users to monitor logs in an efficient manner.

A new TTL cache is added between the Ceph Manager and its modules

Big Ceph clusters generate a lot of data, which might overload the cluster and render modules unsuitable.

A new information message is provided on Ceph Dashboard to troubleshoot issues with Grafana

When Grafana is deployed with self-signed TLS certificates instead of certificates signed by a Certificate Authority, most browsers, such as Chrome or Firefox, do not allow the embedded Grafana iframe to be displayed within the Ceph Dashboard.

The number of repaired objects in pools is exposed under Prometheus metrics

Previously, data regarding auto-repaired objects was gathered through log parsing which was inefficient.

Ceph Dashboard now clearly indicates errors on certain CephFS operations

Previously, when a user tried to perform an operation on a filesystem directory, but did not have permission, the Ceph Dashboard reported a generic 500 internal server-side error. However, these errors are actually imputable to users, since permissions are the same for preventing certain actions for given users.

Users can now see new metrics for different storage class in Prometheus

With this release three new metrics, ceph_cluster_by_class_total_bytes, ceph_cluster_by_class_total_used_bytes, and ceph_cluster_by_class_total_used_raw_bytes, are added for different storage class in Prometheus filtered by device class which would help to follow up the performances and the capacity of the infrastructure.

The WAL and DB devices now get filters pre selected automatically

Previously, the user had to manually apply filters to the selected WAL or DB devices, which was a repetitive task.

A new shortcut button for silencing alerts is added

With this release, users can create a silence for every alert in the notification bar on the Ceph Dashboard using the newly created silent shortcut.

Users can now add server side encryption to the Ceph Object Gateway bucket from the Dashboard

Previously, there was no option on the Ceph Dashboard to add server side encryption (SSE) to the Ceph Object Gateway buckets.

Cross origin resource sharing is now allowed

Previously, IBM developers were facing issues with their storage insights product when they tried to ping the REST API using their front-end because of the tight cross origin resource sharing (CORS) policies setup in the REST API.

Example

[ceph: root@host01 /]# ceph config set mgr mgr/dashboard/cross_origin_url http://localhost:4200

Users can now set and manage quotas on subvolume group

Previously, the user could only apply quotas to individual subvolumes.

The Ceph File System client can now track average read, write, and metadata latencies

Previously, the Ceph File System client would track only the cumulative read, write, and metadata latencies. However, average read, write, and metadata latencies are more useful to the user.

The cephfs-top utility is improved with the support of multiple file systems

Previously, the cephfs-top utility with multiple file systems was not reliable. Moreover, there was no option to display the metrics for only the selected file system.

Users can now use the fs volume info command to display basic details about a volume

Previously, there was no command in Ceph File System to list only the basic details about a volume.

Users can list the in-progress or pending clones for a subvolume snapshot

Previously, there was no way of knowing the set of clone operations in-progress or pending for a subvolume snapshot, unless the user knew the clone subvolume’s name and used clone status command to infer the details.

Users can now use the --human-readable flag with the fs volume info command

Previously, all the sizes were displayed only in bytes on running the fs volume info command.

New S3 bucket lifecycle notifications are now generated

With this release, S3 bucket notifications are generated for current and non-current versions, delete-marker expiration generated by lifecycle processing. This capability is potentially useful for application workflow, among other potential uses.

The objects are transitioned to the S3 cloud endpoint as per the set lifecycle rules

In Red Hat Ceph Storage, we use a special storage class of tier type cloud-s3 to configure the remote cloud S3 object store service to which the data needs to be transitioned. These are defined in terms of zonegroup placement targets and unlike regular storage classes, do not need a data pool.

The Ceph Object Gateway S3 policy errors are now more useful

Previously, Ceph Object Gateway S3 policy error messages were opaque and not very useful. The initial issue with not being able to access data in the buckets after upgrading versions seemed to be the result of an accepted but invalid principal being ignored silently on ingest but rejected on use later due to a code change.

Level 20 Ceph Object Gateway log messages are reduced when updating bucket indices

With this release the Ceph Object Gateway level 20 log messages are reduced when updating bucket indices to remove messages that do not add value and to reduce size of logs.

Lifecycle policy now runs on all zones in multi-site configurations

With this release, lifecycle policy runs on all zones in multi-site Red Hat Ceph Storage configurations, which makes lifecycle processing more resilient in these configurations. The changes are made to also permit new features, such as conditional processing in archive zones.

Multi-site configuration supports dynamic bucket index resharding

Previously, only manual resharding of the buckets for multi-site configurations was supported.

Sites can now customize STS max-session-duration parameter with the role interface

Previously, the max-session-duration parameter controlling duration of STS sessions could not be configured because it was not exposed on the interface.

Kafka SASL/SCRAM security mechanism is added to bucket notifications

With this release, Kafka SASL/SCRAM security mechanism is added to bucket notifications.

Low-level log messages are introduced to warn user about hitting throttle limits

Previously, there was a lack of low-level logging indication that throttle limits were hit, causing these occurrences to incorrectly have the appearance of a networking issue.

The user is warned on Filestore deprecation through ceph status and ceph health detail commands

BlueStore is the default and widely used objectstore.

User can now take advantage of a tunable KernelDevice buffer in BlueStore

With this release, users can now configure custom alignment for read buffers using bdev_read_buffer_alignment command in BlueStore. This removes the limitations imposed by the default 4 KiB alignment space, when buffers are intended to be backed up by huge pages.

OSDs report the slow operation details in an aggregated format to the Ceph cluster log

Previously, slow requests would overwhelm a cluster log with too many details, filling up the monitor database.

Users can now blocklist a CIDR range

With this release, you can blocklist a CIDR range, in addition to individual client instances and IPs. In certain circumstances, you would want to blocklist all clients in an entire data center or rack instead of specifying individual clients to blocklist.

librbd SSD-based persistent write-back cache to reduce latency now fully supported.

With this release, the pwl_cache librbd plugin provides a log-structured write-back cache targeted at SSD devices. Just as with the already provided log-structured write-back cache targeted at PMEM devices, the updates to the image are batched and flushed in-order, retaining the actual image in a crash-consistent state. The benefits and use cases remain the same, but users no longer need to procure more expensive PMEM devices to take advantage of them.

Layered client-side encryption is now supported

With this release, cloned images can be encrypted, each with its own encryption format and passphrase, potentially different from that of the parent image. The efficient copy-on-write semantics used for unformatted regular cloned images are retained.

The PID limit is removed and workloads in the container no longer crash

Previously, in Red Hat Enterprise Linux 9 deployments, pid limits were enforced which limited the number of processes able to run inside the container. Due to this, certain operations, such as Ceph Object Gateway sync, would crash.

Cephadm no longer randomly temporarily removes config and keyring files

Previously, due to incorrect timing on when to calculate the client conf and keyrings, cephadm would calculate that there should be no config and keyrings placed on any host and would subsequently remove all of them.

The Ceph Object Gateway daemons now bind to loopback addresses correctly

Previously, cephadm excluded loopback interfaces when looking for a valid IP address on a host to bind the Ceph Object Gateway daemon, thereby the daemons would not bind to the loopback addresses.

Cephadm now splits the device information into multiple entries after exceeding the Ceph monitor store size limit

Previously, cephadm was unable to refresh the hosts and complete most operations when the device information exceeded the monitor store default maximum size limit of 64K. This caused an entry size error. As a result, users had to raise the default limit if they had hosts with a large number of disks.

Crash daemon now correctly records crash events and reports them to the storage cluster

Previously, crash daemon would not authenticate properly when sending crash reports to the storage cluster, which caused it to not correctly record crash events to send to the cluster.

Log rotation of the cephadm.log should no longer cause issues

Previously, the logrotate command would cause issues if the /var/log/ceph directory was created by something other than cephadm, for example ceph-common or ceph-ansible. As a consequence, the cephadm.log could not be rotated.

Cephadm logging configurations are updated.

Previously, cephadm scripts were logging all output to stderr. As a result, cephadm bootstrap logs signifying successful deployment were also being sent to stderr instead of stdout.

The network check no longer causes the hosts to be excluded from the monitor network

Previously, the network check would fail because cephadm would look for the exact match between the host network and some of the configured public networks. This caused the hosts with valid network configuration, which are the hosts with an interface that belonged to public_network, to be excluded from the monitor network.

cephadm no longer removes osd_memory_target config settings at host level

Previously, if osd_memory_target_autotune was turned off globally, cephadm would remove the values that the user set for osd_memory_target at the host level. Additionally, for hosts with FQDN name, even though the crush map uses a short name, cephadm would still set the configuration option using the FQDN. Due to this, users could not manually set osd_memory_target at the host level and osd_memory_target auto tuning would not work with FQDN hosts.

cephadm rewrites Ceph OSD configuration files

Previously, while redeploying OSDs, cephadm would not write configuration used for Ceph OSDs, thereby the OSDs would not get the updated monitor configuration in its configuration file when the Ceph Monitor daemons were either added or removed.

Users can now drain hosts that are listed in explicit placements

Previously, draining hosts that were listed as part of an explicit placement would cause the hosts not to be properly drained and tracebacks would be logged until the drain was stopped or hosts were removed from any explicit placement .

cephadm returns non-zero code when --apply-spec option fails during bootstrap

Previously, cephadm bootstrap always returned code 0 if the operation was complete. If there were any failures in the deployment using the --apply-spec option, it would not reflect any failures in the return code.

Complex OSD deployment or replacement with shared DB devices now does not need to be done all at once

Previously, devices already used as db devices for previous OSDs were filtered out as unavailable devices, when cephadm created OSDs. As a result, complex OSD deployment where all the OSDs that were meant to use a device as their DB, but were not deployed at once, would not work as the DB device would be filtered out when creating subsequent OSDs even though they should not be according to the OSD specification.

Proper errors are raised if invalid tuned-profile specification is detected by cephadm

Previously, cephadm would not validate YAML specification for tuned-profile and consequently would not return any error or warning while applying invalid and missing data in invalid tuned-profile specification.

The host device Life Expectancy column now shows the correct value on the Ceph Dashboard

Previously, the host device Life Expectancy column would show an empty value because the column had no default value.

Users are now able to change the Ceph Object Gateway subuser permissions

Previously, the users could not change the Ceph Object Gateway subuser permissions as the request was not implemented properly.

The overall performance graphs of the pools show correct values on the Ceph Dashboard

Previously, there was an issue in the query related to the Ceph Dashboard pools’ overall performance graphs and showed multiple entries of the same pool in the pool overview.

The Ceph Dashboard is now aligned with the command line interface’s (CLI) way of NFS exports creation

Previously, the squash field for the NFS exports creation was shown as a mandatory field in the edit form. Additionally, if exports were created from the backend and specified a different kind of squash name, the form would return an empty field.

Pool count shows correct values on the Ceph Dashboard

Previously, there was an issue in the query related to pool count. In the Overall performance tab, the Ceph Dashboard showed pool count as fractional value.

Validation is required when creating a new service name

Previously, there was no validation when creating a new service on the Ceph Dashboard, as a result, users were allowed to create a new service with an existing name. This would overwrite existing services and cause the user to lose a current running service on the hosts.

External snapshot creation in Grafana now disabled by default

Previously, creating external Grafana snapshots would generate broken links. This would make the infrastructure vulnerable to DDoS attacks ,as someone could gain insights into the environment by looking at the metric patterns.

The services can now be safely put to the unmanaged mode on the Ceph Dashboard

Previously, when a user tried to create or modify the services, such as ingress or SNMP, in the unmanaged mode, the form would return a 500 error message and would fail to create the services. This happened because the form would not show some fields that needed to be filled even if the service was going directly to unmanaged.

The Ceph Dashboard now securely connects with the hostname instead of IP address

Previously, an attempt to access the Object Gateway section of the Ceph Dashboard would throw a 500 - internal server error. This error was as a result of the Ceph Dashboard trying to establish the HTTPS connection to the Ceph Object Gateway daemons by IP address instead of hostname, which requires that the hostname of the server matches the hostname in the TLS certificate.

ceph-dashboard now prompts users when creating an ingress service

Previously, ingress service creation would fail with a 500 internal server error, when the form was submitted without specifying frontend and monitor port values. As a result, an ingress service could not be created from the Ceph Dashboard.

The service instance column of the host table on the Ceph Dashboard now shows all the services deployed on the particular host

Previously, the service instance column of Cluster → Hosts table only showed ceph services and not cephadm services as the frontend was lacking subscription to some of the services.

The Ceph Dashboard now raises appropriate error message, if user tries to create a snapshot with an existing name

Previously, Ceph Dashboard would not validate Ceph File System snapshot creation with an existing name and would throw a 500 - internal server error.

Ceph node "network packet" drop alerts are shown appropriately on the dashboard

Previously, there was an issue in the query related to Ceph node "network packet" drop alerts. As a consequence, those alerts would be seen frequently on the Ceph Dashboard.

MDS daemon now resets the heartbeat in each thread after each queued work

Previously, a thread would hold the mds_lock for a longtime if it had a lot of work to do. This caused other threads to be starved of resources and be stuck for a longtime, as a result MDS daemon would fail to report the heartbeat to monitor in time and be kicked out of the cluster.

Ceph Metadata Server no longer crashes during concurrent lookup and unlink operations

Previously, an incorrect assumption of an assertion placed in the code, would hit the concurrent lookup and unlink operations from a Ceph client, causing Ceph Metadata Server crash.

A replica MDS is no longer stuck, if a client sends a getattr client request just after it was created

Previously, if a client sent a getattr client request just after the replica MDS was created, the client would make a path of #INODE-NUMBER because the CInode was not linked yet. The replica MDS would keep retrying until the auth MDS flushed the mdlog and the C_MDS_openc_finish and link_primary_inode were called 5 seconds later at most.

Ceph File System subvolume groups created by the user are now displayed when listing subvolume groups

Previously, the Ceph File System (CephFS) subvolume groups listing included CephFS internal groups instead of CephFS subvolume groups created by users.

Saved snap-schedules are reloaded from Ceph storage

Previously, restarting Ceph Managers caused retention policy specifications to get lost because it was not saved to the Ceph storage. As a consequence, retention would stop working.

The API for deleting RADOS objects is updated

Previously, deleting a RADOS object would result in the program crash and would create tracebacks in the logs.

MDS now stores all damaged dentries

Previously, metadata servers (MDS) would only store dentry damage for a dirfrag if dentry damage would not already exist in that dirfrag. As a result, only the first damaged dentry would be stored in the damage table and subsequent damage in the dirfrag would be forgotten.

The ceph-mds daemon no longer crashes during the upgrade

Previously, the Ceph Metadata Server daemons (ceph-mds) would crash during an upgrade due to an incorrect assumption in the Metadata Servers when recovering inodes. It caused ceph-mds to hit an assert during an upgrade.

The standby-replay Metadata Server daemon is no longer unexpectedly removed

Previously, the Ceph Monitor would remove a standby-replay Metadata Server (MDS) daemon from the MDS map under certain conditions. This would cause the standby-replay MDS daemon to get removed from the Metadata Server cluster, which generated cluster warnings.

The subvolume snapshot info command no longer has the size field in the output

Previously, the output of the subvolume snapshot command would return an incorrect snapshot size. This was due to the fact that the snapshot info command relies on rstats to track the snapshot size. The rstats tracks the size of the snapshot from its corresponding subvolume instead of the snapshot itself.

The disk full scenario does not corrupt the configuration file anymore

Previously, the configuration files were being written directly to the disk without using the temporary files, which involved truncating the existing configuration file and writing the configuration data. This led to the empty configuration files when the disk was full as the truncate was successful, however writing new configuration data failed with no space error. Additionally, it led to the failure of all the operations on corresponding subvolumes.

Do not abort MDS in case of unknown messages

Previously, metadata servers (MDS) would abort if it received a message that it did not understand. As a result, any malicious client would crash the server by just sending a message of a new type to the server. Beside malicious clients, this also meant that whenever there was a protocol issue, such as a new client erroneously sending new messages to the server, the whole system would crash instead of just the new client.

Directory listing from a NFS client now works as expected for NFS-Ganesha exports

Previously, Ceph File System (CephFS) Metadata Server (MDS) would not increment the change attribute, (change_attr) of a directory inode during CephFS operations which only changed the directory inode’s ctime. Therefore, an NFS kernel client would not invalidate its readdir cache when it is supposed to. This is because the NFS Ganesha server backed by CephFS would sometimes report incorrect change attribute value of the directory inode. As a result, the NFS client would list stale directory contents for NFS Ganesha exports backed by CephFS.

The CephFS now has the correct directory access

Previously, directory access was denied even to the UID of 0 due to incorrect Discretionary Access Control (DAC) management.

The ceph-volume inventory command no longer fails

Previously, when a physical volume was not a member of any volume group, ceph-volume would not ignore the volume, and instead would try to process it, which caused the ceph-volume inventory command to fail.

Users can now use MD5 for non-cryptographic purposes in a FIPS environment

Previously, in a FIPS enabled environment, the usage of MD5 digest was not allowed by default, unless explicitly excluded for non-cryptographic purposes. Due to this, a segfault occurred during the S3 complete multipart upload operation.

The Ceph Object Gateway no longer crashes on accesses

Previously, the Ceph Object Gateway would crash on some access due to the changes from in-place to allocated buckets as a malformed bucket URL caused a void pointer dereference to a bucket value that was not always initialized.

The code that parses dates z-amz-date format is changed

Previously, the standard format for x-amz-date was changed which caused issues, since the new software uses the new date format. The new software built with the latest go libraries would not talk to the Ceph Object Gateway.

Ceph Object Gateway’s Swift implicit tenant behavior is restored

Previously, a change to Swift tenant parsing caused the failure of Ceph Object Gateway’s Swift implicit tenant processing.

The Ceph Object Gateway no longer crashes after running continuously for an extended period

Previously, an index into a table would become negative after running continuously for an extended period, resulting in the Ceph Object Gateway crash.

Variable access no longer causes undefined program behavior

Previously, a coverity scan would identify two cases, where variables could be used after a move, potentially causing an undefined program behavior to occur.

Roles and role policy now transparently replicate when multi-site is configured

Previously, the logic to replicate S3 roles and role policy was not implemented in Ceph Object Gateway, thereby, roles and role policy created in any zone in a multi-site replicated setup were not transparently replicated to other zones.

Slow progress and high CPU utilization during backfill is resolved

Previously, the worker thread with the smallest index in an OSD shard would return to the main worker loop, instead of waiting until an item could be scheduled from the mClock queue or until notified. This resulted in the busy loop and high CPU utilization.

Renaming large objects no longer fails when using temporary credentials returned by STS

Previously, due to incorrect permission evaluation of iam policies while renaming large objects, renaming large objects would fail when temporary credentials returned by STS were used.

The small writes are deferred

Previously, Ceph would defer writes while allocating units. When the allocation unit was large, like 64 K, no small write was eligible for deferring.

The Ceph Monitor no longer crashes after reducing the number of monitors

Previously, when the user reduced the number of monitors in the quorum using the ceph orch apply mon NUMBER command, cephadm would remove the monitor before shutting it down. This would trigger an assertion because Ceph would assume that the monitor is shutting down before the monitor removal.

The Ceph Manager checks that deals with the initial service map is now relaxed

Previously, when upgrading a cluster, the Ceph Manager would receive several service_map versions from the previously active Ceph manager. This caused the manager daemon to crash, due to an incorrect check in the code, when the newly activated manager received a map with a higher version sent by the previously active manager.

The ceph --help command now shows that yaml formatters are only valid for ceph orch commands

Previously, it was implied by lack of specification in the ceph --help command that the yaml formatter option was valid for any ceph command, including the ceph config dump command.

Corrupted dups entries of a PG Log can be removed by off-line and on-line trimming

Previously, trimming of PG log dups entries could be prevented during the low-level PG split operation, which is used by the PG autoscaler with far higher frequency than by a human operator. Stalling the trimming of dups resulted in significant memory growth of PG log, leading to OSD crashes as it ran out of memory. Restarting an OSD would not solve the problem as the PG log is stored on disk and reloaded to RAM on startup.

The starts message is added to notify that scrub or deep-scrub process has begun

Previously, users were not able to determine when the scrubbing process started for a placement group (PG) because the starts message was missing from the cluster log. This made it difficult to calculate the time taken to scrub or deep-scrub a PG.

The autoscale-status command no longer displays the NEW PG_NUM value if PG-autoscaling is disabled

Previously, the autoscale-status command would show the NEW PG_NUM value even though PG-autoscaling was not enabled. This would mislead the end user by suggesting that PG autoscaler applied NEW PG_NUM value to the pool, which was not the case.

Users can remove cloned objects after upgrading a cluster

Previously, after upgrading a cluster from Red Hat Ceph Storage 4 to Red Hat Ceph Storage 5 , removing snapshots of objects created in earlier versions would leave clones, which could not be removed. This was because the SnapMapper key’s were wrongly converted.

The ceph daemon heap stats command now returns required usage details for daemon

Previously, the ceph daemon osd.x heap stats command would return empty output instead of the current heap usage for a Ceph daemon. Consequently, users were compelled to use the ceph tell heap stats command to get the desired heap usage.

The Prometheus metrics now reflect the correct Ceph version for all Ceph Monitors whenever requested

Previously, the Prometheus metrics reported mismatched Ceph versions for Ceph Monitors when the monitor was upgraded. As a result, the active Ceph Manager daemon needed to be restarted to resolve this inconsistency. Additionally, the Ceph manager used to update the monitor metadata through the handle_mon_map parameter, which gets triggered when the monitors are removed or added from the cluster or the active mgr is restarted or mgr failsover.

The correct set of replicas are used for remapped placement groups

Previously, for remapped placement groups, the wrong set of replicas would be queried for the scrub information causing a failure of the scrub process, after identifying mismatches that would not exist.

The targeted rank_removed no longer gets stuck in live_pinging and dead_pinging states

Previously, in some cases, the paxos_size of the Monitor Map would get updated before the rank of the monitor was changed. For example, paxos_size would get reduced from 5 to 4, but the highest rank of the Monitors was still 4, thus the old code would skip deleting the rank from dead_pinging state. This would cause the targeted rank to remain in dead_pinging forever, which would then cause strange peer_tracker scores in election strategy: 3.

Ceph Monitors are not stuck during failover of a site

Previously, the removed_ranks variable would not discard its content for every update of the Monitor map. Thus it would replace monitors in a 2-site stretch cluster and fail over of one of the site would cause connection scores, including ranks associated with the scores, to be inconsistent.

Users are now able to set the replica size to 1

Previously, users were unable to set the pool size to 1. The check_pg_num() function would incorrectly calculate the projected placement group number of the pool, which resulted in an underflow. Because of the false result, it appeared that the pg_num was larger than the maximum limit.

Ceph cluster issues a health warning if the require-osd-release flag is not set to the appropriate release after a cluster upgrade.

Previously, the logic in the code that detects the require-osd-release flag mismatch after an upgrade was inadvertently removed during a code refactoring effort. Since the warning was not raised in the ceph -s output post an upgrade, any change made to the cluster without setting the flag to the appropriate release resulted in issues, such as, placement groups (PG) stuck in certain states, excessive Ceph process memory consumption, slow requests, among many other issues.

The NFS-Ganesha package is based on 4.0.8 version

With this release, the nfs-ganesha package is now based on the upstream version 4.0.8, which provides a number of bug fixes and enhancements from the previous version.

The Cephadm fails the upgrade of the Red Hat Ceph Storage version if the OS is unsupported

Currently, Cephadm does not manage the host operating system (OS), therefore, during an upgrade, it does not verify if the Red Hat Ceph Storage version the user is upgrading to is supported on the OS of the Ceph cluster nodes.

Resharding a bucket removes the bucket’s metadata

Resharding a bucket removes the bucket’s metadata if the bucket was created with bucket_index_max_shards as 0. You can recover the affected buckets by restoring the bucket index.

The Red Hat Ceph StorageDashboard shows NaN undefined in some fields in the host table

Currently, when a new host is added, it takes some time to load its daemons, devices and other stats. During this time delay, data may not be available for some fields in the host table, as a result, during expansion, the host adds NAN undefined for those fields.

”Throughput-optimized” option is recommended for clusters containing SSD and NVMe devices

Previously, whenever the cluster had either only SSD devices or both SSDs and NVMe devices, the “Throughput-optimized” option would be recommended, even though it should not be and it had no impact either on the user or the cluster.

Multi-site replication may stop during upgrade

Multi-site replication may stop if clusters are on different versions during the process of an upgrade. We would need to suspend sync until both clusters are upgraded to the same version.

The mclock_scheduler has performance issues with small object workloads and OSDs created on HDD devices

The mclock_scheduler has performance issues with small object workloads and with OSDs created on HDD devices. Due to this, with small object workloads, client throughput is impacted due to on-going recovery operations.

The Ceph OSD benchmark test might get skipped

Currently, the Ceph OSD benchmark test boot-up might sometimes not run even with the osd_mclock_force_run_benchmark_on_init parameter set to true. As a consequence, the osd_mclock_max_capacity_iops_[hdd,ssd] parameter value is not overridden with the default values.

RBD iSCSI gateway support is now retired.

With this release onwards, Red Hat Ceph Storage will no longer ship with iSCSI gateway components. RBD iSCSI gateway has been dropped in favor of the future RBD NVMEoF gateway.