Chapter 6. Bug fixes
This section describes bugs with significant impact on users that were fixed in this release of Red Hat Ceph Storage. In addition, the section includes descriptions of fixed known issues found in previous versions.
6.1. The Ceph Ansible utility
The /targets
URL is showing Prometheus as down
Previously, the Prometheus target URL was configured with the localhost
value. The localhost
value was causing the Prometheus service not to listen on the localhost
address when the target status was considered down. With this release the Prometheus configuration file was updated to use the IP address for the target URL value. As a result, the Prometheus target status is reporting correctly.
ceph-volume
might cause metadata corruption when creating OSDs during a Ceph deployment
Previously, when ceph-volume
issued LVM commands like creating volume groups, logical volumes, setting tags, etc it could result in corrupted metadata when creating OSDs during a Ceph deployment. With this release, users can conditionally disable the lvmetad
service by setting the lvmetad_disabled
parameter to true
in the group_vars/all.yml
file on the host thereby the metadata corruption is avoided.
The role ceph-dashboard
in ceph-ansible
enforces the common name of the self-signed certificate to ceph-dashboard
Previously, when using the self-signed certificates generated by ceph-ansible
, it enforced the common name (CN) to ceph-dashboard
thereby causing applications like Prometheus to error out due to the mismatch in the hostname of the node sending certificate to the clients.
With this release, ceph-ansible
sets the CN with proper values and Prometheus works as expected.
Rolling upgrade fails when Ceph containers are collocated
The rolling_update.yml
Ansible playbook fails when the Ceph Monitor and Ceph Object Gateway daemons are collocated with containers, and when the multi-site Ceph Object Gateway is enabled. This failure was caused by the radosgw-admin
commands not able to execute because of the Ceph Monitor container is stopped during the upgrade process. With this release, the multi-site Ceph Object Gateway code within the ceph-handler
role is skipped during the upgrade process. As a result, the rolling_update.yml
Ansible playbook runs successfully.
Ceph Monitor quorum check fails when switching to a containerized daemons
A regression bug was introduced in the switch-from-non-containerized-to-containerized-ceph-daemons.yml
Ansible playbook. This regression bug caused the Ceph Monitor quorum check to fail because the current node’s host name was not tested. With this release, the ansible_hostname
fact from the current Ceph Monitor node is used correctly. As a result, the Ceph Monitor quorum check is successful.
Adding a new Ceph Object Gateway instance when upgrading fails
The radosgw_frontend_port
option did not consider more than one Ceph Object Gateway instance, and configured port 8080
to all instances. With this release, the radosgw_frontend_port
option is increased for each Ceph Object Gateway instance, allowing you to use more than one Ceph Object Gateway instance.
Ceph Ansible removes any socket file and enables redeployment of cluster
Previously, *.asok files were left on completion of purge playbook causing failure during a redeployment of a cluster. With this update, ceph-ansible
removes any socket file that may be present and the clusters can be redeployed safely.
Added support for log rotation for the tcmu-runner process in containerized Red Hat Ceph Storage deployments
Previously, when Red Hat Ceph Storage with iSCSI was deployed in containers, there was no log rotation for the tcmu-runner process thereby consuming all the space in the containers. With this release, log rotation support is added to the tcmu-runner process and the log files are periodically rotated resulting in lesser space consumption.
The FileStore to BlueStore migration process can fail for OSD nodes that have a mix of FileStore OSDs and BlueStore OSDs
Previously, if deployments running Red Hat Ceph Storage versions earlier than 3.2 never had osd_objectstore
explicitly set in either group_vars
, host_vars
, or inventory
, the deployment had FileStore OSDs. FileStore was the default prior to Red Hat Ceph Storage 3.2.
After upgrading the deployed storage cluster to Red Hat Ceph Storage 3.2, new OSDs added to an existing OSD node would use the BlueStore backend because it became the new default. This resulted in a mix of FileStore and BlueStore OSDs on the same node. In some specific cases, a FileStore OSD might share a journal or DB device with a BlueStore OSD. In such cases, redeploying all the OSDs causes ceph-volume
errors, either because partitions cannot be passed in lvm batch
or because of the GPT header.
With this release, there are two options for migrating OSDs with a mix of FileStore and BlueStore configurations:
-
Set the extra variable
force_filestore_to_bluestore
totrue
when running thefilestore-to-bluestore.yml
playbook. This setting forces the playbook to automatically migrate all OSDs, even those that already use BlueStore. -
Run the
filestore-to-bluestore.yml
playbook without settingforce_filestore_to_bluestore
(the default isfalse
). This causes the playbook to automatically skip the migration on nodes where there is a mix of FileStore and BlueStore OSDs. It will migrate the nodes that have only FileStore OSDs. At the end of the playbook execution, a report displays to show which nodes were skipped.
Before upgrading from Red Hat Ceph Storage 3 to 4, manually examine each node that has been skipped in order to determine the best method for migrating the OSDs.
Special characters can be set in the Docker registry password
Previously, special characters set in the Docker registry password were not handled correctly. With this release, the Ansible playbook does not fail when special characters are set in the Docker registry password. Special characters can now be used in the Docker registry password and the Ansible playbook works as expected
ceph-volume
Ansible module reports correct information on logical volumes and volume groups
Previously, when applying the command ceph-volume lvm zap --destroy
command on a Red Hat Enterprise Linux 7 host with Red Hat Enterprise Linux 8 based containers on an OSD, the lvm cache was not refreshed for the host and still reported the logical volumes and volume groups that are present. With this release,ceph_volume
Ansible module triggers a command on the host to ensure the lvm cache is refreshed and reports correct information on logical volumes and volume groups.
Ceph container logs can be viewed by using the journalctl
command
Previously, Ceph container logs were not present in journald as Podman used Kubernetes file as the default log driver when running containers in detached mode and systemd type forking. With this release, the Ceph containers are configured with journald log driver and the logs are available using the journalctl
command.
Ceph Ansible sets file and directory ownership values to nobody:nobody
Previously, Ceph Ansible set the file and directory ownership values to root:root
. This caused permissions issues for alertmanager
and prometheus
services.
With this release, Ceph Ansible sets ownership to nobody:nobody
. This eliminates the permissions issues.
6.2. The Cockpit Ceph installer
Ceph installation through the Cockpit no longer fails
Previously, when a user provides localhost
as the name of the host in the host selection page, the Ceph installation through the Cockpit failed with an error Systemd must be present
. With this release, the host page UI will display a proper error message and deny the user from using localhost
as a hostname to continue.
Cockpit installer produces incorrect values for number of buckets in rgws.yml
Previously, using the Cockpit installer to generate the rgws.yml
file produced incorrect values for defaults.rgw.buckets.data:pgnum
and rgw_override_bucket_index_max_shards
.
With this release, the Cockpit installer creates correct values in rgws.yml
.
6.3. Ceph Management Dashboard
The prometheus query is fixed to report real-time metrics on CPU usage on the dashboard
Previously, the hosts screen on the Red Hat Ceph Storage dashboard displayed incorrect data about the CPU usage for the nodes in the cluster due to an inaccurate prometheus query. With this release, the prometheus query was fixed and the data on CPU usage gives near real-time metrics.
Removed options which expect future data from Grafana dashboards embedded in Red Hat Ceph Storage dashboard
Previously, when the option ”this week” was selected from “Historical Data” in the overall performance graph, the metrics were not displayed as few historical data options in the Grafana metrics expected future data. With this release,the historical data options that expect future data have been removed and the metrics are displayed as expected.
Cherrypy no longer reveals its versions in header and error pages
Previously, cherrypy revealed its version in header and error pages. Exposing this information caused a potential security vulnerability. With this release, the headers for both dashboard and Prometheus servers now show Ceph-Dashboard and Ceph-Prometheus instead of the cherrypy version. This change eliminates the security vulnerability.
6.4. Ceph File System
Ceph fs status command no longer fails with AttributeError` exception
Previously, Ceph fs status
command failed with AttributeError
exception, caused by incorrect handling of metadata during the rejoin. With this release, Ceph fs status
command returns expected status as unknown objects are now accepted if NoneType is the metadata object type.
6.5. Ceph Manager plugins
The alerts module of Ceph manager works as expected with no health warnings
Previously, Ceph manager would override some module specific config options with default values resulting in the failure of the module with ALERTS_SMTP_ERROR unable to send alert email
as it continued to use the default value smtp_ssl=true
. With this release, the default value handling of the Ceph manager is fixed and the alerts module works as expected with no health warnings.
The restful module API endpoints are now accessible
Previously, the restful module used dict.iteritems
which is no longer available in python 3. As a result a number of restful module API endpoints were not accessible. With this release, the restful module has been updated to use dic.items
and the API endpoints are accessible.
IPv6 addresses were not handled properly in ceph-mgr Prometheus exporter
Previously, some metadata metrics in the ceph-mgr Prometheus exporter output showed incomplete or malformed IPv6 addresses. With this release, the exporter now handles IPv6 addresses correctly.
6.6. The Ceph Volume utility
Executing the ceph-volume
command sent debugging output to /var/log/ceph/ceph-volume.log
Previously, executing the ceph-volume
command always sent debugging-level output to /var/log/ceph/ceph-volume.log
regardless of the level set in the --log-level
option. With this release, executing the ceph-volume
command sends output at the level that the --log-level
option specifies.
ceph volume lvm batch
reports SSD devices correctly and deploys correct configuration
Previously, ceph-volume lvm batch
command caused a race condition with udev
resulting in SSD devices incorrectly reported as HDD, causing unexpected deployment configuration. With the code update in this release, ceph-volume lvm batch
command reports SSDs correctly and deploys the expected configuration of the cluster.
Stack update fails after adding new SSD OSDs to the storage cluster
Previously, using stack update to add new SSD OSDs to the same volume group (VG) in the storage cluster caused stack update to fail. This occurred because stack update incorrectly viewed the new SSDs as belonging to different logical volumes (LVs), instead of belonging to the same LV. With this release, stack update views newly added OSDs as belonging to the same LV, and it no longer fails.
(BZ#1892441)
6.7. Ceph Object Gateway
Set-lifecycle and delete-lifecycle actions against upgraded OSDs now work as expected
Previously, during an upgrade from Red Hat Ceph Storage 3 to Red Hat Ceph Storage 4.2z2, installing a legacy lifecycle policy against upgraded OSDs yielded a structure decoding error although the set-lifecycle action would succeed. With this release, the changes required to decode bucket lifecycle status entries are fixed and the upgrade of daemons works as expected.
The --reset-stats
option updates buckets in groups for users with large numbers of buckets
Previously, the radosgw-admin
user --reset-stats
option simultaneously updated the stats for all buckets owned by a user. For users with very large numbers of buckets, the time required to make the updates could exceed the length of the associated RADOS operation. This could cause Ceph to mark OSDs as down, and could cause the OSDs to flap.
With this release, the --reset-stats
option updates the stats in groups of 1000 buckets. This allows large numbers of buckets to update without resulting in OSD flapping.
gc perf counters increments when gc entries are purged from the system
Previously, gc perf counters did not increment when gc entries were purged from the system. With this code update, the correct value of gc perf counter is observed in accordance with the number of gc entries that have been deleted from the system.
Listing of entries in the last GC object does not enter a loop
Previously, the listing of entries in the last GC object entered a loop because the marker was reset every time for the last GC object. With this release, the truncated flag is updated which does not cause the marker to be reset and the listing works as expected.
The Ceph Object Gateway syncs bucket cache information during the bucket creation process
Previously, the Ceph Object Gateway did not sync the cache of the bucket information upon bucket creation. This led to the condition when a user tried to access a bucket that did not exist from one RGW and then created that bucket from another Ceph Object Gateway, accessing the bucket from the first Ceph Object Gateway caused a 404 error, stating that the bucket did not exist, although it did. With this update, RGW syncs the cache during the bucket creation process, so that each Ceph Object Gateway can access the bucket.
KafkaConnect sends objects from a Kafka topic to the RGW S3 bucket
Previously, sending objects from a Kafka topic to the RGW S3 bucket failed because the chunked-encoding object signature was not calculated correctly.
This produced the following error in the RADOS Gateway log:
20 AWSv4ComplMulti: ERROR: chunk signature mismatch
With this release, the chunked-encoding object signature is calculated correctly, allowing KafkaConnect to send objects successfully.
6.8. Multi-site Ceph Object Gateway
Bucket index do not collect entries after syncing on a bucket has been disabled
Previously, the use of radosgw-admin bucket check --fix …
variable on a bucket in which multi-site syncing has been disabled, would set an incorrect flag indicating syncing has not been disabled. Data would be added to bucket index logs that would not be used or trimmed, thereby consuming more storage over time. With this release, the syncing flag is now copied correctly when running the radosgw-admin bucket check --fix …
command. Bucket index logs do not collect entries after syncing on a bucket is disabled.
6.9. RADOS
The Progress module is no longer stuck for an indefinite time
Previously, the progress vents in Ceph status were stuck for an indefinite time. This was caused by the Progress module checking the PG state early and not syncing with the epoch of the OSDMap. With this release, progress events now pop up as expected.
Progress module causing Ceph Monitor crashes
During backfill and recovery operations, the progress module can generate negative progress events. With large storage clusters, too many negative progress events can lead to large memory allocation on the Ceph Monitor nodes, causing cascading Ceph Monitor crashes. With this release, the code ensures that progress events are not negative. As a result, the progress module does not cause the Ceph Monitors to crash.
Ceph Monitors now show caution when handling forwarded OSD failure reports
Previously, Ceph Monitors would incorrectly report slow operations to administrators and logging systems. With this release, Ceph Monitors now handle forwarded OSD failure reports with caution, so many fewer inaccurate slow operations will occur.
Monitor trims osdmaps appropriately
Previously, the monitor failed to trim stale osdmaps although they were used by “out” OSDs because the monitor considered both “in“ and “out” OSDs when trimming osdmaps. With this release, the “out” OSDs are not considered and the osdsmaps are trimmed appropriately.
BlueStore and FileStore OSDs list objects in the same order in a mixed cluster
Previously, in a cluster with both BlueStore and FileStore OSDs, deep scrub and backfill could report missing objects due to an inconsistency of the sorting mechanism in the backend. With this upgrade, a feature flag OSD_FIXED_COLLECTION_LIST
has been added to ensure that the collection_list
method in BlueStore lists objects in the same order as FileStore.
Log files were created with incorrect permissions
Previously, a code addition changed the order in which relevant functions were called. This caused some daemons to create log files with incorrect permissions. With this release, functions are called in the correct order, and the daemons create log files with the correct permissions.
Enabling bluefs_buffered_io
prevents performance degradation
Previously, the option bluefs_buffered_io
was disabled which led to slow operations of RocksDB and OMAP interactions in certain scenarios.With this release the option bluefs_buffered_io
is set to True
, thereby preventing performance degradation.
Memory growth because of onodes is now controlled
Previously, the default value for the option bluestore_cache_trim_max_skip_pinned
was 64 which was very low for large clusters. Since this option controlled the rate of trimming onodes and with current default value, it could have led to a buildup of onodes causing memory growth. With this release, the default value of bluestore_cache_trim_max_skip_pinned` is 1000 and the memory growth is controlled.
6.10. RADOS Block Devices (RBD)
Applications using librbd works when client-side QoS throttling is enabled
Previously, when client-side QoS throttling was enabled in librbd, it could crash because the data path was not properly protected with a lock. With this release, the missing lock is added and the applications using librbd for IO works as expected when client-side QoS throttling is enabled.
6.11. NFS Ganesha
All file layouts in READDIR return results
Previously, some file layouts caused a loop in READDIR and never returned results. With this update, READDIR works as expected and returns the results correctly.