Chapter 6. Bug fixes

The /targets URL is showing Prometheus as down

Previously, the Prometheus target URL was configured with the localhost value. The localhost value was causing the Prometheus service not to listen on the localhost address when the target status was considered down. With this release the Prometheus configuration file was updated to use the IP address for the target URL value. As a result, the Prometheus target status is reporting correctly.

ceph-volume might cause metadata corruption when creating OSDs during a Ceph deployment

Previously, when ceph-volume issued LVM commands like creating volume groups, logical volumes, setting tags, etc it could result in corrupted metadata when creating OSDs during a Ceph deployment. With this release, users can conditionally disable the lvmetad service by setting the lvmetad_disabled parameter to true in the group_vars/all.yml file on the host thereby the metadata corruption is avoided.

The role ceph-dashboard in ceph-ansible enforces the common name of the self-signed certificate to ceph-dashboard

Previously, when using the self-signed certificates generated by ceph-ansible, it enforced the common name (CN) to ceph-dashboard thereby causing applications like Prometheus to error out due to the mismatch in the hostname of the node sending certificate to the clients.

Rolling upgrade fails when Ceph containers are collocated

The rolling_update.yml Ansible playbook fails when the Ceph Monitor and Ceph Object Gateway daemons are collocated with containers, and when the multi-site Ceph Object Gateway is enabled. This failure was caused by the radosgw-admin commands not able to execute because of the Ceph Monitor container is stopped during the upgrade process. With this release, the multi-site Ceph Object Gateway code within the ceph-handler role is skipped during the upgrade process. As a result, the rolling_update.yml Ansible playbook runs successfully.

Ceph Monitor quorum check fails when switching to a containerized daemons

A regression bug was introduced in the switch-from-non-containerized-to-containerized-ceph-daemons.yml Ansible playbook. This regression bug caused the Ceph Monitor quorum check to fail because the current node’s host name was not tested. With this release, the ansible_hostname fact from the current Ceph Monitor node is used correctly. As a result, the Ceph Monitor quorum check is successful.

Adding a new Ceph Object Gateway instance when upgrading fails

The radosgw_frontend_port option did not consider more than one Ceph Object Gateway instance, and configured port 8080 to all instances. With this release, the radosgw_frontend_port option is increased for each Ceph Object Gateway instance, allowing you to use more than one Ceph Object Gateway instance.

Ceph Ansible removes any socket file and enables redeployment of cluster

Previously, *.asok files were left on completion of purge playbook causing failure during a redeployment of a cluster. With this update, ceph-ansible removes any socket file that may be present and the clusters can be redeployed safely.

Added support for log rotation for the tcmu-runner process in containerized Red Hat Ceph Storage deployments

Previously, when Red Hat Ceph Storage with iSCSI was deployed in containers, there was no log rotation for the tcmu-runner process thereby consuming all the space in the containers. With this release, log rotation support is added to the tcmu-runner process and the log files are periodically rotated resulting in lesser space consumption.

The FileStore to BlueStore migration process can fail for OSD nodes that have a mix of FileStore OSDs and BlueStore OSDs

Previously, if deployments running Red Hat Ceph Storage versions earlier than 3.2 never had osd_objectstore explicitly set in either group_vars, host_vars, or inventory, the deployment had FileStore OSDs. FileStore was the default prior to Red Hat Ceph Storage 3.2.

Special characters can be set in the Docker registry password

Previously, special characters set in the Docker registry password were not handled correctly. With this release, the Ansible playbook does not fail when special characters are set in the Docker registry password. Special characters can now be used in the Docker registry password and the Ansible playbook works as expected

ceph-volume Ansible module reports correct information on logical volumes and volume groups

Previously, when applying the command ceph-volume lvm zap --destroy command on a Red Hat Enterprise Linux 7 host with Red Hat Enterprise Linux 8 based containers on an OSD, the lvm cache was not refreshed for the host and still reported the logical volumes and volume groups that are present. With this release,ceph_volume Ansible module triggers a command on the host to ensure the lvm cache is refreshed and reports correct information on logical volumes and volume groups.

Ceph container logs can be viewed by using the journalctl command

Previously, Ceph container logs were not present in journald as Podman used Kubernetes file as the default log driver when running containers in detached mode and systemd type forking. With this release, the Ceph containers are configured with journald log driver and the logs are available using the journalctl command.

Ceph Ansible sets file and directory ownership values to nobody:nobody

Previously, Ceph Ansible set the file and directory ownership values to root:root. This caused permissions issues for alertmanager and prometheus services.

Ceph installation through the Cockpit no longer fails

Previously, when a user provides localhost as the name of the host in the host selection page, the Ceph installation through the Cockpit failed with an error Systemd must be present. With this release, the host page UI will display a proper error message and deny the user from using localhost as a hostname to continue.

Cockpit installer produces incorrect values for number of buckets in rgws.yml

Previously, using the Cockpit installer to generate the rgws.yml file produced incorrect values for defaults.rgw.buckets.data:pgnum and rgw_override_bucket_index_max_shards.

The prometheus query is fixed to report real-time metrics on CPU usage on the dashboard

Previously, the hosts screen on the Red Hat Ceph Storage dashboard displayed incorrect data about the CPU usage for the nodes in the cluster due to an inaccurate prometheus query. With this release, the prometheus query was fixed and the data on CPU usage gives near real-time metrics.

Removed options which expect future data from Grafana dashboards embedded in Red Hat Ceph Storage dashboard

Previously, when the option ”this week” was selected from “Historical Data” in the overall performance graph, the metrics were not displayed as few historical data options in the Grafana metrics expected future data. With this release,the historical data options that expect future data have been removed and the metrics are displayed as expected.

Cherrypy no longer reveals its versions in header and error pages

Previously, cherrypy revealed its version in header and error pages. Exposing this information caused a potential security vulnerability. With this release, the headers for both dashboard and Prometheus servers now show Ceph-Dashboard and Ceph-Prometheus instead of the cherrypy version. This change eliminates the security vulnerability.

Ceph fs status command no longer fails with AttributeError` exception

Previously, Ceph fs status command failed with AttributeError exception, caused by incorrect handling of metadata during the rejoin. With this release, Ceph fs status command returns expected status as unknown objects are now accepted if NoneType is the metadata object type.

The alerts module of Ceph manager works as expected with no health warnings

Previously, Ceph manager would override some module specific config options with default values resulting in the failure of the module with ALERTS_SMTP_ERROR unable to send alert email as it continued to use the default value smtp_ssl=true. With this release, the default value handling of the Ceph manager is fixed and the alerts module works as expected with no health warnings.

The restful module API endpoints are now accessible

Previously, the restful module used dict.iteritems which is no longer available in python 3. As a result a number of restful module API endpoints were not accessible. With this release, the restful module has been updated to use dic.items and the API endpoints are accessible.

IPv6 addresses were not handled properly in ceph-mgr Prometheus exporter

Previously, some metadata metrics in the ceph-mgr Prometheus exporter output showed incomplete or malformed IPv6 addresses. With this release, the exporter now handles IPv6 addresses correctly.

Executing the ceph-volume command sent debugging output to /var/log/ceph/ceph-volume.log

Previously, executing the ceph-volume command always sent debugging-level output to /var/log/ceph/ceph-volume.log regardless of the level set in the --log-level option. With this release, executing the ceph-volume command sends output at the level that the --log-level option specifies.

ceph volume lvm batch reports SSD devices correctly and deploys correct configuration

Previously, ceph-volume lvm batch command caused a race condition with udev resulting in SSD devices incorrectly reported as HDD, causing unexpected deployment configuration. With the code update in this release, ceph-volume lvm batch command reports SSDs correctly and deploys the expected configuration of the cluster.

Stack update fails after adding new SSD OSDs to the storage cluster

Previously, using stack update to add new SSD OSDs to the same volume group (VG) in the storage cluster caused stack update to fail. This occurred because stack update incorrectly viewed the new SSDs as belonging to different logical volumes (LVs), instead of belonging to the same LV. With this release, stack update views newly added OSDs as belonging to the same LV, and it no longer fails.

Set-lifecycle and delete-lifecycle actions against upgraded OSDs now work as expected

Previously, during an upgrade from Red Hat Ceph Storage 3 to Red Hat Ceph Storage 4.2z2, installing a legacy lifecycle policy against upgraded OSDs yielded a structure decoding error although the set-lifecycle action would succeed. With this release, the changes required to decode bucket lifecycle status entries are fixed and the upgrade of daemons works as expected.

The --reset-stats option updates buckets in groups for users with large numbers of buckets

Previously, the radosgw-admin user --reset-stats option simultaneously updated the stats for all buckets owned by a user. For users with very large numbers of buckets, the time required to make the updates could exceed the length of the associated RADOS operation. This could cause Ceph to mark OSDs as down, and could cause the OSDs to flap.

gc perf counters increments when gc entries are purged from the system

Previously, gc perf counters did not increment when gc entries were purged from the system. With this code update, the correct value of gc perf counter is observed in accordance with the number of gc entries that have been deleted from the system.

Listing of entries in the last GC object does not enter a loop

Previously, the listing of entries in the last GC object entered a loop because the marker was reset every time for the last GC object. With this release, the truncated flag is updated which does not cause the marker to be reset and the listing works as expected.

The Ceph Object Gateway syncs bucket cache information during the bucket creation process

Previously, the Ceph Object Gateway did not sync the cache of the bucket information upon bucket creation. This led to the condition when a user tried to access a bucket that did not exist from one RGW and then created that bucket from another Ceph Object Gateway, accessing the bucket from the first Ceph Object Gateway caused a 404 error, stating that the bucket did not exist, although it did. With this update, RGW syncs the cache during the bucket creation process, so that each Ceph Object Gateway can access the bucket.

KafkaConnect sends objects from a Kafka topic to the RGW S3 bucket

Previously, sending objects from a Kafka topic to the RGW S3 bucket failed because the chunked-encoding object signature was not calculated correctly.

Bucket index do not collect entries after syncing on a bucket has been disabled

Previously, the use of radosgw-admin bucket check --fix …​ variable on a bucket in which multi-site syncing has been disabled, would set an incorrect flag indicating syncing has not been disabled. Data would be added to bucket index logs that would not be used or trimmed, thereby consuming more storage over time. With this release, the syncing flag is now copied correctly when running the radosgw-admin bucket check --fix …​ command. Bucket index logs do not collect entries after syncing on a bucket is disabled.

The Progress module is no longer stuck for an indefinite time

Previously, the progress vents in Ceph status were stuck for an indefinite time. This was caused by the Progress module checking the PG state early and not syncing with the epoch of the OSDMap. With this release, progress events now pop up as expected.

Progress module causing Ceph Monitor crashes

During backfill and recovery operations, the progress module can generate negative progress events. With large storage clusters, too many negative progress events can lead to large memory allocation on the Ceph Monitor nodes, causing cascading Ceph Monitor crashes. With this release, the code ensures that progress events are not negative. As a result, the progress module does not cause the Ceph Monitors to crash.

Ceph Monitors now show caution when handling forwarded OSD failure reports

Previously, Ceph Monitors would incorrectly report slow operations to administrators and logging systems. With this release, Ceph Monitors now handle forwarded OSD failure reports with caution, so many fewer inaccurate slow operations will occur.

Monitor trims osdmaps appropriately

Previously, the monitor failed to trim stale osdmaps although they were used by “out” OSDs because the monitor considered both “in“ and “out” OSDs when trimming osdmaps. With this release, the “out” OSDs are not considered and the osdsmaps are trimmed appropriately.

BlueStore and FileStore OSDs list objects in the same order in a mixed cluster

Previously, in a cluster with both BlueStore and FileStore OSDs, deep scrub and backfill could report missing objects due to an inconsistency of the sorting mechanism in the backend. With this upgrade, a feature flag OSD_FIXED_COLLECTION_LIST has been added to ensure that the collection_list method in BlueStore lists objects in the same order as FileStore.

Log files were created with incorrect permissions

Previously, a code addition changed the order in which relevant functions were called. This caused some daemons to create log files with incorrect permissions. With this release, functions are called in the correct order, and the daemons create log files with the correct permissions.

Enabling bluefs_buffered_io prevents performance degradation

Previously, the option bluefs_buffered_io was disabled which led to slow operations of RocksDB and OMAP interactions in certain scenarios.With this release the option bluefs_buffered_io is set to True , thereby preventing performance degradation.

Memory growth because of onodes is now controlled

Previously, the default value for the option bluestore_cache_trim_max_skip_pinned was 64 which was very low for large clusters. Since this option controlled the rate of trimming onodes and with current default value, it could have led to a buildup of onodes causing memory growth. With this release, the default value of bluestore_cache_trim_max_skip_pinned` is 1000 and the memory growth is controlled.

Applications using librbd works when client-side QoS throttling is enabled

Previously, when client-side QoS throttling was enabled in librbd, it could crash because the data path was not properly protected with a lock. With this release, the missing lock is added and the applications using librbd for IO works as expected when client-side QoS throttling is enabled.

All file layouts in READDIR return results

Previously, some file layouts caused a loop in READDIR and never returned results. With this update, READDIR works as expected and returns the results correctly.