Chapter 6. Asynchronous errata updates

Grafana now binds to an IP within a specific network on a host, rather than always binding to 0.0.0.0

With this enhancement, using a Grafana specification file that includes both the networks section with the network that Grafana binds to an IP on, and only_bind_port_on_networks: true included in the spec section of the specification file. Cephadm configures the Grafana daemon to bind to an IP within that network rather than 0.0.0.0. This enables users to use the same port that Grafana uses for another service but on a different IP on the host. If it is a specification update that does not cause them all to be moved, ceph orch redeploy grafana can be run to pick up the changes to the settings. Grafana specification file:

ceph orch set-unmanaged and ceph orch set-managed commands are now available

Previously, moving services between managed and unmanaged states required manual modification of the specification files.

It is now possible to remove offline hosts from maintenance mode

Previously, if users removed a host from cephadm control without exiting maintenance mode, a health warning persisted indicating that the host was still in maintenance mode. This occurred when a host in maintenance mode became permanently offline due to hardware or other issues.

The vrrp_interface_network and use_keepalived_multicast settings for keepalive of the ingress service are now add

Previously, users were not able to specify options for keepalive to use.

Health warnings for a standby-replay MDS are no longer included

Previously, all inode and stray counters health warnings were displayed during a standby-replay MDS.

Increased efficiency for ordered bucket listings with namespaced bucket index entries

Previously, when ignoring namespaced bucket index entries, the code would still access the ignored entry. As a result, there would be an unnecessary latency in the ignored listings.

New clone creation no longer slows down due to parallel clone limit

Previously, upon reaching the limit of parallel clones, the rest of the clones would queue up, slowing down the cloning.

The Python librados supports iterating object omap key/values

Previously, the iteration would break whenever a binary/unicode key was encountered.

Improved temporary file placement and error messages for the with the ‘/usr/bin/rgw-restore-bucket-index' tool

Previously, the ‘/usr/bin/rgw-restore-bucket-index' tool only placed temporary files into the ‘/tmp’ directory. This could lead to issues if the directory ran out of space, resulting in the following error messing being emitted: "ln: failed to access '/tmp/rgwrbi-object-list.XXX': No such file or directory".

S3 requests are no longer cut off in the middle of transmission during shutdown

Previously, a few clients faced issues with the S3 request being cut off in the middle of transmission during shutdown without waiting.

New ‘mon_cluster_log_level’ command option to control the cluster log level verbosity for external entities

Previously, debug verbosity logs were sent to all external logging systems regardless of their level settings. As a result, the ‘/var/’ filesystem would rapidly fill up.

All bootstrap CLI parameters are now made available for usage in the cephadm-ansible module

Previously, only a subset of the bootstrap CLI parameters were available and it was limiting the module usage.

RBD diff-iterate now executes locally if exclusive lock is available

Previously, when diffing against the beginning of time (fromsnapname == NULL) in fast-diff mode (whole_object == true with fast-diff image feature enabled and valid), RBD diff-iterate wasn’t guaranteed to execute locally.

Snapshot scheduling support is now provided for subvolumes

With this enhancement, snapshot scheduling support is provided for subvolumes. All snapshot scheduling commands accept --subvol and --group arguments to refer to appropriate subvolumes and subvolume groups. If a subvolume is specified without a subvolume group argument, then the default subvolume group is considered. Also, a valid path need not be specified when referring to subvolumes and just a placeholder string is sufficient due to the nature of the argument parsing employed.

Example

ceph fs snap-schedule add - 15m --subvol sv1 --group g1
ceph fs snap-schedule status - --subvol sv1 --group g1

# ceph fs snap-schedule add - 15m --subvol sv1 --group g1
# ceph fs snap-schedule status - --subvol sv1 --group g1

MDS dynamic metadata balancer is off by default.

With this enhancement, MDS dynamic metadata balancer is off by default to improve the poor balancer behavior that could fragment trees in undesirable or unintended ways simply by increasing the max_mds file system setting.

The resident segment size perf counter in the MDS is tracked with a higher priority.

With this enhancement, the MDS resident segment size (or RSS) perf counter is tracked with a higher priority to allow callers to consume its value to generate useful warnings which is useful for rook to ascertain the MDS RSS size and act accordingly.

ceph auth commands give a message when permissions in MDS are incorrect

With this enhancement, permissions in MDS capability now either start with r, rw, `* or all. This results in the ceph auth commands like ceph auth add, ceph auth caps, ceph auth get-or-create and ceph auth get-or-create-key to generate a clear message when the permissions in the MDS caps are incorrect..

The radosgw-admin bucket stats command prints bucket versioning

With this enhancement, the ` radosgw-admin bucket stats` command prints the versioning status for buckets as enabled or off since versioning can be enabled or disabled after creation.

The snap schedule module now supports a new retention specification

With this enhancement, users can define a new retention specification to retain the number of snapshots.

Laggy clients are now evicted only if there are no laggy OSDs

Previously, monitoring performance dumps from the MDS would sometimes show that the OSDs were laggy, objecter.op_laggy and objecter.osd_laggy, causing laggy clients (cannot flush dirty data for cap revokes).

rgw-restore-bucket-index tool can now restore the bucket indices for versioned buckets

With this enhancement, the rgw-restore-bucket-index tool now works as broadly as possible, with the ability to restore the bucket indices for un-versioned as well as for versioned buckets.

NFS Ganesha version updated to V5.6

With this enhancement of updated version of NFS Ganesha, following issues have been fixed: * FSAL’s state_free` function called by free_state did not actually free. * CEPH: Fixed cmount_path. * CEPH: Currently, the client_oc true was broken, forced it to false.

New reports available for sub-events for delayed operations

Previously, slow operations were marked as delayed but without a detailed description.

Turning the noautoscale flag on/off now retains each pool’s original autoscale mode configuration.

Previously, the pg_autoscaler did not persist in each pool’s autoscale mode configuration when the noautoscale flag was set. Due to this, after turning the noautoscale flag on/off, the user would have to go back and set the autoscale mode for each pool again.

BlueStore instance cannot be opened twice

Previously, when using containers, it was possible to create unrelated inodes that targeted the same block device mknod b causing multiple containers to think that they have exclusive access.

Additional features and enhancements are added to rgw-gap-list and rgw-orphan-list scripts to enhance end-users' experience

With this enhancement, to improve the end-users' experience with rgw-gap-list and rgw-orphan-list scripts, a number of features and enhancements have been added, including internal checks, more command-line options, and enhanced output.

Realm, zone group, and/or zone can be specified when running the rgw-restore-bucket-index command

Previously, the tool could only work with the default realm, zone group, and zone.

Original multipart uploads can now be identified in multi-site configurations

Previously, a data corruption bug was fixed in 6.1z1 release that effected multi-part uploads with server-side encryption in multi-site configurations.

Dashboard host loading speed is improved and pages now load faster

Previously, large clusters of five or more hosts have a linear increase in load time on hosts page and main page.

Switch the unfair Mutex lock to fair mutex

Previously, the implementations of the Mutex, for example, std::mutex in C++, would not guarantee fairness and would not guarantee that the lock would be acquired by threads in the order called lock(). In most cases, this worked well but in an overloaded case, the client requests handling thread and submit thread would always successfully acquire the submit_mutex in a long time, causing MDLog::trim() to get stuck. That meant the MDS daemons would fill journal logs into the metadata pool, but could not trim the expired segments in time.

The bucket listing feature enables the rgw-restore-bucket-index tool to complete reindexing

Previously, the rgw-restore-bucket-index tool would restore the bucket’s index partially until the next user listed out the bucket. Due to this, the bucket’s statistics would report incorrectly until the reindexing completed.

Lifecycle transition no longer fails for objects with modified metadata

Previously, setting an ACL on an existing object would change its mtime due to which lifecycle transition failed for such objects.

Blocksize is changed to 4K

Previously, Ceph Object Gateway GC processing would consume excessive time due to the use of a 1K blocksize that would consume the GC queue. This caused slower processing of large GC queues.

Object map for the snapshot accurately reflects the contents of the snapshot

Previously, due to an implementation defect, a stale snapshot context would be used when handling a write-like operation. Due to this, the object map for the snapshot was not guaranteed to accurately reflect the contents of the snapshot in case the snapshot was taken without quiescing the workload. In differential backup and snapshot-based mirroring, use cases with object-map and/or fast-diff features enabled, the destination image could get corrupted.

public_network parameter can now have configuration options, such as global or mon

Previously, in cephadm, the public_network parameter was always set as a part of the mon configuration section during a cluster bootstrap without providing any configuration option to alter this behavior.

The Cephadm commands that are run on the host from the cephadm Manager module now have timeouts

Previously, one of the Cephadm commands would occasionally hang indefinitely, and it was difficult for users to notice and sort the issue.

cephadm support for CA signed keys is implemented

Previously, CA signed keys worked as a deployment setup in Red Hat Ceph Storage 5, although their working was accidental, untested, and broken in changes from Red Hat Ceph Storage 5 to Red Hat Ceph Storage 6.

Deleting objects in versioned buckets causes statistics mismatch

Due to versioned buckets having a mix of current and non-current objects, deleting objects might cause bucket and user statistics discrepancies on local and remote sites. This does not cause object leaks on either site, just statistics mismatch.

Multisite replication may stop during upgrade

Multi-site replication may stop if clusters are on different versions during the process of an upgrade. We would need to suspend sync until both clusters are upgraded to the same version.