Chapter 3. New features

Users can now configure various NFS options in idmap.conf

With this enhancement, the ability to configure NFS options, such as "Domain", "Nobody-User", "Nobody-Group" and the like, in idmap.conf is introduced.

Client IP restriction is now possible over the new haproxy protocol mode for NFS

Previously, client IP restriction did not work in setups using haproxy over NFS.

Users must now enter a username and password to access the Grafana API URL

Previously, anyone who could connect to the Grafana API URL would have access to it without needing any credentials.

Ingress service with NFS backend can now be set up to use only keepalived to create a virtual IP (VIP) for the NFS daemon to bind to, without the HAProxy layer involved

With this enhancement, ingress service with an NFS backend can be set up to only use keepalived to create a virtual IP for the NFS daemon to bind to, without the HAProxy layer involved. This is useful in cases where the NFS daemon is moved around and clients need not use a different IP to connect to it.

The HAProxy daemon binds to its front-end port only on the VIP created by the accompanying keepalived

With this enhancement, the HAProxy daemon will bind to its front-end port only on the VIP created by the accompanying keepalived, rather than on 0.0.0.0. Cephadm deployed HAProxy will bind its front-end port to the VIP, allowing other services, such as an NFS daemon, to potentially bind to port 2049 on other IPs on the same node.

Haproxy health check interval for ingress service is now customizable

Previously, in some cases, the two second default health check interval was too frequent and it caused unnecessary traffic.

Grafana now binds to an IP within a specific network on a host, rather that always binding to 0.0.0.0

With this enhancement, using a Grafana specification file that includes both the networks' section with the network that Grafana binds to an IP on, and only_bind_port_on_networks: true included in the "spec" section of the specification, Cephadm configures the Grafana daemon to bind to an IP within that network rather than 0.0.0.0. This enables users to use the same port that Grafana uses for another service but on a different IP on the host. If it is a specification update that does not cause them all to be moved, ceph orch redeploy grafana can be run to pick up the changes to the settings.

All bootstrap CLI parameters are now made available for usage in the cephadm-ansible module

Previously, only a subset of the bootstrap CLI parameters were available and it was limiting the module usage.

Prometheus scrape configuration is added to the nfs-ganesha exporter

With this enhancement, the Prometheus scrape configuration is added to the nfs-ganesha exporter. This is done to scrape the metrics exposed by nfs-ganesha prometheus exporter into the Prometheus instance running in Ceph, which would be further consumed by Grafana Dashboards.

Prometheus now binds to an IP within a specific network on a host, rather that always binding to 0.0.0.0

With this enhancement, using a Prometheus specification file that includes both the networks section with the network that Prometheus binds to an IP on, and only_bind_port_on_networks: true included in the "spec" section of the specification, Cephadm configures the Prometheus daemon to bind to an IP within that network rather than 0.0.0.0. This enables users to use the same port that Prometheus uses for another service but on a different IP on the host. If it is a specification update that does not cause them all to be moved, ceph orch redeploy prometheus can be run to pick up the changes to the settings.

Users can now mount snapshots (exports within .snap directory)

With this enhancement, users can mount snapshots (exports within .snap directory) to look at in a RO mode. NFS exports created with the NFS MGR module now include the cmount_path setting (this cannot be configured and should be left as "/") which allows snapshots to be mounted.

Zonegroup hostnames can now be set using the specification file provided in the ceph rgw realm bootstrap…​ command

With this release, in continuation to the automation of Ceph Object gateway multi-site setup, users can now set zonegroup hostnames through the initial specification file passed in the bootstrap command ceph rgw realm bootstrap…​ instead of requiring additional steps.

CephFS snapshot schedules management on the Ceph dashboard

Previously, CephFS snapshot schedules could only be managed through the command-line interface.

Ceph dashboard now supports NFSv3-based exports in Ceph dashboard

With this enhancement, support is enabled for NFSv3-based export management in the Ceph dashboard.

Ability to manage Ceph users for CephFS is added

With this enhancement, the ability to manage the Ceph users for CephFS is added. This provides the ability to manage the users' permissions for volumes, subvolume groups, and subvolumes from the File System view.

A new API endpoint for multi-site sync status is added

Previously, multi-site sync status was available only via the CLI command.

Improved monitoring of NVMe-oF gateway

With this enhancement, to improve monitoring of NVMe-oF gateway, alerts of NVMe-oF gateway are added based on the metrics emitted and also, metrics from the embedded prometheus exporter are scraped in the NVMe-oF gateway.

CephFS clone management in Ceph dashboard

With this enhancement, CephFS clone management functionality is provided in the Ceph dashboard. Users can create and delete subvolume clone through the Ceph dashboard.

CephFS snapshot management in Ceph dashboard

With this enhancement, CephFS snapshot management functionality is provided in the Ceph dashboard. Users can create and delete subvolume snapshot through the Ceph dashboard.

Labeled Performance Counters per user/bucket

With this enhancement, users can not only obtain information on the operations happening per Ceph Object Gateway node, but can also view the Ceph Object Gateway performance counters per-user and per-bucket in the Ceph dashboard.

Labeled Sync Performance Counters into Prometheus

With this enhancement, users can gather real-time information from Prometheus about the replication health between zones for increased observability of the Ceph Object Gateway multi-site sync operations.

Add and edit bucket in Ceph dashboard

With this enhancement, as part of the Ceph Object Gateway improvements to the Ceph dashboard, the capability to apply, list and edit Buckets from the Ceph dashboard is added.

Add, List, Delete, and Apply bucket policies in Ceph dashboard

With this enhancement, as part of the Ceph Object Gateway improvements to the Ceph dashboard, the capability to add, list, delete, and apply bucket policies from the Ceph dashboard is added.

MDS dynamic metadata balancer is off by default

Previously, poor balancer behavior would fragment trees in undesirable ways by increasing the max_mds file system setting.

CephFS supports quiescing of subvolumes or directory trees

Previously, multiple clients would interleave reads and writes across a consistent snapshot barrier where out-of-band communication existed between clients. This communication led to clients wrongly believing they have reached a checkpoint that is mutually recoverable via a snapshot.

MDS Resident Segment Size (RSS) performance counter is tracked with a higher priority

With this enhancement, the MDS Resident Segment Size performance counter is tracked with a higher priority to allow callers to consume its value to generate useful warnings. This allows Rook to identify the MDS RSS size and act accordingly.

Laggy clients are now evicted only if there are no laggy OSDs

Previously, monitoring performance dumps from the MDS would sometimes show that the OSDs were laggy, objecter.op_laggy and objecter.osd_laggy, causing laggy clients (dirty data could not be flushed for cap revokes).

cephfs-mirror daemon exports snapshot synchronization performance counters via perf dump command

ceph-mds daemon export per-client performance counters included in the already existing perf dump command.

A new dump dir command is introduced to dump the directory information

With this enhancement, the dump dir command is introduced to dump the directory information and print the output.

Snapshot scheduling support for subvolumes

With this enhancement, snapshot scheduling support is provided for subvolumes. All snapshot scheduling commands accept --subvol and --group arguments to refer to appropriate subvolumes and subvolume groups. If a subvolume is specified without a subvolume group argument, then the default subvolume group is considered. Also, a valid path need not be specified when referring to subvolumes and just a placeholder string is sufficient due to the nature of argument parsing employed.

Ceph commands that add or modify MDS caps give an explanation about why the MDS caps passed by user was rejected

Previously, Ceph commands that add or modify MDS caps printed "Error EINVAL: mds capability parse failed, stopped at 'allow w' of 'allow w'".

Admin interface is now added to manage bucket notification

Previously, the S3 REST APIs were used to manage bucket notifications. However, if an admin wanted to override them, there was no easy way to do that over the radosgw-admin tool.

RGW labeled user and bucket operation counters are now in different sections when the ceph counter dump is run

Previously, all RGW labeled operation counters were in the rgw_op` section of the output of the ceph counter dump command but would either have a user label or a bucket label.

Users can now place temporary files into a directory using the -t command-line option

Previously, the /usr/bin/rgw-restore-bucket-index tool just used /tmp and that directory sometimes did not have enough free space to hold all the temporary files.

Copying of encrypted objects using copy-object APIs is now supported

Previously, in Ceph Object gateway, copying of encrypted objects using copy-object APIs was unsupported since the inception of its server-side encryption support.

A new Ceph Object Gateway admin-ops capability is added to allow reading user metadata but not their associated authorization keys

With this enhancement, a new Ceph Object Gateway admin-ops capability is added to allow reading Ceph Object gateway user metadata but not their associated authorization keys. This is to reduce the privileges of automation and reporting tools and to avoid impersonating users or view their keys.

Cloud Transition: add new supported S3-compatible platforms

With this release, to be able to move object storage to the cloud or other on-premise S3 endpoints, the current lifecycle transition and storage class model is extended. S3-compatible platforms, such as IBM Cloud Object Store (COS) and IBM Storage Ceph are now supported for the cloud archival feature.

NFS with RGW backend

With this release, NFS with Ceph Object Gateway backend is re-GAed with the existing functionalities.

A retry mechanism is introduced in the radosgw-admin sync status command

Previously, when the multisite sync sent requests to a remote zone, it used a round robin strategy to choose one of its zone endpoints. If that endpoint was not available, the http client logic used by the radosgw-admin sync status command would not provide a retry mechanism, and thus report input/output error.

NewerNoncurrentVersions, ObjectSizeGreaterThan, and ObjectSizeLessThan filters are added to the lifecycle

With this enhancement, support for NewerNoncurrentVersions, ObjectSizeGreaterThan, and ObjectSizeLessThan filters are added to the lifecycle.

User S3 replication APIs are now supported

With this enhancement, user S3 replication APIs are now supported. With these APIs, users can set replication policies at bucket-level. The API is extended to include additional parameters to specify source and destination zone names.

Bucket Granular Sync Replication GA (Part 3)

With this release, the ability to replicate a bucket or a group of buckets to a different Red Hat Ceph Storage cluster is added with bucket granular support. The usability requirements are as Ceph Object Gateway multi-site.

Setting the noautoscale flag on/off retains each pool’s original autoscale mode configuration

Previously, the pg_autoscaler did not persist in each pool’s autoscale mode configuration when the noautoscale flag was set. Due to this, whenever the noautoscale flag was set, the autoscale mode had to be set for each pool repeatedly.

reset_purged_snaps_last OSD command is introduced

With this enhancement, reset_purged_snaps_last OSD command is introduced to resolve cases in which the purged_snaps keys (PSN) are missing in the OSD and exist in the monitor. The purged_snaps_last command will be zeroed and as a result, the monitor will share all its purged_snaps information with the OSD on the next boot.

BlueStore’s RocksDB compression enabled

With this enhancement, to ensure that the metadata (especially OMAP) takes less space, RocksDB configuration is modified to enable internal compression of its data.

OSD is now more resilient to fatal corruption

Previously, special OSD layer object "superblock" would be overwritten due to being located at the beginning of the disk, resulting in a fatal corruption.

Improved rbd_diff_iterate2() API performance

Previously, RBD diff-iterate was not guaranteed to execute locally if exclusive lock was available when diffing against the beginning of time (fromsnapname == NULL) in fast-diff mode (whole_object == true with fast-diff image feature enabled and valid).