Release Notes

cephadm supports colocating multiple daemons on the same host

With this release, multiple daemons, such as Ceph Object Gateway and Ceph Metadata Server (MDS), can be deployed on the same host thereby providing an additional performance benefit.

Example

service_type: rgw
placement:
  label: rgw
  count-per-host: 2

Configuration of NFS-RGW using Cephadm is now supported

In Red Hat Ceph Storage 5.0 configuration, use of NFS-RGW required the use of dashboard as a workaround and it was recommended for such users to delay upgrade until Red Hat Ceph Storage 5.1

Users can now bootstrap their storage clusters with custom monitoring stack images

Previously, users had to adjust the image used for their monitoring stack daemons manually after bootstrapping the cluster

Syntax

[mgr]
mgr/cephadm/container_image_grafana = GRAFANA_IMAGE_NAME
mgr/cephadm/container_image_alertmanager = ALERTMANAGER_IMAGE_NAME
mgr/cephadm/container_image_prometheus = PROMETHEUS_IMAGE_NAME
mgr/cephadm/container_image_node_exporter = NODE_EXPORTER_IMAGE_NAME

cephadm enables automated adjustment of osd_memory_target

With this release, cephadm enables automated adjustment of osd_memory_target configuration parameter by default.

Users can now specify CPU limits for the daemons by service

With this release, you can customize the CPU limits for all daemons within any given service by adding the CPU limit to the service specification file via the extra_container_args field.

Example

service_type: mon
service_name: mon
placement:
hosts:
  - host01
  - host02
  - host03
extra_container_args:
  - "--cpus=2"

service_type: osd
service_id: osd_example
placement:
  hosts:
    - host01
extra_container_args:
  - "--cpus=2"
spec:
  data_devices:
    paths:
    - /dev/sdb

cephadm now supports IPv6 networks for Ceph Object Gateway deployment

With this release, cephadm supports specifying an IPv6 network for Ceph Object Gateway specifications. An example of a service configuration file for deploying Ceph Object Gateway is:

Example

service_type: rgw
service_id: rgw
placement:
  count: 3
networks:
- fd00:fd00:3000::/64

The ceph nfs export create rgw command now supports exporting Ceph Object Gateway users

Previously, the ceph nfs export create rgw command would only create Ceph Object Gateway exports at the bucket level.

Syntax

ceph nfs export create rgw --cluster-id CLUSTER_ID --pseudo-path PSEUDO_PATH --user-id USER_ID [--readonly] [--client_addr VALUE...] [--squash VALUE]

Example

[ceph: root@host01 /]# ceph nfs export create rgw --cluster-id mynfs --pseudo-path /bucketdata --user-id myuser --client_addr 192.168.10.0/24

Users can now view the HAProxy metrics on the Red Hat Ceph Storage Dashboard

With this release, Red Hat introduces the new Grafana dashboard for ingress service used for Ceph Object Gateway endpoints. You can now view the four HAProxy metrics under Ceph Object Gateway Daemons Overall Performance such as Total responses by HTTP code, Total requests/responses, Total number of connections, and Current total of incoming/outgoing bytes.

User can view mfa_ids on the Red Hat Ceph Storage Dashboard

With this release, you can view the mfa_ids for a user configured with multi-factor authentication (MFA) for the Ceph Object Gateway user in the User Details section on the Red Hat Ceph Storage Dashboard.

The global recovery event in the progress module is now optimized

With this release, computing for the progress of global recovery events is optimized for a large number of placement groups in a large storage cluster by using the C++ code instead of the python module thereby reducing the CPU utilization.

The lvm commands does not cause metadata corruption when run within the containers

Previously, when the lvm commands were run directly within the containers, it would cause LVM metadata corruption.

Lock contention messages from the Ceph Object Gateway reshard queue are marked as informational

Previously, when the Ceph Object Gateway failed to get a lock on a reshard queue, the output log entry would appear to be an error causing concern to customers.

The support for OIDC JWT token validation using modulus and exponent is available

With this release, OIDC JSON web token (JWT) validation supports the usage of modulus and exponent for signature calculation. It also extends the support for available methods to validate OIDC JWT validation.

The role name and role session fields are now available in ops log for temporary credentials

Previously, the role name and role session were not available and it was difficult for the administrator to know which role was being assumed and which session was active for the temporary credentials being used.

Users can now use the --bucket argument to process bucket lifecycles

With this release, you can provide a --bucket=BUCKET_NAME argument to the radosgw-admin lc process command to process the lifecycle for the corresponding bucket. This is convenient for debugging lifecycle problems that affect specific buckets and for backfilling lifecycle processing for specific buckets that have fallen behind.

Multi-site configuration supports dynamic bucket index resharding

Previously, only manual resharding of the buckets for multi-site configurations was supported.

Use the noautoscale flag to manage the PG autoscaler

With this release, pg_autoscaler can be turned on or off globally using the noautoscale flag. This flag is set to off by default. When this flag is set, then all the pools have pg_autoscale_mode as off

Users can now create pools with the --bulk flag

With this release, you can create pools with the --bulk flag. It uses a profile of the pg_autoscaler and provides better performance from the start and has a full complement of placement groups (PGs) and scales down only when the usage ratio across the pool is not even.

Syntax

ceph osd pool create POOL_NAME --bulk

Syntax

ceph osd pool set POOL_NAME bulk TRUE/FALSE/1/0
ceph osd pool unset POOL_NAME bulk TRUE/FALSE/1/0

Syntax

ceph osd pool get POOL_NAME --bulk

Reintegrating stray entries does not fail when the destination directory is full

Previously, Ceph Metadata Servers reintegrated an unlinked file if it had a reference to it, that is, if the deleted file had hard links or was a part of a snapshot. The reintegration, which is essentially an internal rename operation, failed if the destination directory was full. This resulted in Ceph Metadata Server failing to reintegrate stray or deleted entries.

MDS daemons no longer crash when receiving metrics from new clients

Previously, in certain scenarios, newer clients were being used for old CephFS clusters. While upgrading old CephFS, cephadm or mgr used newer clients to perform checks, tests or configurations with an old Ceph cluster. Due to this, the MDS daemons crashed when receiving unknown metrics from newer clients.

Ceph Metadata Server no longer crashes during concurrent lookup and unlink operations

Previously, an incorrect assumption of an assert placed in the code, which gets hit on concurrent lookup and unlink operations from a Ceph client, caused Ceph Metadata Server crash.

Usage of MD5 for non-cryptographic purposes in a FIPS environment is allowed

Previously, in a FIPS enabled environment, the usage of MD5 digest was not allowed by default, unless explicitly excluded for non-cryptographic purposes. Due to this, a segfault occurred during the S3 complete multipart upload operation.

ceph-objectstore-tool command allows manual trimming of the accumulated PG log dups entries

Previously, trimming of PG log dups entries was prevented during the low-level PG split operation which is used by the PG autoscaler with far higher frequency than by a human operator. Stalling the trimming of dups resulted in significant memory growth of PG log, leading to OSD crashes as it ran out of memory. Restarting an OSD did not solve the problem as the PG log is stored on disk and reloaded to RAM on startup.

Snapshot-based mirroring process no longer gets cancelled

Previously, as a result of an internal race condition, the rbd mirror snapshot schedule add command would be cancelled out. The snapshot-based mirroring process for the affected image would not start, if no other existing schedules were applicable.

Existing schedules take effect when an image is promoted to primary

Previously, due to an ill-considered optimization, existing schedules would not take effect following an image’s promotion to primary resulting in the snapshot-based mirroring process to not start for a recently promoted image.

rbd-mirror daemon no longer acquires exclusive lock

Previously, due to a logic error, rbd-mirror daemon could acquire the exclusive lock on a de-facto primary image. Due to this, the snapshot-based mirroring process for the affected image would stop, reporting a "failed to unlink local peer from remote image" error.

Mirror snapshot queue used by rbd-mirror is extended and is no longer removed

Previously, as a result of an internal race condition, the mirror snapshot used by the rbd-mirror daemon on the secondary cluster would be removed causing the snapshot-based mirroring process for the affected image to stop, reporting a "split-brain" error.

Adoption playbook can now install cephadm on OSD nodes

Previously, due to the tools repository being disabled on OSD nodes, you could not install cephadm OSD nodes resulting in the failure of the adoption playbook.

Removal of legacy directory ensures error-free cluster post adoption

Previously, cephadm displayed an unexpected behaviour with its 'config inferring' function whenever a legacy directory, such as /var/lib/ceph/mon was found. Due to this behaviour, post adoption, the cluster was left with the following error "CEPHADM_REFRESH_FAILED: failed to probe daemons or devices".

Use the yes-i-know flag to bootstrap the Red Hat Ceph Storage cluster 5.1

Users are warned against configuring multi-site in a Red Hat Ceph Storage cluster 5.1.

Syntax

sudo --image IMAGE_NAME bootstrap --mon-ip IP_ADDRESS --yes-i-know

Users are warned while upgrading to Red Hat Ceph Storage 5.1

Previously, buckets resharded in Red Hat Ceph Storage 5.0, might not have been understandable by a Red Hat Ceph Storage 5.1 Ceph Object Gateway daemon, due to which an upgrade warning or blocker was added to make sure all users upgrading to Red Hat Ceph Storage 5.1 are aware of the issue and can downgrade if they were previously using Red Hat Ceph Storage 5.1.