Red Hat Summit Red Hat SummitSupportConsoleDevelopersStart a trialContact

Chapter 3. Encryption and Key Management

The Red Hat Ceph Storage cluster typically resides in its own network security zone, especially when using a private storage cluster network.

Important

Security zone separation may be insufficient for protection if an attacker gains access to Ceph clients on the public network.

There are situations where there is a security requirement to assure the confidentiality or integrity of network traffic, and where Red Hat Ceph Storage uses encryption and key management, including:

  • SSH
  • SSL Termination
  • Encryption in Transit
  • Encryption at Rest

3.1. SSH

All nodes in the RHCS cluster use SSH as part of deploying the cluster. This means that on each node:

  • An Ansible user exists with password-less root privileges.
  • The SSH service is enabled and by extension port 22 is open.
  • A copy of the Ansible user’s public SSH key is available.
Important

Any person with access to the Ansible user by extension has permission to exercise CLI commands as root on any node in the RHCS cluster.

See Creating an Ansible User with sudo Access and Enabling Password-less SSH for Ansible for additional details.

3.2. SSL Termination

The Ceph Object Gateway may be deployed in conjunction with HAProxy and keepalived for load balancing and failover. The object gateway Red Hat Ceph Storage version 2 and 3 uses Civetweb. Earlier versions of Civetweb do not support SSL and later versions support SSL with some performance limitations. When using HAProxy and keepalived to terminate SSL connections, the HAProxy and keepalived components use encryption keys.

When using HAProxy and keepalived to terminate SSL, the connection between the load balancer and the Ceph Object Gateway is NOT encrypted.

See Using SSL with Civetweb and HAProxy/keepalived Configuration for details.

3.3. Encryption in transit

Starting with Red Hat Ceph Storage 4 and later, you can enable encryption for all Ceph traffic over the network with the introduction of the messenger version 2 protocol. The secure mode setting for messenger v2 encrypts communication between Ceph daemons and Ceph clients, giving you end-to-end encryption.

Messenger v2 Protocol

The second version of Ceph’s on-wire protocol, msgr2, includes several new features:

  • A secure mode encrypting all data moving through the network.
  • Encapsulation improvement of authentication payloads.
  • Improvements to feature advertisement and negotiation.

The Ceph daemons bind to multiple ports allowing both the legacy, v1-compatible, and the new, v2-compatible, Ceph clients to connect to the same storage cluster. Ceph clients or other Ceph daemons connecting to the Ceph Monitor daemon will try to use the v2 protocol first, if possible, but if not, then the legacy v1 protocol will be used. By default, both messenger protocols, v1 and v2, are enabled. The new v2 port is 3300, and the legacy v1 port is 6789, by default.

The msgr2 protocol supports two connection modes:

  • crc

    • Provides strong initial authentication when a connection is established with cephx.
    • Provides a crc32c integrity check to protect against bit flips.
    • Does not provide protection against a malicious man-in-the-middle attack.
    • Does not prevent an eavesdropper from seeing all post-authentication traffic.

  • secure

    • Provides strong initial authentication when a connection is established with cephx.
    • Provides full encryption of all post-authentication traffic.
    • Provides a cryptographic integrity check.

The default mode is crc.

Ceph Object Gateway Encryption

Also, the Ceph Object Gateway supports encryption with customer-provided keys using its S3 API.

Important

To comply with regulatory compliance standards requiring strict encryption in transit, administrators MUST deploy the Ceph Object Gateway with client-side encryption.

Ceph Block Device Encryption

System administrators integrating Ceph as a backend for Red Hat OpenStack Platform 13 MUST encrypt Ceph block device volumes using dm_crypt for RBD Cinder to ensure on-wire encryption within the Ceph storage cluster.

Important

To comply with regulatory compliance standards requiring strict encryption in transit, system administrators MUST use dmcrypt for RBD Cinder to ensure on-wire encryption within the Ceph storage cluster.

Additional Resources

3.3.1. Enabling the messenger v2 protocol

For new installations of Red Hat Ceph Storage 4, the messenger v2 protocol, msgr2, is enabled by default. For Red Hat Ceph Storage 3 or earlier, the Ceph Monitors bind to the legacy v1 port 6789. After upgrading you can enable the msgr2 protocol to take advantage of its new features. Once the Ceph Monitors bind to the msgr2 protocol they start advertising a v2 address, after restarting the Ceph Monitor service. The msgr2 protocol’s default connection mode is crc

Ensure that you consider cluster CPU requirements when you plan the Red Hat Ceph Storage cluster, to include encryption overhead.

Important

Using secure mode is currently supported by Ceph kernel clients, such as CephFS and krbd on Red Hat Enterprise Linux 8.2. Using secure mode is supported by Ceph clients using librbd, such as OpenStack Nova, Glance, and Cinder.

Prerequisites

  • A running Red Hat Ceph Storage 4 cluster.
  • Open TCP port 3300 in the firewall.
  • Root-level access to the Ceph Monitor nodes.

Procedure

  1. Open the Ceph configuration file, by default /etc/ceph/ceph.conf, for editing.

  2. Under the [global] section added the following to a new line: 

    ms_bind_msgr2 = true

    1. Optionally, to enable on-wire encryption between the Ceph daemons and to also enable on-wire encryption between Ceph clients and daemons, add the following options under the [global] sections of the Ceph configuration file: 

      [global]
ms_cluster_mode=secure
ms_service_mode=secure
ms_client_mode=secure
  3. Save the changes to the Ceph configuration file.
  4. Copy the updated Ceph configuration file to all nodes in the Red Hat Ceph Storage cluster.

  5. Enable the msgr2 protocol: 

    [root@mon ~]# ceph mon enable-msgr2

  6. Restart the Ceph Monitor service on each Ceph Monitor node: 

    [root@mon ~]# systemctl restart ceph-mon.target

Additional Resources

3.4. Encryption at Rest

Red Hat Ceph Storage supports encryption at rest in a few scenarios:

  1. Ceph Storage Cluster: The Ceph Storage Cluster supports Linux Unified Key Setup or LUKS encryption of OSDs and their corresponding journals, write-ahead logs, and metadata databases. In this scenario, Ceph will encrypt all data at rest irrespective of whether the client is a Ceph Block Device, Ceph Filesystem, Ceph Object Storage cluster or a custom application built on librados.
  2. Ceph Object Gateway: The Ceph Storage Cluster supports encryption of client objects. When the Ceph Object Gateway encrypts objects, they are encrypted independently of the Red Hat Ceph Storage cluster. Additionally, the data transmitted is between the Ceph Object Gateway and the Ceph Storage Cluster is in encrypted form.

Ceph Storage Cluster Encryption

The Ceph Storage Cluster supports encrypting data stored on OSDs. Red Hat Ceph Storage can encrypt logical volumes with lvm by specifying dmcrypt; that is, lvm, invoked by ceph-volume, encrypts an OSD’s logical volume not its physical volume, and may encrypt non-LVM devices like partitions using the same OSD key. Encrypting logical volumes allows for more configuration flexibility.

Ceph uses LUKS v1 rather than LUKS v2, because LUKS v1 has the broadest support among Linux distributions.

When creating an OSD, lvm will generate a secret key and pass the key to the Ceph monitors securely in a JSON payload via stdin. The attribute name for the encryption key is dmcrypt_key.

Important

System administrators must explicitly enable encryption.

By default, Ceph does not encrypt data stored in OSDs. System administrators must enable dmcrypt in Ceph Ansible. See the appendix on OSD Ansible settings in the Red Hat Ceph Storage Installation Guide for details on setting the dmcrypt option in the group_vars/osds.yml file.

Note

LUKS and dmcrypt only address encryption for data at rest, not encryption for data in transit.

Ceph Object Gateway Encryption

The Ceph Object Gateway supports encryption with customer-provided keys using its S3 API. When using customer-provided keys, the S3 client passes an encryption key along with each request to read or write encrypted data. It is the customer’s responsibility to manage those keys. Customers must remember which key the Ceph Object Gateway used to encrypt each object.

See S3 API server-side encryption in the Red Hat Ceph Storage Developer Guide for details.

Red Hat logoGithubRedditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

© 2024 Red Hat, Inc.