Chapter 8. Security


As a storage administrator, securing the storage cluster environment is important. Red Hat Ceph Storage provides encryption and key management to secure the Ceph Object Gateway access point.

Prerequisites

  • A healthy running Red Hat Ceph Storage cluster.
  • Installation of the Ceph Object Gateway software.

8.1. Server-Side Encryption (SSE)

The Ceph Object Gateway supports server-side encryption of uploaded objects for the S3 application programming interface (API). Server-side encryption means that the S3 client sends data over HTTP in its unencrypted form, and the Ceph Object Gateway stores that data in the Red Hat Ceph Storage cluster in encrypted form.

Note
  • Red Hat does NOT support S3 object encryption of Static Large Object (SLO) or Dynamic Large Object (DLO).
  • Currently, none of the Server-Side Encryption (SSE) modes have implemented support for CopyObject. It is currently being developed [BZ#2149450].
Important

Server-side encryption is not compatible with multi-site replication due to a known issue. This issue will be resolved in a future release. See Known issues- Mult-site Object Gateway for more details.

Important

To use encryption, client requests MUST send requests over an SSL connection. Red Hat does not support S3 encryption from a client unless the Ceph Object Gateway uses SSL. However, for testing purposes, administrators can disable SSL during testing by setting the rgw_crypt_require_ssl configuration setting to false at runtime, using the ceph config set client.rgw command, and then restarting the Ceph Object Gateway instance.

In a production environment, it might not be possible to send encrypted requests over SSL. In such a case, send requests using HTTP with server-side encryption.

For information about how to configure HTTP with server-side encryption, see the Additional Resources section below.

There are three options for the management of encryption keys:

Customer-provided Keys

When using customer-provided keys, the S3 client passes an encryption key along with each request to read or write encrypted data. It is the customer’s responsibility to manage those keys. Customers must remember which key the Ceph Object Gateway used to encrypt each object.

Ceph Object Gateway implements the customer-provided key behavior in the S3 API according to the Amazon SSE-C specification.

Since the customer handles the key management and the S3 client passes keys to the Ceph Object Gateway, the Ceph Object Gateway requires no special configuration to support this encryption mode.

Key Management Service

When using a key management service, the secure key management service stores the keys and the Ceph Object Gateway retrieves them on demand to serve requests to encrypt or decrypt data.

Ceph Object Gateway implements the key management service behavior in the S3 API according to the Amazon SSE-KMS specification.

Important

Currently, the only tested key management implementations are HashiCorp Vault, and OpenStack Barbican. However, OpenStack Barbican is a Technology Preview and is not supported for use in production systems.

SSE-S3

When using SSE-S3, the keys are stored in vault, but they are automatically created and deleted by Ceph and retrieved as required to serve requests to encrypt or decrypt data.

Ceph Object Gateway implements the SSE-S3 behavior in the S3 API according to the Amazon SSE-S3 specification.

8.1.1. Setting the default encryption for an existing S3 bucket

As a storage administrator, you can set the default encryption for an existing Amazon S3 bucket so that all objects are encrypted when they are stored in a bucket. You can use Bucket Encryption APIs to support server-side encryption with Amazon S3-managed keys (SSE-S3) or Amazon KMS customer master keys (SSE-KMS).

Note

SSE-KMS is supported only from Red Hat Ceph Storage 5.x, not for Red Hat Ceph Storage 4.x.

You can manage default encryption for an existing Amazon S3 bucket using the PutBucketEncryption API. All files uploaded to this bucket will have this encryption by defining the default encryption at the bucket level.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Installation of the Ceph Object Gateway.
  • An S3 bucket created.
  • An S3 user created with user access.
  • Access to a Ceph Object Gateway client with the AWS CLI package installed.

Procedure

  1. Create a JSON file for the encryption configuration:

    Example

    [user@client ~]$ vi bucket-encryption.json

  2. Add the encryption configuration rules to the file:

    Example

    {
    	"Rules": [
            {
    	    "ApplyServerSideEncryptionByDefault": {
                        "SSEAlgorithm": "AES256"
    	    }
           }
        ]
    }

  3. Set the default encryption for the bucket:

    Syntax

    aws --endpoint-url=pass:q[_RADOSGW_ENDPOINT_URL_]:pass:q[_PORT_] s3api put-bucket-encryption --bucket pass:q[_BUCKET_NAME_] --server-side-encryption-configuration pass:q[_file://PATH_TO_BUCKET_ENCRYPTION_CONFIGURATION_FILE/BUCKET_ENCRYPTION_CONFIGURATION_FILE.json_]

    Example

    [user@client ~]$ aws --endpoint-url=http://host01:80 s3api put-bucket-encryption --bucket testbucket --server-side-encryption-configuration file://bucket-encryption.json

Verification

  • Retrieve the bucket encryption configuration for the bucket:

    Syntax

    aws --endpoint-url=pass:q[_RADOSGW_ENDPOINT_URL_]:pass:q[_PORT_] s3api get-bucket-encryption --bucket BUCKET_NAME

    Example

    [user@client ~]$  aws --profile ceph --endpoint=http://host01:80 s3api get-bucket-encryption --bucket testbucket
    
    {
        "ServerSideEncryptionConfiguration": {
            "Rules": [
                {
                    "ApplyServerSideEncryptionByDefault": {
                        "SSEAlgorithm": "AES256"
                    }
                }
            ]
        }
    }

Note

If the bucket does not have a default encryption configuration, the get-bucket-encryption command returns ServerSideEncryptionConfigurationNotFoundError.

8.1.2. Deleting the default bucket encryption

You can delete the default bucket encryption for a specified bucket using the s3api delete-bucket-encryption command.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Installation of the Ceph Object Gateway.
  • An S3 bucket created.
  • An S3 user created with user access.
  • Access to a Ceph Object Gateway client with the AWS CLI package installed.

Procedure

  • Delete a bucket encryption configuration:

    Syntax

    aws --endpoint-url=RADOSGW_ENDPOINT_URL:PORT s3api delete-bucket-encryption --bucket BUCKET_NAME

    Example

    [user@client ~]$ aws --endpoint-url=http://host01:80 s3api delete-bucket-encryption --bucket testbucket

Verification

  • Retrieve the bucket encryption configuration for the bucket:

    Syntax

    aws --endpoint-url=RADOSGW_ENDPOINT_URL:PORT s3api get-bucket-encryption --bucket BUCKET_NAME

    Example

    [user@client ~]$  aws --endpoint=http://host01:80 s3api get-bucket-encryption --bucket testbucket
    
    An error occurred (ServerSideEncryptionConfigurationNotFoundError) when calling the GetBucketEncryption operation:
    The server side encryption configuration was not found

8.2. Server-side encryption requests

In a production environment, clients often contact the Ceph Object Gateway through a proxy. This proxy is referred to as a load balancer because it connects to multiple Ceph Object Gateways. When the client sends requests to the Ceph Object Gateway, the load balancer routes those requests to the multiple Ceph Object Gateways, thus distributing the workload.

In this type of configuration, it is possible that SSL terminations occur both at a load balancer and between the load balancer and the multiple Ceph Object Gateways. Communication occurs using HTTP only. To set up the Ceph Object Gateways to accept the server-side encryption requests, see Configuring server-side encryption.

8.3. Configuring server-side encryption

You can set up server-side encryption to send requests to the Ceph Object Gateway using HTTP, in cases where it might not be possible to send encrypted requests over SSL.

This procedure uses HAProxy as proxy and load balancer.

Prerequisites

  • Root-level access to all nodes in the storage cluster.
  • A running Red Hat Ceph Storage cluster.
  • Installation of the Ceph Object Gateway software.
  • Installation of the HAProxy software.

Procedure

  1. Edit the haproxy.cfg file:

    Example

    frontend http_web *:80
        mode http
        default_backend rgw
    
    frontend rgw­-https
      bind *:443 ssl crt /etc/ssl/private/example.com.pem
      default_backend rgw
    
    backend rgw
        balance roundrobin
        mode http
        server  rgw1 10.0.0.71:8080 check
        server  rgw2 10.0.0.80:8080 check

  2. Comment out the lines that allow access to the http front end and add instructions to direct HAProxy to use the https front end instead:

    Example

    #     frontend http_web *:80
    #     mode http
    #     default_backend rgw
    
    frontend rgw­-https
      bind *:443 ssl crt /etc/ssl/private/example.com.pem
      http-request set-header X-Forwarded-Proto https if { ssl_fc }
      http-request set-header X-Forwarded-Proto https
    # here we set the incoming HTTPS port on the load balancer (eg : 443)
      http-request set-header X-Forwarded-Port 443
      default_backend rgw
    
    backend rgw
        balance roundrobin
        mode http
        server  rgw1 10.0.0.71:8080 check
        server  rgw2 10.0.0.80:8080 check

  3. Set the rgw_trust_forwarded_https option to true:

    Example

    [ceph: root@host01 /]# ceph config set client.rgw rgw_trust_forwarded_https true

  4. Enable and start HAProxy:

    [root@host01 ~]# systemctl enable haproxy
    [root@host01 ~]# systemctl start haproxy

Additional Resources

8.4. The HashiCorp Vault

As a storage administrator, you can securely store keys, passwords, and certificates in the HashiCorp Vault for use with the Ceph Object Gateway. The HashiCorp Vault provides a secure key management service for server-side encryption used by the Ceph Object Gateway.

Ceph Vault Integration Diagram

The basic workflow:

  1. The client requests the creation of a secret key from the Vault based on an object’s key ID.
  2. The client uploads an object with the object’s key ID to the Ceph Object Gateway.
  3. The Ceph Object Gateway then requests the newly created secret key from the Vault.
  4. The Vault replies to the request by returning the secret key to the Ceph Object Gateway.
  5. Now the Ceph Object Gateway can encrypt the object using the new secret key.
  6. After encryption is done the object is then stored on the Ceph OSD.
Important

Red Hat works with our technology partners to provide this documentation as a service to our customers. However, Red Hat does not provide support for this product. If you need technical assistance for this product, then contact Hashicorp for support.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Installation of the Ceph Object Gateway software.
  • Installation of the HashiCorp Vault software.

8.4.1. Secret engines for Vault

The HashiCorp Vault provides several secret engines to generate, store, or encrypt data. The application programming interface (API) sends data calls to the secret engine asking for action on that data, and the secret engine returns a result of that action request.

The Ceph Object Gateway supports two of the HashiCorp Vault secret engines:

  • Key/Value version 2
  • Transit

Key/Value version 2

The Key/Value secret engine stores random secrets within the Vault, on disk. With version 2 of the kv engine, a key can have a configurable number of versions. The default number of versions is 10. Deleting a version does not delete the underlying data, but marks the data as deleted, allowing deleted versions to be undeleted. You can use the API endpoint or the destroy command to permanently remove a version’s data. To delete all versions and metadata for a key, you can use the metadata command or the API endpoint. The key names must be strings, and the engine will convert non-string values into strings when using the command line interface. To preserve non-string values, provide a JSON file or use the HTTP application programming interface (API).

Note

For access control list (ACL) policies, the Key/Value secret engine recognizes the distinctions between the create and update capabilities.

Transit

The Transit secret engine performs cryptographic functions on in-transit data. The Transit secret engine can generate hashes, can be a source of random bytes, and can also sign and verify data. The Vault does not store data when using the Transit secret engine. The Transit secret engine supports key derivation, by allowing the same key to be used for multiple purposes. Also, the transit secret engine supports key versioning. The Transit secret engine supports these key types:

aes128-gcm96
AES-GCM with a 128-bit AES key and a 96-bit nonce; supports encryption, decryption, key derivation, and convergent encryption
aes256-gcm96
AES-GCM with a 256-bit AES key and a 96-bit nonce; supports encryption, decryption, key derivation, and convergent encryption (default)
chacha20-poly1305
ChaCha20-Poly1305 with a 256-bit key; supports encryption, decryption, key derivation, and convergent encryption
ed25519
Ed25519; supports signing, signature verification, and key derivation
ecdsa-p256
ECDSA using curve P-256; supports signing and signature verification
ecdsa-p384
ECDSA using curve P-384; supports signing and signature verification
ecdsa-p521
ECDSA using curve P-521; supports signing and signature verification
rsa-2048
2048-bit RSA key; supports encryption, decryption, signing, and signature verification
rsa-3072
3072-bit RSA key; supports encryption, decryption, signing, and signature verification
rsa-4096
4096-bit RSA key; supports encryption, decryption, signing, and signature verification

Additional Resources

8.4.2. Authentication for Vault

The HashiCorp Vault supports several types of authentication mechanisms. The Ceph Object Gateway currently supports the Vault agent method. The Ceph Object Gateway uses the rgw_crypt_vault_auth, and rgw_crypt_vault_addr options to configure the use of the HashiCorp Vault.

Important

Red Hat supports the usage of Vault agent as the authentication method for containers and the usage of token authentication is not supported on containers.

Vault Agent

The Vault agent is a daemon that runs on a client node and provides client-side caching, along with token renewal. The Vault agent typically runs on the Ceph Object Gateway node. Run the Vault agent and refresh the token file. When the Vault agent is used in this mode, you can use file system permissions to restrict who has access to the usage of tokens. Also, the Vault agent can act as a proxy server, that is, Vault will add a token when required and add it to the requests passed to it before forwarding them to the actual server. The Vault agent can still handle token renewal just as it would when storing a token in the Filesystem. It is required to secure the network that Ceph Object Gateways uses to connect with the Vault agent, for example, the Vault agent listens to only the localhost.

Additional Resources

  • See the Vault Agent documentation on Vault’s project site for more information.

8.4.3. Namespaces for Vault

Using HashiCorp Vault as an enterprise service provides centralized management for isolated namespaces that teams within an organization can use. These isolated namespace environments are known as tenants, and teams within an organization can utilize these tenants to isolate their policies, secrets, and identities from other teams. The namespace features of Vault help support secure multi-tenancy from within a single infrastructure.

Additional Resources

8.4.4. Transit engine compatibility support

There is compatibility support for the previous versions of Ceph which used the Transit engine as a simple key store. You can use the compat option in the Transit engine to configure the compatibility support. You can disable previous support with the following command:

Example

[ceph: root@host03 /]# ceph config set client.rgw rgw_crypt_vault_secret_engine transit compat=0

Note

This is the default for future versions and you can use the current version for new installations.

The normal default with the current version is:

Example

[ceph: root@host03 /]# ceph config set client.rgw rgw_crypt_vault_secret_engine transit compat=1

This enables the new engine for newly created objects and still allows the old engine to be used for the old objects. To access old and new objects, the Vault token must have both the old and new transit policies.

You can force use only the old engine with the following command:

Example

[ceph: root@host03 /]# ceph config set client.rgw rgw_crypt_vault_secret_engine transit compat=2

This mode is selected by default if the Vault ends in export/encryption-key.

Important

After configuring the client.rgw options, you need to restart the Ceph Object Gateway daemons for the new values to take effect.

Additional Resources

  • See the Vault Agent documentation on Vault’s project site for more information.

8.4.5. Creating token policies for Vault

A token policy specifies the powers that all Vault tokens have. One token can have multiple policies. You should use the required policy for the configuration.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Installation of the HashiCorp Vault software.
  • Root-level access to the HashiCorp Vault node.

Procedure

  1. Create a token policy:

    1. For the Key/Value secret engine:

      Example

      [root@vault ~]# vault policy write rgw-kv-policy -<<EOF
       path "secret/data/*" {
         capabilities = ["read"]
       }
      EOF

    2. For the Transit engine:

      Example

      [root@vault ~]# vault policy write rgw-transit-policy -<<EOF
        path "transit/keys/*" {
          capabilities = [ "create", "update" ]
          denied_parameters = {"exportable" = [], "allow_plaintext_backup" = [] }
        }
      
        path "transit/keys/*" {
          capabilities = ["read", "delete"]
        }
      
        path "transit/keys/" {
          capabilities = ["list"]
        }
      
        path "transit/keys/+/rotate" {
          capabilities = [ "update" ]
        }
      
        path "transit/*" {
          capabilities = [ "update" ]
        }
      EOF

      Note

      If you have used the Transit secret engine on an older version of Ceph, the token policy is:

      Example

      [root@vault ~]# vault policy write old-rgw-transit-policy -<<EOF
        path "transit/export/encryption-key/*" {
          capabilities = ["read"]
        }
      EOF

If you are using both SSE-KMS and SSE-S3, you should point each to separate containers. You could either use separate Vault instances or separately mount transit instances or different branches under a common transit point. If you are not using separate Vault instances, you can point SSE-KMS or SSE-S3 to serparate containers using rgw_crypt_vault_prefix and rgw_crypt_sse_s3_vault_prefix. When granting Vault permissions to SSE-KMS bucket owners, you should not give them permission to SSE-S3 keys; only Ceph should have access to the SSE-S3 keys.

8.4.6. Configuring the Ceph Object Gateway to use SSE-KMS with Vault

To configure the Ceph Object Gateway to use the HashiCorp Vault with SSE-KMS for key management, it must be set as the encryption key store. Currently, the Ceph Object Gateway supports two different secret engines, and two different authentication methods.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Installation of the Ceph Object Gateway software.
  • Root-level access to a Ceph Object Gateway node.

Procedure

  1. Use the ceph config set client.rgw OPTION VALUE command to enable Vault as the encryption key store:

    Syntax

    ceph config set client.rgw rgw_crypt_s3_kms_backend vault

  2. Add the following options and values:

    Syntax

    ceph config set client.rgw rgw_crypt_vault_auth agent
    ceph config set client.rgw rgw_crypt_vault_addr http://VAULT_SERVER:8100

  3. Customize the policy as per the use case.
  4. Get the role-id:

    Syntax

    vault read auth/approle/role/rgw-ap/role-id -format=json | \ jq -r .data.role_id > PATH_TO_FILE

  5. Get the secret-id:

    Syntax

    vault read auth/approle/role/rgw-ap/role-id -format=json | \ jq -r .data.secret_id > PATH_TO_FILE

  6. Create the configuration for the Vault agent:

    Example

    pid_file = "/run/kv-vault-agent-pid"
    auto_auth {
      method "AppRole" {
        mount_path = "auth/approle"
        config = {
          role_id_file_path ="/root/vault_configs/kv-agent-role-id"
          secret_id_file_path ="/root/vault_configs/kv-agent-secret-id"
          remove_secret_id_file_after_reading ="false"
        }
      }
    }
    cache {
      use_auto_auth_token = true
    }
    listener "tcp" {
      address = "127.0.0.1:8100"
      tls_disable = true
    }
    vault {
      address = "http://10.8.128.9:8200"
    }

  7. Use systemctl to run the persistent daemon:

    Example

    [root@host03 ~]# /usr/local/bin/vault agent -config=/usr/local/etc/vault/rgw-agent.hcl

  8. A token file is populated with a valid token when the Vault agent runs.
  9. Select a Vault secret engine, either Key/Value or Transit.

    1. If using Key/Value, then add the following line:

      Example

      [ceph: root@host03 /]# ceph config set client.rgw rgw_crypt_vault_secret_engine kv

    2. If using Transit, then add the following line:

      Example

      [ceph: root@host03 /]# ceph config set client.rgw rgw_crypt_vault_secret_engine transit

  10. Use the ceph config set client.rgw OPTION VALUE command to set the Vault namespace to retrieve the encryption keys:

    Example

    [ceph: root@host03 /]# ceph config set client.rgw rgw_crypt_vault_namespace testnamespace1

  11. Restrict where the Ceph Object Gateway retrieves the encryption keys from the Vault by setting a path prefix:

    Example

    [ceph: root@host03 /]# ceph config set client.rgw rgw_crypt_vault_prefix /v1/secret/data

    1. For exportable Transit keys, set the prefix path as follows:

      Example

      [ceph: root@host03 /]# ceph config set client.rgw rgw_crypt_vault_prefix /v1/transit/export/encryption-key

      Assuming the domain name of the Vault server is vault-server, the Ceph Object Gateway will fetch encrypted transit keys from the following URL:

      Example

      http://vault-server:8200/v1/transit/export/encryption-key

  12. Restart the Ceph Object Gateway daemons.

    1. To restart the Ceph Object Gateway on an individual node in the storage cluster:

      Syntax

      systemctl restart ceph-CLUSTER_ID@SERVICE_TYPE.ID.service

      Example

      [root@host03 ~]# systemctl restart ceph-c4b34c6f-8365-11ba-dc31-529020a7702d@rgw.realm.zone.host01.gwasto.service

    2. To restart the Ceph Object Gateways on all nodes in the storage cluster:

      Syntax

      ceph orch restart SERVICE_TYPE

      Example

      [ceph: root@host03 /]# ceph orch restart rgw

Additional Resources

8.4.7. Configuring the Ceph Object Gateway to use SSE-S3 with Vault

To configure the Ceph Object Gateway to use the HashiCorp Vault with SSE-S3 for key management, it must be set as the encryption key store. Currently, the Ceph Object Gateway only uses agent authentication method.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Installation of the Ceph Object Gateway software.
  • Root-level access to a Ceph Object Gateway node.

Procedure

  1. Log into the Cephadm shell

    Example

    [root@host01 ~]# cephadm shell

  2. Enable Vault as the secrets engine to retrieve SSE-S3 encryption keys:

    Syntax

    ceph config set client.rgw rgw_crypt_sse_s3_backend vault

  3. To set the authentication method to use with SSE-S3 and Vault, configure the following settings:

    Syntax

    ceph config set client.rgw rgw_crypt_sse_s3_vault_auth agent
    ceph config set client.rgw rgw_crypt_sse_s3_vault_addr http://VAULT_AGENT:VAULT_AGENT_PORT

    Example

    [ceph: root@host01 ~]# ceph config set client.rgw rgw_crypt_sse_s3_vault_auth agent
    [ceph: root@host01 ~]# ceph config set client.rgw rgw_crypt_sse_s3_vault_addr http://vaultagent:8100

    1. Customize the policy as per your use case to set up a Vault agent.
    2. Get the role-id:

      Syntax

      vault read auth/approle/role/rgw-ap/role-id -format=json | \ jq -r .rgw-ap-role-id > PATH_TO_FILE

    3. Get the secret-id:

      Syntax

      vault read auth/approle/role/rgw-ap/role-id -format=json | \ jq -r .rgw-ap-secret-id > PATH_TO_FILE

    4. Create the configuration for the Vault agent:

      Example

      pid_file = "/run/rgw-vault-agent-pid"
      auto_auth {
        method "AppRole" {
          mount_path = "auth/approle"
          config = {
            role_id_file_path ="/usr/local/etc/vault/.rgw-ap-role-id"
            secret_id_file_path ="/usr/local/etc/vault/.rgw-ap-secret-id"
            remove_secret_id_file_after_reading ="false"
          }
        }
      }
      cache {
        use_auto_auth_token = true
      }
      listener "tcp" {
        address = "127.0.0.1:8100"
        tls_disable = true
      }
      vault {
        address = "https://vaultserver:8200"
      }

    5. Use systemctl to run the persistent daemon:

      Example

      [root@host01 ~]# /usr/local/bin/vault agent -config=/usr/local/etc/vault/rgw-agent.hcl

    6. A token file is populated with a valid token when the Vault agent runs.
  4. Set the Vault secret engine to use to retrieve encryption keys, either Key/Value or Transit.

    1. If using Key/Value, set the following:

      Example

      [ceph: root@host01 /]# ceph config set client.rgw rgw_crypt_sse_s3_vault_secret_engine kv

    2. If using Transit, set the following:

      Example

      [ceph: root@host01 /]# ceph config set client.rgw rgw_crypt_sse_s3_vault_secret_engine transit

  5. Optional: Configure the Ceph Object Gateway to access Vault within a particular namespace to retrieve the encryption keys:

    Example

    [ceph: root@host01 /]# ceph config set client.rgw rgw_crypt_sse_s3_vault_namespace company/testnamespace1

    Note

    Vault namespaces allow teams to operate within isolated environments known as tenants. The Vault namespaces feature is only available in the Vault Enterprise version.

  6. Optional: Restrict access to a particular subset of the Vault secret space by setting a URL path prefix, where the Ceph Object Gateway retrieves the encryption keys from:

    1. If using Key/Value, set the following:

      Example

      [ceph: root@host01 /]# ceph config set client.rgw rgw_crypt_sse_s3_vault_prefix /v1/secret/data

    2. If using Transit, set the following:

      Example

      [ceph: root@host01 /]# ceph config set client.rgw rgw_crypt_sse_s3_vault_prefix /v1/transit

      Assuming the domain name of the Vault server is vaultserver, the Ceph Object Gateway will fetch encrypted transit keys from the following URL:

      Example

      http://vaultserver:8200/v1/transit

  7. Optional: To use custom SSL certification to authenticate with Vault, configure the following settings:

    Syntax

    ceph config set client.rgw rgw_crypt_sse_s3_vault_verify_ssl true
    ceph config set client.rgw rgw_crypt_sse_s3_vault_ssl_cacert PATH_TO_CA_CERTIFICATE
    ceph config set client.rgw rgw_crypt_sse_s3_vault_ssl_clientcert PATH_TO_CLIENT_CERTIFICATE
    ceph config set client.rgw rgw_crypt_sse_s3_vault_ssl_clientkey PATH_TO_PRIVATE_KEY

    Example

    [ceph: root@host01 /]# ceph config set client.rgw rgw_crypt_sse_s3_vault_verify_ssl true
    [ceph: root@host01 /]# ceph config set client.rgw rgw_crypt_sse_s3_vault_ssl_cacert /etc/ceph/vault.ca
    [ceph: root@host01 /]# ceph config set client.rgw rgw_crypt_sse_s3_vault_ssl_clientcert /etc/ceph/vault.crt
    [ceph: root@host03 /]# ceph config set client.rgw rgw_crypt_sse_s3_vault_ssl_clientkey /etc/ceph/vault.key

  8. Restart the Ceph Object Gateway daemons.

    1. To restart the Ceph Object Gateway on an individual node in the storage cluster:

      Syntax

      systemctl restart ceph-CLUSTER_ID@SERVICE_TYPE.ID.service

      Example

      [root@host01 ~]# systemctl restart ceph-c4b34c6f-8365-11ba-dc31-529020a7702d@rgw.realm.zone.host01.gwasto.service

    2. To restart the Ceph Object Gateways on all nodes in the storage cluster:

      Syntax

      ceph orch restart SERVICE_TYPE

      Example

      [ceph: root@host01 /]# ceph orch restart rgw

Additional Resources

8.4.8. Creating a key using the kv engine

Configure the HashiCorp Vault Key/Value secret engine (kv) so you can create a key for use with the Ceph Object Gateway. Secrets are stored as key-value pairs in the kv secret engine.

Important

Keys for server-side encryption must be 256-bits long and encoded using base64.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Installation of the HashiCorp Vault software.
  • Root-level access to the HashiCorp Vault node.

Procedure

  1. Enable the Key/Value version 2 secret engine:

    Example

    vault secrets enable -path secret kv-v2

  2. Create a new key:

    Syntax

    vault kv put secret/PROJECT_NAME/BUCKET_NAME key=$(openssl rand -base64 32)

    Example

    [root@vault ~]# vault kv put secret/myproject/mybucketkey key=$(openssl rand -base64 32)
    
    ====== Metadata ======
    Key              Value
    ---              ----
    created_time     2020-02-21T17:01:09.095824999Z
    deletion_time    n/a
    destroyed        false
    version          1

8.4.9. Creating a key using the transit engine

Configure the HashiCorp Vault Transit secret engine (transit) so you can create a key for use with the Ceph Object Gateway. Creating keys with the Transit secret engine must be exportable in order to be used for server-side encryption with the Ceph Object Gateway.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Installation of the HashiCorp Vault software.
  • Root-level access to the HashiCorp Vault node.

Procedure

  1. Enable the Transit secret engine:

    [root@vault ~]# vault secrets enable transit
  2. Create a new exportable key:

    Syntax

    vault write -f transit/keys/BUCKET_NAME exportable=true

    Example

    [root@vault ~]# vault write -f transit/keys/mybucketkey exportable=true

    Note

    By default the above command creates a aes256-gcm96 type key.

  3. Verify the creation of the key:

    Syntax

    vault read transit/export/encryption-key/BUCKET_NAME/VERSION_NUMBER

    Example

    [root@vault ~]# vault read transit/export/encryption-key/mybucketkey/1
    
    Key     Value
    ---     -----
    keys    map[1:-gbTI9lNpqv/V/2lDcmH2Nq1xKn6FPDWarCmFM2aNsQ=]
    name    mybucketkey
    type    aes256-gcm96

    Note

    Providing the full key path, including the key version, is required.

8.4.10. Uploading an object using AWS and the Vault

When uploading an object to the Ceph Object Gateway, the Ceph Object Gateway will fetch the key from the Vault, and then encrypt and store the object in a bucket. When a request is made to download the object, the Ceph Object Gateway will automatically retrieve the corresponding key from the Vault and decrypt the object. To upload an object, the Ceph object Gateway fetches the key from the Vault and then encrypts the object and stores it in the bucket. The Ceph Object Gateway retrieves the corresponding key from the Vault and decrypts the object when there is a request to download the object.

Note

The URL is constructed using the base address, set by the rgw_crypt_vault_addr option, and the path prefix, set by the rgw_crypt_vault_prefix option.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Installation of the Ceph Object Gateway software.
  • Installation of the HashiCorp Vault software.
  • Access to a Ceph Object Gateway client node.
  • Access to Amazon Web Services (AWS).

Procedure

  1. Upload an object using the AWS command line client and provide the Secure Side Encryption (SSE) key ID in the request:

    1. For the Key/Value secret engine:

      Example (with SSE-KMS)

      [user@client ~]$ aws --endpoint=http://radosgw:8000 s3 cp plaintext.txt s3://mybucket/encrypted.txt --sse=aws:kms --sse-kms-key-id myproject/mybucketkey

      Example (with SSE-S3)

      [user@client ~]$ aws s3api --endpoint http://rgw_host:8080 put-object --bucket my-bucket --key obj1 --body test_file_to_upload --server-side-encryption AES256

      Note

      In the example, the Ceph Object Gateway would fetch the secret from http://vault-server:8200/v1/secret/data/myproject/mybucketkey

    2. For the Transit engine:

      Example (with SSE-KMS)

      [user@client ~]$ aws --endpoint=http://radosgw:8000 s3 cp plaintext.txt s3://mybucket/encrypted.txt --sse=aws:kms --sse-kms-key-id mybucketkey

      Example (with SSE-S3)

      [user@client ~]$ aws s3api --endpoint http://rgw_host:8080 put-object --bucket my-bucket --key obj1 --body test_file_to_upload --server-side-encryption AES256

      Note

      In the example, the Ceph Object Gateway would fetch the secret from http://vaultserver:8200/v1/transit/mybucketkey

Additional Resources

  • See the Install Vault documentation on Vault’s project site for more information.

8.5. The Ceph Object Gateway and multi-factor authentication

As a storage administrator, you can manage time-based one time password (TOTP) tokens for Ceph Object Gateway users.

8.5.1. Multi-factor authentication

When a bucket is configured for object versioning, a developer can optionally configure the bucket to require multi-factor authentication (MFA) for delete requests. Using MFA, a time-based one time password (TOTP) token is passed as a key to the x-amz-mfa header. The tokens are generated with virtual MFA devices like Google Authenticator, or a hardware MFA device like those provided by Gemalto.

Use radosgw-admin to assign time-based one time password tokens to a user. You must set a secret seed and a serial ID. You can also use radosgw-admin to list, remove, and resynchronize tokens.

Important

In a multi-site environment it is advisable to use different tokens for different zones, because, while MFA IDs are set on the user’s metadata, the actual MFA one time password configuration resides on the local zone’s OSDs.

Table 8.1. Terminology
TermDescription

TOTP

Time-based One Time Password.

Token serial

A string that represents the ID of a TOTP token.

Token seed

The secret seed that is used to calculate the TOTP. It can be hexadecimal or base32.

TOTP seconds

The time resolution used for TOTP generation.

TOTP window

The number of TOTP tokens that are checked before and after the current token when validating tokens.

TOTP pin

The valid value of a TOTP token at a certain time.

8.5.2. Creating a seed for multi-factor authentication

To set up multi-factor authentication (MFA), you must create a secret seed for use by the one-time password generator and the back-end MFA system.

Prerequisites

  • A Linux system.
  • Access to the command line shell.

Procedure

  1. Generate a 30 character seed from the urandom Linux device file and store it in the shell variable SEED:

    Example

    [user@host01 ~]$ SEED=$(head -10 /dev/urandom | sha512sum | cut -b 1-30)

  2. Print the seed by running echo on the SEED variable:

    Example

    [user@host01 ~]$ echo $SEED
    492dedb20cf51d1405ef6a1316017e

    Configure the one-time password generator and the back-end MFA system to use the same seed.

Additional Resources

8.5.3. Creating a new multi-factor authentication TOTP token

Create a new multi-factor authentication (MFA) time-based one time password (TOTP) token.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Ceph Object Gateway is installed.
  • You have root access on a Ceph Monitor node.
  • A secret seed for the one-time password generator and Ceph Object Gateway MFA was generated.

Procedure

  • Create a new MFA TOTP token:

    Syntax

    radosgw-admin mfa create --uid=USERID --totp-serial=SERIAL --totp-seed=SEED --totp-seed-type=SEED_TYPE --totp-seconds=TOTP_SECONDS --totp-window=TOTP_WINDOW

    Set USERID to the user name to set up MFA on, set SERIAL to a string that represents the ID for the TOTP token, and set SEED to a hexadecimal or base32 value that is used to calculate the TOTP. The following settings are optional: Set the SEED_TYPE to hex or base32, set TOTP_SECONDS to the timeout in seconds, or set TOTP_WINDOW to the number of TOTP tokens to check before and after the current token when validating tokens.

    Example

    [root@host01 ~]# radosgw-admin mfa create --uid=johndoe --totp-serial=MFAtest --totp-seed=492dedb20cf51d1405ef6a1316017e

Additional Resources

8.5.4. Test a multi-factor authentication TOTP token

Test a multi-factor authentication (MFA) time-based one time password (TOTP) token.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Ceph Object Gateway is installed.
  • You have root access on a Ceph Monitor node.
  • An MFA TOTP token was created using radosgw-admin mfa create.

Procedure

  • Test the TOTP token PIN to verify that TOTP functions correctly:

    Syntax

    radosgw-admin mfa check --uid=USERID --totp-serial=SERIAL --totp-pin=PIN

    Set USERID to the user name MFA is set up on, set SERIAL to the string that represents the ID for the TOTP token, and set PIN to the latest PIN from the one-time password generator.

    Example

    [root@host01 ~]# radosgw-admin mfa check  --uid=johndoe --totp-serial=MFAtest --totp-pin=870305
    ok

    If this is the first time you have tested the PIN, it may fail. If it fails, resynchronize the token. See Resynchronizing a multi-factor authentication token in the Red Hat Ceph Storage Object Gateway Configuration and Administration Guide.

Additional Resources

8.5.5. Resynchronizing a multi-factor authentication TOTP token

Resynchronize a multi-factor authentication (MFA) time-based one time password token.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Ceph Object Gateway is installed.
  • You have root access on a Ceph Monitor node.
  • An MFA TOTP token was created using radosgw-admin mfa create.

Procedure

  1. Resynchronize a multi-factor authentication TOTP token in case of time skew or failed checks.

    This requires passing in two consecutive pins: the previous pin, and the current pin.

    Syntax

    radosgw-admin mfa resync --uid=USERID --totp-serial=SERIAL --totp-pin=PREVIOUS_PIN --totp=pin=CURRENT_PIN

    Set USERID to the user name MFA is set up on, set SERIAL to the string that represents the ID for the TOTP token, set PREVIOUS_PIN to the user’s previous PIN, and set CURRENT_PIN to the user’s current PIN.

    Example

    [root@host01 ~]# radosgw-admin mfa resync --uid=johndoe --totp-serial=MFAtest --totp-pin=802021 --totp-pin=439996

  2. Verify the token was successfully resynchronized by testing a new PIN:

    Syntax

    radosgw-admin mfa check --uid=USERID --totp-serial=SERIAL --totp-pin=PIN

    Set USERID to the user name MFA is set up on, set SERIAL to the string that represents the ID for the TOTP token, and set PIN to the user’s PIN.

    Example

    [root@host01 ~]# radosgw-admin mfa check  --uid=johndoe --totp-serial=MFAtest --totp-pin=870305
    ok

Additional Resources

8.5.6. Listing multi-factor authentication TOTP tokens

List all multi-factor authentication (MFA) time-based one time password (TOTP) tokens that a particular user has.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Ceph Object Gateway is installed.
  • You have root access on a Ceph Monitor node.
  • An MFA TOTP token was created using radosgw-admin mfa create.

Procedure

  • List MFA TOTP tokens:

    Syntax

    radosgw-admin mfa list --uid=USERID

    Set USERID to the user name MFA is set up on.

    Example

    [root@host01 ~]# radosgw-admin mfa list --uid=johndoe
    {
        "entries": [
            {
                "type": 2,
                "id": "MFAtest",
                "seed": "492dedb20cf51d1405ef6a1316017e",
                "seed_type": "hex",
                "time_ofs": 0,
                "step_size": 30,
                "window": 2
            }
        ]
    }

Additional Resources

8.5.7. Display a multi-factor authentication TOTP token

Display a specific multi-factor authentication (MFA) time-based one time password (TOTP) token by specifying its serial.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Ceph Object Gateway is installed.
  • You have root access on a Ceph Monitor node.
  • An MFA TOTP token was created using radosgw-admin mfa create.

Procedure

  • Show the MFA TOTP token:

    Syntax

    radosgw-admin mfa get --uid=USERID --totp-serial=SERIAL

    Set USERID to the user name MFA is set up on and set SERIAL to the string that represents the ID for the TOTP token.

Additional Resources

8.5.8. Deleting a multi-factor authentication TOTP token

Delete a multi-factor authentication (MFA) time-based one time password (TOTP) token.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Ceph Object Gateway is installed.
  • You have root access on a Ceph Monitor node.
  • An MFA TOTP token was created using radosgw-admin mfa create.

Procedure

  1. Delete an MFA TOTP token:

    Syntax

    radosgw-admin mfa remove --uid=USERID --totp-serial=SERIAL

    Set USERID to the user name MFA is set up on and set SERIAL to the string that represents the ID for the TOTP token.

    Example

    [root@host01 ~]# radosgw-admin mfa remove --uid=johndoe --totp-serial=MFAtest

  2. Verify the MFA TOTP token was deleted:

    Syntax

    radosgw-admin mfa get --uid=USERID --totp-serial=SERIAL

    Set USERID to the user name MFA is set up on and set SERIAL to the string that represents the ID for the TOTP token.

    Example

    [root@host01 ~]# radosgw-admin mfa get --uid=johndoe --totp-serial=MFAtest
    MFA serial id not found

Additional Resources

Red Hat logoGithubRedditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

© 2024 Red Hat, Inc.