Red Hat Summit Red Hat SummitSupportConsoleDevelopersStart a trialContact

Manage Secrets with OpenStack Key Manager

Red Hat OpenStack Platform 13

How to integrate OpenStack Key Manager (Barbican) with your OpenStack deployment.

OpenStack Documentation Team

Abstract

How to integrate OpenStack Key Manager (Barbican) with your OpenStack deployment.

Making open source more inclusive
Copy link

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. We are beginning with these four terms: master, slave, blacklist, and whitelist. Because of the enormity of this endeavor, these changes will be implemented gradually over several upcoming releases. For more details, see our CTO Chris Wright’s message.

Chapter 1. Overview
Copy link

OpenStack Key Manager (barbican) is the secrets manager for Red Hat OpenStack Platform. You can use the barbican API and command line to centrally manage the certificates, keys, and passwords used by OpenStack services. Barbican currently supports the following use cases described in this guide:

  • Symmetric encryption keys - used for Block Storage (cinder) volume encryption, ephemeral disk encryption, and Object Storage (swift) encryption, among others.
  • Asymmetric keys and certificates - used for glance image signing and verification, among others.

In this release, barbican offers integration with the Block Storage (cinder) and Compute (nova) components.

Chapter 2. Choosing a backend
Copy link

Secrets (such as certificates, API keys, and passwords) can either be stored as an encrypted blob in the barbican database, or directly in a secure storage system.

To store the secrets as an encrypted blob in the barbican database, the following options are available.

  • Simple crypto plugin - The simple crypto plugin is enabled by default and uses a single symmetric key to encrypt the blob of secrets. This key is stored in plain text in the barbican.conf file.
Note

The simple crypto plugin is currently the only plugin supported by Red Hat.

  • PKCS#11 crypto plugin - The PKCS#11 crypto plugin encrypts secrets with project-specific key encryption keys (KEK), which are stored in the barbican database. These project-specific KEKs are encrypted by a master KEK, which is stored in a hardware security module (HSM). All encryption and decryption operations take place in the HSM, rather than in-process memory. The PKCS#11 plugin communicates with the HSM through the PKCS#11 protocol. Because the encryption is done in secure hardware, and a different KEK is used per project, this option is more secure than the simple crypto plugin.
Note

Regarding high availability (HA) options: The barbican service runs within Apache and is configured by director to use HAProxy for high availability. HA options for the back end layer will depend on the back end being used. For example, for simple crypto, all the barbican instances have the same encryption key in the config file, resulting in a simple HA configuration.

2.1. Migrating between backends
Copy link

Barbican allows you to define a different backend for a project. If no mapping exists for a project, then secrets are stored in the global default backend. This means that multiple backends can be configured, but there must be at least one global backend defined. The heat templates supplied for the different backends contain the parameters that set each backend as the default.

If you do store secrets in a certain backend and then decide to migrate to a new backend, you can keep the old backend available while enabling the new backend as the global default (or as a project-specific backend). As a result, the old secrets remain available through the old backend.

Chapter 3. Installing Barbican
Copy link

Barbican is not enabled by default in Red Hat OpenStack Platform. This procedure describes how you can deploy barbican in an existing OpenStack deployment. Barbican runs as a containerized service, so this procedure also describes how to prepare and upload the new container images:

Note

This procedure configures barbican to use the simple_crypto backend. Additional backends are available, such as PKCS11 and DogTag, however they are not supported in this release.

  1. On the undercloud node, create an environment file for barbican. This will instruct director to install barbican (when its included in openstack overcloud deploy […​]) 

    $ cat /home/stack/configure-barbican.yaml
	parameter_defaults:
	  BarbicanSimpleCryptoGlobalDefault: true
    Copy to Clipboard Toggle word wrap
    • BarbicanSimpleCryptoGlobalDefault - Sets this plugin as the global default plugin.

    • Further options are also configurable:

      • BarbicanPassword - Sets a password for the barbican service account.
      • BarbicanWorkers - Sets the number of workers for barbican::wsgi::apache. Uses '%{::processorcount}' by default.
      • BarbicanDebug - Enables debugging.
      • BarbicanPolicies - Defines policies to configure for barbican. Uses a hash value, for example: { barbican-context_is_admin: { key: context_is_admin, value: 'role:admin' } }. This entry is then added to /etc/barbican/policy.json. Policies are described in detail in a later section.
      • BarbicanSimpleCryptoKek - The Key Encryption Key (KEK) is generated by director, if none is specified.

  2. This step prepares new container images for barbican. You will need to include the configure-barbican.yaml and all the relevant template files. Change the following example to suit your deployment: 

    $ openstack overcloud container image prepare \
	--namespace example.lab.local:5000/rhosp13 \
	--tag 2018-06-06.1 \
	--push-destination 192.168.100.1:8787 \
	--output-images-file ~/container-images-with-barbican.yaml \
	-e /home/stack/virt/config_lvm.yaml \
	-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
	-e /home/stack/virt/network/network-environment.yaml \
	-e /home/stack/virt/hostnames.yml \
	-e /home/stack/virt/nodes_data.yaml \
	-e /home/stack/virt/extra_templates.yaml \
	-e /home/stack/virt/docker-images.yaml \
	-e /usr/share/openstack-tripleo-heat-templates/environments/services/barbican.yaml \
	-e /usr/share/openstack-tripleo-heat-templates/environments/barbican-backend-simple-crypto.yaml \
	-e /home/stack/configure-barbican.yaml
    Copy to Clipboard Toggle word wrap

  3. Upload the new container images to the undercloud registry: 

    $ openstack overcloud container image upload --debug --config-file container-images-with-barbican.yaml
    Copy to Clipboard Toggle word wrap

  4. Prepare the new environment file: 

    $ openstack overcloud container image prepare \
  --tag 2018-06-06.1 \
  --namespace 192.168.100.1:8787/rhosp13 \
  --output-env-file ~/container-parameters-with-barbican.yaml \
  -e /home/stack/virt/config_lvm.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
  -e /home/stack/virt/network/network-environment.yaml \
  -e /home/stack/virt/hostnames.yml \
  -e /home/stack/virt/nodes_data.yaml \
  -e /home/stack/virt/extra_templates.yaml \
  -e /home/stack/virt/docker-images.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/services/barbican.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/barbican-backend-simple-crypto.yaml \
  -e /home/stack/configure-barbican.yaml
    Copy to Clipboard Toggle word wrap

  5. To apply these changes to your deployment: update the overcloud and specify all the heat template files that you used in your previous openstack overcloud deploy […​]. For example: 

    $ openstack overcloud deploy \
    --timeout 100 \
    --templates /usr/share/openstack-tripleo-heat-templates \
    --stack overcloud \
    --libvirt-type kvm \
    --ntp-server clock.redhat.com \
    -e /home/stack/virt/config_lvm.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
    -e /home/stack/virt/network/network-environment.yaml \
    -e /home/stack/virt/hostnames.yml \
    -e /home/stack/virt/nodes_data.yaml \
    -e /home/stack/virt/extra_templates.yaml \
    -e /home/stack/container-parameters-with-barbican.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/services/barbican.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/barbican-backend-simple-crypto.yaml \
    -e /home/stack/configure-barbican.yaml \
    --log-file overcloud_deployment_38.log
    Copy to Clipboard Toggle word wrap

3.1. Add users to the creator role on Overcloud
Copy link

Users must be members of the creator role in order to create and edit barbican secrets, or to create encrypted volumes that store their secret in barbican.

  1. Retrieve the id of the creator role: 

    openstack role show creator
+-----------+----------------------------------+
| Field     | Value                            |
+-----------+----------------------------------+
| domain_id | None                             |
| id        | 4e9c560c6f104608948450fbf316f9d7 |
| name      | creator                          |
+-----------+----------------------------------+
    Copy to Clipboard Toggle word wrap
    Note

    You will not see the creator role unless OpenStack Key Manager (barbican) is installed.

  2. Assign a user to the creator role and specify the relevant project. In this example, a user named user1 in the project_a project is added to the creator role: 

    openstack role add --user user1 --project project_a 4e9c560c6f104608948450fbf316f9d7
    Copy to Clipboard Toggle word wrap

3.1.1. Test barbican functionality
Copy link

This section describes how to test that barbican is working correctly.

  1. Create a test secret. For example: 

    $ openstack secret store --name testSecret --payload 'TestPayload'
+---------------+------------------------------------------------------------------------------------+
| Field         | Value                                                                              |
+---------------+------------------------------------------------------------------------------------+
| Secret href   | https://192.168.123.163/key-manager/v1/secrets/4cc5ffe0-eea2-449d-9e64-b664d574be53 |
| Name          | testSecret                                                                         |
| Created       | None                                                                               |
| Status        | None                                                                               |
| Content types | None                                                                               |
| Algorithm     | aes                                                                                |
| Bit length    | 256                                                                                |
| Secret type   | opaque                                                                             |
| Mode          | cbc                                                                                |
| Expiration    | None                                                                               |
+---------------+------------------------------------------------------------------------------------+
    Copy to Clipboard Toggle word wrap

  2. Retrieve the payload for the secret you just created: 

    openstack secret get https://192.168.123.163/key-manager/v1/secrets/4cc5ffe0-eea2-449d-9e64-b664d574be53 --payload
+---------+-------------+
| Field   | Value       |
+---------+-------------+
| Payload | TestPayload |
+---------+-------------+
    Copy to Clipboard Toggle word wrap

3.2. Understanding policies
Copy link

Barbican uses policies to determine which users are allowed to perform actions against the secrets, such as adding or deleting keys. To implement these controls, keystone project roles (such as creator you created earlier) are mapped to barbican internal permissions. As a result, users assigned to those project roles receive the corresponding barbican permissions.

3.2.1. Viewing the default policy
Copy link

The default policy is defined in code and typically does not require any amendments. You can view the default policy by generating it from the barbican source code:

  1. Perform the following steps on a non-production system, because additional components may be downloaded and installed. This example switches to the queens branch, so you must adapt this if using a different version. 

    git clone https://github.com/openstack/barbican
cd /home/stack/barbican
git checkout origin/stable/queens
tox -e genpolicy
    Copy to Clipboard Toggle word wrap

    This generates a policy file within a subdirectory that contains the default settings: etc/barbican/policy.yaml.sample. Note that this path refers to a subdirectory within the repository, not your system’s /etc directory. The contents of this file are explained in the following step.

  2. The policy.yaml.sample file you generated describes the policies used by barbican. The policy is implemented by four different roles that define how a user interacts with secrets and secret metadata. A user receives these permissions by being assigned to a particular role:

    • admin - Can delete, create/edit, and read secrets.
    • creator - Can create/edit, and read secrets. Can not delete secrets.
    • observer - Can only read data.

    • audit - Can only read metadata. Can not read secrets.

      For example, the following entries list the admin, observer, and creator keystone roles for each project. On the right, notice that they are assigned the role:admin, role:observer, and role:creator permissions: 

      #
#"admin": "role:admin"

#
#"observer": "role:observer"

#
#"creator": "role:creator"
      Copy to Clipboard Toggle word wrap

      These roles can also be grouped together by barbican. For example, rules that specify admin_or_creator can apply to members of either rule:admin or rule:creator.

  3. Further down in the file, there are secret:put and secret:delete actions. To their right, notice which roles have permissions to execute these actions. In the following example, secret:delete means that only admin and creator role members can delete secret entries. In addition, the rule states that users in the admin or creator role for that project can delete a secret in that project. The project match is defined by the secret_project_match rule, which is also defined in the policy. 

    secret:delete": "rule:admin_or_creator and rule:secret_project_match"
    Copy to Clipboard Toggle word wrap

Chapter 4. Managing secrets in barbican
Copy link

4.1. Listing secrets
Copy link

Secrets are identified by their URI, indicated as a href value. This example shows the secret you created in the previous step: 

$ openstack secret list
+------------------------------------------------------------------------------------+------+---------------------------+--------+-------------------------------------------+-----------+------------+-------------+------+------------+
| Secret href                                                                        | Name | Created                   | Status | Content types                             | Algorithm | Bit length | Secret type | Mode | Expiration |
+------------------------------------------------------------------------------------+------+---------------------------+--------+-------------------------------------------+-----------+------------+-------------+------+------------+
| https://192.168.123.169:9311/v1/secrets/24845e6d-64a5-4071-ba99-0fdd1046172e | None | 2018-01-22T02:23:15+00:00 | ACTIVE | {u'default': u'application/octet-stream'} | aes       |        256 | symmetric   | None | None       |
+------------------------------------------------------------------------------------+------+---------------------------+--------+-------------------------------------------+-----------+------------+-------------+------+------------+
Copy to Clipboard Toggle word wrap

4.2. Adding new secrets
Copy link

Create a test secret. For example: 

$ openstack secret store --name testSecret --payload 'TestPayload'
+---------------+------------------------------------------------------------------------------------+
| Field         | Value                                                                              |
+---------------+------------------------------------------------------------------------------------+
| Secret href   | https://192.168.123.163:9311/v1/secrets/ecc7b2a4-f0b0-47ba-b451-0f7d42bc1746 |
| Name          | testSecret                                                                         |
| Created       | None                                                                               |
| Status        | None                                                                               |
| Content types | None                                                                               |
| Algorithm     | aes                                                                                |
| Bit length    | 256                                                                                |
| Secret type   | opaque                                                                             |
| Mode          | cbc                                                                                |
| Expiration    | None                                                                               |
+---------------+------------------------------------------------------------------------------------+
Copy to Clipboard Toggle word wrap

4.3. Updating secrets
Copy link

You cannot change the payload of a secret (other than deleting the secret), but if you initially created a secret without specifying a payload, you can later add a payload to it by using the update function. For example: 

$ openstack secret update https://192.168.123.163:9311/v1/secrets/ca34a264-fd09-44a1-8856-c6e7116c3b16 'TestPayload-updated'
$
Copy to Clipboard Toggle word wrap

4.4. Deleting secrets
Copy link

You can delete a secret by specifying its URI. For example: 

$ openstack secret delete https://192.168.123.163:9311/v1/secrets/ecc7b2a4-f0b0-47ba-b451-0f7d42bc1746
$
Copy to Clipboard Toggle word wrap

4.5. Generate a symmetric key
Copy link

Symmetric keys are suitable for certain tasks, such as nova disk encryption and swift object encryption.

  1. Generate a new 256-bit key using order create and store it in barbican. For example: 

    $ openstack secret order create --name swift_key --algorithm aes --mode ctr --bit-length 256 --payload-content-type=application/octet-stream key
+----------------+-----------------------------------------------------------------------------------+
| Field          | Value                                                                             |
+----------------+-----------------------------------------------------------------------------------+
| Order href     | https://192.168.123.173:9311/v1/orders/043383fe-d504-42cf-a9b1-bc328d0b4832 |
| Type           | Key                                                                               |
| Container href | N/A                                                                               |
| Secret href    | None                                                                              |
| Created        | None                                                                              |
| Status         | None                                                                              |
| Error code     | None                                                                              |
| Error message  | None                                                                              |
+----------------+-----------------------------------------------------------------------------------+
    Copy to Clipboard Toggle word wrap
    • --mode - Generated keys can be configured to use a particular mode, such as ctr or cbc. For more information, see NIST SP 800-38A.

  2. View the details of the order to identify the location of the generated key, shown here as the Secret href value: 

    $ openstack secret order get https://192.168.123.173:9311/v1/orders/043383fe-d504-42cf-a9b1-bc328d0b4832
+----------------+------------------------------------------------------------------------------------+
| Field          | Value                                                                              |
+----------------+------------------------------------------------------------------------------------+
| Order href     | https://192.168.123.173:9311/v1/orders/043383fe-d504-42cf-a9b1-bc328d0b4832  |
| Type           | Key                                                                                |
| Container href | N/A                                                                                |
| Secret href    | https://192.168.123.173:9311/v1/secrets/efcfec49-b9a3-4425-a9b6-5ba69cb18719 |
| Created        | 2018-01-24T04:24:33+00:00                                                          |
| Status         | ACTIVE                                                                             |
| Error code     | None                                                                               |
| Error message  | None                                                                               |
+----------------+------------------------------------------------------------------------------------+
    Copy to Clipboard Toggle word wrap

  3. Retrieve the details of the secret: 

    $ openstack secret get https://192.168.123.173:9311/v1/secrets/efcfec49-b9a3-4425-a9b6-5ba69cb18719
+---------------+------------------------------------------------------------------------------------+
| Field         | Value                                                                              |
+---------------+------------------------------------------------------------------------------------+
| Secret href   | https://192.168.123.173:9311/v1/secrets/efcfec49-b9a3-4425-a9b6-5ba69cb18719 |
| Name          | swift_key                                                                     |
| Created       | 2018-01-24T04:24:33+00:00                                                          |
| Status        | ACTIVE                                                                             |
| Content types | {u'default': u'application/octet-stream'}                                          |
| Algorithm     | aes                                                                                |
| Bit length    | 256                                                                                |
| Secret type   | symmetric                                                                          |
| Mode          | ctr                                                                                |
| Expiration    | None                                                                               |
+---------------+------------------------------------------------------------------------------------+
    Copy to Clipboard Toggle word wrap

4.6. Backup and Restore Keys
Copy link

The process for backup and restore of encryption keys will vary depending on the type of back end:

Two separate components need to be backed up for simple crypto back end: the KEK and the database. It is recommended that you regularly test your backup and restore process.

4.6.1.1. Backup and restore the KEK
Copy link

For the simple crypto back end, you need to backup the barbican.conf file that contains the master KEK is written. This file must be backed up to a security hardened location. The actual data is stored in the Barbican database in an encrypted state, described in the next section.

  • To restore the key from a backup, you need to copy the restored barbican.conf over the existing barbican.conf.
4.6.1.2. Backup and restore the back end database
Copy link

This procedure describes how to backup and restore a barbican database for the simple crypto back end. To demonstrate this, you will generate a key and upload the secrets to barbican. You will then backup the barbican database, and delete the secrets you created. You will then restore the database and confirm that the secrets you created earlier have been recovered.

Note

Be sure you are also backing up the KEK, as this is also an important requirement. This is described in the previous section.

4.6.1.2.1. Create the test secret
Copy link

  1. On the overcloud, generate a new 256-bit key using order create and store it in barbican. For example: 

    (overcloud) [stack@undercloud-0 ~]$ openstack secret order create --name swift_key --algorithm aes --mode ctr --bit-length 256 --payload-content-type=application/octet-stream key
+----------------+-----------------------------------------------------------------------+
| Field          | Value                                                                 |
+----------------+-----------------------------------------------------------------------+
| Order href     | http://10.0.0.104:9311/v1/orders/2a11584d-851c-4bc2-83b7-35d04d3bae86 |
| Type           | Key                                                                   |
| Container href | N/A                                                                   |
| Secret href    | None                                                                  |
| Created        | None                                                                  |
| Status         | None                                                                  |
| Error code     | None                                                                  |
| Error message  | None                                                                  |
+----------------+-----------------------------------------------------------------------+
    Copy to Clipboard Toggle word wrap

  2. Create a test secret: 

    (overcloud) [stack@undercloud-0 ~]$ openstack secret store --name testSecret --payload 'TestPayload'
+---------------+------------------------------------------------------------------------+
| Field         | Value                                                                  |
+---------------+------------------------------------------------------------------------+
| Secret href   | http://10.0.0.104:9311/v1/secrets/93f62cfd-e008-401f-be74-bf057c88b04a |
| Name          | testSecret                                                             |
| Created       | None                                                                   |
| Status        | None                                                                   |
| Content types | None                                                                   |
| Algorithm     | aes                                                                    |
| Bit length    | 256                                                                    |
| Secret type   | opaque                                                                 |
| Mode          | cbc                                                                    |
| Expiration    | None                                                                   |
+---------------+------------------------------------------------------------------------+
    Copy to Clipboard Toggle word wrap

  3. Confirm that the secrets were created: 

    (overcloud) [stack@undercloud-0 ~]$ openstack secret list
+------------------------------------------------------------------------+------------+---------------------------+--------+-------------------------------------------+-----------+------------+-------------+------+------------+
| Secret href                                                            | Name       | Created                   | Status | Content types                             | Algorithm | Bit length | Secret type | Mode | Expiration |
+------------------------------------------------------------------------+------------+---------------------------+--------+-------------------------------------------+-----------+------------+-------------+------+------------+
| http://10.0.0.104:9311/v1/secrets/93f62cfd-e008-401f-be74-bf057c88b04a | testSecret | 2018-06-19T18:25:25+00:00 | ACTIVE | {u'default': u'text/plain'}               | aes       |        256 | opaque      | cbc  | None       |
| http://10.0.0.104:9311/v1/secrets/f664b5cf-5221-47e5-9887-608972a5fefb | swift_key  | 2018-06-19T18:24:40+00:00 | ACTIVE | {u'default': u'application/octet-stream'} | aes       |        256 | symmetric   | ctr  | None       |
+------------------------------------------------------------------------+------------+---------------------------+--------+-------------------------------------------+-----------+------------+-------------+------+------------+
    Copy to Clipboard Toggle word wrap
4.6.1.2.2. Backup the barbican database
Copy link

Run these steps while logged in to the controller-0 node.

Note

Only the user barbican has access to the barbican database. So the barbican user password is required to backup or restore the database.

  1. Retrieve barbican user password. For example: 

    [heat-admin@controller-0 ~]$ sudo grep -r "barbican::db::mysql::password" /etc/puppet/hieradata
/etc/puppet/hieradata/service_configs.json:    "barbican::db::mysql::password": "seDJRsMNRrBdFryCmNUEFPPev",
    Copy to Clipboard Toggle word wrap

  2. Backup the barbican database: 

    [heat-admin@controller-0 ~]$ mysqldump -u barbican -p"seDJRsMNRrBdFryCmNUEFPPev" barbican > barbican_db_backup.sql
    Copy to Clipboard Toggle word wrap

  3. Database backup is stored in /home/heat-admin 

    [heat-admin@controller-0 ~]$ ll
total 36
-rw-rw-r--. 1 heat-admin heat-admin 36715 Jun 19 18:31 barbican_db_backup.sql
    Copy to Clipboard Toggle word wrap
4.6.1.2.3. Delete the test secrets
Copy link

  1. On the overcloud, delete the secrets you created previously, and verify they no longer exist. For example: 

    (overcloud) [stack@undercloud-0 ~]$ openstack secret delete http://10.0.0.104:9311/v1/secrets/93f62cfd-e008-401f-be74-bf057c88b04a
(overcloud) [stack@undercloud-0 ~]$ openstack secret delete http://10.0.0.104:9311/v1/secrets/f664b5cf-5221-47e5-9887-608972a5fefb
(overcloud) [stack@undercloud-0 ~]$ openstack secret list

(overcloud) [stack@undercloud-0 ~]$
    Copy to Clipboard Toggle word wrap
4.6.1.2.4. Restore the databases
Copy link

Run these steps while logged in to the controller-0 node.

  1. Make sure you have the barbican database on the controller which grants access to the barbican user for database restoration: 

    [heat-admin@controller-0 ~]$ mysql -u barbican -p"seDJRsMNRrBdFryCmNUEFPPev"
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 3799
Server version: 10.1.20-MariaDB MariaDB Server

Copyright (c) 2000, 2016, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> SHOW DATABASES;
+--------------------+
| Database           |
+--------------------+
| barbican           |
| information_schema |
+--------------------+
2 rows in set (0.00 sec)

MariaDB [(none)]> exit
Bye
[heat-admin@controller-0 ~]$
    Copy to Clipboard Toggle word wrap

9) Restore the backup file to the barbican database:

+ 

[heat-admin@controller-0 ~]$ sudo mysql -u barbican -p"seDJRsMNRrBdFryCmNUEFPPev" barbican < barbican_db_backup.sql
[heat-admin@controller-0 ~]$
Copy to Clipboard Toggle word wrap
4.6.1.2.5. Verify the restore process
Copy link

  1. On the overcloud, verify that the test secrets were restored successfully: 

    (overcloud) [stack@undercloud-0 ~]$ openstack secret list
+------------------------------------------------------------------------+------------+---------------------------+--------+-------------------------------------------+-----------+------------+-------------+------+------------+
| Secret href                                                            | Name       | Created                   | Status | Content types                             | Algorithm | Bit length | Secret type | Mode | Expiration |
+------------------------------------------------------------------------+------------+---------------------------+--------+-------------------------------------------+-----------+------------+-------------+------+------------+
| http://10.0.0.104:9311/v1/secrets/93f62cfd-e008-401f-be74-bf057c88b04a | testSecret | 2018-06-19T18:25:25+00:00 | ACTIVE | {u'default': u'text/plain'}               | aes       |        256 | opaque      | cbc  | None       |
| http://10.0.0.104:9311/v1/secrets/f664b5cf-5221-47e5-9887-608972a5fefb | swift_key  | 2018-06-19T18:24:40+00:00 | ACTIVE | {u'default': u'application/octet-stream'} | aes       |        256 | symmetric   | ctr  | None       |
+------------------------------------------------------------------------+------------+---------------------------+--------+-------------------------------------------+-----------+------------+-------------+------+------------+
(overcloud) [stack@undercloud-0 ~]$
    Copy to Clipboard Toggle word wrap

Chapter 5. Encrypting cinder volumes
Copy link

You can use barbican to manage your Block Storage (cinder) encryption keys. This configuration uses LUKS to encrypt the disks attached to your instances, including boot disks. Key management is transparent to the user; when you create a new volume using luks as the encryption type, cinder generates a symmetric key secret for the volume and stores it in barbican. When booting the instance (or attaching an encrypted volume), nova retrieves the key from barbican and stores the secret locally as a Libvirt secret on the Compute node.

Important

Nova formats encrypted volumes during their first use if they are unencrypted. The resulting block device is then presented to the Compute node.

Note

If you intend to update any configuration files, be aware that certain OpenStack services now run within containers; this applies to keystone, nova, and cinder, among others. As a result, there are administration practices to consider:

  • Do not update any configuration file you might find on the physical node’s host operating system, for example, /etc/cinder/cinder.conf. The containerized service does not reference this file.

  • Do not update the configuration file running within the container. Changes are lost once you restart the container.

    Instead, if you must change containerized services, update the configuration file in /var/lib/config-data/puppet-generated/, which is used to generate the container.

    For example:

  • keystone: /var/lib/config-data/puppet-generated/keystone/etc/keystone/keystone.conf
  • cinder: /var/lib/config-data/puppet-generated/cinder/etc/cinder/cinder.conf

  • nova: /var/lib/config-data/puppet-generated/nova_libvirt/etc/nova/nova.conf

    Changes are applied after you restart the container.

  1. On nodes running the cinder-volume and nova-compute services, confirm that nova and cinder are both configured to use barbican for key management: 

    $ crudini --get /var/lib/config-data/puppet-generated/cinder/etc/cinder/cinder.conf key_manager backend
castellan.key_manager.barbican_key_manager.BarbicanKeyManager

$ crudini --get /var/lib/config-data/puppet-generated/nova_libvirt/etc/nova/nova.conf key_manager backend
castellan.key_manager.barbican_key_manager.BarbicanKeyManager
    Copy to Clipboard Toggle word wrap

  2. Create a volume template that uses encryption. When you create new volumes they can be modeled off the settings you define here: 

    $ openstack volume type create --encryption-provider nova.volume.encryptors.luks.LuksEncryptor --encryption-cipher aes-xts-plain64 --encryption-key-size 256 --encryption-control-location front-end LuksEncryptor-Template-256
+-------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field       | Value                                                                                                                                                                              |
+-------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| description | None                                                                                                                                                                               |
| encryption  | cipher='aes-xts-plain64', control_location='front-end', encryption_id='9df604d0-8584-4ce8-b450-e13e6316c4d3', key_size='256', provider='nova.volume.encryptors.luks.LuksEncryptor' |
| id          | 78898a82-8f4c-44b2-a460-40a5da9e4d59                                                                                                                                               |
| is_public   | True                                                                                                                                                                               |
| name        | LuksEncryptor-Template-256                                                                                                                                                         |
+-------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
    Copy to Clipboard Toggle word wrap

  3. Create a new volume and specify that it uses the LuksEncryptor-Template-256 settings:

    Note

    Ensure that the user creating the encrypted volume has the creator barbican role on the project. For more information, see the Grant user access to the creator role section. 

    $ openstack volume create --size 1 --type LuksEncryptor-Template-256 'Encrypted-Test-Volume'
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| attachments         | []                                   |
| availability_zone   | nova                                 |
| bootable            | false                                |
| consistencygroup_id | None                                 |
| created_at          | 2018-01-22T00:19:06.000000           |
| description         | None                                 |
| encrypted           | True                                 |
| id                  | a361fd0b-882a-46cc-a669-c633630b5c93 |
| migration_status    | None                                 |
| multiattach         | False                                |
| name                | Encrypted-Test-Volume                |
| properties          |                                      |
| replication_status  | None                                 |
| size                | 1                                    |
| snapshot_id         | None                                 |
| source_volid        | None                                 |
| status              | creating                             |
| type                | LuksEncryptor-Template-256           |
| updated_at          | None                                 |
| user_id             | 0e73cb3111614365a144e7f8f1a972af     |
+---------------------+--------------------------------------+
    Copy to Clipboard Toggle word wrap

    The resulting secret is automatically uploaded to the barbican backend.

  4. Use barbican to confirm that the disk encryption key is present. In this example, the timestamp matches the LUKS volume creation time: 

    $ openstack secret list
+------------------------------------------------------------------------------------+------+---------------------------+--------+-------------------------------------------+-----------+------------+-------------+------+------------+
| Secret href                                                                        | Name | Created                   | Status | Content types                             | Algorithm | Bit length | Secret type | Mode | Expiration |
+------------------------------------------------------------------------------------+------+---------------------------+--------+-------------------------------------------+-----------+------------+-------------+------+------------+
| https://192.168.123.169:9311/v1/secrets/24845e6d-64a5-4071-ba99-0fdd1046172e | None | 2018-01-22T02:23:15+00:00 | ACTIVE | {u'default': u'application/octet-stream'} | aes       |        256 | symmetric   | None | None       |
+------------------------------------------------------------------------------------+------+---------------------------+--------+-------------------------------------------+-----------+------------+-------------+------+------------+
    Copy to Clipboard Toggle word wrap

  5. Attach the new volume to an existing instance. For example: 

    $ openstack server add volume testInstance Encrypted-Test-Volume
    Copy to Clipboard Toggle word wrap

    The volume is then presented to the guest operating system and can be mounted using the built-in tools.

5.1. Migrate existing volume keys to Barbican
Copy link

Previously, deployments might have used ConfKeyManager to manage disk encryption keys. This meant that a fixed key was generated and then stored in the nova and cinder configuration files. The key IDs can be migrated to barbican using the following procedure. This utility works by scanning the databases for encryption_key_id entries within scope for migration to barbican. Each entry gets a new barbican key ID and the existing ConfKeyManager secret is retained.

Note

Previously, you could reassign ownership for volumes encrypted using ConfKeyManager. This is not possible for volumes that have their keys managed by barbican.

Note

Activating barbican will not break your existing keymgr volumes.

After it is enabled, the migration process runs automatically, but it requires some configuration, described in the next section. The actual migration runs in the cinder-volume and cinder-backup process, and you can track the progress in the cinder log files.

  • cinder-volume - migrates keys stored in cinder’s Volumes and Snapshots tables.
  • cinder-backup - migrates keys in the Backups table.

5.1.1. Overview of the migration steps
Copy link

  1. Deploy the barbican service.

  2. Add the creator role to the cinder service. For example: 

    #openstack role create creator
#openstack role add --user cinder creator  --project service
    Copy to Clipboard Toggle word wrap
  3. Restart the cinder-volume and cinder-backup services.
  4. cinder-volume and cinder-backup automatically begin the migration process.
  5. Monitor the logs for the message indicating migration has finished and check that no more volumes are using the ConfKeyManager all-zeros encryption key ID.
  6. Remove the fixed_key option from cinder.conf and nova.conf. You must determine which nodes have this setting configured.
  7. Remove the creator role from the cinder service.

5.1.2. Behavioral differences
Copy link

Barbican-managed encrypted volumes behave differently than volumes that use ConfKeyManager:

  • You cannot transfer ownership of encrypted volumes, because it is not currently possible to transfer ownership of the barbican secret.
  • Barbican is more restrictive about who is allowed to read and delete secrets, which can affect some cinder volume operations. For example, a user cannot attach, detach, or delete a different user’s volumes.

5.1.3. Reviewing the migration process
Copy link

This section describes how you can view the status of the migration tasks. After you start the process, one of these entries appears in the logs. This indicates whether the migration started correctly, or it identifies the issue it encountered:

  • Not migrating encryption keys because the ConfKeyManager is still in use.
  • Not migrating encryption keys because the ConfKeyManager's fixed_key is not in use.
  • Not migrating encryption keys because migration to the 'XXX' key_manager backend is not supported. - This message is unlikely to appear; it is a safety check to handle the code ever encountering another Key Manager backend other than barbican. This is because the code only supports one migration scenario: From ConfKeyManager to barbican.
  • Not migrating encryption keys because there are no volumes associated with this host. - This may occur when cinder-volume is running on multiple hosts, and a particular host has no volumes associated with it. This arises because every host is responsible for handling its own volumes.
  • Starting migration of ConfKeyManager keys.

  • Migrating volume <UUID> encryption key to Barbican - During migration, all of the host’s volumes are examined, and if a volume is still using the ConfKeyManager’s key ID (identified by the fact that it’s all zeros (00000000-0000-0000-0000-000000000000)), then this message appears.

    • For cinder-backup, this message uses slightly different capitalization: Migrating Volume [...] or Migrating Backup [...]

  • After each host examines all of its volumes, the host displays a summary status message: 

    `No volumes are using the ConfKeyManager's encryption_key_id.`
`No backups are known to be using the ConfKeyManager's encryption_key_id.`
    Copy to Clipboard Toggle word wrap

    You may also see the following entries:

    There are still %d volume(s) using the ConfKeyManager's all-zeros encryption key ID.There are still %d backup(s) using the ConfKeyManager’s all-zeros encryption key ID. Note that both of these messages can appear in the cinder-volume and cinder-backup logs. Whereas each service only handles the migration of its own entries, the service is aware of the the other’s status. As a result, cinder-volume knows if cinder-backup still has backups to migrate, and cinder-backup knows if the cinder-volume service has volumes to migrate.

    Although each host migrates only its own volumes, the summary message is based on a global assessment of whether any volume still requires migration This allows you to confirm that migration for all volumes is complete. Once you receive confirmation, remove the fixed_key setting from cinder.conf and nova.conf. See the Clean up the fixed keys section below for more information.

5.1.4. Troubleshooting the migration process
Copy link

5.1.4.1. Role assignment
Copy link

The barbican secret can only be created when the requestor has the creator role. This means that the cinder service itself requires the creator role, otherwise a log sequence similar to this will occur:

  1. Starting migration of ConfKeyManager keys.
  2. Migrating volume <UUID> encryption key to Barbican
  3. Error migrating encryption key: Forbidden: Secret creation attempt not allowed - please review your user/project privileges
  4. There are still %d volume(s) using the ConfKeyManager's all-zeros encryption key ID.

The key message is the third one: Secret creation attempt not allowed. To fix the problem, update the cinder account’s privileges:

  1. Run openstack role add --project service --user cinder creator
  2. Restart the cinder-volume and cinder-backup services.

As a result, the next attempt at migration should succeed.

5.1.5. Clean up the fixed keys
Copy link

Important

The encryption_key_id was only recently added to the Backup table, as part of the Queens release. As a result, pre-existing backups of encrypted volumes are likely to exist. The all-zeros encryption_key_id is stored on the backup itself, but it won’t appear in the Backup database. As such, it is impossible for the migration process to know for certain whether a backup of an encrypted volume exists that still relies on the all-zeros ConfKeyMgr key ID.

After migrating your key IDs into barbican, the fixed key remains in the configuration files. This may present a security concern to some users, because the fixed_key value is not encrypted in the .conf files. To address this, you can manually remove the fixed_key values from your nova and cinder configurations. However, first complete testing and review the output of the log file before you proceed, because disks that are still dependent on this value will not be accessible.

  1. Review the existing fixed_key values. The values must match for both services. 

    crudini --get /var/lib/config-data/puppet-generated/cinder/etc/cinder/cinder.conf keymgr fixed_key
crudini --get /var/lib/config-data/puppet-generated/nova_libvirt/etc/nova/nova.conf keymgr fixed_key
    Copy to Clipboard Toggle word wrap
  2. IMPORTANT: Make a backup of the existing fixed_key values. This allows you to restore the value if something goes wrong, or if you need to restore a backup that uses the old encryption key.

  3. Delete the fixed_key values: 

    crudini --del /var/lib/config-data/puppet-generated/cinder/etc/cinder/cinder.conf keymgr fixed_key
crudini --del /var/lib/config-data/puppet-generated/nova_libvirt/etc/nova/nova.conf keymgr fixed_key
    Copy to Clipboard Toggle word wrap

Chapter 6. Encrypt at-rest swift objects
Copy link

By default, objects uploaded to Object Storage are stored unencrypted. Because of this, it is possible to access objects directly from the file system. This can present a security risk if disks are not properly erased before they are discarded. When you have barbican enabled, the Object Storage service (swift) can transparently encrypt and decrypt your stored (at-rest) objects. At-rest encryption is distinct from in-transit encryption in that it refers to the objects being encrypted while being stored on disk.

Swift performs these encryption tasks transparently, with the objects being automatically encrypted when uploaded to swift, then automatically decrypted when served to a user. This encryption and decryption is done using the same (symmetric) key, which is stored in barbican.

Note

You cannot disable encryption after you have enabled encryption and added data to the swift cluster, because the data is now stored in an encrypted state. Consequently, the data will not be readable if encryption is disabled, until you re-enable encryption with the same key.

6.1. Enable at-rest encryption for swift
Copy link

  1. You can enable the swift encryption capabilities by including SwiftEncryptionEnabled: True in your environment file, then re-running openstack overcloud deploy using /home/stack/overcloud_deploy.sh. Note that you still need to enable barbican, as described in the Install Barbican chapter.

  2. Confirm that swift is configured to use at-rest encryption: 

    $ crudini --get /var/lib/config-data/puppet-generated/swift/etc/swift/proxy-server.conf pipeline-main pipeline

pipeline = catch_errors healthcheck proxy-logging cache ratelimit bulk tempurl formpost authtoken keystone staticweb copy container_quotas account_quotas slo dlo versioned_writes kms_keymaster encryption proxy-logging proxy-server
    Copy to Clipboard Toggle word wrap

    The result should include an entry for encryption.

Chapter 7. Validate glance images
Copy link

After enabling Barbican, you can configure the Image Service (glance) to verify that an uploaded image has not been tampered with. In this implementation, the image is first signed with a key that is stored in barbican. The image is then uploaded to glance, along with the accompanying signing information. As a result, the image’s signature is verified before each use, with the instance build process failing if the signature does not match.

Barbican’s integration with glance means that you can use the openssl command with your private key to sign glance images before uploading them.

7.1. Enable glance image validation
Copy link

In your environment file, enable image verification with the VerifyGlanceSignatures: True setting. You must re-run the openstack overcloud deploy command for this setting to take effect.

To verify that glance image validation is enabled, run the following command on an overcloud Compute node: 

$ sudo crudini --get /var/lib/config-data/puppet-generated/nova_libvirt/etc/nova/nova.conf glance verify_glance_signatures
Copy to Clipboard Toggle word wrap
Note

If you use Ceph as the back end for the Image and Compute services, a CoW clone is created. Therefore, Image signing verification cannot be performed.

7.2. Validate an image
Copy link

To configure a glance image for validation, complete the following steps:

  1. Confirm that glance is configured to use barbican: 

    $ sudo crudini --get /var/lib/config-data/puppet-generated/glance_api/etc/glance/glance-api.conf key_manager backend
castellan.key_manager.barbican_key_manager.BarbicanKeyManager
    Copy to Clipboard Toggle word wrap

  2. Generate a certificate: 

    openssl genrsa -out private_key.pem 1024
openssl rsa -pubout -in private_key.pem -out public_key.pem
openssl req -new -key private_key.pem -out cert_request.csr
openssl x509 -req -days 14 -in cert_request.csr -signkey private_key.pem -out x509_signing_cert.crt
    Copy to Clipboard Toggle word wrap

  3. Add certificate to the barbican secret store: 

    $ source ~/overcloudrc
$ openstack secret store --name signing-cert --algorithm RSA --secret-type certificate --payload-content-type "application/octet-stream" --payload-content-encoding base64  --payload "$(base64 x509_signing_cert.crt)" -c 'Secret href' -f value
https://192.168.123.170:9311/v1/secrets/5df14c2b-f221-4a02-948e-48a61edd3f5b
    Copy to Clipboard Toggle word wrap
    Note

    Record the resulting UUID for use in a later step. In this example, the certificate’s UUID is 5df14c2b-f221-4a02-948e-48a61edd3f5b.

  4. Use private_key.pem to sign the image and generate the .signature file. For example: 

    $ openssl dgst -sha256 -sign private_key.pem -sigopt rsa_padding_mode:pss -out cirros-0.4.0.signature cirros-0.4.0-x86_64-disk.img
    Copy to Clipboard Toggle word wrap

  5. Convert the resulting .signature file into base64 format: 

    $ base64 -w 0 cirros-0.4.0.signature  > cirros-0.4.0.signature.b64
    Copy to Clipboard Toggle word wrap

  6. Load the base64 value into a variable to use it in the subsequent command: 

    $ cirros_signature_b64=$(cat cirros-0.4.0.signature.b64)
    Copy to Clipboard Toggle word wrap

  7. Upload the signed image to glance. For img_signature_certificate_uuid, you must specify the UUID of the signing key you previously uploaded to barbican: 

     openstack image create \
 --container-format bare --disk-format qcow2 \
 --property img_signature="$cirros_signature_b64" \
 --property img_signature_certificate_uuid="5df14c2b-f221-4a02-948e-48a61edd3f5b" \
 --property img_signature_hash_method="SHA-256" \
 --property img_signature_key_type="RSA-PSS" cirros_0_4_0_signed \
 --file cirros-0.4.0-x86_64-disk.img
+--------------------------------+----------------------------------------------------------------------------------+
| Property                       | Value                                                                            |
+--------------------------------+----------------------------------------------------------------------------------+
| checksum                       | 25c96942084f61e008d593b6c2cfda00                                                 |
| container_format               | bare                                                                             |
| created_at                     | 2018-01-23T05:37:31Z                                                             |
| disk_format                    | qcow2                                                                            |
| id                             | d3396fa0-2ea2-4832-8a77-d36fa3f2ab27                                             |
| img_signature                  | lcI7nGgoKxnCyOcsJ4abbEZEpzXByFPIgiPeiT+Otjz0yvW00KNN3fI0AA6tn9EXrp7fb2xBDE4UaO3v |
|                                | IFquV/s3mU4LcCiGdBAl3pGsMlmZZIQFVNcUPOaayS1kQYKY7kxYmU9iq/AZYyPw37KQI52smC/zoO54 |
|                                | zZ+JpnfwIsM=                                                                     |
| img_signature_certificate_uuid | ba3641c2-6a3d-445a-8543-851a68110eab                                             |
| img_signature_hash_method      | SHA-256                                                                          |
| img_signature_key_type         | RSA-PSS                                                                          |
| min_disk                       | 0                                                                                |
| min_ram                        | 0                                                                                |
| name                           | cirros_0_4_0_signed                                                              |
| owner                          | 9f812310df904e6ea01e1bacb84c9f1a                                                 |
| protected                      | False                                                                            |
| size                           | None                                                                             |
| status                         | active                                                                           |
| tags                           | []                                                                               |
| updated_at                     | 2018-01-23T05:37:31Z                                                             |
| virtual_size                   | None                                                                             |
| visibility                     | shared                                                                           |
+--------------------------------+----------------------------------------------------------------------------------+
    Copy to Clipboard Toggle word wrap

  8. You can view glance’s image validation activities in the Compute log: /var/log/containers/nova/nova-compute.log. For example, you can expect the following entry when the instance is booted: 

    2018-05-24 12:48:35.256 1 INFO nova.image.glance [req-7c271904-4975-4771-9d26-cbea6c0ade31 b464b2fd2a2140e9a88bbdacf67bdd8c a3db2f2beaee454182c95b646fa7331f - default default] Image signature verification succeeded for image d3396fa0-2ea2-4832-8a77-d36fa3f2ab27
    Copy to Clipboard Toggle word wrap

Legal Notice
Copy link

Copyright © 2018 Red Hat, Inc.
The text of and illustrations in this document are licensed by Red Hat under a Creative Commons Attribution–Share Alike 3.0 Unported license ("CC-BY-SA"). An explanation of CC-BY-SA is available at http://creativecommons.org/licenses/by-sa/3.0/. In accordance with CC-BY-SA, if you distribute this document or an adaptation of it, you must provide the URL for the original version.
Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert, Section 4d of CC-BY-SA to the fullest extent permitted by applicable law.
Portions adapted from Barbican’s developer documentation. See "Manage Secrets with OpenStack Key Manager" in Red Hat OpenStack Platform Licenses for Documentation.
Red Hat, Red Hat Enterprise Linux, the Shadowman logo, JBoss, MetaMatrix, Fedora, the Infinity Logo, and RHCE are trademarks of Red Hat, Inc., registered in the United States and other countries.
Linux® is the registered trademark of Linus Torvalds in the United States and other countries.
Java® is a registered trademark of Oracle and/or its affiliates.
XFS® is a trademark of Silicon Graphics International Corp. or its subsidiaries in the United States and/or other countries.
MySQL® is a registered trademark of MySQL AB in the United States, the European Union and other countries.
Node.js® is an official trademark of Joyent. Red Hat Software Collections is not formally related to or endorsed by the official Joyent Node.js open source or commercial project.
The OpenStack® Word Mark and OpenStack Logo are either registered trademarks/service marks or trademarks/service marks of the OpenStack Foundation, in the United States and other countries and are used with the OpenStack Foundation's permission. We are not affiliated with, endorsed or sponsored by the OpenStack Foundation, or the OpenStack community.
All other trademarks are the property of their respective owners.
Back to top
Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Theme

© 2025 Red Hat