Chapter 34. Encrypting Data at Datastore Layer
34.1. Overview
This topic reviews how to enable and configure encryption of secret data at the datastore layer. While the examples use the secrets resource, any resource can be encrypted, such as configmaps.
etcd v3 or later is required in order to use this feature.
34.2. Configuration and Determining Whether Encryption Is Already Enabled
To activate data encryption, pass the --experimental-encryption-provider-config argument to the Kubernetes API server:
Excerpt of master-config.yaml
kubernetesMasterConfig:
apiServerArguments:
experimental-encryption-provider-config:
- /path/to/encryption-config.yaml
For more information about master-config.yaml and its format, see the Master Configuration Files topic.
34.3. Understanding the Encryption Configuration
Encryption configuration file with all available providers
kind: EncryptionConfig apiVersion: v1 resources: 1 - resources: 2 - secrets providers: 3 - aescbc: 4 keys: - name: key1 5 secret: c2VjcmV0IGlzIHNlY3VyZQ== 6 - name: key2 secret: dGhpcyBpcyBwYXNzd29yZA== - secretbox: keys: - name: key1 secret: YWJjZGVmZ2hpamtsbW5vcHFyc3R1dnd4eXoxMjM0NTY= - aesgcm: keys: - name: key1 secret: c2VjcmV0IGlzIHNlY3VyZQ== - name: key2 secret: dGhpcyBpcyBwYXNzd29yZA== - identity: {}
- 1
- Each
resourcesarray item is a separate configuration and contains a complete configuration. - 2
- The
resources.resourcesfield is an array of Kubernetes resource names (resourceorresource.group) that should be encrypted. - 3
- The
providersarray is an ordered list of the possible encryption providers. Only one provider type can be specified per entry (identityoraescbccan be provided, but not both in the same item). - 4
- The first provider in the list is used to encrypt resources going into storage.
- 5
- Arbitrary name of the secret.
- 6
- Base64 encoded random key. Different providers have different key lengths. See instructions on how to generate the key.
When reading resources from storage, each provider that matches the stored data attempts to decrypt the data in order. If no provider can read the stored data due to a mismatch in format or secret key, an error is returned, which prevents clients from accessing that resource.
If any resource is not readable via the encryption configuration (because keys were changed), the only recourse is to delete that key from the underlying etcd directly. Calls attempting to read that resource will fail until it is deleted or a valid decryption key is provided.
34.3.1. Available Providers
| Name | Encryption | Strength | Speed | Key Length | Other Considerations |
|---|---|---|---|---|---|
| identity | None | N/A | N/A | N/A | Resources written as-is without encryption. When set as the first provider, the resource will be decrypted as new values are written. |
| aescbc | AES-CBC with PKCS#7 padding | Strongest | Fast | 32-byte | The recommended choice for encryption, but may be slightly slower than secretbox. |
| secretbox | XSalsa20 and Poly1305 | Strong | Faster | 32-byte | A newer standard and may not be considered acceptable in environments that require high levels of review. |
| aesgcm | AES-GCM with a random initialization vector (IV) | Must be rotated every 200,000 writes | Fastest | 16, 24, or 32-byte | Is not recommended for use except when an automated key rotation scheme is implemented. |
Each provider supports multiple keys. The keys are tried in order for decryption. If the provider is the first provider, the first key is used for encryption.
Kubernetes has no proper nonce generator and uses a random IV as nonce for AES-GCM. Since AES-GCM requires a proper nonce to be secure, AES-GCM is not recommended. The 200,000 write limit just limits the possibility of a fatal nonce misuse to a reasonable low margin.
34.4. Encrypting Data
Create a new encryption configuration file.
kind: EncryptionConfig
apiVersion: v1
resources:
- resources:
- secrets
providers:
- aescbc:
keys:
- name: key1
secret: <BASE 64 ENCODED SECRET>
- identity: {}To create a new secret:
Generate a 32-byte random key and base64 encode it. For example, on Linux and macOS use:
$ head -c 32 /dev/urandom | base64
ImportantThe encryption key must be generated with an appropriate cryptographically secure random number generator like /dev/urandom. For example,
math/randomfrom Golang orrandom.random()from Python are not suitable.-
Place that value in the
secretfield. Restart the API server:
# systemctl restart atomic-openshift-master-api
The encryption provider configuration file contains keys that can decrypt content in etcd, so you must properly restrict permissions on masters so only the user who runs the master API server can read it.
34.5. Verifying that Data is Encrypted
Data is encrypted when written to etcd. After restarting the API server, any newly created or updated secrets should be encrypted when stored. To check, you can use the etcdctl command line program to retrieve the contents of your secret.
Create a new secret called
secret1in thedefaultnamespace:$ oc create secret generic secret1 -n default --from-literal=mykey=mydata
Using the
etcdctlcommand line, read that secret out of etcd:$ ETCDCTL_API=3 etcdctl get /kubernetes.io/secrets/default/secret1 -w fields [...] | grep Value
[…]must be the additional arguments for connecting to the etcd server.The final command will look similar to:
$ ETCDCTL_API=3 etcdctl get /kubernetes.io/secrets/default/secret1 -w fields \ --cacert=/var/lib/origin/openshift.local.config/master/ca.crt \ --key=/var/lib/origin/openshift.local.config/master/master.etcd-client.key \ --cert=/var/lib/origin/openshift.local.config/master/master.etcd-client.crt \ --endpoints 'https://127.0.0.1:4001' | grep Value
- Verify that the output of the command above is prefixed with k8s:enc:aescbc:v1: which indicates the aescbc provider has encrypted the resulting data.
Verify the secret is correctly decrypted when retrieved via the API:
$ oc get secret secret1 -n default -o yaml | grep mykey
This should match mykey: bXlkYXRh.
34.6. Ensure All Secrets are Encrypted
Since secrets are encrypted when written, performing an update on a secret will encrypt that content.
$ oc adm migrate storage --include=secrets --confirm
This command reads all secrets, then updates them to apply server-side encryption. If an error occurs due to a conflicting write, retry the command.
For larger clusters, you can subdivide the secrets by namespace or script an update.
34.7. Rotating a Decryption Key
Changing the secret without incurring downtime requires a multi-step operation, especially in the presence of a highly available deployment where multiple API servers are running.
- Generate a new key and add it as the second key entry for the current provider on all servers.
Restart all API servers to ensure each server can decrypt using the new key.
NoteIf using a single API server, you can skip this step.
# systemctl restart atomic-openshift-master-api
-
Make the new key the first entry in the
keysarray so that it is used for encryption in the configuration. Restart all API servers to ensure each server now encrypts using the new key.
# systemctl restart atomic-openshift-master-api
Run the following to encrypt all existing secrets with the new key:
$ oc adm migrate storage --include=secrets --confirm
- After you back up etcd with the new key in use and update all secrets, remove the old decryption key from the configuration.
34.8. Decrypting Data
To disable encryption at the datastore layer:
- Place the identity provider as the first entry in the configuration:
kind: EncryptionConfig
apiVersion: v1
resources:
- resources:
- secrets
providers:
- identity: {}
- aescbc:
keys:
- name: key1
secret: <BASE 64 ENCODED SECRET>Restart all API servers:
# systemctl restart atomic-openshift-master-api
Run the following to force all secrets to be decrypted:
$ oc adm migrate storage --include=secrets --confirm
