Chapter 6. External encryption key management

Procedure

1. Deploy the KMS plugin on each control plane host as a static pod:

   • Configure the plugin to listen at unix:///var/run/kmsplugin/kms.sock
   • For your static pod, mount /var/run/kmsplugin as hostPath
   • Configure KMS provider connection details and authentication credentials

   The following steps show how to deploy the HashiCorp Vault KMS plugin as a static pod.

2. Create a static pod manifest file:

   $ cat > /tmp/vault-kms-plugin.yaml <<'EOF'
   apiVersion: v1
   kind: Pod
   metadata:
     name: vault-kms-plugin
     namespace: kube-system
     labels:
       app: vault-kms-plugin
       tier: control-plane
   spec:
     priorityClassName: system-node-critical
     hostNetwork: true
     containers:
     - name: vault-kms-plugin
       image: quay.io/redhat-isv-containers/698df066f8d1ddf179c15ef9:<version>
       command:
       - /vault-kubernetes-kms
       - -vault-address=<https://vault.example.com:8200>
       - -<vault_namespace>=admin
       - -auth-method=userpass
       - -userpass-username=<vault_username>
       - -userpass-password=<vault_password>
       - -transit-mount=transit
       - -transit-key=kms-key
       - -socket=unix:///var/run/kmsplugin/kms.sock
       volumeMounts:
       - name: kmsplugin
         mountPath: /var/run/kmsplugin
       resources:
         requests:
           cpu: 100m
           memory: 128Mi
         limits:
           cpu: 500m
           memory: 512Mi
       securityContext:
         privileged: true
     volumes:
     - name: kmsplugin
       hostPath:
         path: /var/run/kmsplugin
         type: DirectoryOrCreate
   EOF

   Replace the following values to match your environment:

   <version>

   The Vault KMS plugin image version tag. Use 0.1.0-beta-ubi.

   <vault_username>

   Your Vault username for authentication.

   <vault_password>

   Your Vault password for authentication.

   https://vault.example.com:8200

   The Vault address. Update this field to match your Vault server URL.

   <vault_namespace>

   Optional field.

   The manifest uses the Red Hat certified container image from Quay.io (quay.io/redhat-isv-containers/698df066f8d1ddf179c15ef9), which is the recommended image for OpenShift Container Platform.

   Note

   Alternatively, you can use the HashiCorp image from Docker Hub by replacing the image field with docker.io/hashicorp/vault-kube-kms:0.1.0-beta-ubi.

3. Deploy the static pod manifest to each control plane node by running the following commands:

   $ MANIFEST=$(cat /tmp/vault-kms-plugin.yaml | base64)
   $ for node in $(oc get nodes --selector=node-role.kubernetes.io/master -o name | cut -d/ -f2); do
       echo "Deploying to $node..."
       oc debug node/$node -- chroot /host bash -c \
         "echo '$MANIFEST' | base64 -d > /etc/kubernetes/manifests/vault-kms-plugin.yaml"
     done

   The kubelet automatically detects and starts static pods from /etc/kubernetes/manifests/.

4. Verify the static pods are running by entering the following command:

   $ oc get pods -n kube-system -o wide | grep vault-kms

   Example output

   vault-kms-plugin-ip-10-0-16-7.compute.internal    1/1  Running  0  2m  10.0.16.7
   vault-kms-plugin-ip-10-0-32-93.compute.internal   1/1  Running  0  2m  10.0.32.93
   vault-kms-plugin-ip-10-0-69-106.compute.internal  1/1  Running  0  2m  10.0.69.106

   Static pod names include the node name as a suffix. You should see one pod per control plane node.

5. Verify the socket exists on a control plane node by entering the following command:

   $ oc debug node/<node_name> -- chroot /host ls -la /var/run/kmsplugin/kms.sock

   Example output

   srwxr-xr-x. 1 root root 0 <timestamp> /var/run/kmsplugin/kms.sock

   Note

   Static pods are managed by kubelet on each node and cannot be deleted with oc delete pod. To remove a static pod, delete the manifest file from /etc/kubernetes/manifests/ on each control plane node.

6. Edit the APIServer custom resource by entering the following command:

   $ oc edit apiserver cluster

7. Add the KMS configuration to the spec.encryption section:

   apiVersion: config.openshift.io/v1
   kind: APIServer
   metadata:
     name: cluster
   spec:
     encryption:
       type: KMS

8. Save and exit.

   Migration begins automatically. The kube-apiserver, openshift-apiserver and oauth-apiserver Operators will restart and roll out new revisions.

   Note

   The openshift-apiserver and authentication operators typically complete migration in 5-10 minutes. The kube-apiserver operator uses a conservative rollout strategy, updating one control plane node at a time and waiting for health checks before proceeding to the next node. This process can take 30 minutes or longer depending on cluster load.

Verification

1. Verify the encryption type by entering the following command:

   $ oc get apiserver cluster -o jsonpath='{.spec.encryption.type}'

   Output should show KMS.

2. Monitor the kube-apiserver rollout progress by entering the following command:

   $ oc get kubeapiserver cluster -o jsonpath='{.status.nodeStatuses}' | jq -r '.[] | "\(.nodeName | split(".")[0]): current=\(.currentRevision) target=\(.targetRevision)"'

   Example output during rollout

   ip-10-0-16-166: current=10 target=0
   ip-10-0-32-93: current=9 target=10
   ip-10-0-69-106: current=9 target=0

   The operator rolls out one node at a time. When all nodes show the same current revision and target is 0, the rollout is complete.

3. Check the encryption migration status by entering the following command:

   $ oc get kubeapiserver cluster -o jsonpath='{.status.conditions[?(@.type=="Encrypted")]}' | jq .

   Example output when complete

   {
     "lastTransitionTime": "2026-05-15T17:39:02Z",
     "message": "All resources encrypted: secrets, configmaps",
     "reason": "EncryptionCompleted",
     "status": "True",
     "type": "Encrypted"
   }

   Wait for reason to show EncryptionCompleted before proceeding to verify encryption in etcd.

4. Verify secrets are encrypted in etcd:

   Warning

   Wait for the kube-apiserver rollout to complete on all control plane nodes before verifying encryption. During the rollout, API requests are distributed across nodes, and secrets created while some nodes are still on an earlier revision will not be encrypted with Kubernetes KMS v2.

   1. Create a test secret by entering the following command:

      $ oc create secret generic test-secret --from-literal=key=value -n default

   2. Get an etcd pod name by entering the following command:

      $ oc get pods -n openshift-etcd -l app=etcd -o name | head -1

   3. Check the secret data in etcd by entering the following command:

      $ oc exec -n openshift-etcd <etcd_pod_name> -- etcdctl get /kubernetes.io/secrets/default/test-secret --print-value-only | hexdump -C | head -1

      Output should begin with k8s:enc:kms:v2: followed by encrypted binary data.

   4. Delete the test secret by entering the following command:

      $ oc delete secret test-secret -n default

Procedure

1. Rotate the Vault encryption key by entering the following command:

   $ vault write -f transit/keys/kms-key/rotate

   Vault creates a new key version while maintaining earlier versions for decryption. The API server automatically uses the correct key version for each secret.

2. Verify the new key version by entering the following command:

   $ vault read transit/keys/kms-key

   The latest_version field shows the current key version number.

Procedure

1. Back up the current etcd encryption configuration by entering the following command:

   $ oc get apiserver cluster -o yaml > apiserver-backup.yaml

2. Edit the APIServer custom resource by entering the following command:

   $ oc edit apiserver cluster

3. Change the encryption type from aescbc or aesgcm to KMS:

   apiVersion: config.openshift.io/v1
   kind: APIServer
   metadata:
     name: cluster
   spec:
     encryption:
       type: KMS

4. Save and exit.

   Migration starts automatically and typically takes several minutes.

5. Verify migration completion for all API servers by running the following commands:

   $ oc get openshiftapiserver -o=jsonpath='{range .items[0].status.conditions[?(@.type=="Encrypted")]}{.reason}{"\n"}{end}'

   $ oc get kubeapiserver -o=jsonpath='{range .items[0].status.conditions[?(@.type=="Encrypted")]}{.reason}{"\n"}{end}'

   $ oc get authentication.operator.openshift.io -o=jsonpath='{range .items[0].status.conditions[?(@.type=="Encrypted")]}{.reason}{"\n"}{end}'

   All outputs should show EncryptionCompleted.

Procedure

1. Check the kube-apiserver operator logs for KMS-related events by entering the following command:

   $ oc logs -n openshift-kube-apiserver-operator deploy/kube-apiserver-operator | grep -i kms

2. View API server logs for KMS-related events by entering the following command:

   $ oc logs -n openshift-kube-apiserver -l apiserver=true --tail=100 | grep -i kms

3. Verify the KMS encryption configuration by entering the following command:

   $ oc get apiserver cluster -o jsonpath='{.spec.encryption}' | jq

Procedure

1. Edit the APIServer custom resource by entering the following command:

   $ oc edit apiserver cluster

2. Change the encryption configuration:

   apiVersion: config.openshift.io/v1
   kind: APIServer
   metadata:
     name: cluster
   spec:
     encryption:
       type: <encryption>

   Replace <encryption> with aescbc, aesgcm, or identity.

3. Save and exit.

   OpenShift Container Platform automatically migrates etcd data. Migration time depends on etcd size and secret count.

4. Monitor the migration progress by entering the following command:

   $ oc get openshiftapiserver -o=jsonpath='{range .items[0].status.conditions[?(@.type=="Encrypted")]}{.reason}{"\n"}{.message}{"\n"}{end}'

   Wait until the output shows EncryptionCompleted.

5. Verify kube-apiserver pods rolled out by entering the following command:

   $ oc get pods -n openshift-kube-apiserver -l app=openshift-kube-apiserver

6. Remove the static pod manifest from each control plane node by running the following command:

   $ for node in $(oc get nodes --selector=node-role.kubernetes.io/master -o name | cut -d/ -f2); do
       echo "Removing static pod from $node..."
       oc debug node/$node -- chroot /host \
         rm -f /etc/kubernetes/manifests/vault-kms-plugin.yaml
     done

   The kubelet automatically stops static pods when their manifest is removed from /etc/kubernetes/manifests/.

7. Clean up the socket directory by running the following command:

   $ for node in $(oc get nodes --selector=node-role.kubernetes.io/master -o name | cut -d/ -f2); do
       oc debug node/$node -- chroot /host rm -rf /var/run/kmsplugin
     done

8. After backup retention period passes, decommission the KMS key.

Verification

1. Verify the encryption type by entering the following command:

   $ oc get apiserver cluster -o jsonpath='{.spec.encryption.type}'

2. Verify that a test secret uses the new encryption type:

   1. Create a test secret by entering the following command:

      $ oc create secret generic encryption-test --from-literal=key=value -n default

   2. Get an etcd pod name by entering the following command:

      $ oc get pods -n openshift-etcd -l app=etcd -o name | head -1

   3. Check the encryption prefix by entering the following command:

      $ oc exec -n openshift-etcd <etcd_pod_name> -- etcdctl get /kubernetes.io/secrets/default/encryption-test --print-value-only | hexdump -C | head -1

      Output should begin with k8s:enc:aescbc:v1:, k8s:enc:aesgcm:v1:, or show readable JSON for identity.

   4. Delete the test secret by entering the following command:

      $ oc delete secret encryption-test -n default