Chapter 17. Network-Bound Disk Encryption (NBDE)

Procedure

1. Locate and access the directory that stores the Tang server key. This is usually the /var/db/tang directory. Check the currently advertised key thumbprint:

   # tang-show-keys 7500

   Example output

   36AHjNH3NZDSnlONLz1-V4ie6t8

2. Enter the Tang server key directory:

   # cd /var/db/tang/

3. List the current Tang server keys:

   # ls -A1

   Example output

   36AHjNH3NZDSnlONLz1-V4ie6t8.jwk
   gJZiNPMLRBnyo_ZKfK4_5SrnHYo.jwk

   During normal Tang server operations, there are two .jwk files in this directory: one for signing and verification, and another for key derivation.

4. Disable advertisement of the old keys:

   # for key in *.jwk; do \
     mv -- "$key" ".$key"; \
   done

   New clients setting up Network-Bound Disk Encryption (NBDE) or requesting keys will no longer see the old keys. Existing clients can still access and use the old keys until they are deleted. The Tang server reads but does not advertise keys stored in UNIX hidden files, which start with the . character.

5. Generate a new key:

   # /usr/libexec/tangd-keygen /var/db/tang

6. List the current Tang server keys to verify the old keys are no longer advertised, as they are now hidden files, and new keys are present:

   # ls -A1

   Example output

   .36AHjNH3NZDSnlONLz1-V4ie6t8.jwk
   .gJZiNPMLRBnyo_ZKfK4_5SrnHYo.jwk
   Bp8XjITceWSN_7XFfW7WfJDTomE.jwk
   WOjQYkyK7DxY_T5pMncMO5w0f6E.jwk

   Tang automatically advertises the new keys.

   Note

   More recent Tang server installations include a helper /usr/libexec/tangd-rotate-keys directory that takes care of disabling advertisement and generating the new keys simultaneously.

7. If you are running multiple Tang servers behind a load balancer that share the same key material, ensure the changes made here are properly synchronized across the entire set of servers before proceeding.

Verification

1. Verify that the Tang server is advertising the new key, and not advertising the old key:

   # tang-show-keys 7500

   Example output

   WOjQYkyK7DxY_T5pMncMO5w0f6E

2. Verify that the old key, while not advertised, is still available to decryption requests:

   # clevis decrypt </tmp/encrypted.oldkey

Procedure

1. Create a DaemonSet object based on the following template. This template sets up three redundant Tang servers, but can be easily adapted to other situations. Change the Tang server URLs and thumbprints in the NEW_TANG_PIN environment to suit your environment:

   apiVersion: apps/v1
   kind: DaemonSet
   metadata:
     name: tang-rekey
     namespace: openshift-machine-config-operator
   spec:
     selector:
       matchLabels:
         name: tang-rekey
     template:
       metadata:
         labels:
           name: tang-rekey
       spec:
         containers:
         - name: tang-rekey
           image: registry.access.redhat.com/ubi9/ubi-minimal:latest
           imagePullPolicy: IfNotPresent
           command:
           - "/sbin/chroot"
           - "/host"
           - "/bin/bash"
           - "-ec"
           args:
           - |
             rm -f /tmp/rekey-complete || true
             echo "Current tang pin:"
             clevis-luks-list -d $ROOT_DEV -s 1
             echo "Applying new tang pin: $NEW_TANG_PIN"
             clevis-luks-edit -f -d $ROOT_DEV -s 1 -c "$NEW_TANG_PIN"
             echo "Pin applied successfully"
             touch /tmp/rekey-complete
             sleep infinity
           readinessProbe:
             exec:
               command:
               - cat
               - /host/tmp/rekey-complete
             initialDelaySeconds: 30
             periodSeconds: 10
           env:
           - name: ROOT_DEV
             value: /dev/disk/by-partlabel/root
           - name: NEW_TANG_PIN
             value: >-
               {"t":1,"pins":{"tang":[
                 {"url":"http://tangserver01:7500","thp":"WOjQYkyK7DxY_T5pMncMO5w0f6E"},
                 {"url":"http://tangserver02:7500","thp":"I5Ynh2JefoAO3tNH9TgI4obIaXI"},
                 {"url":"http://tangserver03:7500","thp":"38qWZVeDKzCPG9pHLqKzs6k1ons"}
               ]}}
           volumeMounts:
           - name: hostroot
             mountPath: /host
           securityContext:
             privileged: true
         volumes:
         - name: hostroot
           hostPath:
             path: /
         nodeSelector:
           kubernetes.io/os: linux
         priorityClassName: system-node-critical
         restartPolicy: Always
         serviceAccount: machine-config-daemon
         serviceAccountName: machine-config-daemon

   In this case, even though you are rekeying tangserver01, you must specify not only the new thumbprint for tangserver01, but also the current thumbprints for all other Tang servers. Failure to specify all thumbprints for a rekeying operation opens up the opportunity for a man-in-the-middle attack.

2. To distribute the daemon set to every cluster that must be rekeyed, run the following command:

   $ oc apply -f tang-rekey.yaml

   However, to run at scale, wrap the daemon set in an ACM policy. This ACM configuration must contain one policy to deploy the daemon set, a second policy to check that all the daemon set pods are READY, and a placement rule to apply it to the appropriate set of clusters.

Procedure

1. Restart the pod that performs the rekeying operation using the normal Kubernetes pod restart policy.
2. If any of the associated Tang servers are unavailable, try rekeying until all the servers are back online.

Procedure

1. Validate the Tang server configuration by performing a simple encrypt and decrypt operation on each Tang server’s configuration as defined in the daemon set.

   This is an example of an encryption and decryption attempt with a bad thumbprint:

   $ echo "okay" | clevis encrypt tang \
     '{"url":"http://tangserver02:7500","thp":"badthumbprint"}' | \
     clevis decrypt

   Example output

   Unable to fetch advertisement: 'http://tangserver02:7500/adv/badthumbprint'!

   This is an example of an encryption and decryption attempt with a good thumbprint:

   $ echo "okay" | clevis encrypt tang \
     '{"url":"http://tangserver03:7500","thp":"goodthumbprint"}' | \
     clevis decrypt

   Example output

   okay

2. After you identify the root cause, remedy the underlying situation:

   1. Delete the non-working daemon set.

   2. Edit the daemon set definition to fix the underlying issue. This might include any of the following actions:

      • Edit a Tang server entry to correct the URL and thumbprint.
      • Remove a Tang server that is no longer in service.
      • Add a new Tang server that is a replacement for a decommissioned server.
3. Distribute the updated daemon set again.

1. Locate the name of the container that was created by the daemon set:

   $ oc get pods -A | grep tang-rekey

   Example output

   openshift-machine-config-operator  tang-rekey-7ks6h  1/1  Running   20 (8m39s ago)  89m

2. Print the logs from the container. The following log is from a completed successful rekeying operation:

   $ oc logs tang-rekey-7ks6h

   Example output

   Current tang pin:
   1: sss '{"t":1,"pins":{"tang":[{"url":"http://10.46.55.192:7500"},{"url":"http://10.46.55.192:7501"},{"url":"http://10.46.55.192:7502"}]}}'
   Applying new tang pin: {"t":1,"pins":{"tang":[
     {"url":"http://tangserver01:7500","thp":"WOjQYkyK7DxY_T5pMncMO5w0f6E"},
     {"url":"http://tangserver02:7500","thp":"I5Ynh2JefoAO3tNH9TgI4obIaXI"},
     {"url":"http://tangserver03:7500","thp":"38qWZVeDKzCPG9pHLqKzs6k1ons"}
   ]}}
   Updating binding...
   Binding edited successfully
   Pin applied successfully