Record Encryption filter guide

Procedure

1. Establish a naming convention for keys to ensure that the filter’s keys remain separate from those used by other systems.

   In this example, a KEK- prefix is used for filter key names. Adjust the naming convention as needed.

2. Create a symmetric key:

   vault write -f transit/keys/KEK-trades type=aes256-gcm96 auto_rotate_period=90d

   Copy to Clipboard Toggle word wrap
   • The key type must be aes256-gcm96, which is the default type for the Transit Engine.
   • (Optional) The auto_rotate_period parameter enables automatic 90-day key rotation. Adjust the rotation period if required.

Procedure

1. Create a key and apply the alias:

   KEY_ALIAS="KEK-<name>"
   KEY_ID=$(aws kms create-key | jq -r '.KeyMetadata.KeyId')
   # the create key command will produce JSON output including the KeyId
   aws kms create-alias --alias-name alias/${KEY_ALIAS} --target-key-id ${KEY_ID}

   Copy to Clipboard Toggle word wrap
   Note

   You cannot use keys from other AWS accounts. For more information on this limitation, see the issue for AWS KMS serde improvements.

2. (Optional) Enable a key rotation policy:

   aws kms enable-key-rotation --key-id ${KEY_ID} --rotation-period-in-days 180

   Copy to Clipboard Toggle word wrap
   Tip

   It is recommended to use a key rotation policy.

Procedure

1. Create a resource group

   Before you create a key vault, you must create a resource group to contain it.

   If you deploy Streams for Apache Kafka Proxy into multiple different environments, you must have separate resource groups and key vaults for each environment.

   Note

   Resource groups are tied to the region they were created with. While resources (such as key vaults) within a resource group can be deployed to any region, Microsoft recommends that they should be located in the same region as the resource group. Keep this in mind when choosing a region for your resource groups.

   If using the Azure CLI, you can create a resource group with a command like this:

   az group create --name "my-resource-group" --location "eastus"

   Copy to Clipboard Toggle word wrap

2. Create a key vault with the Azure CLI:

   KEY_VAULT_NAME="kroxylicious-key-vault"
   RESOURCE_GROUP_NAME="my-resource-group"
   az keyvault create --name ${KEY_VAULT_NAME} --resource-group ${RESOURCE_GROUP_NAME}

   Copy to Clipboard Toggle word wrap
   • Key vault names must be unique.

   • RESOURCE_GROUP_NAME should match the name of the resource group created in the previous section.

     There are several methods to create key vault resources in Azure. This step describes how to do this using the Azure CLI. The Microsoft Azure documentation has further instructions on how to create key vaults via the Azure portal, Azure CLI, and Azure PowerShell.

     Note

     Microsoft’s best practice guide for Azure Key Vault recommends having one key vault per application, per environment, per region. For example, if you have Streams for Apache Kafka Proxy deployed in your development, test, and production environments, and each of those environments is deployed across two different regions, you would create six key vaults for Streams for Apache Kafka Proxy — one for each Streams for Apache Kafka Proxy deployment in each environment in each region.

3. Create a key management user.

   To use the filter, a user such as the Key Vault Owner or any user with the Key Vault Crypto Officer RBAC role must create the encryption keys in the key vault. These keys are used by the envelope encryption process.

   When you create a key vault, only the creator (the Key Vault Owner) can manage it. Access to key vaults is controlled through RBAC roles. A user with either the Role Based Access Control Administrator role or the User Access Administrator role can grant those roles to others. For information on exceptions, see the Microsoft documentation.

   The following Azure CLI commands demonstrate creating a user in Microsoft Entra ID and granting key management RBAC privileges. This example illustrates using the domain example.com but you must use your own domain, such as a *.onmicrosoft.com domain. Substitute the value in the USER_NAME field (kroxylicious-user) for a different username if preferred.

   KEY_VAULT_NAME="kroxylicious-key-vault"
   RESOURCE_GROUP_NAME="my-resource-group"
   
   SUBSCRIPTION_ID=$(az account show --query id --output tsv)
   PORTAL_URL="$(az cloud show --query endpoints.portal --output tsv)/$(az account show --query tenantId --output tsv)"
   
   USER_NAME="kroxylicious-user"
   DEFAULT_DOMAIN=$(az rest --method get --url https://graph.microsoft.com/v1.0/domains --query "value[?isDefault].id" -o tsv)
   USER_PRINCIPAL_NAME="${USER_NAME}@${DEFAULT_DOMAIN}"
   INITIAL_PASSWORD=$(tr -dc 'A-Za-z0-9!?%=' < /dev/urandom | head -c 16)
   
   # Create user
   az ad user create \
     --display-name "${USER_NAME}" \
     --user-principal-name "${USER_PRINCIPAL_NAME}" \
     --password "${INITIAL_PASSWORD}" \
     --force-change-password-next-sign-in true
   
   # Create RBAC role assignment
   az role assignment create \
     --role "Key Vault Crypto Officer" \
     --assignee ${USER_PRINCIPAL_NAME} \
     --scope "/subscriptions/${SUBSCRIPTION_ID}/resourceGroups/${RESOURCE_GROUP_NAME}/providers/Microsoft.KeyVault/vaults/${KEY_VAULT_NAME}"
   
   echo "Now log in at ${PORTAL_URL} with user name ${USER_PRINCIPAL_NAME} and password \"${INITIAL_PASSWORD}\" and change the password."

   Copy to Clipboard Toggle word wrap
   • View all subscriptions for the logged-in user with az account list, and change the active subscription with az account set.
   • The Azure portal URL varies depending on your current Azure cloud.
   • The Key Vault Crypto Officer role lets a user perform any action on the keys within the key vault identified in the --scope, except managing permissions.
   • The scope in the role assignment command specifies the full scope hierarchy of the key vault. For more information on this format, see the Microsoft documentation.

4. Establish an authentication mechanism for the filter.

   The filter must authenticate to Azure Key Vault in order to perform envelope encryption operations, such as generating and decrypting DEKs. See the following procedures for details of the authentication methods available.

Procedure

1. Establish a naming convention for keys within key vault for easy identification and use with the Record Encryption filter. Use separate key vaults per application, environment, and region. This ensures separation between the filter’s keys and those used by other systems.

2. Select a key type.

   Note

   Available key types depend on your Azure Key Vault service tier. See the Azure Key Vault documentation for background information on key types.

   The filter accepts the following key types and their associated wrapping algorithms:

   • Symmetric Keys (oct or oct-HSM): Uses the quantum-resistant A256GCM (256-bit AES GCM) wrapping algorithm, which requires a Managed HSM subscription.
   • Asymmetric Keys (RSA or RSA-HSM): Uses the RSA-OAEP-256 (RSAES-OAEP with SHA-256 hash and MGF1/SHA-256 mask) wrapping algorithm, which is not quantum-resistant. This wrapping algorithm does not require a Managed HSM subscription to use (however, RSA-HSM is not available without a premium Azure Key Vault subscription).

3. Create a key with the Azure CLI:

   KEY_VAULT_NAME="kroxylicious-key-vault"
   KEY_NAME="KEK-<name>"
   KEY_TYPE="rsa"
   az keyvault key create --vault-name ${KEY_VAULT_NAME} --name ${KEY_NAME} --kty ${KEY_TYPE} --ops wrapKey unwrapKey

   Copy to Clipboard Toggle word wrap
   • Change KEY_TYPE if you want to use a key type other than RSA.
   • The wrapKey and unwrapKey operations must be enabled.

4. (Optional) Enable a key rotation policy.

   1. Save your policy to a JSON file. You can refer to an example policy in the Azure CLI documentation.

   2. Apply the rotation policy:

      az keyvault key rotation-policy update --vault-name ${VAULT_NAME} --name ${KEY_NAME} --value path/to/my_policy.json

      Copy to Clipboard Toggle word wrap
      Tip

      It is recommended to use a key rotation policy.

1. Use your KMS console or API to check if the missing key is scheduled for deletion.
2. If so, cancel the deletion to restore the key.

1. Use your KMS’s backup and restore tools to recover the KEK.
2. Ensure that you also restore the original key metadata, such as the key identifier. The Record Encryption filter uses the identifier to reference the KEK in cipher text records.

1. Identify the earliest offset after which all records can be successfully decrypted.

   Proxy instances may not switch to a new KEK at the same time, so records encrypted with different keys might appear together in the log. As a result, there may be no single offset where encryption clearly transitions from one KEK to the next.

   Use kafka-console-consumer.sh with a binary search approach to find the lowest offset for each affected topic partition where decryption succeeds. Domain-specific knowledge can help narrow the search.

2. Use the new starting offset for each affected topic partition to do one of the following:

   • Delete records using kafka-delete-records.sh
     This tool deletes all records up to the specified offset, including any that may still be readable.
   • Advance consumer group offsets using kafka-consumer-groups.sh
     You must reset offsets for every consumer group that must skip the records that cannot be decrypted.