Using the Streams for Apache Kafka Proxy

Deploying Vault using the example Helm deployment configuration

1. Download and extract the Streams for Apache Kafka Proxy installation artifacts.

   The proxy is available from Streams for Apache Kafka software downloads page.

   The files contain the deployment configuration required for deploying Vault.

2. Create a root token and make a note of it:

   cat /dev/urandom | LC_ALL=C tr -dc 'a-zA-Z0-9' | fold -w 32 | head -n 1 > vault.root.token
   export VAULT_TOKEN=$(cat vault.root.token)

3. Install Vault using Helm:

   helm repo add hashicorp https://helm.releases.hashicorp.com
   helm install vault hashicorp/vault \
     --create-namespace --namespace=vault \
     --version <helm_version> \
     --values vault/helm-dev-values.yaml \
     --set server.dev.devRootToken=${VAULT_TOKEN} \
     --wait

   The root token is used for the Vault instance.

4. Check the status of the deployment:

   oc get pods -n vault

   Output shows the deployment name and readiness

   NAME                      READY  STATUS   RESTARTS
   vault-0                   1/1    Running  0

   A pod ID identifies the pod created.

   With the default deployment, you install a single proxy pod.

   READY shows the number of replicas that are ready/expected. The deployment is successful when the STATUS displays as Running.

5. Create a Vault address (VAULT_ADDR) environment variable to point to the new Vault instance:

   export VAULT_ADDR=$(oc get route -n vault vault --template='https://{{.spec.host}}')

6. Login to Vault as an administrator and enable the Vault Transit secrets engine:

   vault secrets enable transit

   If the secrets engine is already enabled, ignore the error.

7. Create an environment variable to point to the Vault Transit address:

   export VAULT_TRANSIT_URL=${VAULT_ADDR}/v1/transit

   The address is used in the proxy deployment configuration.

8. Create a Vault policy and token.

1. Create a Vault address environment variable (VAULT_ADDR and VAULT_NAMESPACE, if using Enterprise) to point to the Vault instance:

   export VAULT_ADDR=https://<vault server>:8200
   export VAULT_NAMESPACE=<namespaces>

2. Login to Vault as an administrator and enable the Vault Transit secrets engine:

   vault secrets enable transit

   If the secrets engine is already enabled, ignore the error.

3. Create an environment variable to point to the Vault Transit address:

   export VAULT_TRANSIT_URL=${VAULT_ADDR}/v1/${VAULT_NAMESPACE}/transit

   The address is used in the proxy deployment configuration.

4. Update the proxy deployment configuration to refer to your Vault instance:

   sed -i "s/\(vaultTransitEngineUrl:\).*$/\1 ${VAULT_TRANSIT_URL}/" */proxy/proxy-config.yaml

5. Create a Vault policy and token.

1. Create a Vault policy:

   vault policy write amqstreams_proxy_encryption_filter_policy vault/amqstreams_proxy_encryption_filter_policy.hcl

   Write the policy to Vault using the HashiCorp policy definition file (.hcl) provided with Streams for Apache Kafka Proxy. The policy is named amqstreams_proxy_encryption_filter_policy.

2. Create a Vault token:

   vault token create \
     -display-name "amqstreams-proxy encryption filter" \
     -policy=amqstreams_proxy_encryption_filter_policy \
     -no-default-policy \
     -orphan \
     -field=token > vault.encryption.token

   The command creates a token with the specified policy, and with no associated parent token or default policies.

3. Create a secret containing the token:

   oc create secret generic proxy-encryption-vault-token \
     -n proxy \
     --from-file=encryption-vault-token.txt=vault.encryption.token \
     --dry-run=client \
     -o yaml > base/proxy/proxy-encryption-vault-token-secret.yaml

   The command stores the Vault token in the secret and creates the secret as a YAML file in the proxy namespace.

   The proxy-encryption-vault-token-secret.yaml secret is applied to the OpenShift cluster when deploying Streams for Apache Kafka Proxy with the Record Encryption filter.

Procedure

1. Download and extract the Streams for Apache Kafka Proxy installation artifacts.

   The artifacts are included with installation and example files available from the Streams for Apache Kafka software downloads page.

   The files contain the deployment configuration required for connecting through a cluster-ip or loadbalancer type listener.

2. Create a topic in the Kafka cluster:

   oc run -n <my_proxy_namespace> -ti proxy-producer \
     --image=registry.redhat.io/amq-streams/kafka-37-rhel9:2.7.0 \
     --rm=true \
     --restart=Never \
     -- bin/kafka-topics.sh \
     --bootstrap-server proxy-service:9092 \
     --create -topic my-topic

   In this example, we create a topic named my-topic through an interactive pod container.

3. Create a key for my-topic in Vault:

   vault write -f transit/keys/KEK_my-topic

4. Edit the ConfigMap that provides the filter configuration for the proxy.

   The Record Encryption filter config requires credentials for the HashiCorp Vault KMS.

   Example Record Encryption filter configuration

   filters:
     - type: RecordEncryption
       config:
         kms: VaultKmsService 

   1

   
         kmsConfig:
           vaultTransitEngineUrl: http://vault.vault.svc.cluster.local:8200/v1/transit 

   2

   
           vaultToken:
             passwordFile: /opt/proxy/encryption/token.txt 

   3

   
         selector: TemplateKekSelector 

   4

   
         selectorConfig:
           template: "KEK_${topicName}" 

   5

   
   	 # ...

   1

   The type of KMS (key management system) used. In this case, HashiCorp Vault.

   2

   The URL of the Vault Transit Engine service.

   3

   The file containing the token required to access the Vault service. If this location changes, equivalent changes are required in the proxy deployment configuration.

   4

   The Key Encryption Key (KEK) selector to use. The ${topicName} is a literal understood by the proxy.

   5

   The template for deriving the KEK, based on a specific topic name.

5. Deploy Streams for Apache Kafka Proxy with the Record Encryption filter and the appropriate listener to your OpenShift cluster:

   Deploying the proxy with a cluster-ip listener

   cd /examples/proxy/record-encryption/
   oc apply -k cluster-ip

   Deploying the proxy with a loadbalancer listener

   cd /examples/proxy/record-encryption/
   oc apply -k loadbalancer

6. If you are using a loadbalancer listener, update the proxy configuration to use the address of loadbalancer service that was created.

   1. Get the external address of the proxy service:

      LOAD_BALANCER_ADDRESS=$(oc get service -n <my_proxy_namespace> proxy-service --template='{{(index .status.loadBalancer.ingress 0).hostname}}')

   2. Update the brokerAddressPattern property in the proxy service configuration to use the broker address:

      sed -i "s/\(brokerAddressPattern:\).*$/\1 ${LOAD_BALANCER_ADDRESS}/" load-balancer/proxy/proxy-config.yaml

   3. Apply the change to the proxy configuration and restart the proxy pod.

       oc apply -k load-balancer && oc delete pod -n <my_proxy_namespace> --all

7. Check the status of the deployment:

   oc get pods -n <my_proxy_namespace>

   Output shows the deployment name and readiness

   NAME                      READY  STATUS   RESTARTS
   my-cluster-kafka-0        1/1    Running  0
   my-cluster-kafka-1        1/1    Running  0
   my-cluster-kafka-2        1/1    Running  0
   my-cluster-proxy-<pod_id> 1/1    Running  0

   my-cluster-proxy is the name of the proxy.

   A pod ID identifies the pod created.

   With the default deployment, you install a single proxy pod.

   READY shows the number of replicas that are ready/expected. The deployment is successful when the STATUS displays as Running.

8. Verify that the encryption has been applied to the specified topics by producing messages through the proxy and then consuming directly and indirectly from the Kafka cluster.