Chapter 3. Deploying Streams for Apache Kafka Proxy with the Record Encryption filter

Procedure

1. Download and extract the Streams for Apache Kafka Proxy installation artifacts.

   The artifacts are included with installation and example files available from the Streams for Apache Kafka software downloads page.

   The files contain the deployment configuration required for connecting through a cluster-ip or loadbalancer type listener.

2. Create a topic in the Kafka cluster:

   oc run -n <my_proxy_namespace> -ti proxy-producer \
     --image=registry.redhat.io/amq-streams/kafka-37-rhel9:2.7.0 \
     --rm=true \
     --restart=Never \
     -- bin/kafka-topics.sh \
     --bootstrap-server proxy-service:9092 \
     --create -topic my-topic

   In this example, we create a topic named my-topic through an interactive pod container.

3. Create a key for my-topic in Vault:

   vault write -f transit/keys/KEK_my-topic

4. Edit the ConfigMap that provides the filter configuration for the proxy.

   The Record Encryption filter config requires credentials for the HashiCorp Vault KMS.

   Example Record Encryption filter configuration

   filters:
     - type: RecordEncryption
       config:
         kms: VaultKmsService 

   1

   
         kmsConfig:
           vaultTransitEngineUrl: http://vault.vault.svc.cluster.local:8200/v1/transit 

   2

   
           vaultToken:
             passwordFile: /opt/proxy/encryption/token.txt 

   3

   
         selector: TemplateKekSelector 

   4

   
         selectorConfig:
           template: "KEK_${topicName}" 

   5

   
   	 # ...

   1

   The type of KMS (key management system) used. In this case, HashiCorp Vault.

   2

   The URL of the Vault Transit Engine service.

   3

   The file containing the token required to access the Vault service. If this location changes, equivalent changes are required in the proxy deployment configuration.

   4

   The Key Encryption Key (KEK) selector to use. The ${topicName} is a literal understood by the proxy.

   5

   The template for deriving the KEK, based on a specific topic name.

5. Deploy Streams for Apache Kafka Proxy with the Record Encryption filter and the appropriate listener to your OpenShift cluster:

   Deploying the proxy with a cluster-ip listener

   cd /examples/proxy/record-encryption/
   oc apply -k cluster-ip

   Deploying the proxy with a loadbalancer listener

   cd /examples/proxy/record-encryption/
   oc apply -k loadbalancer

6. If you are using a loadbalancer listener, update the proxy configuration to use the address of loadbalancer service that was created.

   1. Get the external address of the proxy service:

      LOAD_BALANCER_ADDRESS=$(oc get service -n <my_proxy_namespace> proxy-service --template='{{(index .status.loadBalancer.ingress 0).hostname}}')

   2. Update the brokerAddressPattern property in the proxy service configuration to use the broker address:

      sed -i "s/\(brokerAddressPattern:\).*$/\1 ${LOAD_BALANCER_ADDRESS}/" load-balancer/proxy/proxy-config.yaml

   3. Apply the change to the proxy configuration and restart the proxy pod.

       oc apply -k load-balancer && oc delete pod -n <my_proxy_namespace> --all

7. Check the status of the deployment:

   oc get pods -n <my_proxy_namespace>

   Output shows the deployment name and readiness

   NAME                      READY  STATUS   RESTARTS
   my-cluster-kafka-0        1/1    Running  0
   my-cluster-kafka-1        1/1    Running  0
   my-cluster-kafka-2        1/1    Running  0
   my-cluster-proxy-<pod_id> 1/1    Running  0

   my-cluster-proxy is the name of the proxy.

   A pod ID identifies the pod created.

   With the default deployment, you install a single proxy pod.

   READY shows the number of replicas that are ready/expected. The deployment is successful when the STATUS displays as Running.

8. Verify that the encryption has been applied to the specified topics by producing messages through the proxy and then consuming directly and indirectly from the Kafka cluster.