Chapter 3. Deploying Service Registry storage in AMQ Streams

Procedure

1. In the OpenShift Container Platform web console, log in using an account with cluster administrator privileges.
2. Change to the OpenShift project in which you want to install AMQ Streams. For example, from the Project drop-down, select my-project.
3. In the left navigation menu, click Operators and then OperatorHub.
4. In the Filter by keyword text box, enter AMQ Streams to find the Red Hat Integration - AMQ Streams Operator.
5. Read the information about the Operator, and click Install to display the Operator subscription page.

6. Select your subscription settings, for example:

   • Update Channel and then amq-streams-2.6.x

   • Installation Mode: Select one of the following:

     • All namespaces on the cluster (default)
     • A specific namespace on the cluster > my-project
   • Approval Strategy: Select Automatic or Manual
7. Click Install, and wait a few moments until the Operator is ready for use.

Procedure

1. In the OpenShift Container Platform web console, log in using an account with cluster administrator privileges.

2. If you do not already have a Kafka cluster configured, create a new Kafka cluster using AMQ Streams. For example, in the OpenShift OperatorHub:

   1. Click Installed Operators and then Red Hat Integration - AMQ Streams.
   2. Under Provided APIs and then Kafka, click Create Instance to create a new Kafka cluster.

   3. Edit the custom resource definition as appropriate, and click Create.

      Warning

      The default example creates a cluster with 3 Zookeeper nodes and 3 Kafka nodes with ephemeral storage. This temporary storage is suitable for development and testing only, and not for production. For more details, see Deploying and Managing AMQ Streams on OpenShift.

3. After the cluster is ready, click Provided APIs > Kafka > my-cluster > YAML.

4. In the status block, make a copy of the bootstrapServers value, which you will use later to deploy Service Registry. For example:

   status:
     ...
     conditions:
     ...
     listeners:
       - addresses:
           - host: my-cluster-kafka-bootstrap.my-project.svc
             port: 9092
         bootstrapServers: 'my-cluster-kafka-bootstrap.my-project.svc:9092'
         type: plain
     ...

5. Click Installed Operators > Red Hat Integration - Service Registry > ApicurioRegistry > Create ApicurioRegistry.

6. Paste in the following custom resource definition, but use your bootstrapServers value that you copied earlier:

   apiVersion: registry.apicur.io/v1
   kind: ApicurioRegistry
   metadata:
     name: example-apicurioregistry-kafkasql
   spec:
     configuration:
       persistence: 'kafkasql'
       kafkasql:
         bootstrapServers: 'my-cluster-kafka-bootstrap.my-project.svc:9092'

7. Click Create and wait for the Service Registry route to be created on OpenShift.

8. Click Networking > Route to access the new route for the Service Registry web console. For example:

   http://example-apicurioregistry-kafkasql.my-project.my-domain-name.com/

9. To configure the Kafka topic that Service Registry uses to store data, click Installed Operators > Red Hat Integration - AMQ Streams > Provided APIs > Kafka Topic > kafkasql-journal > YAML. For example:

   apiVersion: kafka.strimzi.io/v1beta2
   kind: KafkaTopic
   metadata:
     name: kafkasql-journal
     labels:
       strimzi.io/cluster: my-cluster
     namespace: ...
   spec:
     partitions: 3
     replicas: 3
     config:
       cleanup.policy: compact

   Warning

   You must configure the Kafka topic used by Service Registry (named kafkasql-journal by default) with a compaction cleanup policy, otherwise a data loss might occur.

Procedure

1. In the OpenShift web console, click Installed Operators, select the AMQ Streams Operator details, and then the Kafka tab.
2. Click Create Kafka to provision a new Kafka cluster for Service Registry storage.

3. Configure the authorization and tls fields to use TLS authentication for the Kafka cluster, for example:

   apiVersion: kafka.strimzi.io/v1beta2
   kind: Kafka
   metadata:
     name: my-cluster
     namespace: registry-example-kafkasql-tls
     # Change or remove the explicit namespace
   spec:
     kafka:
       config:
         offsets.topic.replication.factor: 3
         transaction.state.log.replication.factor: 3
         transaction.state.log.min.isr: 2
         log.message.format.version: '2.7'
         inter.broker.protocol.version: '2.7'
       version: 2.7.0
       storage:
         type: ephemeral
       replicas: 3
       listeners:
         - name: tls
           port: 9093
           type: internal
           tls: true
           authentication:
             type: tls
       authorization:
         type: simple
     entityOperator:
       topicOperator: {}
       userOperator: {}
     zookeeper:
       storage:
         type: ephemeral
       replicas: 3

   The default Kafka topic name automatically created by Service Registry to store data is kafkasql-journal. You can override this behavior or the default topic name by setting environment variables. The default values are as follows:

   • REGISTRY_KAFKASQL_TOPIC_AUTO_CREATE=true
   • REGISTRY_KAFKASQL_TOPIC=kafkasql-journal

   If you decide not to create the Kafka topic manually, skip the next step.

4. Click the Kafka Topic tab, and then Create Kafka Topic to create the kafkasql-journal topic:

   apiVersion: kafka.strimzi.io/v1beta1
   kind: KafkaTopic
   metadata:
     name: kafkasql-journal
     labels:
       strimzi.io/cluster: my-cluster
     namespace: registry-example-kafkasql-tls
   spec:
     partitions: 2
     replicas: 1
     config:
       cleanup.policy: compact

5. Create a Kafka User resource to configure authentication and authorization for the Service Registry user. You can specify a user name in the metadata section or use the default my-user.

   apiVersion: kafka.strimzi.io/v1beta1
   kind: KafkaUser
   metadata:
     name: my-user
     labels:
       strimzi.io/cluster: my-cluster
     namespace: registry-example-kafkasql-tls
   spec:
     authentication:
       type: tls
     authorization:
       acls:
         - operation: All
           resource:
             name: '*'
             patternType: literal
             type: topic
         - operation: All
           resource:
             name: '*'
             patternType: literal
             type: cluster
         - operation: All
           resource:
             name: '*'
             patternType: literal
             type: transactionalId
         - operation: All
           resource:
             name: '*'
             patternType: literal
             type: group
       type: simple

   Note

   This simple example assumes admin permissions and creates the Kafka topic automatically. You must configure the authorization section specifically for the topics and resources that the Service Registry requires.

   The following example shows the minimum configuration required when the Kafka topic is created manually:

    ...
     authorization:
       acls:
       - operations:
           - Read
           - Write
         resource:
           name: kafkasql-journal
           patternType: literal
           type: topic
       - operations:
           - Read
           - Write
         resource:
           name: apicurio-registry-
           patternType: prefix
           type: group
       type: simple

6. Click Workloads and then Secrets to find two secrets that AMQ Streams creates for Service Registry to connect to the Kafka cluster:

   • my-cluster-cluster-ca-cert - contains the PKCS12 truststore for the Kafka cluster

   • my-user - contains the user’s keystore

     Note

     The name of the secret can vary based on your cluster or user name.

7. If you create the secrets manually, they must contain the following key-value pairs:

   • my-cluster-ca-cert

     • ca.p12 - truststore in PKCS12 format
     • ca.password - truststore password

   • my-user

     • user.p12 - keystore in PKCS12 format
     • user.password - keystore password

8. Configure the following example configuration to deploy the Service Registry.

   apiVersion: registry.apicur.io/v1
   kind: ApicurioRegistry
   metadata:
     name: example-apicurioregistry-kafkasql-tls
   spec:
     configuration:
       persistence: "kafkasql"
       kafkasql:
         bootstrapServers: "my-cluster-kafka-bootstrap.registry-example-kafkasql-tls.svc:9093"
         security:
           tls:
             keystoreSecretName: my-user
             truststoreSecretName: my-cluster-cluster-ca-cert

Procedure

1. In the OpenShift web console, click Installed Operators, select the AMQ Streams Operator details, and then the Kafka tab.
2. Click Create Kafka to provision a new Kafka cluster for Service Registry storage.

3. Configure the authorization and tls fields to use SCRAM-SHA-512 authentication for the Kafka cluster, for example:

   apiVersion: kafka.strimzi.io/v1beta2
   kind: Kafka
   metadata:
     name: my-cluster
     namespace: registry-example-kafkasql-scram
     # Change or remove the explicit namespace
   spec:
     kafka:
       config:
         offsets.topic.replication.factor: 3
         transaction.state.log.replication.factor: 3
         transaction.state.log.min.isr: 2
         log.message.format.version: '2.7'
         inter.broker.protocol.version: '2.7'
       version: 2.7.0
       storage:
         type: ephemeral
       replicas: 3
       listeners:
         - name: tls
           port: 9093
           type: internal
           tls: true
           authentication:
             type: scram-sha-512
       authorization:
         type: simple
     entityOperator:
       topicOperator: {}
       userOperator: {}
     zookeeper:
       storage:
         type: ephemeral
       replicas: 3

   The default Kafka topic name automatically created by Service Registry to store data is kafkasql-journal. You can override this behavior or the default topic name by setting environment variables. The default values are as follows:

   • REGISTRY_KAFKASQL_TOPIC_AUTO_CREATE=true
   • REGISTRY_KAFKASQL_TOPIC=kafkasql-journal

   If you decide not to create the Kafka topic manually, skip the next step.

4. Click the Kafka Topic tab, and then Create Kafka Topic to create the kafkasql-journal topic:

   apiVersion: kafka.strimzi.io/v1beta1
   kind: KafkaTopic
   metadata:
     name: kafkasql-journal
     labels:
       strimzi.io/cluster: my-cluster
     namespace: registry-example-kafkasql-scram
   spec:
     partitions: 2
     replicas: 1
     config:
       cleanup.policy: compact

5. Create a Kafka User resource to configure SCRAM authentication and authorization for the Service Registry user. You can specify a user name in the metadata section or use the default my-user.

   apiVersion: kafka.strimzi.io/v1beta1
   kind: KafkaUser
   metadata:
     name: my-user
     labels:
       strimzi.io/cluster: my-cluster
     namespace: registry-example-kafkasql-scram
   spec:
     authentication:
       type: scram-sha-512
     authorization:
       acls:
         - operation: All
           resource:
             name: '*'
             patternType: literal
             type: topic
         - operation: All
           resource:
             name: '*'
             patternType: literal
             type: cluster
         - operation: All
           resource:
             name: '*'
             patternType: literal
             type: transactionalId
         - operation: All
           resource:
             name: '*'
             patternType: literal
             type: group
       type: simple

   Note

   This simple example assumes admin permissions and creates the Kafka topic automatically. You must configure the authorization section specifically for the topics and resources that the Service Registry requires.

   The following example shows the minimum configuration required when the Kafka topic is created manually:

    ...
     authorization:
       acls:
       - operations:
           - Read
           - Write
         resource:
           name: kafkasql-journal
           patternType: literal
           type: topic
       - operations:
           - Read
           - Write
         resource:
           name: apicurio-registry-
           patternType: prefix
           type: group
       type: simple

6. Click Workloads and then Secrets to find two secrets that AMQ Streams creates for Service Registry to connect to the Kafka cluster:

   • my-cluster-cluster-ca-cert - contains the PKCS12 truststore for the Kafka cluster

   • my-user - contains the user’s keystore

     Note

     The name of the secret can vary based on your cluster or user name.

7. If you create the secrets manually, they must contain the following key-value pairs:

   • my-cluster-ca-cert

     • ca.p12 - the truststore in PKCS12 format
     • ca.password - truststore password

   • my-user

     • password - user password

8. Configure the following example settings to deploy the Service Registry:

   apiVersion: registry.apicur.io/v1
   kind: ApicurioRegistry
   metadata:
     name: example-apicurioregistry-kafkasql-scram
   spec:
     configuration:
       persistence: "kafkasql"
       kafkasql:
         bootstrapServers: "my-cluster-kafka-bootstrap.registry-example-kafkasql-scram.svc:9093"
         security:
           scram:
             truststoreSecretName: my-cluster-cluster-ca-cert
             user: my-user
             passwordSecretName: my-user