Chapter 1. Overview of AMQ Streams
AMQ Streams simplifies the process of running Apache Kafka in an OpenShift cluster.
1.1. Kafka capabilities
The underlying data stream-processing capabilities and component architecture of Kafka can deliver:
- Microservices and other applications to share data with extremely high throughput and low latency
- Message ordering guarantees
- Message rewind/replay from data storage to reconstruct an application state
- Message compaction to remove old records when using a key-value log
- Horizontal scalability in a cluster configuration
- Replication of data to control fault tolerance
- Retention of high volumes of data for immediate access
1.2. Kafka use cases
Kafka’s capabilities make it suitable for:
- Event-driven architectures
- Event sourcing to capture changes to the state of an application as a log of events
- Message brokering
- Website activity tracking
- Operational monitoring through metrics
- Log collection and aggregation
- Commit logs for distributed systems
- Stream processing so that applications can respond to data in real time
1.3. How AMQ Streams supports Kafka
AMQ Streams provides container images and Operators for running Kafka on OpenShift. AMQ Streams Operators are fundamental to the running of AMQ Streams. The Operators provided with AMQ Streams are purpose-built with specialist operational knowledge to effectively manage Kafka.
Operators simplify the process of:
- Deploying and running Kafka clusters
- Deploying and running Kafka components
- Configuring access to Kafka
- Securing access to Kafka
- Upgrading Kafka
- Managing brokers
- Creating and managing topics
- Creating and managing users
1.4. Operators
AMQ Streams provides Operators for managing a Kafka cluster running within an OpenShift cluster.
- Cluster Operator
- Deploys and manages Apache Kafka clusters, Kafka Connect, Kafka MirrorMaker, Kafka Bridge, Kafka Exporter, and the Entity Operator
- Entity Operator
- Comprises the Topic Operator and User Operator
- Topic Operator
- Manages Kafka topics
- User Operator
- Manages Kafka users
The Cluster Operator can deploy the Topic Operator and User Operator as part of an Entity Operator configuration at the same time as a Kafka cluster.
Operators within the AMQ Streams architecture
1.5. AMQ Streams installation methods
There are two ways to install AMQ Streams on OpenShift.
| Installation method | Description | Supported versions |
|---|---|---|
| Installation artifacts (YAML files) |
Download the | OpenShift 3.11 and later |
| OperatorHub | Use the AMQ Streams Operator in the OperatorHub to deploy the Cluster Operator to a single namespace or all namespaces. | OpenShift 4.x only |
For the greatest flexibility, choose the installation artifacts method. Choose the OperatorHub method if you want to install AMQ Streams to OpenShift 4 in a standard configuration using the OpenShift 4 web console. The OperatorHub also allows you to take advantage of automatic updates.
In the case of both methods, the Cluster Operator is deployed to your OpenShift cluster, ready for you to deploy the other components of AMQ Streams, starting with a Kafka cluster, using the YAML example files provided.
AMQ Streams installation artifacts
The AMQ Streams installation artifacts contain various YAML files that can be deployed to OpenShift, using oc, to create custom resources, including:
- Deployments
- Custom resource definitions (CRDs)
- Roles and role bindings
- Service accounts
YAML installation files are provided for the Cluster Operator, Topic Operator, User Operator, and the Strimzi Admin role.
OperatorHub
In OpenShift 4, the Operator Lifecycle Manager (OLM) helps cluster administrators to install, update, and manage the lifecycle of all Operators and their associated services running across their clusters. The OLM is part of the Operator Framework, an open source toolkit designed to manage Kubernetes-native applications (Operators) in an effective, automated, and scalable way.
The OperatorHub is part of the OpenShift 4 web console. Cluster administrators can use it to discover, install, and upgrade Operators. Operators can be pulled from the OperatorHub, installed on the OpenShift cluster to a single (project) namespace or all (projects) namespaces, and managed by the OLM. Engineering teams can then independently manage the software in development, test, and production environments using the OLM.
The OperatorHub is not available in versions of OpenShift earlier than version 4.
AMQ Streams Operator
The AMQ Streams Operator is available to install from the OperatorHub. Once installed, the AMQ Streams Operator deploys the Cluster Operator to your OpenShift cluster, along with the necessary CRDs and role-based access control (RBAC) resources.
Additional resources
Installing AMQ Streams using the installation artifacts:
Installing AMQ Streams from the OperatorHub:
- Section 2.3.6, “Deploying the Cluster Operator from the OperatorHub”
- Operators guide in the OpenShift documentation.
1.6. Document Conventions
Replaceables
In this document, replaceable text is styled in monospace and italics.
For example, in the following code, you will want to replace my-namespace with the name of your namespace:
sed -i 's/namespace: .*/namespace: my-namespace/' install/cluster-operator/*RoleBinding*.yaml