红帽全球峰会此内容没有您所选择的语言版本。
Chapter 1. Key features
Streams for Apache Kafka simplifies the process of running Apache Kafka within an OpenShift cluster.
This guide serves as an introduction to Streams for Apache Kafka, outlining key Kafka concepts that are central to operating Streams for Apache Kafka. It briefly explains Kafka’s components, their purposes, and configuration points, including security and monitoring options. Streams for Apache Kafka provides the necessary files to deploy and manage a Kafka cluster, along with example configuration files for monitoring your deployment.
1.1. Kafka capabilities
Kafka’s data stream-processing capabilities and component architecture offer:
- High-throughput, low-latency data sharing for microservices and other applications
- Guaranteed message ordering
- Message rewind/replay from data storage to reconstruct application state
- Message compaction to remove outdated records in a key-value log
- Horizontal scalability within a cluster
- Data replication to enhance fault tolerance
- High-volume data retention for immediate access
1.2. Kafka use cases
Kafka’s capabilities make it ideal for:
- Event-driven architectures
- Event sourcing to log application state changes
- Message brokering
- Website activity tracking
- Operational monitoring through metrics
- Log collection and aggregation
- Commit logs for distributed systems
- Stream processing for real-time data responses
1.3. How Streams for Apache Kafka supports Kafka
Streams for Apache Kafka provides container images and operators for running Kafka on OpenShift. These operators are designed with specialized operational knowledge to efficiently manage Kafka on OpenShift.
Streams for Apache Kafka operators simplify:
- Deploying and running Kafka clusters
- Deploying and managing Kafka components
- Configuring Kafka access
- Securing Kafka access
- Upgrading Kafka
- Managing brokers
- Creating and managing topics
- Creating and managing users
For detailed information and instructions on using operators to perform these operations, see the guide for Deploying and Managing Streams for Apache Kafka.
Running Kafka on OpenShift without native support from Streams for Apache Kafka can be complex. While deploying Kafka directly with standard resources like StatefulSet and Service is possible, the process is often error-prone and time-consuming. This is especially true for operations like upgrades and configuration updates. Streams for Apache Kafka dramatically reduces this complexity, providing the following advantages:
- Native OpenShift integration
-
Streams for Apache Kafka transforms Kafka into an OpenShift-native application. It extends the Kubernetes API with Custom Resources (CRs) like
Kafka,KafkaTopic, andKafkaUser. Custom resources offer a stable and highly configurable way to manage Kafka. This allows you to define Kafka components at a high level, while the Streams for Apache Kafka operators automatically manage the underlying OpenShift resources for you. This native approach lowers the barrier to adoption and reduces operational overhead, making it easier to deploy and manage Kafka with less manual effort. - Declarative cluster management
- Manage the lifecycle of your Kafka resources declaratively. Declarative control allows you to manage resources like topics and users directly in YAML. This supports an Infrastructure-as-Code (IaC) workflow where Kafka configuration can be version-controlled, audited, and deployed through automated pipelines for a more consistent and repeatable setup.
- Support for upgrade, scaling, and recovery
- Streams for Apache Kafka operators automate rolling upgrades and recovery of Kafka components, helping to reduce manual intervention and downtime. They also support scaling of Kafka clusters through node pools, automated partition reassignment using Cruise Control to maintain balanced workloads, and safe node removal using the Streams for Apache Kafka Drain Cleaner.
- Integrated support for data streaming pipelines
- When Streams for Apache Kafka is installed, you can deploy and manage Kafka clusters alongside supporting components such as Kafka Connect, MirrorMaker 2, and Kafka Bridge, all using OpenShift-native custom resources.
- Integrated security
-
Streams for Apache Kafka enables fine-grained access control through listener-level authentication, cluster-wide authorization, and network policies. It simplifies certificate management to support TLS encryption with configurable protocols and cipher suites, and manages secure client access through
KafkaUserresources with ACLs, quotas, and handling of credentials.
'%20d='M201.5%20305.5c-13.8%200-24.9-11.1-24.9-24.6%200-13.8%2011.1-24.9%2024.9-24.9%2013.6%200%2024.6%2011.1%2024.6%2024.9%200%2013.6-11.1%2024.6-24.6%2024.6zM504%20256c0%20137-111%20248-248%20248S8%20393%208%20256%20119%208%20256%208s248%20111%20248%20248zm-132.3-41.2c-9.4%200-17.7%203.9-23.8%2010-22.4-15.5-52.6-25.5-86.1-26.6l17.4-78.3%2055.4%2012.5c0%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.3%2024.9-24.9s-11.1-24.9-24.9-24.9c-9.7%200-18%205.8-22.1%2013.8l-61.2-13.6c-3-.8-6.1%201.4-6.9%204.4l-19.1%2086.4c-33.2%201.4-63.1%2011.3-85.5%2026.8-6.1-6.4-14.7-10.2-24.1-10.2-34.9%200-46.3%2046.9-14.4%2062.8-1.1%205-1.7%2010.2-1.7%2015.5%200%2052.6%2059.2%2095.2%20132%2095.2%2073.1%200%20132.3-42.6%20132.3-95.2%200-5.3-.6-10.8-1.9-15.8%2031.3-16%2019.8-62.5-14.9-62.5zM302.8%20331c-18.2%2018.2-76.1%2017.9-93.6%200-2.2-2.2-6.1-2.2-8.3%200-2.5%202.5-2.5%206.4%200%208.6%2022.8%2022.8%2087.3%2022.8%20110.2%200%202.5-2.2%202.5-6.1%200-8.6-2.2-2.2-6.1-2.2-8.3%200zm7.7-75c-13.6%200-24.6%2011.1-24.6%2024.9%200%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.1%2024.9-24.6%200-13.8-11-24.9-24.9-24.9z'/%3e%3c/svg%3e){kind=small}