红帽全球峰会此内容没有您所选择的语言版本。
Chapter 30. Finding information on Kafka restarts
After the Cluster Operator restarts a Kafka pod in an OpenShift cluster, it emits an OpenShift event into the pod’s namespace explaining why the pod restarted. For help in understanding cluster behavior, you can check restart events from the command line.
You can export and monitor restart events using metrics collection tools like Prometheus. Use the metrics tool with an event exporter that can export the output in a suitable format.
30.1. Reasons for a restart event
The Cluster Operator initiates a restart event for a specific reason. You can check the reason by fetching information on the restart event.
| Event | Description |
|---|---|
| CaCertHasOldGeneration | The pod is still using a server certificate signed with an old CA, so needs to be restarted as part of the certificate update. |
| CaCertRemoved | Expired CA certificates have been removed, and the pod is restarted to run with the current certificates. |
| CaCertRenewed | CA certificates have been renewed, and the pod is restarted to run with the updated certificates. |
| ClusterCaCertKeyReplaced | The key used to sign the cluster’s CA certificates has been replaced, and the pod is being restarted as part of the CA renewal process. |
| ConfigChangeRequiresRestart | Some Kafka configuration properties are changed dynamically, but others require that the broker be restarted. |
| FileSystemResizeNeeded | The file system size has been increased, and a restart is needed to apply it. |
| KafkaCertificatesChanged | One or more TLS certificates used by the Kafka broker have been updated, and a restart is needed to use them. |
| ManualRollingUpdate |
A user annotated the pod, or the |
| PodForceRestartOnError | An error occurred that requires a pod restart to rectify. |
| PodHasOldRevision |
A disk was added or removed from the Kafka volumes, and a restart is needed to apply the change. When using |
| PodHasOldRevision |
The |
| PodStuck | The pod is still pending, and is not scheduled or cannot be scheduled, so the operator has restarted the pod in a final attempt to get it running. |
| PodUnresponsive | Streams for Apache Kafka was unable to connect to the pod, which can indicate a broker not starting correctly, so the operator restarted it in an attempt to resolve the issue. |
30.2. Restart event filters
When checking restart events from the command line, you can specify a field-selector to filter on OpenShift event fields.
The following fields are available when filtering events with field-selector.
regardingObject.kind-
The resource that owns the Pod being restarted, and for restart events, the kind is always
Kafka. regarding.namespace- The namespace that the resource belongs to.
regardingObject.name-
The resource’s name, for example,
strimzi-cluster. regardingObject.uid- The unique ID of the resource.
reason-
The reason the Pod was restarted, for example,
JbodVolumesChanged. reportingController-
The reporting component is always
strimzi.io/cluster-operatorfor Streams for Apache Kafka restart events. source-
sourceis an older version ofreportingController. The reporting component is alwaysstrimzi.io/cluster-operatorfor Streams for Apache Kafka restart events. type-
The event type, which is either
WarningorNormal. For Streams for Apache Kafka restart events, the type isNormal.
In older versions of OpenShift, the fields using the regarding prefix might use an involvedObject prefix instead. reportingController was previously called reportingComponent.
30.3. Checking Kafka restarts
Use a oc command to list restart events initiated by the Cluster Operator. Filter restart events emitted by the Cluster Operator by setting the Cluster Operator as the reporting component using the reportingController or source event fields.
Prerequisites
- The Cluster Operator is running in the OpenShift cluster.
Procedure
Get all restart events emitted by the Cluster Operator:
oc -n kafka get events --field-selector reportingController=strimzi.io/cluster-operatorExample showing events returned
LAST SEEN TYPE REASON OBJECT MESSAGE 2m Normal CaCertRenewed kafka/strimzi-cluster Rolling Pod strimzi-cluster-kafka-0 due to CA certificate renewed 58m Normal PodForceRestartOnError kafka/strimzi-cluster Rolling Pod strimzi-cluster-kafka-1 due to Pod needs to be forcibly restarted due to an error 5m47s Normal ManualRollingUpdate kafka/strimzi-cluster Rolling Pod strimzi-cluster-kafka-2 due to Pod was manually annotated to be rolledYou can also specify a
reasonor otherfield-selectoroptions to constrain the events returned.Here, a specific reason is added:
oc -n kafka get events --field-selector reportingController=strimzi.io/cluster-operator,reason=PodForceRestartOnErrorUse an output format, such as YAML, to return more detailed information about one or more events.
oc -n kafka get events --field-selector reportingController=strimzi.io/cluster-operator,reason=PodForceRestartOnError -o yamlExample showing detailed events output
apiVersion: v1 items: - action: StrimziInitiatedPodRestart apiVersion: v1 eventTime: "2022-05-13T00:22:34.168086Z" firstTimestamp: null involvedObject: kind: Kafka name: strimzi-cluster namespace: kafka kind: Event lastTimestamp: null message: Rolling Pod strimzi-cluster-kafka-1 due to Pod needs to be forcibly restarted due to an error metadata: creationTimestamp: "2022-05-13T00:22:34Z" generateName: strimzi-event name: strimzi-eventwppk6 namespace: kafka resourceVersion: "432961" uid: 29fcdb9e-f2cf-4c95-a165-a5efcd48edfc reason: PodForceRestartOnError related: kind: Pod name: strimzi-cluster-kafka-1 namespace: kafka reportingController: strimzi.io/cluster-operator reportingInstance: strimzi-cluster-operator-6458cfb4c6-6bpdp source: {} type: Normal kind: List metadata: resourceVersion: "" selfLink: ""
The following fields are deprecated, so they are not populated for these events:
-
firstTimestamp -
lastTimestamp -
source
'%20d='M201.5%20305.5c-13.8%200-24.9-11.1-24.9-24.6%200-13.8%2011.1-24.9%2024.9-24.9%2013.6%200%2024.6%2011.1%2024.6%2024.9%200%2013.6-11.1%2024.6-24.6%2024.6zM504%20256c0%20137-111%20248-248%20248S8%20393%208%20256%20119%208%20256%208s248%20111%20248%20248zm-132.3-41.2c-9.4%200-17.7%203.9-23.8%2010-22.4-15.5-52.6-25.5-86.1-26.6l17.4-78.3%2055.4%2012.5c0%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.3%2024.9-24.9s-11.1-24.9-24.9-24.9c-9.7%200-18%205.8-22.1%2013.8l-61.2-13.6c-3-.8-6.1%201.4-6.9%204.4l-19.1%2086.4c-33.2%201.4-63.1%2011.3-85.5%2026.8-6.1-6.4-14.7-10.2-24.1-10.2-34.9%200-46.3%2046.9-14.4%2062.8-1.1%205-1.7%2010.2-1.7%2015.5%200%2052.6%2059.2%2095.2%20132%2095.2%2073.1%200%20132.3-42.6%20132.3-95.2%200-5.3-.6-10.8-1.9-15.8%2031.3-16%2019.8-62.5-14.9-62.5zM302.8%20331c-18.2%2018.2-76.1%2017.9-93.6%200-2.2-2.2-6.1-2.2-8.3%200-2.5%202.5-2.5%206.4%200%208.6%2022.8%2022.8%2087.3%2022.8%20110.2%200%202.5-2.2%202.5-6.1%200-8.6-2.2-2.2-6.1-2.2-8.3%200zm7.7-75c-13.6%200-24.6%2011.1-24.6%2024.9%200%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.1%2024.9-24.6%200-13.8-11-24.9-24.9-24.9z'/%3e%3c/svg%3e){kind=small}