Red Hat Summit Red Hat Summitサポートコンソール開発者トライアルの開始お問い合わせ

このコンテンツは選択した言語では利用できません。

Chapter 5. Operations

This section documents the operational aspects of using the record encryption filter.

5.1. Handling lost KEKs
リンクのコピー

This section describes how to recover or mitigate the loss of a Key Encryption Key (KEK) required for decryption.

A KEK is considered lost if it is no longer usable for decryption even though the Key Management System (KMS) remains accessible to the proxy. For example, the key might be scheduled for deletion or in an invalid state.

Warning

Do not delete KEKs from your KMS. Determining which KEKs are still required for decryption is complex and error-prone. If a KEK is deleted while encrypted records still depend on it, those records become unrecoverable. As a result, consuming applications will encounter errors and stop processing unless additional action is taken. Only follow the procedures in this section if absolutely necessary.

When a consumer attempts to fetch a record that cannot be decrypted, the proxy returns an error. The exact error depends on the Kafka client library:

Apache Kafka client
Unexpected error code 91 while fetching at offset n from topic-partition <topic>-<partition>
librdkafka-based client
Fetch from broker 0 failed at offset n (leader epoch 0): Broker: Request illegally referred to resource that does not exist

These errors indicate that the KEK required for decryption is missing. Error code 91 (RESOURCE_NOT_FOUND) is returned by the Record Encryption filter when the KEK is unavailable.

To confirm the issue, check the proxy logs for entries like the following: 

Failed to decrypt record in topic-partition <topic>-<partition> owing to key not found condition.
This will be reported to the client as a RESOURCE_NOT_FOUND(91).
Client may see a message like 'Unexpected error code 91 while fetching at offset' (java) or or 'Request illegally referred to resource that does not exist' (librdkafka).
Cause message: key 'd691a642-d8b4-4445-b668-d390df7000bb' is not found (AWS error: ErrorResponse{type='KMSInvalidStateException', message='arn:aws:kms:us-east-1:000000000000:key/d691a642-d8b4-4445-b668-d390df7000bb is pending deletion.'}).
Raise log level to DEBUG to see the stack.
Copy to Clipboard Toggle word wrap

If you confirm that a KEK is lost, take one of the following actions:

  • Cancel key deletion
  • Restore key from backup
  • Delete or skip affected records

The actions are listed in recommended order to help restore record consumption. After applying any of the strategies, restart all proxy instances to resume consuming records.

5.1.1. Cancel key deletion
リンクのコピー

Some KMS providers schedule keys for deletion instead of deleting them immediately. During this time, the key appears unavailable but can still be recovered:

  1. Use your KMS console or API to check if the missing key is scheduled for deletion.
  2. If so, cancel the deletion to restore the key.

Refer to the documentation of the KMS for more details.

5.1.2. Restore key from backup
リンクのコピー

If the key was backed up, restore it from the backup:

  1. Use your KMS’s backup and restore tools to recover the KEK.
  2. Ensure that you also restore the original key metadata, such as the key identifier. The Record Encryption filter uses the identifier to reference the KEK in cipher text records.
Important

Restoring the key material alone does not ensure compatibility with encrypted records. You must also recover related metadata, such as the key identifier, to resume successful decryption.

5.1.3. Delete or skip affected records
リンクのコピー

If the KEK cannot be recovered, you must do one of the following:

  • Delete the encrypted records
  • Advance consumer group offsets to skip the affected records

The process is as follows:

  1. Identify the earliest offset after which all records can be successfully decrypted.

    Proxy instances may not switch to a new KEK at the same time, so records encrypted with different keys might appear together in the log. As a result, there may be no single offset where encryption clearly transitions from one KEK to the next.

    Use kafka-console-consumer.sh with a binary search approach to find the lowest offset for each affected topic partition where decryption succeeds. Domain-specific knowledge can help narrow the search.

  2. Use the new starting offset for each affected topic partition to do one of the following:

    • Delete records using kafka-delete-records.sh
      This tool deletes all records up to the specified offset, including any that may still be readable.
    • Advance consumer group offsets using kafka-consumer-groups.sh
      You must reset offsets for every consumer group that must skip the records that cannot be decrypted.
Red Hat logoGithubredditYoutubeTwitter

詳細情報

試用、購入および販売

コミュニティー

Red Hat ドキュメントについて

Red Hat をお使いのお客様が、信頼できるコンテンツが含まれている製品やサービスを活用することで、イノベーションを行い、目標を達成できるようにします。 最新の更新を見る.

多様性を受け入れるオープンソースの強化

Red Hat では、コード、ドキュメント、Web プロパティーにおける配慮に欠ける用語の置き換えに取り組んでいます。このような変更は、段階的に実施される予定です。詳細情報: Red Hat ブログ.

会社概要

Red Hat は、企業がコアとなるデータセンターからネットワークエッジに至るまで、各種プラットフォームや環境全体で作業を簡素化できるように、強化されたソリューションを提供しています。

Theme

© 2026 Red Hatトップに戻る