Red Hat Summit Red Hat Summit지원콘솔개발자평가판 시작연락처

이 콘텐츠는 선택한 언어로 제공되지 않습니다.

Chapter 5. Operations

This section documents the operational aspects of using the record encryption filter.

5.1. Handling lost KEKs
링크 복사

This section describes how to recover or mitigate the loss of a Key Encryption Key (KEK) required for decryption.

A KEK is considered lost if it is no longer usable for decryption even though the Key Management System (KMS) remains accessible to the proxy. For example, the key might be scheduled for deletion or in an invalid state.

Warning

Do not delete KEKs from your KMS. Determining which KEKs are still required for decryption is complex and error-prone. If a KEK is deleted while encrypted records still depend on it, those records become unrecoverable. As a result, consuming applications will encounter errors and stop processing unless additional action is taken. Only follow the procedures in this section if absolutely necessary.

When a consumer attempts to fetch a record that cannot be decrypted, the proxy returns an error. The exact error depends on the Kafka client library:

Apache Kafka client
Unexpected error code 91 while fetching at offset n from topic-partition <topic>-<partition>
librdkafka-based client
Fetch from broker 0 failed at offset n (leader epoch 0): Broker: Request illegally referred to resource that does not exist

These errors indicate that the KEK required for decryption is missing. Error code 91 (RESOURCE_NOT_FOUND) is returned by the Record Encryption filter when the KEK is unavailable.

To confirm the issue, check the proxy logs for entries like the following: 

Failed to decrypt record in topic-partition <topic>-<partition> owing to key not found condition.
This will be reported to the client as a RESOURCE_NOT_FOUND(91).
Client may see a message like 'Unexpected error code 91 while fetching at offset' (java) or or 'Request illegally referred to resource that does not exist' (librdkafka).
Cause message: key 'd691a642-d8b4-4445-b668-d390df7000bb' is not found (AWS error: ErrorResponse{type='KMSInvalidStateException', message='arn:aws:kms:us-east-1:000000000000:key/d691a642-d8b4-4445-b668-d390df7000bb is pending deletion.'}).
Raise log level to DEBUG to see the stack.
Copy to Clipboard Toggle word wrap

If you confirm that a KEK is lost, take one of the following actions:

  • Cancel key deletion
  • Restore key from backup
  • Delete or skip affected records

The actions are listed in recommended order to help restore record consumption. After applying any of the strategies, restart all proxy instances to resume consuming records.

5.1.1. Cancel key deletion
링크 복사

Some KMS providers schedule keys for deletion instead of deleting them immediately. During this time, the key appears unavailable but can still be recovered:

  1. Use your KMS console or API to check if the missing key is scheduled for deletion.
  2. If so, cancel the deletion to restore the key.

Refer to the documentation of the KMS for more details.

5.1.2. Restore key from backup
링크 복사

If the key was backed up, restore it from the backup:

  1. Use your KMS’s backup and restore tools to recover the KEK.
  2. Ensure that you also restore the original key metadata, such as the key identifier. The Record Encryption filter uses the identifier to reference the KEK in cipher text records.
Important

Restoring the key material alone does not ensure compatibility with encrypted records. You must also recover related metadata, such as the key identifier, to resume successful decryption.

5.1.3. Delete or skip affected records
링크 복사

If the KEK cannot be recovered, you must do one of the following:

  • Delete the encrypted records
  • Advance consumer group offsets to skip the affected records

The process is as follows:

  1. Identify the earliest offset after which all records can be successfully decrypted.

    Proxy instances may not switch to a new KEK at the same time, so records encrypted with different keys might appear together in the log. As a result, there may be no single offset where encryption clearly transitions from one KEK to the next.

    Use kafka-console-consumer.sh with a binary search approach to find the lowest offset for each affected topic partition where decryption succeeds. Domain-specific knowledge can help narrow the search.

  2. Use the new starting offset for each affected topic partition to do one of the following:

    • Delete records using kafka-delete-records.sh
      This tool deletes all records up to the specified offset, including any that may still be readable.
    • Advance consumer group offsets using kafka-consumer-groups.sh
      You must reset offsets for every consumer group that must skip the records that cannot be decrypted.
Red Hat logoGithubredditYoutubeTwitter

자세한 정보

평가판, 구매 및 판매

커뮤니티

Red Hat 문서 정보

Red Hat을 사용하는 고객은 신뢰할 수 있는 콘텐츠가 포함된 제품과 서비스를 통해 혁신하고 목표를 달성할 수 있습니다. 최신 업데이트를 확인하세요.

보다 포괄적 수용을 위한 오픈 소스 용어 교체

Red Hat은 코드, 문서, 웹 속성에서 문제가 있는 언어를 교체하기 위해 최선을 다하고 있습니다. 자세한 내용은 다음을 참조하세요.Red Hat 블로그.

Red Hat 소개

Red Hat은 기업이 핵심 데이터 센터에서 네트워크 에지에 이르기까지 플랫폼과 환경 전반에서 더 쉽게 작업할 수 있도록 강화된 솔루션을 제공합니다.

Theme

© 2026 Red Hat맨 위로 이동