10.3. Enabling log based bucket replication


When creating a bucket replication policy, you can use logs so that recent data is replicated more quickly, while the default scan-based replication works on replicating the rest of the data.

重要

This feature requires setting up bucket logs on AWS or Azure.For more information about setting up AWS logs, see Enabling Amazon S3 server access logging. The AWS logs bucket needs to be created in the same region as the source NamespaceStore AWS bucket.

注記

This feature is only supported in buckets that are backed by a NamespaceStore. Buckets backed by BackingStores cannot utilized log-based replication.

You can optimize replication by using the event logs of the Amazon Web Service(AWS) cloud environment. You enable log based bucket replication for new namespace buckets using the web console during the creation of namespace buckets.

Prerequisites

  • Ensure that object logging is enabled in AWS. For more information, see the “Using the S3 console” section in Enabling Amazon S3 server access logging.
  • Administrator access to OpenShift Web Console.

Procedure

  1. In the OpenShift Web Console, navigate to Storage Object Storage Object Bucket Claims.
  2. Click Create ObjectBucketClaim.
  3. Enter the name of ObjectBucketName and select StorageClass and BucketClass.
  4. Select the Enable replication check box to enable replication.
  5. In the Replication policy section, select the Optimize replication using event logs checkbox.
  6. Enter the name of the bucket that will contain the logs under Event log Bucket.

    If the logs are not stored in the root of the bucket, provide the full path without s3://

  7. Enter a prefix to replicate only the objects whose name begins with the given prefix.

10.3.2. Enabling log based bucket replication for existing namespace buckets using YAML

You can enable log based bucket replication for the existing buckets that are created using the command line interface or by applying an YAML, and not the buckets that are created using AWS S3 commands.

Procedure

  • Edit the YAML of the bucket’s OBC to enable log based bucket replication. Add the following under spec:

    replicationPolicy: '{"rules":[{"rule_id":"<RULE ID>", "destination_bucket":"<DEST>", "filter": {"prefix": "<PREFIX>"}}], "log_replication_info": {"logs_location": {"logs_bucket": "<LOGS_BUCKET>"}}}'
注記

It is also possible to add this to the YAML of an OBC before it is created.

rule_id
Specify an ID of your choice for identifying the rule
destination_bucket
Specify the name of the target MCG bucket that the objects are copied to
(optional) {"filter": {"prefix": <>}}
Specify a prefix string that you can set to filter the objects that are replicated
log_replication_info
Specify an object that contains data related to log-based replication optimization. {"logs_location": {"logs_bucket": <>}} is set to the location of the AWS S3 server access logs.

10.3.3. Enabling log based bucket replication in Microsoft Azure

Prerequisites

  • Refer to Microsoft Azure documentation and ensure that you have completed the following tasks in the Microsoft Azure portal:

    1. Ensure that have created a new application and noted down the name, application (client) ID, and directory (tenant) ID.

      For information, see Register an application.

    2. Ensure that a new a new client secret is created and the application secret is noted down.
    3. Ensure that a new Log Analytics workspace is created and its name and workspace ID is noted down.

      For information, see Create a Log Analytics workspace.

    4. Ensure that the Reader role is assigned under Access control and members are selected and the name of the application that you registered in the previous step is provided.

      For more information, see Assign Azure roles using the Azure portal.

    5. Ensure that a new storage account is created and the Access keys are noted down.
    6. In the Monitoring section of the storage account created, select a blob and in the Diagnostic settings screen, select only StorageWrite and StorageDelete, and in the destination details add the Log Analytics workspace that you created earlier. Ensure that a blob is selected in the Diagnostic settings screen of the Monitoring section of the storage account created. Also, ensure that only StorageWrite and StorageDelete is selected and in the destination details, the Log Analytics workspace that you created earlier is added.

      For more information, see Diagnostic settings in Azure Monitor.

    7. Ensure that two new containers for object source and object destination are created.
  • Administrator access to OpenShift Web Console.

Procedure

  1. Create a secret with credentials to be used by the namespacestores.

    apiVersion: v1
    kind: Secret
    metadata:
      name: <namespacestore-secret-name>
    type: Opaque
    data:
        TenantID: <AZURE TENANT ID ENCODED IN BASE64>
        ApplicationID: <AZURE APPLICATIOM ID ENCODED IN BASE64>
        ApplicationSecret: <AZURE APPLICATION SECRET ENCODED IN BASE64>
        LogsAnalyticsWorkspaceID: <AZURE LOG ANALYTICS WORKSPACE ID ENCODED IN BASE64>
        AccountName: <AZURE ACCOUNT NAME ENCODED IN BASE64>
        AccountKey: <AZURE ACCOUNT KEY ENCODED IN BASE64>
  2. Create a NamespaceStore backed by a container created in Azure.

    For more information, see Adding a namespace bucket using the OpenShift Container Platform user interface.

  3. Create a new Namespace-Bucketclass and OBC that utilizes it.
  4. Check the object bucket name by looking in the YAML of target OBC, or by listing all S3 buckets, for example, - s3 ls.
  5. Use the following template to apply an Azure replication policy on your source OBC by adding the following in its YAML, under .spec:

    replicationPolicy:'{"rules":[ {"rule_id":"ID goes here", "sync_deletions": "<true or false>"", "destination_bucket":object bucket name"}
     ], "log_replication_info":{"endpoint_type":"AZURE"}}'
    sync_deletion
    Specify a boolean value, true or false.
    destination_bucket
    Make sure to use the name of the object bucket, and not the claim. The name can be retrieved using the s3 ls command, or by looking for the value in an OBC’s YAML.

Verification steps

  1. Write objects to the source bucket.
  2. Wait until MCG replicates them.
  3. Delete the objects from the source bucket.
  4. Verify the objects were removed from the target bucket.

10.3.4. Enabling log-based bucket replication deletion

Prerequisites

  • Administrator access to OpenShift Web Console.
  • AWS Server Access Logging configured for the desired bucket.

Procedure

  1. In the OpenShift Web Console, navigate to Storage Object Storage Object Bucket Claims.
  2. Click Create new Object bucket claim.
  3. (Optional) In the Replication rules section, select the Sync deletion checkbox for each rule separately.
  4. Enter the name of the bucket that will contain the logs under Event log Bucket.

    If the logs are not stored in the root of the bucket, provide the full path without s3://

  5. Enter a prefix to replicate only the objects whose name begins with the given prefix.
Red Hat logoGithubredditYoutubeTwitter

詳細情報

試用、購入および販売

コミュニティー

会社概要

Red Hat は、企業がコアとなるデータセンターからネットワークエッジに至るまで、各種プラットフォームや環境全体で作業を簡素化できるように、強化されたソリューションを提供しています。

多様性を受け入れるオープンソースの強化

Red Hat では、コード、ドキュメント、Web プロパティーにおける配慮に欠ける用語の置き換えに取り組んでいます。このような変更は、段階的に実施される予定です。詳細情報: Red Hat ブログ.

Red Hat ドキュメントについて

Legal Notice

Theme

© 2026 Red Hat
トップに戻る