Chapter 17. Geo-replication

Procedure

1. Update your config.yaml file to include the storage engines to which data will be replicated. You must list all storage engines to be used:

   # ...
   FEATURE_STORAGE_REPLICATION: true
   # ...
   DISTRIBUTED_STORAGE_CONFIG:
       usstorage:
           - RHOCSStorage
           - access_key: <access_key>
             bucket_name: <example_bucket>
             hostname: my.noobaa.hostname
             is_secure: false
             port: "443"
             secret_key: <secret_key>
             storage_path: /datastorage/registry
       eustorage:
           - S3Storage
           - host: s3.amazon.com
             port: "443"
             s3_access_key: <access_key>
             s3_bucket: <example bucket>
             s3_secret_key: <secret_key>
             storage_path: /datastorage/registry
   DISTRIBUTED_STORAGE_DEFAULT_LOCATIONS: []
   DISTRIBUTED_STORAGE_PREFERENCE:
       - usstorage
       - eustorage
   # ...

2. Optional. If complete replication of all images to all storage engines is required, you can replicate images to the storage engine by manually setting the DISTRIBUTED_STORAGE_DEFAULT_LOCATIONS field. This ensures that all images are replicated to that storage engine. For example:

   # ...
   DISTRIBUTED_STORAGE_DEFAULT_LOCATIONS:
       - usstorage
       - eustorage
   # ...

   Note

   To enable per-namespace replication, contact Red Hat Quay support.

3. After adding storage and enabling Replicate to storage engine by default for geo-replication, you must sync existing image data across all storage. To do this, you must execute into the container by running the following command:

   $ podman exec -it <container_id>

4. To sync the content after adding new storage, enter the following commands:

   # scl enable python27 bash

   # python -m util.backfillreplication

   Note

   This is a one time operation to sync content after adding new storage.

1. Copy the config.yaml to all machines running Red Hat Quay

2. For each machine in each region, add a QUAY_DISTRIBUTED_STORAGE_PREFERENCE environment variable with the preferred storage engine for the region in which the machine is running.

   For example, for a machine running in Europe with the config directory on the host available from $QUAY/config:

   $ sudo podman run -d --rm -p 80:8080 -p 443:8443  \
      --name=quay \
      -v $QUAY/config:/conf/stack:Z \
      -e QUAY_DISTRIBUTED_STORAGE_PREFERENCE=europestorage \
      registry.redhat.io/quay/quay-rhel8:v3.13.1

   Note

   The value of the environment variable specified must match the name of a Location ID as defined in the config panel.

3. Restart all Red Hat Quay containers

Procedure

1. Sync the blobs between all of your defined sites by running the following command:

   $ python -m util.backfillreplication

   Warning

   Prior to removing storage engines from your Red Hat Quay config.yaml file, you must ensure that all blobs are synced between all defined sites. Complete this step before proceeding.

2. In your Red Hat Quay config.yaml file for site usstorage, remove the DISTRIBUTED_STORAGE_CONFIG entry for the eustorage site.

3. Enter the following command to obtain a list of running containers:

   $ podman ps

   Example output

   CONTAINER ID  IMAGE                                                                     COMMAND         CREATED         STATUS             PORTS                                        NAMES
   92c5321cde38  registry.redhat.io/rhel8/redis-5:1                                        run-redis       11 days ago     Up 11 days ago     0.0.0.0:6379->6379/tcp                       redis
   4e6d1ecd3811  registry.redhat.io/rhel8/postgresql-13:1-109                              run-postgresql  33 seconds ago  Up 34 seconds ago  0.0.0.0:5432->5432/tcp                       postgresql-quay
   d2eadac74fda  registry-proxy.engineering.redhat.com/rh-osbs/quay-quay-rhel8:v3.9.0-131  registry        4 seconds ago   Up 4 seconds ago   0.0.0.0:80->8080/tcp, 0.0.0.0:443->8443/tcp  quay

4. Enter the following command to execute a shell inside of the PostgreSQL container:

   $ podman exec -it postgresql-quay -- /bin/bash

5. Enter psql by running the following command:

   bash-4.4$ psql

6. Enter the following command to reveal a list of sites in your geo-replicated deployment:

   quay=# select * from imagestoragelocation;

   Example output

    id |       name
   ----+-------------------
     1 | usstorage
     2 | eustorage

7. Enter the following command to exit the postgres CLI to re-enter bash-4.4:

   \q

8. Enter the following command to permanently remove the eustorage site:

   Important

   The following action cannot be undone. Use with caution.

   bash-4.4$ python -m util.removelocation eustorage

   Example output

   WARNING: This is a destructive operation. Are you sure you want to remove eustorage from your storage locations? [y/n] y
   Deleted placement 30
   Deleted placement 31
   Deleted placement 32
   Deleted placement 33
   Deleted location eustorage

Procedure

1. Deploy a postgres instance for Red Hat Quay.

2. Login to the database by entering the following command:

   psql -U <username> -h <hostname> -p <port> -d <database_name>

3. Create a database for Red Hat Quay named quay. For example:

   CREATE DATABASE quay;

4. Enable pg_trm extension inside the database

   \c quay;
   CREATE EXTENSION IF NOT EXISTS pg_trgm;

5. Deploy a Redis instance:

   Note
   • Deploying a Redis instance might be unnecessary if your cloud provider has its own service.
   • Deploying a Redis instance is required if you are leveraging Builders.
   1. Deploy a VM for Redis
   2. Verify that it is accessible from the clusters where Red Hat Quay is running
   3. Port 6379/TCP must be open

   4. Run Redis inside the instance

      sudo dnf install -y podman
      podman run -d --name redis -p 6379:6379 redis

6. Create two object storage backends, one for each cluster. Ideally, one object storage bucket will be close to the first, or primary, cluster, and the other will run closer to the second, or secondary, cluster.
7. Deploy the clusters with the same config bundle, using environment variable overrides to select the appropriate storage backend for an individual cluster.
8. Configure a load balancer to provide a single entry point to the clusters.

Procedure

1. Sync the blobs between all of your defined sites by running the following command:

   $ python -m util.backfillreplication

   Warning

   Prior to removing storage engines from your Red Hat Quay config.yaml file, you must ensure that all blobs are synced between all defined sites.

   When running this command, replication jobs are created which are picked up by the replication worker. If there are blobs that need replicated, the script returns UUIDs of blobs that will be replicated. If you run this command multiple times, and the output from the return script is empty, it does not mean that the replication process is done; it means that there are no more blobs to be queued for replication. Customers should use appropriate judgement before proceeding, as the allotted time replication takes depends on the number of blobs detected.

   Alternatively, you could use a third party cloud tool, such as Microsoft Azure, to check the synchronization status.

   This step must be completed before proceeding.

2. In your Red Hat Quay config.yaml file for site usstorage, remove the DISTRIBUTED_STORAGE_CONFIG entry for the eustorage site.

3. Enter the following command to identify your Quay application pods:

   $ oc get pod -n <quay_namespace>

   Example output

   quay390usstorage-quay-app-5779ddc886-2drh2
   quay390eustorage-quay-app-66969cd859-n2ssm

4. Enter the following command to open an interactive shell session in the usstorage pod:

   $ oc rsh quay390usstorage-quay-app-5779ddc886-2drh2

5. Enter the following command to permanently remove the eustorage site:

   Important

   The following action cannot be undone. Use with caution.

   sh-4.4$ python -m util.removelocation eustorage

   Example output

   WARNING: This is a destructive operation. Are you sure you want to remove eustorage from your storage locations? [y/n] y
   Deleted placement 30
   Deleted placement 31
   Deleted placement 32
   Deleted placement 33
   Deleted location eustorage