Chapter 4. Geo-replication

Procedure

1. Deploy a PostgreSQL instance for Red Hat Quay.

2. Login to the database by entering the following command:

   psql -U <username> -h <hostname> -p <port> -d <database_name>

3. Create a database for Red Hat Quay named quay. For example:

   CREATE DATABASE quay;

4. Enable pg_trm extension inside the database

   \c quay;
   CREATE EXTENSION IF NOT EXISTS pg_trgm;

5. Deploy a Redis instance:

   Note
   • Deploying a Redis instance might be unnecessary if your cloud provider has its own service.
   • Deploying a Redis instance is required if you are leveraging Builders.
   1. Deploy a VM for Redis
   2. Verify that it is accessible from the clusters where Red Hat Quay is running
   3. Port 6379/TCP must be open

   4. Run Redis inside the instance

      sudo dnf install -y podman
      podman run -d --name redis -p 6379:6379 redis

6. Create two object storage backends, one for each cluster. Ideally, one object storage bucket will be close to the first, or primary, cluster, and the other will run closer to the second, or secondary, cluster.
7. Deploy the clusters with the same config bundle, using environment variable overrides to select the appropriate storage backend for an individual cluster.
8. Configure a load balancer to provide a single entry point to the clusters.

Procedure

1. Create a config.yaml file that is shared between clusters. This config.yaml file contains the details for the common PostgreSQL, Redis and storage backends:

   Geo-replication config.yaml file

   SERVER_HOSTNAME: <georep.quayteam.org or any other name> 

   1

   
   DB_CONNECTION_ARGS:
     autorollback: true
     threadlocals: true
   DB_URI: postgresql://postgres:password@10.19.0.1:5432/quay
   BUILDLOGS_REDIS:
     host: 10.19.0.2
     port: 6379
   USER_EVENTS_REDIS:
     host: 10.19.0.2
     port: 6379
   DATABASE_SECRET_KEY: 0ce4f796-c295-415b-bf9d-b315114704b8
   DISTRIBUTED_STORAGE_CONFIG:
     usstorage:
       - GoogleCloudStorage
       - access_key: GOOGQGPGVMASAAMQABCDEFG
         bucket_name: georep-test-bucket-0
         secret_key: AYWfEaxX/u84XRA2vUX5C987654321
         storage_path: /quaygcp
     eustorage:
       - GoogleCloudStorage
       - access_key: GOOGQGPGVMASAAMQWERTYUIOP
         bucket_name: georep-test-bucket-1
         secret_key: AYWfEaxX/u84XRA2vUX5Cuj12345678
         storage_path: /quaygcp
   DISTRIBUTED_STORAGE_DEFAULT_LOCATIONS:
     - usstorage
     - eustorage
   DISTRIBUTED_STORAGE_PREFERENCE:
     - usstorage
     - eustorage
   FEATURE_STORAGE_REPLICATION: true

   where:

   SERVER_HOSTNAME:: Specifies the hostname of the global load balancer. Must match the hostname of the global load balancer.

2. Create the configBundleSecret custom resource (CR) by entering the following command:

   $ oc create secret generic --from-file config.yaml=./config.yaml georep-config-bundle

3. In each of the clusters, set the configBundleSecret and use the QUAY_DISTRIBUTED_STORAGE_PREFERENCE environmental variable override to configure the appropriate storage for that cluster. For example:

   Note

   The config.yaml file between both deployments must match. If making a change to one cluster, it must also be changed in the other.

   US cluster QuayRegistry example

   apiVersion: quay.redhat.com/v1
   kind: QuayRegistry
   metadata:
     name: example-registry
     namespace: quay-enterprise
   spec:
     configBundleSecret: georep-config-bundle
     components:
       - kind: objectstorage
         managed: false
       - kind: route
         managed: true
       - kind: tls
         managed: false
       - kind: postgres
         managed: false
       - kind: clairpostgres
         managed: false
       - kind: redis
         managed: false
       - kind: quay
         managed: true
         overrides:
           env:
           - name: QUAY_DISTRIBUTED_STORAGE_PREFERENCE
             value: usstorage
       - kind: mirror
         managed: true
         overrides:
           env:
           - name: QUAY_DISTRIBUTED_STORAGE_PREFERENCE
             value: usstorage

   Note

   Because SSL/TLS is unmanaged, and the route is managed, you must supply the certificates directly in the config bundle. For more information, see Configuring SSL/TLS and Routes.

   European cluster

   apiVersion: quay.redhat.com/v1
   kind: QuayRegistry
   metadata:
     name: example-registry
     namespace: quay-enterprise
   spec:
     configBundleSecret: georep-config-bundle
     components:
       - kind: objectstorage
         managed: false
       - kind: route
         managed: true
       - kind: tls
         managed: false
       - kind: postgres
         managed: false
       - kind: clairpostgres
         managed: false
       - kind: redis
         managed: false
       - kind: quay
         managed: true
         overrides:
           env:
           - name: QUAY_DISTRIBUTED_STORAGE_PREFERENCE
             value: eustorage
       - kind: mirror
         managed: true
         overrides:
           env:
           - name: QUAY_DISTRIBUTED_STORAGE_PREFERENCE
             value: eustorage

   Note

   Because SSL/TLS is unmanaged, and the route is managed, you must supply the certificates directly in the config bundle. For more information, see Configuring SSL and TLS for Red Hat Quay.

Procedure

1. On System B and System C, scale down your Red Hat Quay registry. This is done by disabling auto scaling and overriding the replica county for Red Hat Quay, mirror workers, and Clair if it is managed. Use the following quayregistry.yaml file as a reference:

   apiVersion: quay.redhat.com/v1
   kind: QuayRegistry
   metadata:
     name: registry
     namespace: ns
   spec:
     components:
       …
       - kind: horizontalpodautoscaler
         managed: false
       - kind: quay
         managed: true
         overrides:
           replicas: 0
       - kind: clair
         managed: true
         overrides:
           replicas: 0
       - kind: mirror
         managed: true
         overrides:
           replicas: 0
       …

   where:

   managed: false:: Disables auto scaling of Quay, Clair and Mirroring workers overrides:: Sets the replica count to 0 for components accessing the database and objectstorage

   Note

   You must keep the Red Hat Quay registry running on System A. Do not update the quayregistry.yaml file on System A.

2. Wait for the registry-quay-app, registry-quay-mirror, and registry-clair-app pods to disappear. Enter the following command to check their status:

   oc get pods -n <quay-namespace>

   Example output

   quay-operator.v3.7.1-6f9d859bd-p5ftc               1/1     Running     0             12m
   quayregistry-clair-postgres-7487f5bd86-xnxpr       1/1     Running     1 (12m ago)   12m
   quayregistry-quay-app-upgrade-xq2v6                0/1     Completed   0             12m
   quayregistry-quay-redis-84f888776f-hhgms           1/1     Running     0             12m

3. On System A, initiate a Red Hat Quay upgrade to the latest y-stream version. This is a manual process. For more information about upgrading installed Operators, see Upgrading installed Operators. For more information about Red Hat Quay upgrade paths, see Upgrading the Red Hat Quay Operator.
4. After the new Red Hat Quay registry is installed, the necessary upgrades on the cluster are automatically completed. Afterwards, new Red Hat Quay pods are started with the latest y-stream version. Additionally, new Quay pods are scheduled and started.

5. Confirm that the update has properly worked by navigating to the Red Hat Quay UI:

   1. In the OpenShift console, navigate to Operators Installed Operators, and click the Registry Endpoint link.

      Important

      Do not execute the following step until the Red Hat Quay UI is available. Do not upgrade the Red Hat Quay registry on System B and on System C until the UI is available on System A.

6. Confirm that the update has properly worked on System A, initiate the Red Hat Quay upgrade on System B and on System C. The Operator upgrade results in an upgraded Red Hat Quay installation, and the pods are restarted.

   Note

   Because the database schema is correct for the new y-stream installation, the new pods on System B and on System C should quickly start.

7. After updating, revert the changes made in step 1 of this procedure by removing overrides for the components. For example:

   apiVersion: quay.redhat.com/v1
   kind: QuayRegistry
   metadata:
     name: registry
     namespace: ns
   spec:
     components:
       …
       - kind: horizontalpodautoscaler
         managed: true
       - kind: quay
         managed: true
       - kind: clair
         managed: true
       - kind: mirror
         managed: true
       …

   where:

   kind: horizontalpodautoscaler:: Set this resource to True if the horizontalpodautoscaler resource was set to True before the upgrade procedure, or if you want Red Hat Quay to scale in case of a resource shortage.

Procedure

1. Sync the blobs between all of your defined sites by running the following command:

   $ python -m util.backfillreplication

   Warning

   Prior to removing storage engines from your Red Hat Quay config.yaml file, you must ensure that all blobs are synced between all defined sites.

   When running this command, replication jobs are created which are picked up by the replication worker. If there are blobs that need replicated, the script returns UUIDs of blobs that will be replicated. If you run this command multiple times, and the output from the return script is empty, it does not mean that the replication process is done; it means that there are no more blobs to be queued for replication. Customers should use appropriate judgement before proceeding, as the allotted time replication takes depends on the number of blobs detected.

   Alternatively, you could use a third party cloud tool, such as Microsoft Azure, to check the synchronization status.

   This step must be completed before proceeding.

2. In your Red Hat Quay config.yaml file for site usstorage, remove the DISTRIBUTED_STORAGE_CONFIG entry for the eustorage site.

3. Identify your Red Hat Quay application pods by entering the following command:

   $ oc get pod -n <quay_namespace>

   Example output

   quay390usstorage-quay-app-5779ddc886-2drh2
   quay390eustorage-quay-app-66969cd859-n2ssm

4. Open an interactive shell session in the usstorage pod by entering the following command:

   $ oc rsh quay390usstorage-quay-app-5779ddc886-2drh2

5. Permanently remove the eustorage site by entering the following command:

   Important

   The following action cannot be undone. Use with caution.

   sh-4.4$ python -m util.removelocation eustorage

   Example output

   WARNING: This is a destructive operation. Are you sure you want to remove eustorage from your storage locations? [y/n] y
   Deleted placement 30
   Deleted placement 31
   Deleted placement 32
   Deleted placement 33
   Deleted location eustorage