Chapter 6. Geo-replication

1. Create the configBundleSecret by entering the following command:

   $ oc create secret generic --from-file config.yaml=./config.yaml georep-config-bundle

   Copy to Clipboard Toggle word wrap

2. In each of the clusters, set the configBundleSecret and use the QUAY_DISTRIBUTED_STORAGE_PREFERENCE environmental variable override to configure the appropriate storage for that cluster. For example:

   Note

   The config.yaml file between both deployments must match. If making a change to one cluster, it must also be changed in the other.

   US cluster QuayRegistry example

   apiVersion: quay.redhat.com/v1
   kind: QuayRegistry
   metadata:
     name: example-registry
     namespace: quay-enterprise
   spec:
     configBundleSecret: georep-config-bundle
     components:
       - kind: objectstorage
         managed: false
       - kind: route
         managed: true
       - kind: tls
         managed: false
       - kind: postgres
         managed: false
       - kind: clairpostgres
         managed: false
       - kind: redis
         managed: false
       - kind: quay
         managed: true
         overrides:
           env:
           - name: QUAY_DISTRIBUTED_STORAGE_PREFERENCE
             value: usstorage
       - kind: mirror
         managed: true
         overrides:
           env:
           - name: QUAY_DISTRIBUTED_STORAGE_PREFERENCE
             value: usstorage

   Copy to Clipboard Toggle word wrap

   Note

   <<<<<<< HEAD Because SSL/TLS is unmanaged, and the route is managed, you must supply the certificates directly in the config bundle. For more information, see Configuring TLS and routes.

1. On System B and System C, scale down your Red Hat Quay registry. This is done by disabling auto scaling and overriding the replica county for Red Hat Quay, mirror workers, and Clair if it is managed. Use the following quayregistry.yaml file as a reference:

   apiVersion: quay.redhat.com/v1
   kind: QuayRegistry
   metadata:
     name: registry
     namespace: ns
   spec:
     components:
       …
       - kind: horizontalpodautoscaler
         managed: false 

   1

   
       - kind: quay
         managed: true
         overrides: 

   2

   
           replicas: 0
       - kind: clair
         managed: true
         overrides:
           replicas: 0
       - kind: mirror
         managed: true
         overrides:
           replicas: 0
       …

   Copy to Clipboard Toggle word wrap

   1

   Disable auto scaling of Quay, Clair and Mirroring workers

   2

   Set the replica count to 0 for components accessing the database and objectstorage

1. Wait for the registry-quay-app, registry-quay-mirror, and registry-clair-app pods to disappear. Enter the following command to check their status:

   oc get pods -n <quay-namespace>

   Copy to Clipboard Toggle word wrap

   Example output

   quay-operator.v3.7.1-6f9d859bd-p5ftc               1/1     Running     0             12m
   quayregistry-clair-postgres-7487f5bd86-xnxpr       1/1     Running     1 (12m ago)   12m
   quayregistry-quay-app-upgrade-xq2v6                0/1     Completed   0             12m
   quayregistry-quay-redis-84f888776f-hhgms           1/1     Running     0             12m

   Copy to Clipboard Toggle word wrap

2. On System A, initiate a Red Hat Quay upgrade to the latest y-stream version. This is a manual process. For more information about upgrading installed Operators, see Upgrading installed Operators. For more information about Red Hat Quay upgrade paths, see Upgrading the Red Hat Quay Operator.
3. After the new Red Hat Quay registry is installed, the necessary upgrades on the cluster are automatically completed. Afterwards, new Red Hat Quay pods are started with the latest y-stream version. Additionally, new Quay pods are scheduled and started.

4. Confirm that the update has properly worked by navigating to the Red Hat Quay UI:

   1. In the OpenShift console, navigate to Operators Installed Operators, and click the Registry Endpoint link.

1. Confirm that the update has properly worked on System A, initiate the Red Hat Quay upgrade on System B and on System C. The Operator upgrade results in an upgraded Red Hat Quay installation, and the pods are restarted.

1. After updating, revert the changes made in step 1 of this procedure by removing overrides for the components. For example:

   apiVersion: quay.redhat.com/v1
   kind: QuayRegistry
   metadata:
     name: registry
     namespace: ns
   spec:
     components:
       …
       - kind: horizontalpodautoscaler
         managed: true 

   1

   
       - kind: quay
         managed: true
       - kind: clair
         managed: true
       - kind: mirror
         managed: true
       …

   Copy to Clipboard Toggle word wrap

   1

   If the horizontalpodautoscaler resource was set to true before the upgrade procedure, or if you want Red Hat Quay to scale in case of a resource shortage, set it to true.

Procedure

1. Sync the blobs between all of your defined sites by running the following command:

   $ python -m util.backfillreplication

   Copy to Clipboard Toggle word wrap

1. In your Red Hat Quay config.yaml file for site usstorage, remove the DISTRIBUTED_STORAGE_CONFIG entry for the eustorage site.

2. Enter the following command to identify your Quay application pods:

   $ oc get pod -n <quay_namespace>

   Copy to Clipboard Toggle word wrap

   Example output

   quay390usstorage-quay-app-5779ddc886-2drh2
   quay390eustorage-quay-app-66969cd859-n2ssm

   Copy to Clipboard Toggle word wrap

3. Enter the following command to open an interactive shell session in the usstorage pod:

   $ oc rsh quay390usstorage-quay-app-5779ddc886-2drh2

   Copy to Clipboard Toggle word wrap
4. Enter the following command to permanently remove the eustorage site:

Procedure

1. Enter the following command to obtain a list of Red Hat Quay pods:

   $ oc get pods -n <namespace>

   Copy to Clipboard Toggle word wrap

   Example output

   example-registry-clair-app-7dc7ff5844-4skw5           0/1     Error                    0             70d
   example-registry-clair-app-7dc7ff5844-nvn4f           1/1     Running                  0             31d
   example-registry-clair-app-7dc7ff5844-x4smw           0/1     ContainerStatusUnknown   6 (70d ago)   70d
   example-registry-clair-app-7dc7ff5844-xjnvt           1/1     Running                  0             60d
   example-registry-clair-postgres-547d75759-75c49       1/1     Running                  0             70d
   example-registry-quay-app-76c8f55467-52wjz            1/1     Running                  0             70d
   example-registry-quay-app-76c8f55467-hwz4c            1/1     Running                  0             70d
   example-registry-quay-app-upgrade-57ghs               0/1     Completed                1             70d
   example-registry-quay-database-7c55899f89-hmnm6       1/1     Running                  0             70d
   example-registry-quay-mirror-6cccbd76d-btsnb          1/1     Running                  0             70d
   example-registry-quay-mirror-6cccbd76d-x8g42          1/1     Running                  0             70d
   example-registry-quay-redis-85cbdf96bf-4vk5m          1/1     Running                  0             70d

   Copy to Clipboard Toggle word wrap

2. Open a remote shell session to the Quay container by entering the following command:

   $ oc rsh example-registry-quay-app-76c8f55467-52wjz

   Copy to Clipboard Toggle word wrap

3. Enter the following command to create the necessary service keys:

   sh-4.4$ python3 tools/generatekeypair.py quay-readonly

   Copy to Clipboard Toggle word wrap

   Example output

   Writing public key to quay-readonly.jwk
   Writing key ID to quay-readonly.kid
   Writing private key to quay-readonly.pem

   Copy to Clipboard Toggle word wrap

Procedure

1. Enter the following command to enter your Red Hat Quay database environment:

   $ oc rsh example-registry-quay-app-76c8f55467-52wjz psql -U <database_username> -d <database_name>

   Copy to Clipboard Toggle word wrap

2. Display the approval types and associated notes of the servicekeyapproval by entering the following command:

   quay=# select * from servicekeyapproval;

   Copy to Clipboard Toggle word wrap

   Example output

    id | approver_id |          approval_type           |       approved_date        | notes
   ----+-------------+----------------------------------+----------------------------+-------
     1 |             | ServiceKeyApprovalType.AUTOMATIC | 2024-05-07 03:47:48.181347 |
     2 |             | ServiceKeyApprovalType.AUTOMATIC | 2024-05-07 03:47:55.808087 |
     3 |             | ServiceKeyApprovalType.AUTOMATIC | 2024-05-07 03:49:04.27095  |
     4 |             | ServiceKeyApprovalType.AUTOMATIC | 2024-05-07 03:49:05.46235  |
     5 |           1 | ServiceKeyApprovalType.SUPERUSER | 2024-05-07 04:05:10.296796 |
   ...

   Copy to Clipboard Toggle word wrap

3. Add the service key to your Red Hat Quay database by entering the following query:

   quay=# INSERT INTO servicekey
     (name, service, metadata, kid, jwk, created_date, expiration_date)
     VALUES ('quay-readonly',
              'quay',
              '{}',
              '{<contents_of_.kid_file>}',
              '{<contents_of_.jwk_file>}',
              '{<created_date_of_read-only>}',
              '{<expiration_date_of_read-only>}');

   Copy to Clipboard Toggle word wrap

   Example output

   INSERT 0 1

   Copy to Clipboard Toggle word wrap

4. Next, add the key approval with the following query:

   quay=# INSERT INTO servicekeyapproval ('approval_type', 'approved_date', 'notes')
     VALUES ("ServiceKeyApprovalType.SUPERUSER", "CURRENT_DATE",
              {include_notes_here_on_why_this_is_being_added});

   Copy to Clipboard Toggle word wrap

   Example output

   INSERT 0 1

   Copy to Clipboard Toggle word wrap

5. Set the approval_id field on the created service key row to the id field from the created service key approval. You can use the following SELECT statements to get the necessary IDs:

   UPDATE servicekey
   SET approval_id = (SELECT id FROM servicekeyapproval WHERE approval_type = 'ServiceKeyApprovalType.SUPERUSER')
   WHERE name = 'quay-readonly';

   Copy to Clipboard Toggle word wrap

   UPDATE 1

   Copy to Clipboard Toggle word wrap

Procedure

1. Enter the following command to read the secret name of your Red Hat Quay on OpenShift Container Platform deployment:

   $ oc get deployment -o yaml <quay_main_app_deployment_name>

   Copy to Clipboard Toggle word wrap

2. Use the base64 command to encode the quay-readonly.kid and quay-readonly.pem files:

   $ base64 -w0 quay-readonly.kid

   Copy to Clipboard Toggle word wrap

   Example output

   ZjUyNDFm...

   Copy to Clipboard Toggle word wrap

   $ base64 -w0 quay-readonly.pem

   Copy to Clipboard Toggle word wrap

   Example output

   LS0tLS1CRUdJTiBSU0E...

   Copy to Clipboard Toggle word wrap

3. Obtain the current configuration bundle and secret by entering the following command:

   $ oc get secret quay-config-secret-name -o json | jq '.data."config.yaml"' | cut -d '"' -f2 | base64 -d -w0 > config.yaml

   Copy to Clipboard Toggle word wrap

4. Edit the config.yaml file and add the following information:

   # ...
   REGISTRY_STATE: readonly
   INSTANCE_SERVICE_KEY_KID_LOCATION: 'conf/stack/quay-readonly.kid'
   INSTANCE_SERVICE_KEY_LOCATION: 'conf/stack/quay-readonly.pem'
   # ...

   Copy to Clipboard Toggle word wrap

5. Save the file and base64 encode it by running the following command:

   $ base64 -w0 quay-config.yaml

   Copy to Clipboard Toggle word wrap

6. Scale down the Red Hat Quay Operator pods to 0. This ensures that the Operator does not reconcile the secret after editing it.

   $ oc scale --replicas=0 deployment quay-operator -n openshift-operators

   Copy to Clipboard Toggle word wrap

7. Edit the secret to include the new content:

   $ oc edit secret quay-config-secret-name -n quay-namespace

   Copy to Clipboard Toggle word wrap

   # ...
   data:
     "quay-readonly.kid": "ZjUyNDFm..."
     "quay-readonly.pem": "LS0tLS1CRUdJTiBSU0E..."
     "config.yaml": "QUNUSU9OX0xPR19..."
   # ...

   Copy to Clipboard Toggle word wrap

   With your Red Hat Quay on OpenShift Container Platform deployment on read-only mode, you can safely manage your registry’s operations and perform such actions as backup and restore.

Procedure

1. Edit the config.yaml file and remove the following information:

   # ...
   REGISTRY_STATE: readonly
   INSTANCE_SERVICE_KEY_KID_LOCATION: 'conf/stack/quay-readonly.kid'
   INSTANCE_SERVICE_KEY_LOCATION: 'conf/stack/quay-readonly.pem'
   # ...

   Copy to Clipboard Toggle word wrap

2. Scale the Red Hat Quay Operator back up by entering the following command:

   oc scale --replicas=1 deployment quay-operator -n openshift-operators

   Copy to Clipboard Toggle word wrap

Procedure

1. To back the QuayRegistry custom resource by exporting it, enter the following command:

   $ oc get quayregistry <quay_registry_name> -n <quay_namespace> -o yaml > quay-registry.yaml

   Copy to Clipboard Toggle word wrap

2. Edit the resulting quayregistry.yaml and remove the status section and the following metadata fields:

     metadata.creationTimestamp
     metadata.finalizers
     metadata.generation
     metadata.resourceVersion
     metadata.uid

   Copy to Clipboard Toggle word wrap
3. Backup the managed keys secret by entering the following command:

1. Edit the resulting managed_secret_keys.yaml file and remove the entry metadata.ownerReferences. Your managed_secret_keys.yaml file should look similar to the following:

   apiVersion: v1
   kind: Secret
   type: Opaque
   metadata:
     name: <quay_registry_name>-quay-registry-managed-secret-keys
     namespace: <quay_namespace>
   data:
     CONFIG_EDITOR_PW: <redacted>
     DATABASE_SECRET_KEY: <redacted>
     DB_ROOT_PW: <redacted>
     DB_URI: <redacted>
     SECRET_KEY: <redacted>
     SECURITY_SCANNER_V4_PSK: <redacted>

   Copy to Clipboard Toggle word wrap

   All information under the data property should remain the same.

2. Redirect the current Quay configuration file by entering the following command:

   $ oc get secret -n <quay-namespace>  $(oc get quayregistry <quay_registry_name> -n <quay_namespace>  -o jsonpath='{.spec.configBundleSecret}') -o yaml > config-bundle.yaml

   Copy to Clipboard Toggle word wrap

3. Backup the /conf/stack/config.yaml file mounted inside of the Quay pods:

   $ oc exec -it quay_pod_name -- cat /conf/stack/config.yaml > quay_config.yaml

   Copy to Clipboard Toggle word wrap

4. Obtain the Quay database name:

   $ oc -n <quay_namespace> rsh $(oc get pod -l app=quay -o NAME -n <quay_namespace> |head -n 1) cat /conf/stack/config.yaml|awk -F"/" '/^DB_URI/ {print $4}'

   Copy to Clipboard Toggle word wrap

   Example output

   quayregistry-quay-database

   Copy to Clipboard Toggle word wrap

Procedure

1. Depending on the version of your Red Hat Quay deployment, scale down your deployment using one of the following options.

   1. For Operator version 3.7 and newer: Scale down the Red Hat Quay deployment by disabling auto scaling and overriding the replica count for Red Hat Quay, mirror workers, and Clair (if managed). Your QuayRegistry resource should look similar to the following:

      apiVersion: quay.redhat.com/v1
      kind: QuayRegistry
      metadata:
        name: registry
        namespace: ns
      spec:
        components:
          …
          - kind: horizontalpodautoscaler
            managed: false 

      1

      
          - kind: quay
            managed: true
            overrides: 

      2

      
              replicas: 0
          - kind: clair
            managed: true
            overrides:
              replicas: 0
          - kind: mirror
            managed: true
            overrides:
              replicas: 0
          …

      Copy to Clipboard Toggle word wrap

      1

      Disable auto scaling of Quay, Clair and Mirroring workers

      2

      Set the replica count to 0 for components accessing the database and objectstorage

   2. For Operator version 3.6 and earlier: Scale down the Red Hat Quay deployment by scaling down the Red Hat Quay registry first and then the managed Red Hat Quay resources:

      $ oc scale --replicas=0 deployment $(oc get deployment -n <quay-operator-namespace>|awk '/^quay-operator/ {print $1}') -n <quay-operator-namespace>

      Copy to Clipboard Toggle word wrap

      $ oc scale --replicas=0 deployment $(oc get deployment -n <quay-namespace>|awk '/quay-app/ {print $1}') -n <quay-namespace>

      Copy to Clipboard Toggle word wrap

      $ oc scale --replicas=0 deployment $(oc get deployment -n <quay-namespace>|awk '/quay-mirror/ {print $1}') -n <quay-namespace>

      Copy to Clipboard Toggle word wrap

      $ oc scale --replicas=0 deployment $(oc get deployment -n <quay-namespace>|awk '/clair-app/ {print $1}') -n <quay-namespace>

      Copy to Clipboard Toggle word wrap

2. Wait for the registry-quay-app, registry-quay-mirror and registry-clair-app pods (depending on which components you set to be managed by the Red Hat Quay Operator) to disappear. You can check their status by running the following command:

   $ oc get pods -n <quay_namespace>

   Copy to Clipboard Toggle word wrap

   Example output:

   $ oc get pod

   Copy to Clipboard Toggle word wrap

   Example output

   quay-operator.v3.7.1-6f9d859bd-p5ftc               1/1     Running     0             12m
   quayregistry-clair-postgres-7487f5bd86-xnxpr       1/1     Running     1 (12m ago)   12m
   quayregistry-quay-app-upgrade-xq2v6                0/1     Completed   0             12m
   quayregistry-quay-database-859d5445ff-cqthr        1/1     Running     0             12m
   quayregistry-quay-redis-84f888776f-hhgms           1/1     Running     0             12m

   Copy to Clipboard Toggle word wrap

Procedure

1. Identify the Quay PostgreSQL pod name:

   $ oc get pod -l quay-component=postgres -n <quay_namespace> -o jsonpath='{.items[0].metadata.name}'

   Copy to Clipboard Toggle word wrap

   Example output:

   quayregistry-quay-database-59f54bb7-58xs7

   Copy to Clipboard Toggle word wrap

2. Download a backup database:

   $ oc -n <quay_namespace> exec quayregistry-quay-database-59f54bb7-58xs7 -- /usr/bin/pg_dump -C quayregistry-quay-database  > backup.sql

   Copy to Clipboard Toggle word wrap

Procedure

1. Decode and export the AWS_ACCESS_KEY_ID by entering the following command:

   $ export AWS_ACCESS_KEY_ID=$(oc get secret -l app=noobaa -n <quay-namespace>  -o jsonpath='{.items[0].data.AWS_ACCESS_KEY_ID}' |base64 -d)

   Copy to Clipboard Toggle word wrap

2. Decode and export the AWS_SECRET_ACCESS_KEY_ID by entering the following command:

   $ export AWS_SECRET_ACCESS_KEY=$(oc get secret -l app=noobaa -n <quay-namespace> -o jsonpath='{.items[0].data.AWS_SECRET_ACCESS_KEY}' |base64 -d)

   Copy to Clipboard Toggle word wrap

3. Create a new directory:

   $ mkdir blobs

   Copy to Clipboard Toggle word wrap

1. Copy all blobs to the directory by entering the following command:

   $ aws s3 sync --no-verify-ssl --endpoint https://$(oc get route s3 -n openshift-storage  -o jsonpath='{.spec.host}')  s3://$(oc get cm -l app=noobaa -n <quay-namespace> -o jsonpath='{.items[0].data.BUCKET_NAME}') ./blobs

   Copy to Clipboard Toggle word wrap

1. Depending on the version of your Red Hat Quay deployment, scale up your deployment using one of the following options.

   1. For Operator version 3.7 and newer: Scale up the Red Hat Quay deployment by re-enabling auto scaling, if desired, and removing the replica overrides for Quay, mirror workers and Clair as applicable. Your QuayRegistry resource should look similar to the following:

      apiVersion: quay.redhat.com/v1
      kind: QuayRegistry
      metadata:
        name: registry
        namespace: ns
      spec:
        components:
          …
          - kind: horizontalpodautoscaler
            managed: true 

      1

      
          - kind: quay 

      2

      
            managed: true
          - kind: clair
            managed: true
          - kind: mirror
            managed: true
          …

      Copy to Clipboard Toggle word wrap

      1

      Re-enables auto scaling of Quay, Clair and Mirroring workers again (if desired)

      2

      Replica overrides are removed again to scale the Quay components back up

   2. For Operator version 3.6 and earlier: Scale up the Red Hat Quay deployment by scaling up the Red Hat Quay registry:

      $ oc scale --replicas=1 deployment $(oc get deployment -n <quay_operator_namespace> | awk '/^quay-operator/ {print $1}') -n <quay_operator_namespace>

      Copy to Clipboard Toggle word wrap

2. Check the status of the Red Hat Quay deployment by entering the following command:

   $ oc wait quayregistry registry --for=condition=Available=true -n <quay_namespace>

   Copy to Clipboard Toggle word wrap

   Example output:

   apiVersion: quay.redhat.com/v1
   kind: QuayRegistry
   metadata:
     ...
     name: registry
     namespace: <quay-namespace>
     ...
   spec:
     ...
   status:
     - lastTransitionTime: '2022-06-20T05:31:17Z'
       lastUpdateTime: '2022-06-20T17:31:13Z'
       message: All components reporting as healthy
       reason: HealthChecksPassing
       status: 'True'
       type: Available

   Copy to Clipboard Toggle word wrap

Procedure

1. Restore the backed up Red Hat Quay configuration by entering the following command:

   $ oc create -f ./config-bundle.yaml

   Copy to Clipboard Toggle word wrap

1. Restore the generated keys from the backup by entering the following command:

   $ oc create -f ./managed-secret-keys.yaml

   Copy to Clipboard Toggle word wrap

2. Restore the QuayRegistry custom resource:

   $ oc create -f ./quay-registry.yaml

   Copy to Clipboard Toggle word wrap

3. Check the status of the Red Hat Quay deployment and wait for it to be available:

   $ oc wait quayregistry registry --for=condition=Available=true -n <quay-namespace>

   Copy to Clipboard Toggle word wrap

Procedure

1. Depending on the version of your Red Hat Quay deployment, scale down your deployment using one of the following options.

   1. For Operator version 3.7 and newer: Scale down the Red Hat Quay deployment by disabling auto scaling and overriding the replica count for Quay, mirror workers and Clair (if managed). Your QuayRegistry resource should look similar to the following:

      apiVersion: quay.redhat.com/v1
      kind: QuayRegistry
      metadata:
        name: registry
        namespace: ns
      spec:
        components:
          …
          - kind: horizontalpodautoscaler
            managed: false 

      1

      
          - kind: quay
            managed: true
            overrides: 

      2

      
              replicas: 0
          - kind: clair
            managed: true
            overrides:
              replicas: 0
          - kind: mirror
            managed: true
            overrides:
              replicas: 0
          …

      Copy to Clipboard Toggle word wrap

      1

      Disable auto scaling of Quay, Clair and Mirroring workers

      2

      Set the replica count to 0 for components accessing the database and objectstorage

   2. For Operator version 3.6 and earlier: Scale down the Red Hat Quay deployment by scaling down the Red Hat Quay registry first and then the managed Red Hat Quay resources:

      $ oc scale --replicas=0 deployment $(oc get deployment -n <quay-operator-namespace>|awk '/^quay-operator/ {print $1}') -n <quay-operator-namespace>

      Copy to Clipboard Toggle word wrap

      $ oc scale --replicas=0 deployment $(oc get deployment -n <quay-namespace>|awk '/quay-app/ {print $1}') -n <quay-namespace>

      Copy to Clipboard Toggle word wrap

      $ oc scale --replicas=0 deployment $(oc get deployment -n <quay-namespace>|awk '/quay-mirror/ {print $1}') -n <quay-namespace>

      Copy to Clipboard Toggle word wrap

      $ oc scale --replicas=0 deployment $(oc get deployment -n <quay-namespace>|awk '/clair-app/ {print $1}') -n <quay-namespace>

      Copy to Clipboard Toggle word wrap

2. Wait for the registry-quay-app, registry-quay-mirror and registry-clair-app pods (depending on which components you set to be managed by Red Hat Quay Operator) to disappear. You can check their status by running the following command:

   $ oc get pods -n <quay-namespace>

   Copy to Clipboard Toggle word wrap

   Example output:

   registry-quay-config-editor-77847fc4f5-nsbbv   1/1     Running            0          9m1s
   registry-quay-database-66969cd859-n2ssm        1/1     Running            0          6d1h
   registry-quay-redis-7cc5f6c977-956g8           1/1     Running            0          5d21h

   Copy to Clipboard Toggle word wrap

Procedure

1. Identify your Quay database pod by entering the following command:

   $ oc get pod -l quay-component=postgres -n  <quay-namespace> -o jsonpath='{.items[0].metadata.name}'

   Copy to Clipboard Toggle word wrap

   Example output:

   quayregistry-quay-database-59f54bb7-58xs7

   Copy to Clipboard Toggle word wrap

2. Upload the backup by copying it from the local environment and into the pod:

   $ oc cp ./backup.sql -n <quay-namespace> registry-quay-database-66969cd859-n2ssm:/tmp/backup.sql

   Copy to Clipboard Toggle word wrap

3. Open a remote terminal to the database by entering the following command:

   $ oc rsh -n <quay-namespace> registry-quay-database-66969cd859-n2ssm

   Copy to Clipboard Toggle word wrap

4. Enter psql by running the following command:

   bash-4.4$ psql

   Copy to Clipboard Toggle word wrap

5. You can list the database by running the following command:

   postgres=# \l

   Copy to Clipboard Toggle word wrap

   Example output

                                                     List of databases
              Name            |           Owner            | Encoding |  Collate   |   Ctype    |   Access privileges
   ----------------------------+----------------------------+----------+------------+------------+-----------------------
   postgres                   | postgres                   | UTF8     | en_US.utf8 | en_US.utf8 |
   quayregistry-quay-database | quayregistry-quay-database | UTF8     | en_US.utf8 | en_US.utf8 |

   Copy to Clipboard Toggle word wrap

6. Drop the database by entering the following command:

   postgres=# DROP DATABASE "quayregistry-quay-database";

   Copy to Clipboard Toggle word wrap

   Example output

   DROP DATABASE

   Copy to Clipboard Toggle word wrap

7. Exit the postgres CLI to re-enter bash-4.4:

   \q

   Copy to Clipboard Toggle word wrap

8. Redirect your PostgreSQL database to your backup database:

   sh-4.4$ psql < /tmp/backup.sql

   Copy to Clipboard Toggle word wrap

9. Exit bash by entering the following command:

   sh-4.4$ exit

   Copy to Clipboard Toggle word wrap

Procedure

1. Export the AWS_ACCESS_KEY_ID by entering the following command:

   $ export AWS_ACCESS_KEY_ID=$(oc get secret -l app=noobaa -n <quay-namespace>  -o jsonpath='{.items[0].data.AWS_ACCESS_KEY_ID}' |base64 -d)

   Copy to Clipboard Toggle word wrap

2. Export the AWS_SECRET_ACCESS_KEY by entering the following command:

   $ export AWS_SECRET_ACCESS_KEY=$(oc get secret -l app=noobaa -n <quay-namespace> -o jsonpath='{.items[0].data.AWS_SECRET_ACCESS_KEY}' |base64 -d)

   Copy to Clipboard Toggle word wrap

3. Upload all blobs to the bucket by running the following command:

   $ aws s3 sync --no-verify-ssl --endpoint https://$(oc get route s3 -n openshift-storage  -o jsonpath='{.spec.host}') ./blobs  s3://$(oc get cm -l app=noobaa -n <quay-namespace> -o jsonpath='{.items[0].data.BUCKET_NAME}')

   Copy to Clipboard Toggle word wrap

1. Depending on the version of your Red Hat Quay deployment, scale up your deployment using one of the following options.

   1. For Operator version 3.7 and newer: Scale up the Red Hat Quay deployment by re-enabling auto scaling, if desired, and removing the replica overrides for Quay, mirror workers and Clair as applicable. Your QuayRegistry resource should look similar to the following:

      apiVersion: quay.redhat.com/v1
      kind: QuayRegistry
      metadata:
        name: registry
        namespace: ns
      spec:
        components:
          …
          - kind: horizontalpodautoscaler
            managed: true 

      1

      
          - kind: quay 

      2

      
            managed: true
          - kind: clair
            managed: true
          - kind: mirror
            managed: true
          …

      Copy to Clipboard Toggle word wrap

      1

      Re-enables auto scaling of Red Hat Quay, Clair and mirroring workers again (if desired)

      2

      Replica overrides are removed again to scale the Red Hat Quay components back up

   2. For Operator version 3.6 and earlier: Scale up the Red Hat Quay deployment by scaling up the Red Hat Quay registry again:

      $ oc scale --replicas=1 deployment $(oc get deployment -n <quay-operator-namespace> | awk '/^quay-operator/ {print $1}') -n <quay-operator-namespace>

      Copy to Clipboard Toggle word wrap

2. Check the status of the Red Hat Quay deployment:

   $ oc wait quayregistry registry --for=condition=Available=true -n <quay-namespace>

   Copy to Clipboard Toggle word wrap

   Example output:

   apiVersion: quay.redhat.com/v1
   kind: QuayRegistry
   metadata:
     ...
     name: registry
     namespace: <quay-namespace>
     ...
   spec:
     ...
   status:
     - lastTransitionTime: '2022-06-20T05:31:17Z'
       lastUpdateTime: '2022-06-20T17:31:13Z'
       message: All components reporting as healthy
       reason: HealthChecksPassing
       status: 'True'
       type: Available

   Copy to Clipboard Toggle word wrap

Procedure

1. On the OpenShift Container Platform console page, select Operators OperatorHub and search for Container Security Operator.
2. Select the Container Security Operator, then select Install to go to the Create Operator Subscription page.
3. Check the settings (all namespaces and automatic approval strategy, by default), and select Subscribe. The Container Security appears after a few moments on the Installed Operators screen.

4. Optional: you can add custom certificates to the CSO. In this example, create a certificate named quay.crt in the current directory. Then, run the following command to add the certificate to the CSO:

   $ oc create secret generic container-security-operator-extra-certs --from-file=quay.crt -n openshift-operators

   Copy to Clipboard Toggle word wrap

1. Navigate to Home Overview. A link to Image Vulnerabilities appears under the status section, with a listing of the number of vulnerabilities found so far. Select the link to see a security breakdown, as shown in the following image:

   

1. You can do one of two things at this point to follow up on any detected vulnerabilities:

   1. Select the link to the vulnerability. You are taken to the container registry, Red Hat Quay or other registry where the container came from, where you can see information about the vulnerability. The following figure shows an example of detected vulnerabilities from a Quay.io registry:

      

   2. Select the namespaces link to go to the Image Manifest Vulnerabilities page, where you can see the name of the selected image and all namespaces where that image is running. The following figure indicates that a particular vulnerable image is running in two namespaces:

      

Procedure

1. Enter the following command to query for detected vulnerabilities:

   $ oc get vuln --all-namespaces

   Copy to Clipboard Toggle word wrap

   Example output

   NAMESPACE     NAME              AGE
   default       sha256.ca90...    6m56s
   skynet        sha256.ca90...    9m37s

   Copy to Clipboard Toggle word wrap

2. Optional. To display details for a particular vulnerability, identify a specific vulnerability and its namespace, and use the oc describe command. The following example shows an active container whose image includes an RPM package with a vulnerability:

   $ oc describe vuln --namespace <namespace> sha256.ac50e3752...

   Copy to Clipboard Toggle word wrap

   Example output

   Name:         sha256.ac50e3752...
   Namespace:    quay-enterprise
   ...
   Spec:
     Features:
       Name:            nss-util
       Namespace Name:  centos:7
       Version:         3.44.0-3.el7
       Versionformat:   rpm
       Vulnerabilities:
         Description: Network Security Services (NSS) is a set of libraries...

   Copy to Clipboard Toggle word wrap

Procedure

1. On the OpenShift Container Platform web console, click Operators Installed Operators.
2. Click the menu kebab of the Container Security Operator.
3. Click Uninstall Operator. Confirm your decision by clicking Uninstall in the popup window.

4. Remove the imagemanifestvulns.secscan.quay.redhat.com custom resource definition by entering the following command:

   $ oc delete customresourcedefinition imagemanifestvulns.secscan.quay.redhat.com

   Copy to Clipboard Toggle word wrap

   Example output

   customresourcedefinition.apiextensions.k8s.io "imagemanifestvulns.secscan.quay.redhat.com" deleted

   Copy to Clipboard Toggle word wrap

Procedure

1. Log in to the Amazon Web Services (AWS) console and navigate to the Identity and Access Management (IAM) console.
2. In the navigation pane, under Access management click Users.

3. Click Create User and enter the following information:

   1. Enter a valid username, for example, quay-user.
   2. For Permissions options, click Add user to group.
4. On the review and create page, click Create user. You are redirected to the Users page.
5. Click the username, for example, quay-user.
6. Copy the ARN of the user, for example, arn:aws:iam::123492922789:user/quay-user.
7. On the same page, click the Security credentials tab.
8. Navigate to Access keys.
9. Click Create access key.
10. On the Access key best practices & alternatives page, click Command Line Interface (CLI), then, check the confirmation box. Then click Next.
11. Optional. On the Set description tag - optional page, enter a description.
12. Click Create access key.
13. Copy and store the access key and the secret access key.

1. Click Done.

Procedure

1. If you are not already, navigate to the IAM dashboard by clicking Dashboard.
2. In the navigation pane, click Roles under Access management.

3. Click Create role.

   • Click Custom Trust Policy, which shows an editable JSON policy. By default, it shows the following information:

     {
     	"Version": "2012-10-17",
     	"Statement": [
     		{
     			"Sid": "Statement1",
     			"Effect": "Allow",
     			"Principal": {},
     			"Action": "sts:AssumeRole"
     		}
     	]
     }

     Copy to Clipboard Toggle word wrap

4. Under the Principal configuration field, add your AWS ARN information. For example:

   {
       "Version": "2012-10-17",
       "Statement": [
      	 {
      		 "Sid": "Statement1",
      		 "Effect": "Allow",
      		 "Principal": {
      		 	"AWS": "arn:aws:iam::123492922789:user/quay-user"
      		 },
      		 "Action": "sts:AssumeRole"
      	 }
       ]
   }

   Copy to Clipboard Toggle word wrap
5. Click Next.
6. On the Add permissions page, type AmazonS3FullAccess in the search box. Check the box to add that policy to the S3 role, then click Next.

7. On the Name, review, and create page, enter the following information:

   1. Enter a role name, for example, example-role.
   2. Optional. Add a description.
8. Click the Create role button. You are navigated to the Roles page. Under Role name, the newly created S3 should be available.

Procedure

1. On the Home page of your OpenShift Container Platform deployment, click Operators Installed Operators.
2. Click Red Hat Quay.
3. Click Quay Registry and then the name of your Red Hat Quay registry.
4. Under Config Bundle Secret, click the name of your registry configuration bundle, for example, quay-registry-config-bundle-qet56.
5. On the configuration bundle page, click Actions to reveal a drop-down menu. Then click Edit Secret.

6. Update your the DISTRIBUTED_STORAGE_CONFIG fields of your config.yaml file with the following information:

   # ...
   DISTRIBUTED_STORAGE_CONFIG:
      default:
       - STSS3Storage
       - sts_role_arn: <role_arn> 

   1

   
         s3_bucket: <s3_bucket_name> 

   2

   
         storage_path: <storage_path> 

   3

   
         s3_region: <region> 

   4

   
         sts_user_access_key: <s3_user_access_key> 

   5

   
         sts_user_secret_key: <s3_user_secret_key> 

   6

   
   # ...

   Copy to Clipboard Toggle word wrap

   1

   The unique Amazon Resource Name (ARN) required when configuring AWS STS

   2

   The name of your s3 bucket.

   3

   The storage path for data. Usually /datastorage.

   4

   Optional. The Amazon Web Services region. Defaults to us-east-1.

   5

   The generated AWS S3 user access key required when configuring AWS STS.

   6

   The generated AWS S3 user secret key required when configuring AWS STS.

7. Click Save.

Verification

1. Tag a sample image, for example, busybox, that will be pushed to the repository. For example:

   $ podman tag docker.io/library/busybox <quay-server.example.com>/<organization_name>/busybox:test

   Copy to Clipboard Toggle word wrap

2. Push the sample image by running the following command:

   $ podman push <quay-server.example.com>/<organization_name>/busybox:test

   Copy to Clipboard Toggle word wrap
3. Verify that the push was successful by navigating to the Organization that you pushed the image to in your Red Hat Quay registry Tags.
4. Navigate to the Amazon Web Services (AWS) console and locate your s3 bucket.
5. Click the name of your s3 bucket.
6. On the Objects page, click datastorage/.

7. On the datastorage/ page, the following resources should seen:

   • sha256/

   • uploads/

     These resources indicate that the push was successful, and that AWS STS is properly configured.

Procedure

1. Log in to Red Hat Quay through the web UI.
2. Select the organization for which the external application will be configured.
3. On the navigation pane, select Applications.
4. Select Create New Application and enter a name for the new application, for example, openshift.
5. On the OAuth Applications page, select your application, for example, openshift.
6. On the navigation pane, select Generate Token.

7. Select the following fields:

   • Administer Organization
   • Administer Repositories
   • Create Repositories
   • View all visible repositories
   • Read/Write to any accessible repositories
   • Administer User
   • Read User Information
8. Review the assigned permissions.
9. Select Authorize Application and then confirm confirm the authorization by selecting Authorize Application.
10. Save the generated access token.

Procedure

1. Open the Administrator perspective of the web console and navigate to Operators OperatorHub on the navigation pane.
2. Search for Quay Bridge Operator, click the Quay Bridge Operator title, and then click Install.
3. Select the version to install, for example, stable-3.7, and then click Install.
4. Click View Operator when the installation finishes to go to the Quay Bridge Operator’s Details page. Alternatively, you can click Installed Operators Red Hat Quay Bridge Operator to go to the Details page.

Procedure

1. Create a quay-integration.yaml:

   $ touch quay-integration.yaml

   Copy to Clipboard Toggle word wrap

2. Use the following configuration for a minimal deployment of the QuayIntegration custom resource:

     apiVersion: quay.redhat.com/v1
     kind: QuayIntegration
     metadata:
       name: example-quayintegration
     spec:
       clusterID: openshift  

   1

   
       credentialsSecret:
         namespace: openshift-operators
         name: quay-integration

   2

   
       quayHostname: https://<QUAY_URL>   

   3

   
       insecureRegistry: false 

   4

   Copy to Clipboard Toggle word wrap

   1

   The clusterID value should be unique across the entire ecosystem. This value is required and defaults to openshift.

   2

   The credentialsSecret property refers to the namespace and name of the secret containing the token that was previously created.

   3

   Replace the QUAY_URL with the hostname of your Red Hat Quay instance.

   4

   If Red Hat Quay is using self signed certificates, set the property to insecureRegistry: true.

1. Create the QuayIntegration custom resource:

   $ oc create -f quay-integration.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Open the Administrator perspective of the web console and navigate to Operators Installed Operators.
2. Click Red Hat Quay Bridge Operator.
3. On the Details page of the Quay Bridge Operator, click Create Instance on the Quay Integration API card.

4. On the Create QuayIntegration page, enter the following required information in either Form view or YAML view:

   • Name: The name that will refer to the QuayIntegration custom resource object.
   • Cluster ID: The ID associated with this cluster. This value should be unique across the entire ecosystem. Defaults to openshift if left unspecified.
   • Credentials secret: Refers to the namespace and name of the secret containing the token that was previously created.
   • Quay hostname: The hostname of the Quay registry.

Procedure

1. Enter the following command to create a new OpenShift Container Platform project called e2e-demo:

   $ oc new-project e2e-demo

   Copy to Clipboard Toggle word wrap
2. After you have created a new project, a new Organization is created in Red Hat Quay. Navigate to the Red Hat Quay registry and confirm that you have created a new Organization named openshift_e2e-demo.

1. On the Red Hat Quay UI, click the name of the new Organization, for example, openshift_e2e-demo.

2. Click Robot Accounts in the navigation pane. As part of new project, the following Robot Accounts should have been created:

   • openshift_e2e-demo+deployer
   • openshift_e2e-demo+default
   • openshift_e2e-demo+builder

3. Enter the following command to confirm three secrets containing Docker configuration associated with the applicable Robot Accounts were created:

   $ oc get secrets builder-quay-openshift deployer-quay-openshift default-quay-openshift

   Copy to Clipboard Toggle word wrap

   Example output

   stevsmit@stevsmit ocp-quay $ oc get secrets builder-quay-openshift deployer-quay-openshift default-quay-openshift
   NAME                      TYPE                             DATA   AGE
   builder-quay-openshift    kubernetes.io/dockerconfigjson   1      77m
   deployer-quay-openshift   kubernetes.io/dockerconfigjson   1      77m
   default-quay-openshift    kubernetes.io/dockerconfigjson   1      77m

   Copy to Clipboard Toggle word wrap

4. Enter the following command to display detailed information about builder ServiceAccount (SA), including its secrets, token expiration, and associated roles and role bindings. This ensures that the project is integrated via the Quay Bridge Operator.

   $ oc describe sa builder default deployer

   Copy to Clipboard Toggle word wrap

   Example output

   ...
   Name:                builder
   Namespace:           e2e-demo
   Labels:              <none>
   Annotations:         <none>
   Image pull secrets:  builder-dockercfg-12345
                        builder-quay-openshift
   Mountable secrets:   builder-dockercfg-12345
                        builder-quay-openshift
   Tokens:              builder-token-12345
   Events:              <none>
   ...

   Copy to Clipboard Toggle word wrap

5. Enter the following command to create and deploy a new application called httpd-template:

   $ oc new-app --template=httpd-example

   Copy to Clipboard Toggle word wrap

   Example output

   --> Deploying template "e2e-demo/httpd-example" to project e2e-demo
   ...
   --> Creating resources ...
       service "httpd-example" created
       route.route.openshift.io "httpd-example" created
       imagestream.image.openshift.io "httpd-example" created
       buildconfig.build.openshift.io "httpd-example" created
       deploymentconfig.apps.openshift.io "httpd-example" created
   --> Success
       Access your application via route 'httpd-example-e2e-demo.apps.quay-ocp.gcp.quaydev.org'
       Build scheduled, use 'oc logs -f buildconfig/httpd-example' to track its progress.
       Run 'oc status' to view your app.

   Copy to Clipboard Toggle word wrap

   After running this command, BuildConfig, ImageStream, Service, Route, and DeploymentConfig resources are created. When the ImageStream resource is created, an associated repository is created in Red Hat Quay. For example:

   

6. The ImageChangeTrigger for the BuildConfig triggers a new Build when the Apache HTTPD image, located in the openshift namespace, is resolved. As the new Build is created, the MutatingWebhookConfiguration automatically rewriters the output to point at Red Hat Quay. You can confirm that the build is complete by querying the output field of the build by running the following command:

   $ oc get build httpd-example-1 --template='{{ .spec.output.to.name }}'

   Copy to Clipboard Toggle word wrap

   Example output

   example-registry-quay-quay-enterprise.apps.quay-ocp.gcp.quaydev.org/openshift_e2e-demo/httpd-example:latest

   Copy to Clipboard Toggle word wrap

7. On the Red Hat Quay UI, navigate to the openshift_e2e-demo Organization and select the httpd-example repository.
8. Click Tags in the navigation pane and confirm that the latest tag has been successfully pushed.

9. Enter the following command to ensure that the latest tag has been resolved:

   $ oc describe is httpd-example

   Copy to Clipboard Toggle word wrap

   Example output

   Name:			httpd-example
   Namespace:		e2e-demo
   Created:		55 minutes ago
   Labels:			app=httpd-example
   			template=httpd-example
   Description:		Keeps track of changes in the application image
   Annotations:		openshift.io/generated-by=OpenShiftNewApp
   			openshift.io/image.dockerRepositoryCheck=2023-10-02T17:56:45Z
   Image Repository:	image-registry.openshift-image-registry.svc:5000/e2e-demo/httpd-example
   Image Lookup:		local=false
   Unique Images:		0
   Tags:			1
   
   latest
     tagged from example-registry-quay-quay-enterprise.apps.quay-ocp.gcp.quaydev.org/openshift_e2e-demo/httpd-example:latest

   Copy to Clipboard Toggle word wrap

10. After the ImageStream is resolved, a new deployment should have been triggered. Enter the following command to generate a URL output:

    $ oc get route httpd-example --template='{{ .spec.host }}'

    Copy to Clipboard Toggle word wrap

    Example output

    httpd-example-e2e-demo.apps.quay-ocp.gcp.quaydev.org

    Copy to Clipboard Toggle word wrap

11. Navigate to the URL. If a sample webpage appears, the deployment was successful.

12. Enter the following command to delete the resources and clean up your Red Hat Quay repository:

    $ oc delete project e2e-demo

    Copy to Clipboard Toggle word wrap

1. After the command completes, navigate to your Red Hat Quay repository and confirm that the openshift_e2e-demo Organization is no longer available.

Procedure

1. In your deployment’s config.yaml file, add the FEATURE_LISTEN_IP_VERSION parameter and set it to IPv6, for example:

   # ...
   FEATURE_GOOGLE_LOGIN: false
   FEATURE_INVITE_ONLY_USER_CREATION: false
   FEATURE_LISTEN_IP_VERSION: IPv6
   FEATURE_MAILING: false
   FEATURE_NONSUPERUSER_TEAM_SYNCING_SETUP: false
   # ...

   Copy to Clipboard Toggle word wrap
2. Start, or restart, your Red Hat Quay deployment.

3. Check that your deployment is listening to IPv6 by entering the following command:

   $ curl <quay_endpoint>/health/instance
   {"data":{"services":{"auth":true,"database":true,"disk_space":true,"registry_gunicorn":true,"service_key":true,"web_gunicorn":true}},"status_code":200}

   Copy to Clipboard Toggle word wrap

Procedure

1. Base64 encode the contents of an SSL/TLS certificate by entering the following command:

   $ cat ca.crt | base64 -w 0

   Copy to Clipboard Toggle word wrap

   Example output

   ...c1psWGpqeGlPQmNEWkJPMjJ5d0pDemVnR2QNCnRsbW9JdEF4YnFSdVd3PT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=

   Copy to Clipboard Toggle word wrap

2. Enter the following kubectl command to edit the quay-enterprise-config-secret file:

   $ kubectl --namespace quay-enterprise edit secret/quay-enterprise-config-secret

   Copy to Clipboard Toggle word wrap

3. Add an entry for the certificate and paste the full base64 encoded stringer under the entry. For example:

     custom-cert.crt:
   c1psWGpqeGlPQmNEWkJPMjJ5d0pDemVnR2QNCnRsbW9JdEF4YnFSdVd3PT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=

   Copy to Clipboard Toggle word wrap

4. Use the kubectl delete command to remove all Red Hat Quay pods. For example:

   $ kubectl delete pod quay-operator.v3.7.1-6f9d859bd-p5ftc quayregistry-clair-postgres-7487f5bd86-xnxpr quayregistry-quay-app-upgrade-xq2v6  quayregistry-quay-database-859d5445ff-cqthr quayregistry-quay-redis-84f888776f-hhgms

   Copy to Clipboard Toggle word wrap

   Afterwards, the Red Hat Quay deployment automatically schedules replace pods with the new certificate data.

1. 3.0.5 3.1.3
2. 3.1.3 3.2.2
3. 3.2.2 3.3.4
4. 3.3.4 3.4.z
5. 3.4.z 3.5.z

Procedure

1. In the OpenShift Container Platform Web Console, navigate to Operators Installed Operators.
2. Click on the Red Hat Quay Operator.
3. Navigate to the Subscription tab.
4. Under Subscription details click Update channel.
5. Select stable-3.12 Save.
6. Check the progress of the new installation under Upgrade status. Wait until the upgrade status changes to 1 installed before proceeding.
7. In your OpenShift Container Platform cluster, navigate to Workloads Pods. Existing pods should be terminated, or in the process of being terminated.
8. Wait for the following pods, which are responsible for upgrading the database and alembic migration of existing data, to spin up: clair-postgres-upgrade, quay-postgres-upgrade, and quay-app-upgrade.
9. After the clair-postgres-upgrade, quay-postgres-upgrade, and quay-app-upgrade pods are marked as Completed, the remaining pods for your Red Hat Quay deployment spin up. This takes approximately ten minutes.
10. Verify that the quay-database and clair-postgres pods now use the postgresql-13 image.
11. After the quay-app pod is marked as Running, you can reach your Red Hat Quay registry.

Procedure

1. Optional. Back up your old PVCs from PostgreSQL 10 by setting POSTGRES_UPGRADE_RETAIN_BACKUP to True your quay-operator Subscription object. For example:

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: quay-operator
     namespace: quay-enterprise
   spec:
     channel: stable-3.8
     name: quay-operator
     source: redhat-operators
     sourceNamespace: openshift-marketplace
     config:
       env:
       - name: POSTGRES_UPGRADE_RETAIN_BACKUP
         value: "true"

   Copy to Clipboard Toggle word wrap
2. In the OpenShift Container Platform Web Console, navigate to Operators Installed Operators.
3. Click on the Red Hat Quay Operator.
4. Navigate to the Subscription tab.
5. Under Subscription details click Update channel.
6. Select stable-3.9 and save the changes.
7. Check the progress of the new installation under Upgrade status. Wait until the upgrade status changes to 1 installed before proceeding.
8. In your OpenShift Container Platform cluster, navigate to Workloads Pods. Existing pods should be terminated, or in the process of being terminated.
9. Wait for the following pods, which are responsible for upgrading the database and alembic migration of existing data, to spin up: clair-postgres-upgrade, quay-postgres-upgrade, and quay-app-upgrade.
10. After the clair-postgres-upgrade, quay-postgres-upgrade, and quay-app-upgrade pods are marked as Completed, the remaining pods for your Red Hat Quay deployment spin up. This takes approximately ten minutes.
11. Verify that the quay-database and clair-postgres pods now use the postgresql-13 image.
12. After the quay-app pod is marked as Running, you can reach your Red Hat Quay registry.