Red Hat Quay Operator features

Procedure

1. Enter the following command to check whether the cluster-monitoring-config ConfigMap object exists:

   $ oc -n openshift-monitoring get configmap cluster-monitoring-config

   Example output

   Error from server (NotFound): configmaps "cluster-monitoring-config" not found

2. Optional: If the ConfigMap object does not exist, create a YAML manifest. In the following example, the file is called cluster-monitoring-config.yaml.

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |

3. Optional: If the ConfigMap object does not exist, create the ConfigMap object:

   $ oc apply -f cluster-monitoring-config.yaml

   Example output

   configmap/cluster-monitoring-config created

4. Ensure that the ConfigMap object exists by running the following command:

   $ oc -n openshift-monitoring get configmap cluster-monitoring-config

   Example output

   NAME                        DATA   AGE
   cluster-monitoring-config   1      12s

Procedure

1. Enter the following command to check whether the user-workload-monitoring-config ConfigMap object exists:

   $ oc -n openshift-user-workload-monitoring get configmap user-workload-monitoring-config

   Example output

   Error from server (NotFound): configmaps "user-workload-monitoring-config" not found

2. If the ConfigMap object does not exist, create a YAML manifest. In the following example, the file is called user-workload-monitoring-config.yaml.

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |

3. Optional: Create the ConfigMap object by entering the following command:

   $ oc apply -f user-workload-monitoring-config.yaml

   Example output

   configmap/user-workload-monitoring-config created

Procedure

1. Enter the following command to check if monitoring for user-defined projects is running:

   $ oc get pods -n openshift-user-workload-monitoring

   Example output

   No resources found in openshift-user-workload-monitoring namespace.

2. Edit the cluster-monitoring-config ConfigMap by entering the following command:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

3. Set enableUserWorkload: true in your config.yaml file to enable monitoring for user-defined projects on the cluster:

   apiVersion: v1
   data:
     config.yaml: |
       enableUserWorkload: true
   kind: ConfigMap
   metadata:
     annotations:

4. Enter the following command to save the file, apply the changes, and ensure that the appropriate pods are running:

   $ oc get pods -n openshift-user-workload-monitoring

   Example output

   NAME                                   READY   STATUS    RESTARTS   AGE
   prometheus-operator-6f96b4b8f8-gq6rl   2/2     Running   0          15s
   prometheus-user-workload-0             5/5     Running   1          12s
   prometheus-user-workload-1             5/5     Running   1          12s
   thanos-ruler-user-workload-0           3/3     Running   0          8s
   thanos-ruler-user-workload-1           3/3     Running   0          8s

Procedure

1. Create a YAML file for the Service object:

   $ cat <<EOF >  quay-service.yaml
   
   apiVersion: v1
   kind: Service
   metadata:
     annotations:
     labels:
       quay-component: monitoring
       quay-operator/quayregistry: example-registry
     name: example-registry-quay-metrics
     namespace: quay-enterprise
   spec:
     ports:
     - name: quay-metrics
       port: 9091
       protocol: TCP
       targetPort: 9091
     selector:
       quay-component: quay-app
       quay-operator/quayregistry: example-registry
     type: ClusterIP
   EOF

2. Create the Service object by entering the following command:

   $  oc apply -f quay-service.yaml

   Example output

   service/example-registry-quay-metrics created

Procedure

1. Create a YAML file for the ServiceMonitor resource:

   $ cat <<EOF >  quay-service-monitor.yaml
   
   apiVersion: monitoring.coreos.com/v1
   kind: ServiceMonitor
   metadata:
     labels:
       quay-operator/quayregistry: example-registry
     name: example-registry-quay-metrics-monitor
     namespace: quay-enterprise
   spec:
     endpoints:
     - port: quay-metrics
     namespaceSelector:
       any: true
     selector:
       matchLabels:
         quay-component: monitoring
   EOF

2. Create the ServiceMonitor resource by entering the following command:

   $ oc apply -f quay-service-monitor.yaml

   Example output

   servicemonitor.monitoring.coreos.com/example-registry-quay-metrics-monitor created

Procedure

1. Log into the OpenShift Container Platform console and select Storage → Persistent Volume Claims.
2. Select the desired PersistentVolumeClaim for either PostgreSQL 13 or Clair PostgreSQL 13, for example, example-registry-quay-postgres-13.
3. From the Action menu, select Expand PVC.

4. Enter the new size of the Persistent Volume Claim and select Expand.

   After a few minutes, the expanded size should reflect in the PVC’s Capacity field.

Procedure

1. The ClusterServiceVersion object is Operator Lifecycle Manager’s representation of a running Operator in the cluster. Find the Red Hat Quay Operator’s ClusterServiceVersion by using a Kubernetes UI or the kubectl/oc CLI tool. For example:

   $ oc get clusterserviceversions -n <your-namespace>

2. Using the UI, oc edit, or another method, modify the Red Hat Quay ClusterServiceVersion to include the environment variables outlined above to point to the override images:

   JSONPath: spec.install.spec.deployments[0].spec.template.spec.containers[0].env

   - name: RELATED_IMAGE_COMPONENT_QUAY
     value: quay.io/projectquay/quay@sha256:c35f5af964431673f4ff5c9e90bdf45f19e38b8742b5903d41c10cc7f6339a6d
   - name: RELATED_IMAGE_COMPONENT_CLAIR
     value: quay.io/projectquay/clair@sha256:70c99feceb4c0973540d22e740659cd8d616775d3ad1c1698ddf71d0221f3ce6
   - name: RELATED_IMAGE_COMPONENT_POSTGRES
     value: centos/postgresql-10-centos7@sha256:de1560cb35e5ec643e7b3a772ebaac8e3a7a2a8e8271d9e91ff023539b4dfb33
   - name: RELATED_IMAGE_COMPONENT_REDIS
     value: centos/redis-32-centos7@sha256:06dbb609484330ec6be6090109f1fa16e936afcf975d1cbc5fff3e6c7cae7542

Procedure

1. Deploy a postgres instance for Red Hat Quay.

2. Login to the database by entering the following command:

   psql -U <username> -h <hostname> -p <port> -d <database_name>

3. Create a database for Red Hat Quay named quay. For example:

   CREATE DATABASE quay;

4. Enable pg_trm extension inside the database

   \c quay;
   CREATE EXTENSION IF NOT EXISTS pg_trgm;

5. Deploy a Redis instance:

   Note
   • Deploying a Redis instance might be unnecessary if your cloud provider has its own service.
   • Deploying a Redis instance is required if you are leveraging Builders.
   1. Deploy a VM for Redis
   2. Verify that it is accessible from the clusters where Red Hat Quay is running
   3. Port 6379/TCP must be open

   4. Run Redis inside the instance

      sudo dnf install -y podman
      podman run -d --name redis -p 6379:6379 redis

6. Create two object storage backends, one for each cluster. Ideally, one object storage bucket will be close to the first, or primary, cluster, and the other will run closer to the second, or secondary, cluster.
7. Deploy the clusters with the same config bundle, using environment variable overrides to select the appropriate storage backend for an individual cluster.
8. Configure a load balancer to provide a single entry point to the clusters.

Procedure

1. Sync the blobs between all of your defined sites by running the following command:

   $ python -m util.backfillreplication

   Warning

   Prior to removing storage engines from your Red Hat Quay config.yaml file, you must ensure that all blobs are synced between all defined sites.

   When running this command, replication jobs are created which are picked up by the replication worker. If there are blobs that need replicated, the script returns UUIDs of blobs that will be replicated. If you run this command multiple times, and the output from the return script is empty, it does not mean that the replication process is done; it means that there are no more blobs to be queued for replication. Customers should use appropriate judgement before proceeding, as the allotted time replication takes depends on the number of blobs detected.

   Alternatively, you could use a third party cloud tool, such as Microsoft Azure, to check the synchronization status.

   This step must be completed before proceeding.

2. In your Red Hat Quay config.yaml file for site usstorage, remove the DISTRIBUTED_STORAGE_CONFIG entry for the eustorage site.

3. Enter the following command to identify your Quay application pods:

   $ oc get pod -n <quay_namespace>

   Example output

   quay390usstorage-quay-app-5779ddc886-2drh2
   quay390eustorage-quay-app-66969cd859-n2ssm

4. Enter the following command to open an interactive shell session in the usstorage pod:

   $ oc rsh quay390usstorage-quay-app-5779ddc886-2drh2

5. Enter the following command to permanently remove the eustorage site:

   Important

   The following action cannot be undone. Use with caution.

   sh-4.4$ python -m util.removelocation eustorage

   Example output

   WARNING: This is a destructive operation. Are you sure you want to remove eustorage from your storage locations? [y/n] y
   Deleted placement 30
   Deleted placement 31
   Deleted placement 32
   Deleted placement 33
   Deleted location eustorage

Procedure

1. Enter the following command to obtain a list of Red Hat Quay pods:

   $ oc get pods -n <namespace>

   Example output

   example-registry-clair-app-7dc7ff5844-4skw5           0/1     Error                    0             70d
   example-registry-clair-app-7dc7ff5844-nvn4f           1/1     Running                  0             31d
   example-registry-clair-app-7dc7ff5844-x4smw           0/1     ContainerStatusUnknown   6 (70d ago)   70d
   example-registry-clair-app-7dc7ff5844-xjnvt           1/1     Running                  0             60d
   example-registry-clair-postgres-547d75759-75c49       1/1     Running                  0             70d
   example-registry-quay-app-76c8f55467-52wjz            1/1     Running                  0             70d
   example-registry-quay-app-76c8f55467-hwz4c            1/1     Running                  0             70d
   example-registry-quay-app-upgrade-57ghs               0/1     Completed                1             70d
   example-registry-quay-database-7c55899f89-hmnm6       1/1     Running                  0             70d
   example-registry-quay-mirror-6cccbd76d-btsnb          1/1     Running                  0             70d
   example-registry-quay-mirror-6cccbd76d-x8g42          1/1     Running                  0             70d
   example-registry-quay-redis-85cbdf96bf-4vk5m          1/1     Running                  0             70d

2. Open a remote shell session to the Quay container by entering the following command:

   $ oc rsh example-registry-quay-app-76c8f55467-52wjz

3. Enter the following command to create the necessary service keys:

   sh-4.4$ python3 tools/generatekeypair.py quay-readonly

   Example output

   Writing public key to quay-readonly.jwk
   Writing key ID to quay-readonly.kid
   Writing private key to quay-readonly.pem

Procedure

1. Enter the following command to enter your Red Hat Quay database environment:

   $ oc rsh example-registry-quay-app-76c8f55467-52wjz psql -U <database_username> -d <database_name>

2. Display the approval types and associated notes of the servicekeyapproval by entering the following command:

   quay=# select * from servicekeyapproval;

   Example output

    id | approver_id |          approval_type           |       approved_date        | notes
   ----+-------------+----------------------------------+----------------------------+-------
     1 |             | ServiceKeyApprovalType.AUTOMATIC | 2024-05-07 03:47:48.181347 |
     2 |             | ServiceKeyApprovalType.AUTOMATIC | 2024-05-07 03:47:55.808087 |
     3 |             | ServiceKeyApprovalType.AUTOMATIC | 2024-05-07 03:49:04.27095  |
     4 |             | ServiceKeyApprovalType.AUTOMATIC | 2024-05-07 03:49:05.46235  |
     5 |           1 | ServiceKeyApprovalType.SUPERUSER | 2024-05-07 04:05:10.296796 |
   ...

3. Add the service key to your Red Hat Quay database by entering the following query:

   quay=# INSERT INTO servicekey
     (name, service, metadata, kid, jwk, created_date, expiration_date)
     VALUES ('quay-readonly',
              'quay',
              '{}',
              '{<contents_of_.kid_file>}',
              '{<contents_of_.jwk_file>}',
              '{<created_date_of_read-only>}',
              '{<expiration_date_of_read-only>}');

   Example output

   INSERT 0 1

4. Next, add the key approval with the following query:

   quay=# INSERT INTO servicekeyapproval ('approval_type', 'approved_date', 'notes')
     VALUES ("ServiceKeyApprovalType.SUPERUSER", "CURRENT_DATE",
              {include_notes_here_on_why_this_is_being_added});

   Example output

   INSERT 0 1

5. Set the approval_id field on the created service key row to the id field from the created service key approval. You can use the following SELECT statements to get the necessary IDs:

   UPDATE servicekey
   SET approval_id = (SELECT id FROM servicekeyapproval WHERE approval_type = 'ServiceKeyApprovalType.SUPERUSER')
   WHERE name = 'quay-readonly';

   UPDATE 1

Procedure

1. Enter the following command to read the secret name of your Red Hat Quay on OpenShift Container Platform deployment:

   $ oc get deployment -o yaml <quay_main_app_deployment_name>

2. Use the base64 command to encode the quay-readonly.kid and quay-readonly.pem files:

   $ base64 -w0 quay-readonly.kid

   Example output

   ZjUyNDFm...

   $ base64 -w0 quay-readonly.pem

   Example output

   LS0tLS1CRUdJTiBSU0E...

3. Obtain the current configuration bundle and secret by entering the following command:

   $ oc get secret quay-config-secret-name -o json | jq '.data."config.yaml"' | cut -d '"' -f2 | base64 -d -w0 > config.yaml

4. Edit the config.yaml file and add the following information:

   # ...
   REGISTRY_STATE: readonly
   INSTANCE_SERVICE_KEY_KID_LOCATION: 'conf/stack/quay-readonly.kid'
   INSTANCE_SERVICE_KEY_LOCATION: 'conf/stack/quay-readonly.pem'
   # ...

5. Save the file and base64 encode it by running the following command:

   $ base64 -w0 quay-config.yaml

6. Scale down the Red Hat Quay Operator pods to 0. This ensures that the Operator does not reconcile the secret after editing it.

   $ oc scale --replicas=0 deployment quay-operator -n openshift-operators

7. Edit the secret to include the new content:

   $ oc edit secret quay-config-secret-name -n quay-namespace

   # ...
   data:
     "quay-readonly.kid": "ZjUyNDFm..."
     "quay-readonly.pem": "LS0tLS1CRUdJTiBSU0E..."
     "config.yaml": "QUNUSU9OX0xPR19..."
   # ...

   With your Red Hat Quay on OpenShift Container Platform deployment on read-only mode, you can safely manage your registry’s operations and perform such actions as backup and restore.

Procedure

1. To back the QuayRegistry custom resource by exporting it, enter the following command:

   $ oc get quayregistry <quay_registry_name> -n <quay_namespace> -o yaml > quay-registry.yaml

2. Edit the resulting quayregistry.yaml and remove the status section and the following metadata fields:

     metadata.creationTimestamp
     metadata.finalizers
     metadata.generation
     metadata.resourceVersion
     metadata.uid

3. Backup the managed keys secret by entering the following command:

   Note

   If you are running a version older than Red Hat Quay 3.7.0, this step can be skipped. Some secrets are automatically generated while deploying Red Hat Quay for the first time. These are stored in a secret called <quay_registry_name>-quay_registry_managed_secret_keys in the namespace of the QuayRegistry resource.

   $ oc get secret -n <quay_namespace> <quay_registry_name>_quay_registry_managed_secret_keys -o yaml > managed_secret_keys.yaml

4. Edit the resulting managed_secret_keys.yaml file and remove the entry metadata.ownerReferences. Your managed_secret_keys.yaml file should look similar to the following:

   apiVersion: v1
   kind: Secret
   type: Opaque
   metadata:
     name: <quayname>_quay_registry_managed_secret_keys>
     namespace: <quay_namespace>
   data:
     CONFIG_EDITOR_PW: <redacted>
     DATABASE_SECRET_KEY: <redacted>
     DB_ROOT_PW: <redacted>
     DB_URI: <redacted>
     SECRET_KEY: <redacted>
     SECURITY_SCANNER_V4_PSK: <redacted>

   All information under the data property should remain the same.

5. Redirect the current Quay configuration file by entering the following command:

   $ oc get secret -n <quay-namespace>  $(oc get quayregistry <quay_registry_name> -n <quay_namespace>  -o jsonpath='{.spec.configBundleSecret}') -o yaml > config-bundle.yaml

6. Backup the /conf/stack/config.yaml file mounted inside of the Quay pods:

   $ oc exec -it quay_pod_name -- cat /conf/stack/config.yaml > quay_config.yaml

Procedure

1. Depending on the version of your Red Hat Quay deployment, scale down your deployment using one of the following options.

   1. For Operator version 3.7 and newer: Scale down the Red Hat Quay deployment by disabling auto scaling and overriding the replica count for Red Hat Quay, mirror workers, and Clair (if managed). Your QuayRegistry resource should look similar to the following:

      apiVersion: quay.redhat.com/v1
      kind: QuayRegistry
      metadata:
        name: registry
        namespace: ns
      spec:
        components:
          …
          - kind: horizontalpodautoscaler
            managed: false 1
          - kind: quay
            managed: true
            overrides: 2
              replicas: 0
          - kind: clair
            managed: true
            overrides:
              replicas: 0
          - kind: mirror
            managed: true
            overrides:
              replicas: 0
          …

      1

      Disable auto scaling of Quay, Clair and Mirroring workers

      2

      Set the replica count to 0 for components accessing the database and objectstorage

   2. For Operator version 3.6 and earlier: Scale down the Red Hat Quay deployment by scaling down the Red Hat Quay registry first and then the managed Red Hat Quay resources:

      $ oc scale --replicas=0 deployment $(oc get deployment -n <quay-operator-namespace>|awk '/^quay-operator/ {print $1}') -n <quay-operator-namespace>

      $ oc scale --replicas=0 deployment $(oc get deployment -n <quay-namespace>|awk '/quay-app/ {print $1}') -n <quay-namespace>

      $ oc scale --replicas=0 deployment $(oc get deployment -n <quay-namespace>|awk '/quay-mirror/ {print $1}') -n <quay-namespace>

      $ oc scale --replicas=0 deployment $(oc get deployment -n <quay-namespace>|awk '/clair-app/ {print $1}') -n <quay-namespace>

2. Wait for the registry-quay-app, registry-quay-mirror and registry-clair-app pods (depending on which components you set to be managed by the Red Hat Quay Operator) to disappear. You can check their status by running the following command:

   $ oc get pods -n <quay_namespace>

   Example output:

   $ oc get pod

   Example output

   quay-operator.v3.7.1-6f9d859bd-p5ftc               1/1     Running     0             12m
   quayregistry-clair-postgres-7487f5bd86-xnxpr       1/1     Running     1 (12m ago)   12m
   quayregistry-quay-app-upgrade-xq2v6                0/1     Completed   0             12m
   quayregistry-quay-database-859d5445ff-cqthr        1/1     Running     0             12m
   quayregistry-quay-redis-84f888776f-hhgms           1/1     Running     0             12m

Procedure

1. Identify the Quay PostgreSQL pod name:

   $ oc get pod -l quay-component=postgres -n <quay_namespace> -o jsonpath='{.items[0].metadata.name}'

   Example output:

   quayregistry-quay-database-59f54bb7-58xs7

2. Obtain the Quay database name:

   $ oc -n <quay_namespace> rsh $(oc get pod -l app=quay -o NAME -n <quay_namespace> |head -n 1) cat /conf/stack/config.yaml|awk -F"/" '/^DB_URI/ {print $4}'
   quayregistry-quay-database

3. Download a backup database:

   $ oc -n <quay_namespace> exec quayregistry-quay-database-59f54bb7-58xs7 -- /usr/bin/pg_dump -C quayregistry-quay-database  > backup.sql

1. Depending on the version of your Red Hat Quay deployment, scale up your deployment using one of the following options.

   1. For Operator version 3.7 and newer: Scale up the Red Hat Quay deployment by re-enabling auto scaling, if desired, and removing the replica overrides for Quay, mirror workers and Clair as applicable. Your QuayRegistry resource should look similar to the following:

      apiVersion: quay.redhat.com/v1
      kind: QuayRegistry
      metadata:
        name: registry
        namespace: ns
      spec:
        components:
          …
          - kind: horizontalpodautoscaler
            managed: true 1
          - kind: quay 2
            managed: true
          - kind: clair
            managed: true
          - kind: mirror
            managed: true
          …

      1

      Re-enables auto scaling of Quay, Clair and Mirroring workers again (if desired)

      2

      Replica overrides are removed again to scale the Quay components back up

   2. For Operator version 3.6 and earlier: Scale up the Red Hat Quay deployment by scaling up the Red Hat Quay registry:

      $ oc scale --replicas=1 deployment $(oc get deployment -n <quay_operator_namespace> | awk '/^quay-operator/ {print $1}') -n <quay_operator_namespace>

2. Check the status of the Red Hat Quay deployment by entering the following command:

   $ oc wait quayregistry registry --for=condition=Available=true -n <quay_namespace>

   Example output:

   apiVersion: quay.redhat.com/v1
   kind: QuayRegistry
   metadata:
     ...
     name: registry
     namespace: <quay-namespace>
     ...
   spec:
     ...
   status:
     - lastTransitionTime: '2022-06-20T05:31:17Z'
       lastUpdateTime: '2022-06-20T17:31:13Z'
       message: All components reporting as healthy
       reason: HealthChecksPassing
       status: 'True'
       type: Available

Procedure

1. Restore the backed up Red Hat Quay configuration by entering the following command:

   $ oc create -f ./config-bundle.yaml

   Important

   If you receive the error Error from server (AlreadyExists): error when creating "./config-bundle.yaml": secrets "config-bundle-secret" already exists, you must delete your existing resource with $ oc delete Secret config-bundle-secret -n <quay-namespace> and recreate it with $ oc create -f ./config-bundle.yaml.

2. Restore the generated keys from the backup by entering the following command:

   $ oc create -f ./managed-secret-keys.yaml

3. Restore the QuayRegistry custom resource:

   $ oc create -f ./quay-registry.yaml

4. Check the status of the Red Hat Quay deployment and wait for it to be available:

   $ oc wait quayregistry registry --for=condition=Available=true -n <quay-namespace>

Procedure

1. Depending on the version of your Red Hat Quay deployment, scale down your deployment using one of the following options.

   1. For Operator version 3.7 and newer: Scale down the Red Hat Quay deployment by disabling auto scaling and overriding the replica count for Quay, mirror workers and Clair (if managed). Your QuayRegistry resource should look similar to the following:

      apiVersion: quay.redhat.com/v1
      kind: QuayRegistry
      metadata:
        name: registry
        namespace: ns
      spec:
        components:
          …
          - kind: horizontalpodautoscaler
            managed: false 1
          - kind: quay
            managed: true
            overrides: 2
              replicas: 0
          - kind: clair
            managed: true
            overrides:
              replicas: 0
          - kind: mirror
            managed: true
            overrides:
              replicas: 0
          …

      1

      Disable auto scaling of Quay, Clair and Mirroring workers

      2

      Set the replica count to 0 for components accessing the database and objectstorage

   2. For Operator version 3.6 and earlier: Scale down the Red Hat Quay deployment by scaling down the Red Hat Quay registry first and then the managed Red Hat Quay resources:

      $ oc scale --replicas=0 deployment $(oc get deployment -n <quay-operator-namespace>|awk '/^quay-operator/ {print $1}') -n <quay-operator-namespace>

      $ oc scale --replicas=0 deployment $(oc get deployment -n <quay-namespace>|awk '/quay-app/ {print $1}') -n <quay-namespace>

      $ oc scale --replicas=0 deployment $(oc get deployment -n <quay-namespace>|awk '/quay-mirror/ {print $1}') -n <quay-namespace>

      $ oc scale --replicas=0 deployment $(oc get deployment -n <quay-namespace>|awk '/clair-app/ {print $1}') -n <quay-namespace>

2. Wait for the registry-quay-app, registry-quay-mirror and registry-clair-app pods (depending on which components you set to be managed by Red Hat Quay Operator) to disappear. You can check their status by running the following command:

   $ oc get pods -n <quay-namespace>

   Example output:

   registry-quay-config-editor-77847fc4f5-nsbbv   1/1     Running            0          9m1s
   registry-quay-database-66969cd859-n2ssm        1/1     Running            0          6d1h
   registry-quay-redis-7cc5f6c977-956g8           1/1     Running            0          5d21h

Procedure

1. Identify your Quay database pod by entering the following command:

   $ oc get pod -l quay-component=postgres -n  <quay-namespace> -o jsonpath='{.items[0].metadata.name}'

   Example output:

   quayregistry-quay-database-59f54bb7-58xs7

2. Upload the backup by copying it from the local environment and into the pod:

   $ oc cp ./backup.sql -n <quay-namespace> registry-quay-database-66969cd859-n2ssm:/tmp/backup.sql

3. Open a remote terminal to the database by entering the following command:

   $ oc rsh -n <quay-namespace> registry-quay-database-66969cd859-n2ssm

4. Enter psql by running the following command:

   bash-4.4$ psql

5. You can list the database by running the following command:

   postgres=# \l

   Example output

                                                     List of databases
              Name            |           Owner            | Encoding |  Collate   |   Ctype    |   Access privileges
   ----------------------------+----------------------------+----------+------------+------------+-----------------------
   postgres                   | postgres                   | UTF8     | en_US.utf8 | en_US.utf8 |
   quayregistry-quay-database | quayregistry-quay-database | UTF8     | en_US.utf8 | en_US.utf8 |

6. Drop the database by entering the following command:

   postgres=# DROP DATABASE "quayregistry-quay-database";

   Example output

   DROP DATABASE

7. Exit the postgres CLI to re-enter bash-4.4:

   \q

8. Redirect your PostgreSQL database to your backup database:

   sh-4.4$ psql < /tmp/backup.sql

9. Exit bash by entering the following command:

   sh-4.4$ exit

Procedure

1. Export the AWS_ACCESS_KEY_ID by entering the following command:

   $ export AWS_ACCESS_KEY_ID=$(oc get secret -l app=noobaa -n <quay-namespace>  -o jsonpath='{.items[0].data.AWS_ACCESS_KEY_ID}' |base64 -d)

2. Export the AWS_SECRET_ACCESS_KEY by entering the following command:

   $ export AWS_SECRET_ACCESS_KEY=$(oc get secret -l app=noobaa -n <quay-namespace> -o jsonpath='{.items[0].data.AWS_SECRET_ACCESS_KEY}' |base64 -d)

3. Upload all blobs to the bucket by running the following command:

   $ aws s3 sync --no-verify-ssl --endpoint https://$(oc get route s3 -n openshift-storage  -o jsonpath='{.spec.host}') ./blobs  s3://$(oc get cm -l app=noobaa -n <quay-namespace> -o jsonpath='{.items[0].data.BUCKET_NAME}')

1. Depending on the version of your Red Hat Quay deployment, scale up your deployment using one of the following options.

   1. For Operator version 3.7 and newer: Scale up the Red Hat Quay deployment by re-enabling auto scaling, if desired, and removing the replica overrides for Quay, mirror workers and Clair as applicable. Your QuayRegistry resource should look similar to the following:

      apiVersion: quay.redhat.com/v1
      kind: QuayRegistry
      metadata:
        name: registry
        namespace: ns
      spec:
        components:
          …
          - kind: horizontalpodautoscaler
            managed: true 1
          - kind: quay 2
            managed: true
          - kind: clair
            managed: true
          - kind: mirror
            managed: true
          …

      1

      Re-enables auto scaling of Red Hat Quay, Clair and mirroring workers again (if desired)

      2

      Replica overrides are removed again to scale the Red Hat Quay components back up

   2. For Operator version 3.6 and earlier: Scale up the Red Hat Quay deployment by scaling up the Red Hat Quay registry again:

      $ oc scale --replicas=1 deployment $(oc get deployment -n <quay-operator-namespace> | awk '/^quay-operator/ {print $1}') -n <quay-operator-namespace>

2. Check the status of the Red Hat Quay deployment:

   $ oc wait quayregistry registry --for=condition=Available=true -n <quay-namespace>

   Example output:

   apiVersion: quay.redhat.com/v1
   kind: QuayRegistry
   metadata:
     ...
     name: registry
     namespace: <quay-namespace>
     ...
   spec:
     ...
   status:
     - lastTransitionTime: '2022-06-20T05:31:17Z'
       lastUpdateTime: '2022-06-20T17:31:13Z'
       message: All components reporting as healthy
       reason: HealthChecksPassing
       status: 'True'
       type: Available

Procedure

1. Create a quay-integration.yaml:

   $ touch quay-integration.yaml

2. Use the following configuration for a minimal deployment of the QuayIntegration custom resource:

     apiVersion: quay.redhat.com/v1
     kind: QuayIntegration
     metadata:
       name: example-quayintegration
     spec:
       clusterID: openshift  1
       credentialsSecret:
         namespace: openshift-operators
         name: quay-integration2
       quayHostname: https://<QUAY_URL>   3
       insecureRegistry: false 4

   1

   The clusterID value should be unique across the entire ecosystem. This value is required and defaults to openshift.

   2

   The credentialsSecret property refers to the namespace and name of the secret containing the token that was previously created.

   3

   Replace the QUAY_URL with the hostname of your Red Hat Quay instance.

   4

   If Red Hat Quay is using self signed certificates, set the property to insecureRegistry: true.

1. Create the QuayIntegration custom resource:

   $ oc create -f quay-integration.yaml

Procedure

1. Open the Administrator perspective of the web console and navigate to Operators → Installed Operators.
2. Click Red Hat Quay Bridge Operator.
3. On the Details page of the Quay Bridge Operator, click Create Instance on the Quay Integration API card.

4. On the Create QuayIntegration page, enter the following required information in either Form view or YAML view:

   • Name: The name that will refer to the QuayIntegration custom resource object.
   • Cluster ID: The ID associated with this cluster. This value should be unique across the entire ecosystem. Defaults to openshift if left unspecified.
   • Credentials secret: Refers to the namespace and name of the secret containing the token that was previously created.
   • Quay hostname: The hostname of the Quay registry.

Procedure

1. In the OpenShift Container Platform Web Console, navigate to Operators → Installed Operators.
2. Click on the Red Hat Quay Operator.
3. Navigate to the Subscription tab.
4. Under Subscription details click Update channel.
5. Select stable-3.13 → Save.
6. Check the progress of the new installation under Upgrade status. Wait until the upgrade status changes to 1 installed before proceeding.
7. In your OpenShift Container Platform cluster, navigate to Workloads → Pods. Existing pods should be terminated, or in the process of being terminated.
8. Wait for the following pods, which are responsible for upgrading the database and alembic migration of existing data, to spin up: clair-postgres-upgrade, quay-postgres-upgrade, and quay-app-upgrade.
9. After the clair-postgres-upgrade, quay-postgres-upgrade, and quay-app-upgrade pods are marked as Completed, the remaining pods for your Red Hat Quay deployment spin up. This takes approximately ten minutes.
10. Verify that the quay-database uses the postgresql-13 image, and clair-postgres pods now uses the postgresql-15 image.
11. After the quay-app pod is marked as Running, you can reach your Red Hat Quay registry.

Procedure

1. Delete the QuayRegistry using either the UI or kubectl:

   $ kubectl delete -n <namespace> quayregistry <quayecosystem-name>

2. If external access was provided using a Route, change the Route to point back to the original Service using the UI or kubectl.