Deploy Red Hat Quay on OpenShift with the Quay Operator

1. Create a namespace, for example, quay-enterprise.
2. Create a secret for the config bundle, if you want to pre-configure any aspects of the deployment

3. Create a QuayRegistry custom resource in a file called quayregistry.yaml

   1. For a minimal deployment, using all the defaults:

      quayregistry.yaml:

      apiVersion: quay.redhat.com/v1
      kind: QuayRegistry
      metadata:
        name: example-registry
        namespace: quay-enterprise

   2. If you want to have some components unmanaged, add this information in the spec field. For example, a minimal deployment might look like:

      quayregistry.yaml:

      apiVersion: quay.redhat.com/v1
      kind: QuayRegistry
      metadata:
        name: example-registry
        namespace: quay-enterprise
      spec:
        components:
          - kind: clair
            managed: false
          - kind: horizontalpodautoscaler
            managed: false
          - kind: mirror
            managed: false
          - kind: monitoring
            managed: false

   3. If you have created a config bundle, for example, init-config-bundle-secret, reference it in the quayregistry.yaml file:

      quayregistry.yaml:

      apiVersion: quay.redhat.com/v1
      kind: QuayRegistry
      metadata:
        name: example-registry
        namespace: quay-enterprise
      spec:
        configBundleSecret: init-config-bundle-secret

4. Create the QuayRegistry in specified namespace:

   $ oc create -f quayregistry.yaml

5. See the section Monitoring and debugging the deployment process for information on how to track the progress of the deployment.

6. Wait until the status.registryEndpoint is populated.

   $ oc get quayregistry -n quay-enterprise example-registry -o jsonpath="{.status.registryEndpoint}" -w

1. Create a namespace, for example, quay-enterprise.
2. Select Operators → Installed Operators, then select the Quay Operator to navigate to the Operator detail view.
3. Click 'Create Instance' on the 'Quay Registry' tile under 'Provided APIs'.
4. Optionally change the 'Name' of the QuayRegistry. This will affect the hostname of the registry. All other fields have been populated with defaults.
5. Click 'Create' to submit the QuayRegistry to be deployed by the Quay Operator.
6. You should be redirected to the QuayRegistry list view. Click on the QuayRegistry you just created to see the details view.
7. Once the 'Registry Endpoint' has a value, click it to access your new Quay registry via the UI. You can now select 'Create Account' to create a user and sign in.

1. Retrieve the secret:

   $ oc get secret -n quay-enterprise example-registry-quay-config-editor-credentials-kk55dc7299 -o yaml
   
   apiVersion: v1
   data:
     password: SkZwQkVKTUN0a1BUZmp4dA==
     username: cXVheWNvbmZpZw==
   kind: Secret

2. Decode the username:

   $ echo 'cXVheWNvbmZpZw==' | base64 --decode
   
   quayconfig

3. Decode the password:

   $ echo 'SkZwQkVKTUN0a1BUZmp4dA==' | base64 --decode
   
   JFpBEJMCtkPTfjxt

1. Using the config editor endpoint, specifying the username and password for the config editor:

   $ curl -k -u quayconfig:JFpBEJMCtkPTfjxt https://example-registry-quay-config-editor-quay-enterprise.apps.docs.quayteam.org/api/v1/config

   {
       "config.yaml": {
           "ALLOW_PULLS_WITHOUT_STRICT_LOGGING": false,
           "AUTHENTICATION_TYPE": "Database",
           ...
           "USER_RECOVERY_TOKEN_LIFETIME": "30m"
       },
       "certs": {
           "extra_ca_certs/service-ca.crt": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURVVENDQWptZ0F3SUJBZ0lJRE9kWFhuUXFjMUF3RFFZSktvWklodmNOQVFFTEJRQXdOakUwTURJR0ExVUUKQXd3cmIzQmxibk5vYVdaMExYTmxjblpwWTJVdGMyVnlkbWx1WnkxemFXZHVaWEpBTVRZek1UYzNPREV3TXpBZQpGdzB5TVRBNU1UWXdOelF4TkRKYUZ..."
       }
   }

2. Using the config bundle secret

   1. Get the secret data:

      $ oc get secret -n quay-enterprise example-registry-quay-config-bundle-jkfhs -o jsonpath='{.data}'

      {
          "config.yaml": "QUxMT1dfUFVMTFNfV0lUSE9VVF9TVFJJQ1RfTE9HR0lORzogZmFsc2UKQVVUSEVOVElDQVRJT05fVFlQRTogRGF0YWJhc2UKQVZBVEFSX0tJTkQ6IGxvY2FsCkRBVEFCQVNFX1NFQ1JFVF9LRVk6IHhlOEc1VDBNbkllaGxNQzNkTjd3MWR5WWxwVmo0a0R2enlxZ3l6Ulp5ZjFpODBmWWU3VDUxU1FPZ3hXelpocFlqYlVxNzRKaDllVVVEVWpyCkRFR
      ...
      OgotIDJ3ClRFQU1fUkVTWU5DX1NUQUxFX1RJTUU6IDYwbQpURVNUSU5HOiBmYWxzZQpVU0VSX1JFQ09WRVJZX1RPS0VOX0xJRkVUSU1FOiAzMG0K",
          "extra_ca_cert_service-ca.crt": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURVVENDQWptZ0F3SUJBZ0lJRE9kWFhuUXFjMUF3RFFZSktvWklodmNOQVFFTEJRQXdOakUwTURJR0ExVUUKQXd3cmIzQmxibk5vYVdaMExYTmxjblpwWTJVdGMyVnlkbWx1WnkxemFXZHVaWEpBTVRZek1UYzNPREV3TXpBZQpGdzB5TVRBNU1UWXdOelF4TkRKYUZ3MHl
      ...
      XSW1jaApkQXZTWGpFUnZOZEZzN3pHK1VzTmZwN0ZIQkJVWkY4L2RZNWJCR2MwWTVaY0J6bFNjQT09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K"
      }

   2. Decode the data:

      $ echo 'QUxMT1dfUFVMTFN...U1FOiAzMG0K' | base64 --decode

      ALLOW_PULLS_WITHOUT_STRICT_LOGGING: false
      AUTHENTICATION_TYPE: Database
      ...
      TAG_EXPIRATION_OPTIONS:
      - 2w
      TEAM_RESYNC_STALE_TIME: 60m
      TESTING: false
      USER_RECOVERY_TOKEN_LIFETIME: 30m

1. Create the secret using embedded data or using files:

   1. Embed the configuration details directly in the Secret resource YAML file, for example:

      custom-ssl-config-bundle.yaml

      apiVersion: v1
      kind: Secret
      metadata:
        name: custom-ssl-config-bundle-secret
        namespace: quay-enterprise
      data:
        config.yaml: |
          ALLOW_PULLS_WITHOUT_STRICT_LOGGING: false
          AUTHENTICATION_TYPE: Database
          ...
        extra_ca_cert_my-custom-ssl.crt: |
          -----BEGIN CERTIFICATE-----
          MIIDsDCCApigAwIBAgIUCqlzkHjF5i5TXLFy+sepFrZr/UswDQYJKoZIhvcNAQEL
          BQAwbzELMAkGA1UEBhMCSUUxDzANBgNVBAgMBkdBTFdBWTEPMA0GA1UEBwwGR0FM
          ....
          -----END CERTIFICATE-----

      Next, create the secret from the YAML file:

      $ oc create  -f custom-ssl-config-bundle.yaml

   2. Alternatively, you can create files containing the desired information, and then create the secret from those files:

      $ oc create secret generic custom-ssl-config-bundle-secret \
        --from-file=config.yaml \
        --from-file=extra_ca_cert_my-custom-ssl.crt=my-custom-ssl.crt

2. Create or update the QuayRegistry YAML file quayregistry.yaml, referencing the created Secret, for example:

   quayregistry.yaml

   apiVersion: quay.redhat.com/v1
   kind: QuayRegistry
   metadata:
     name: example-registry
     namespace: quay-enterprise
   spec:
     configBundleSecret: custom-ssl-config-bundle-secret

3. Deploy or update the registry using the YAML file:

   oc apply -f quayregistry.yaml

Procedure

1. Backup the QuayRegistry custom resource by exporting it:

   $ oc get quayregistry <quay-registry-name> -n <quay-namespace> -o yaml > quay-registry.yaml

2. Edit the resulting quayregistry.yaml and remove the status section and the following metadata fields:

     metadata.creationTimestamp
     metadata.finalizers
     metadata.generation
     metadata.resourceVersion
     metadata.uid

3. Backup the managed keys secret:

   Note

   If you are running a version older than Red Hat Quay 3.7.0, this step can be skipped. Some secrets are automatically generated while deploying Quay for the first time. These are stored in a secret called <quay-registry-name>-quay-registry-managed-secret-keys in the QuayRegistry namespace.

   $ oc get secret -n <quay-namespace> <quay-registry-name>-quay-registry-managed-secret-keys -o yaml > managed-secret-keys.yaml

4. Edit the the resulting managed-secret-keys.yaml file and remove all owner references. Your managed-secret-keys.yaml file should look similar to the following:

   apiVersion: v1
   kind: Secret
   type: Opaque
   metadata:
     name: <quayname>-quay-registry-managed-secret-keys
     namespace: <quay-namespace>
   data:
     CONFIG_EDITOR_PW: <redacted>
     DATABASE_SECRET_KEY: <redacted>
     DB_ROOT_PW: <redacted>
     DB_URI: <redacted>
     SECRET_KEY: <redacted>
     SECURITY_SCANNER_V4_PSK: <redacted>

   All information under the data property should remain the same.

5. Backup the current Quay configuration:

   $ oc get secret -n <quay-namespace>  $(oc get quayregistry <quay-registry-name> -n <quay-namespace>  -o jsonpath='{.spec.configBundleSecret}') -o yaml > config-bundle.yaml

6. Backup the /conf/stack/config.yaml file mounted inside of the Quay pods:

   $ oc exec -it quay-pod-name -- cat /conf/stack/config.yaml > quay-config.yaml

7. Scale down the Quay the Quay Operator:

   $  oc scale --replicas=0 deployment $(oc get deployment -n <quay-operator-namespace> |awk '/^quay-operator/ {print $1}') -n <quay-operator-namespace>

8. Scale down the Quay namespace:

   $ oc scale --replicas=0 deployment $(oc get deployment -n <quay-namespace> -l quay-component=quay -o jsonpath='{.items[0].metadata.name}') -n <quay-namespace>

9. Wait for the registry-quay-app pods to disappear. You can check their status by running the following command:

   $ oc get pods -n <quay-namespace>

   Example output:

   registry-quay-config-editor-77847fc4f5-nsbbv   1/1     Running            0          9m1s
   registry-quay-database-66969cd859-n2ssm        1/1     Running            0          6d1h
   registry-quay-mirror-758fc68ff7-5wxlp          1/1     Running            0          8m29s
   registry-quay-mirror-758fc68ff7-lbl82          1/1     Running            0          8m29s
   registry-quay-redis-7cc5f6c977-956g8           1/1     Running            0          5d21h

10. Identify the Quay PostgreSQL pod name:

    $ oc get pod -l quay-component=postgres -n <quay-namespace> -o jsonpath='{.items[0].metadata.name}'

    Exampe output:

1. Obtain the Quay database name:

   $ oc -n <quay-namespace> rsh $(oc get pod -l app=quay -o NAME -n <quay-namespace> |head -n 1) cat /conf/stack/config.yaml|awk -F"/" '/^DB_URI/ {print $4}'
   quayregistry-quay-database

2. Download a backup database:

   $ oc exec quayregistry-quay-database-59f54bb7-58xs7 -- /usr/bin/pg_dump -C quayregistry-quay-database  > backup.sql

3. Decode and export the AWS_ACCESS_KEY_ID:

   $ export AWS_ACCESS_KEY_ID=$(oc get secret -l app=noobaa -n <quay-namespace>  -o jsonpath='{.items[0].data.AWS_ACCESS_KEY_ID}' |base64 -d)

4. Decode and export the AWS_SECRET_ACCESS_KEY_ID:

   $ export AWS_SECRET_ACCESS_KEY=$(oc get secret -l app=noobaa -n <quay-namespace> -o jsonpath='{.items[0].data.AWS_SECRET_ACCESS_KEY}' |base64 -d)

5. Create a new directory and copy all blobs to it:

   $ mkdir blobs

   $ aws s3 sync --no-verify-ssl --endpoint https://$(oc get route s3 -n openshift-storage  -o jsonpath='{.spec.host}')  s3://$(oc get cm -l app=noobaa -n <quay-namespace> -o jsonpath='{.items[0].data.BUCKET_NAME}') ./blobs

1. Scale up the Quay the Quay Operator:

   $  oc scale --replicas=1 deployment $(oc get deployment -n <quay-operator-namespace> |awk '/^quay-operator/ {print $1}') -n <quay-operator-namespace>

2. Scale up the Quay namespace:

   $ oc scale --replicas=1 deployment $(oc get deployment -n <quay-namespace> -l quay-component=quay -o jsonpath='{.items[0].metadata.name}') -n <quay-namespace>

3. Check the status of the Operator:

   $ oc get quayregistry <quay-registry-name> -n <quay-namespace> -o yaml

   Example output:

   apiVersion: quay.redhat.com/v1
   kind: QuayRegistry
   metadata:
     ...
     name: example-registry
     namespace: <quay-namespace>
     ...
   spec:
     components:
     - kind: quay
       managed: true
     ...
     - kind: clairpostgres
       managed: true
     configBundleSecret: init-config-bundle-secret
   status:
     configEditorCredentialsSecret: example-registry-quay-config-editor-credentials-fg2gdgtm24
     configEditorEndpoint: https://example-registry-quay-config-editor-quay-enterprise.apps.docs.gcp.quaydev.org
     currentVersion: 3.7.0
     lastUpdated: 2022-05-11 13:28:38.199476938 +0000 UTC
     registryEndpoint: https://example-registry-quay-quay-enterprise.apps.docs.gcp.quaydev.org
        0          5d21h

Procedure

1. Restore the backed up Quay configuration and the randomly generated keys:

   $ oc create -f ./config-bundle.yaml

   $ oc create -f ./managed-secret-keys.yaml

   Note

   If you receive the error Error from server (AlreadyExists): error when creating "./config-bundle.yaml": secrets "config-bundle-secret" already exists, you must delete your exist resource with $ oc delete Secret config-bundle-secret -n <quay-namespace> and recreate it with $ oc create -f ./config-bundle.yaml.

2. Restore the QuayRegistry custom resource:

   $ oc create -f ./quay-registry.yaml

3. Scale down the Quay the Quay Operator:

   $  oc scale --replicas=0 deployment $(oc get deployment -n <quay-operator-namespace> |awk '/^quay-operator/ {print $1}') -n <quay-operator-namespace>

4. Scale down the Quay namespace:

   $ oc scale --replicas=0 deployment $(oc get deployment -n <quay-namespace> -l quay-component=quay -o jsonpath='{.items[0].metadata.name}') -n <quay-namespace>

5. Identify your Quay database pod:

   $ oc get pod -l quay-component=postgres -n  <quay-namespace> -o jsonpath='{.items[0].metadata.name}'

   Example output:

   quayregistry-quay-database-59f54bb7-58xs7

6. Upload the backup by copying it from the local environment and into the pod:

   $ oc cp ./backup.sql -n <quay-namespace> registry-quay-database-66969cd859-n2ssm:/tmp/backup.sql

7. Open a remote terminal to the database:

   $ oc rsh -n <quay-namespace> registry-quay-database-66969cd859-n2ssm

8. Enter psql:

   bash-4.4$ psql

9. You can list the database by running the following command:

   postgres=# \l

   Example output:

                                                     List of databases
              Name            |           Owner            | Encoding |  Collate   |   Ctype    |   Access privileges
   ----------------------------+----------------------------+----------+------------+------------+-----------------------
   postgres                   | postgres                   | UTF8     | en_US.utf8 | en_US.utf8 |
   quayregistry-quay-database | quayregistry-quay-database | UTF8     | en_US.utf8 | en_US.utf8 |

10. Drop the database:

    postgres=# DROP DATABASE "quayregistry-quay-database";

    Example output:

    DROP DATABASE

11. Exit the postgres CLI to re-enter bash-4.4:

    \q

12. Redirect your PostgreSQL database to your backup database:

    sh-4.4$ psql < /tmp/backup.sql

13. Exit bash:

    sh-4.4$ exit

14. Export the AWS_ACCESS_KEY_ID:

    $ export AWS_ACCESS_KEY_ID=$(oc get secret -l app=noobaa -n <quay-namespace>  -o jsonpath='{.items[0].data.AWS_ACCESS_KEY_ID}' |base64 -d)

15. Export the AWS_SECRET_ACCESS_KEY:

    $ export AWS_SECRET_ACCESS_KEY=$(oc get secret -l app=noobaa -n <quay-namespace> -o jsonpath='{.items[0].data.AWS_SECRET_ACCESS_KEY}' |base64 -d)

16. Upload all blobs to the bucket by running the following command:

    $ aws s3 sync --no-verify-ssl --endpoint https://$(oc get route s3 -n openshift-storage  -o jsonpath='{.spec.host}') ./blobs  s3://$(oc get cm -l app=noobaa -n <quay-namespace> -o jsonpath='{.items[0].data.BUCKET_NAME}')

17. Scale up the Quay the Quay Operator:

    $  oc scale --replicas=1 deployment $(oc get deployment -n <quay-operator-namespace> |awk '/^quay-operator/ {print $1}') -n <quay-operator-namespace>

18. Scale up the Quay namespace:

    $ oc scale --replicas=1 deployment $(oc get deployment -n <quay-namespace> -l quay-component=quay -o jsonpath='{.items[0].metadata.name}') -n <quay-namespace>

19. Check the status of the Operator and ensure it has come back online:

    $ oc get quayregistry -n <quay-namespace> <registry-name> -o yaml

    Example output:

    apiVersion: quay.redhat.com/v1
    kind: QuayRegistry
    metadata:
      ...
      name: example-registry
      namespace: quay-enterprise
      ...
    spec:
      components:
      - kind: quay
        managed: true
      ...
      - kind: clairpostgres
        managed: true
      configBundleSecret: init-config-bundle-secret
    status:
      configEditorCredentialsSecret: example-registry-quay-config-editor-credentials-fg2gdgtm24
      configEditorEndpoint: https://example-registry-quay-config-editor-quay-enterprise.apps.docs.gcp.quaydev.org
      currentVersion: 3.7.0
      lastUpdated: 2022-05-11 13:28:38.199476938 +0000 UTC
      registryEndpoint: https://example-registry-quay-quay-enterprise.apps.docs.gcp.quaydev.org
         0          5d21h

1. Add "quay-operator/migrate": "true" to the metadata.labels of the QuayEcosystem.

   $ oc edit quayecosystem <quayecosystemname>

   metadata:
     labels:
       quay-operator/migrate: "true"

2. Wait for a QuayRegistry to be created with the same metadata.name as your QuayEcosystem. The QuayEcosystem will be marked with the label "quay-operator/migration-complete": "true".
3. Once the status.registryEndpoint of the new QuayRegistry is set, access Quay and confirm all data and settings were migrated successfully.
4. When you are confident everything worked correctly, you may delete the QuayEcosystem and Kubernetes garbage collection will clean up all old resources.