Chapter 3. Standalone upgrade
In general, Red Hat Quay supports upgrades from a prior (N-1) minor version only. For example, upgrading directly from v3.0.5 to v3.5.7 is not supported. You will need to first upgrade from v3.0.5 to v3.1.3 to v3.2.2 to v3.3.4 to v3.4.latest and then finally to v3.5.7. This is required to ensure that any necessary database migrations are done correctly and in the right order during the upgrade.
This document describes the steps needed to perform each individual upgrade. Determine your current version and then follow the steps in sequential order, starting with your current version and working up to your desired target version.
- Upgrade to v3.5.0 from v3.4.*
- Upgrade to v3.4.3 from v3.3.*
- Upgrade to v3.3.4 from v3.2.z
- Upgrade to v3.2.2 from v3.1.z
- Upgrade to v3.1.3 from v3.0.z
- Upgrade to v3.0.5 from v2.9.5
See the Red Hat Quay Release Notes for information on features for individual releases.
The general procedure for a manual upgrade consists of the following steps:
- Stop the Quay and Clair containers
- Backup the database and image storage (optional but recommended)
- Start Clair using the new version of the image
- Wait until Clair is ready to accept connections before starting the new version of Quay
3.1. Accessing images Copy linkLink copied to clipboard!
Images for Quay 3.4.0 and later are available from registry.redhat.io
and registry.access.redhat.com
, with authentication set up as described in Red Hat Container Registry Authentication.
Images for Quay 3.3.4 and earlier are available from quay.io
, with authentication set up as described in Accessing Red Hat Quay without a CoreOS login.
3.2. Upgrade to v3.5.7 from v3.4.* Copy linkLink copied to clipboard!
3.2.1. Explicitly enabling OCI and Helm support Copy linkLink copied to clipboard!
Support for Helm and OCI artifacts is now enabled by default in Red Hat Quay 3.5. If you need to explicitly enable the feature, for example, if you are upgrading from a version where it is not enabled by default, you need to add two properties in the Quay configuration to enable the use of OCI artifacts:
FEATURE_GENERAL_OCI_SUPPORT: true FEATURE_HELM_OCI_SUPPORT: true
FEATURE_GENERAL_OCI_SUPPORT: true
FEATURE_HELM_OCI_SUPPORT: true
3.2.2. Target images Copy linkLink copied to clipboard!
- Quay: registry.redhat.io/quay/quay-rhel8:v3.5.7
- Clair: registry.redhat.io/quay/clair-rhel8:v3.5.7
- PostgreSQL: registry.redhat.io/rhel8/postgresql-10:1
- Redis: registry.redhat.io/rhel8/redis-5:1
3.3. Upgrade to v3.4.3 from v3.3.* Copy linkLink copied to clipboard!
Upgrading to Quay 3.4 requires a database migration which does not support downgrading back to a prior version of Quay. Please back up your database before performing this migration.
3.3.1. Target images Copy linkLink copied to clipboard!
- Quay: registry.redhat.io/quay/quay-rhel8:v3.5.7
- Clair: registry.redhat.io/quay/clair-rhel8:v3.5.7
- PostgreSQL: registry.redhat.io/rhel8/postgresql-10:1
- Redis: registry.redhat.io/rhel8/redis-5:1
3.4. Upgrade to v3.3.4 from v3.2.z Copy linkLink copied to clipboard!
3.4.1. Target images Copy linkLink copied to clipboard!
- Quay: quay.io/redhat/quay:v3.3.4
- Clair: quay.io/redhat/clair-jwt:v3.3.4
- PostgreSQL: rhscl/postgresql-96-rhel7
- Redis: registry.access.redhat.com/rhscl/redis-32-rhel7
3.5. Upgrade to v3.2.2 from v3.1.z Copy linkLink copied to clipboard!
Once your cluster is running any Red Hat Quay 3.1.z version, to upgrade your cluster to v3.2.2 you must bring down your entire cluster and make a small change to the configuration before bringing it back up with the v3.2.2 version.
Once you set the value of DATABASE_SECRET_KEY in this procedure, do not ever change it. If you do so, then existing robot accounts, API tokens, etc. cannot be used anymore. You would have to create a new robot account and API tokens to use with Quay.
- Take all hosts in the Red Hat Quay cluster out of service.
Generate some random data to use as a database secret key. For example:
openssl rand -hex 48
$ openssl rand -hex 48 2d023adb9c477305348490aa0fd9c
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Add a new DATABASE_SECRET_KEY field to your
config.yaml
file. For example:DATABASE_SECRET_KEY: "2d023adb9c477305348490aa0fd9c"
DATABASE_SECRET_KEY: "2d023adb9c477305348490aa0fd9c"
Copy to Clipboard Copied! Toggle word wrap Toggle overflow NoteFor an OpenShift installation, the
config.yaml
file is stored as a secret.-
Bring up one
Quay
container to complete the migration to v3.2.2 . -
Once the migration is done, make sure the same
config.yaml
is available on all nodes and bring up the new quay v3.2.2 service on those nodes. - Start v3.0.z versions of quay-builder and clair to replace any instances of those containers you want to return to your cluster.
3.5.1. Target images Copy linkLink copied to clipboard!
- Quay: quay.io/redhat/quay:v3.2.2
- Clair: quay.io/redhat/clair-jwt:v3.2.2
- PostgreSQL: rhscl/postgresql-96-rhel7
- Redis: registry.access.redhat.com/rhscl/redis-32-rhel7
3.6. Upgrade to v3.1.3 from v3.0.z Copy linkLink copied to clipboard!
3.6.1. Target images Copy linkLink copied to clipboard!
- Quay: quay.io/redhat/quay:v3.1.3
- Clair: quay.io/redhat/clair-jwt:v3.1.3
- PostgreSQL: rhscl/postgresql-96-rhel7
- Redis: registry.access.redhat.com/rhscl/redis-32-rhel7
3.7. Upgrade to v3.0.5 from v2.9.5 Copy linkLink copied to clipboard!
For the v2.9.5 to v3.0.5 upgrade, you can either do the whole upgrade with Red Hat Quay down (Synchronous) or only bring down Red Hat Quay for a few minutes and have the bulk of the upgrade continue with Red Hat Quay running (Background).
In a background upgrade, it could take much longer to run the upgrade (depending on how many tags need to be processed), but it takes less total downtime. The downside of a background upgrade is that you won’t have access to the latest features until the upgrade completes (the cluster runs from the quay v3 container in v2 compatibility mode until the upgrade is done).
3.7.1. Overview of upgrade Copy linkLink copied to clipboard!
Follow the procedure below if you are starting with a Red Hat Quay v2 cluster. Before upgrading to the latest Red Hat Quay 3.x version, you must first migrate that cluster to v3.0.5, as described here. Once your cluster is running v3.0.5, you can then upgrade to the latest 3.x version by sequentially upgrading to each minor version in turn (3.0 to 3.1 to 3.2, etc…)
Before beginning your Red Hat Quay v2 to v3.0 upgrade, please note the following:
- Synchronous upgrade: For a synchronous upgrade, expect less than one hour of total downtime for small installations. Consider a small installation to contain a few thousand container image tags or fewer. For that size installation, you could probably get by with just a couple hours of scheduled downtime. The entire Red Hat Quay service is down for the duration, so if you were to try a synchronous upgrade on a registry with millions of tags, you could potentially be down for several days.
- Background upgrade: For a background upgrade (also called a compatibility mode upgrade), after a short shutdown your Red Hat Quay cluster upgrade runs in the background. For large Red Hat Quay registries, this could take weeks to complete, but the cluster continues to operate in v2 mode for the duration of the upgrade. As a point of reference, one Red Hat Quay v3 upgrade took four days to process approximately 30 million tags across six machines.
- Full features on completion: Before you have access to features associated with Docker version 2, schema 2 changes (such as support for containers of different architectures), the entire migration must complete. Other v3 features are immediately available when you switch over.
-
Upgrade complete: When the upgrade is complete, you need to set V3_UPGRADE_MODE: complete in the Red Hat Quay
config.yaml
file for the new features to be available. All new Red Hat Quay v3 installations automatically have that set.
3.7.2. Prerequisites Copy linkLink copied to clipboard!
To assure the best results, we recommend the following prerequisites:
- Back up your Red Hat Quay database before starting the upgrade (doing regular backups is a general best practice). A good time to do this is right after you have taken down the Red Hat Quay cluster to do the upgrade.
- Back up your storage (also a general best practice).
Upgrade your current Red Hat Quay 2.y.z setup to the latest 2.9.z version (currently 2.9.5) before starting the v3 upgrade. To do that:
-
While the Red Hat Quay cluster is still running, take one node and change the
Quay
container on that system to aQuay
container that is running the latest 2.9.z version. - Wait for all the database migrations to run, bringing the database up to the latest 2.9.z version. This should only take a few minutes to a half an hour.
-
Once that is done, replace the
Quay
container on all the existing nodes with the same latest 2.9.z version. With the entire Red Hat Quay cluster on the new version, you can proceed to the v3 upgrade.
-
While the Red Hat Quay cluster is still running, take one node and change the
3.7.3. Choosing upgrade type Copy linkLink copied to clipboard!
Choose between a synchronous upgrade (complete the upgrade in downtime) and a background upgrade (complete the upgrade while Red Hat Quay is still running). Both of these major-release upgrades require that the Red Hat Quay cluster be down for at least a short period of time.
Regardless of which upgrade type you choose, during the time that the Red Hat Quay cluster is down, if you are using builder and clair images, you need to also upgrade to those new images:
- The builder image (quay.io/redhat/quay-builder:v3.0.5)
- The clair image (quay.io/redhat/clair-jwt:v3.0.5)
Both of those images are available from the registry.redhat.io/quay repository.
3.7.4. Running a synchronous upgrade Copy linkLink copied to clipboard!
To run a synchronous upgrade, where your whole cluster is down for the entire upgrade, do the following:
- Take down your entire Red Hat Quay cluster, including any quay-builder and clair containers.
Add the following setting to the
config.yaml
file on all nodes:V3_UPGRADE_MODE: complete
Pull and start up the v3 container on a single node and wait for however long it takes to do the upgrade (it should take just a few minutes). Use the following container or later:
quay.io/redhat/quay:v3.0.5
Note that the
Quay
container comes up on ports 8080 and 8443 for v3, instead of 80 and 443, as they did for v2. Therefore, we recommend remapping 8080 and 8443 into 80 and 443, respectively, as shown in this example:Copy to Clipboard Copied! Toggle word wrap Toggle overflow - After the upgrade completes, bring the Red Hat Quay v3 container up on all other nodes.
- Start v3.0.z versions of quay-builder and clair to replace any instances of those containers you want to return to your cluster.
- Verify that Red Hat Quay is working, including pushes and pulls of containers compatible with Docker version 2, schema 2. This can include windows container images and images of different computer architectures (arm, ppc, etc.).
3.7.5. Running a background upgrade Copy linkLink copied to clipboard!
To run a background upgrade, you need only bring down your cluster for a short period of time on two occasions. When you bring the cluster back up after the first downtime, the quay v3 container runs in v2 compatibility mode as it backfills the database. This background process can take hours or even days to complete. Background upgrades are recommended for large installations where downtime of more than a few hours would be a problem.
For this type of upgrade, you put Red Hat Quay into a compatibility mode, where you have a v3 Quay
container running, but it is running on the old data model while the upgrade completes. Here’s what you do:
Pull the Red Hat Quay v3 container to all the nodes. Use the following container or later:
quay.io/redhat/quay:v3.0.5
- Take down your entire Red Hat Quay cluster, including any quay-builder and clair containers.
Edit the
config.yaml
file on each node and set the upgrade mode to background as follows:V3_UPGRADE_MODE: background
Bring the Red Hat Quay v3 container up on a single node and wait for the migrations to complete (should take a few minutes maximum). Here is an example of that command:
Note that the
Quay
container comes up on ports 8080 and 8443 for v3, instead of 80 and 443, as they did for v2. Therefore, we recommend remapping 8080 and 8443 into 80 and 443, respectively, as shown in this example:Copy to Clipboard Copied! Toggle word wrap Toggle overflow - Bring the Red Hat Quay v3 container up on all the other nodes.
-
Monitor the
/upgradeprogress
API endpoint until it reports done enough to move to the next step (the status reaches 99%). For example, viewhttps://myquay.example.com/upgradeprogress
or use some other tool to query the API. - Once the background process is far enough along you have to schedule another maintenance window.
- During your scheduled maintenance, take the entire Red Hat Quay cluster down.
Edit the
config.yaml
file on each node and set the upgrade mode tocomplete
as follows:V3_UPGRADE_MODE: complete
- Bring Red Hat Quay back up on one node to have it do a final check.
- Once the final check is done, bring Red Hat Quay v3 back up on all the other nodes.
- Start v3.0.z versions of quay-builder and clair to replace any instances of those containers you want to return to your cluster.
- Verify Quay is working, including pushes and pulls of containers compatible with Docker version 2, schema 2. This can include windows container images and images of different computer architectures (arm, ppc, etc.).
3.7.6. Target images Copy linkLink copied to clipboard!
- Quay: quay.io/redhat/quay:v3.0.5
- Clair: quay.io/redhat/clair-jwt:v3.0.5
- Redis: registry.access.redhat.com/rhscl/redis-32-rhel7
- PostgreSQL: rhscl/postgresql-96-rhel7
- Builder: quay.io/redhat/quay-builder:v3.0.5