Chapter 5. Running the upgrade to Red Hat Quay v3.0.z
Choose between a synchronous upgrade (complete the upgrade in downtime) and a background upgrade (complete the upgrade while Red Hat Quay is still running). Both of these major-release upgrades require that the Red Hat Quay cluster be down for at least a short period of time.
Regardless of which upgrade type you choose, during the time that the Red Hat Quay cluster is down, if you are using builder and clair images, you need to also upgrade to those new images:
- The builder image (quay.io/redhat/quay-builder:v3.0.5)
- The clair image (quay.io/redhat/clair-jwt:v3.0.5)
Both of those images are available from the quay.io/redhat repository.
5.1. Run a synchronous upgrade
To run a synchronous upgrade, where your whole cluster is down for the entire upgrade, do the following:
- Take down your entire Red Hat Quay cluster, including any quay-builder and clair containers.
Add the following setting to the
config.yamlfile on all nodes:V3_UPGRADE_MODE: complete
Pull and start up the v3 container on a single node and wait for however long it takes to do the upgrade (it should take just a few minutes). Use the following container or later:
quay.io/redhat/quay:v3.0.5
Note that the quay container comes up on ports 8080 and 8443 for v3, instead of 80 and 443, as they did for v2. Therefore, we recommend remapping 8080 and 8443 into 80 and 443, respectively, as shown in this example:
# docker run --restart=always -p 443:8443 -p 80:8080 \ --sysctl net.core.somaxconn=4096 \ --privileged=true \ -v /mnt/quay/config:/conf/stack:Z \ -v /mnt/quay/storage:/datastorage:Z \ -d quay.io/redhat/quay:v3.0.5
- After the upgrade completes, bring the Red Hat Quay v3 container up on all other nodes.
- Start v3.0.z versions of quay-builder and clair to replace any instances of those containers you want to return to your cluster.
- Verify that Red Hat Quay is working, including pushes and pulls of containers compatible with Docker version 2, schema 2. This can include windows container images and images of different computer architectures (arm, ppc, etc.).
5.2. Run a background upgrade
To run a background upgrade, you need only bring down your cluster for a short period of time on two occasions. When you bring the cluster back up after the first downtime, the quay v3 container runs in v2 compatibility mode as it backfills the database. This background process can take hours or even days to complete. Background upgrades are recommended for large installations where downtime of more than a few hours would be a problem.
For this type of upgrade, you put Red Hat Quay into a compatibility mode, where you have a v3 quay container running, but it is running on the old data model while the upgrade completes. Here’s what you do:
Pull the Red Hat Quay v3 container to all the nodes. Use the following container or later:
quay.io/redhat/quay:v3.0.5
- Take down your entire Red Hat Quay cluster, including any quay-builder and clair containers.
Edit the
config.yamlfile on each node and set the upgrade mode to background as follows:V3_UPGRADE_MODE: background
Bring the Red Hat Quay v3 container up on a single node and wait for the migrations to complete (should take a few minutes maximum). Here is an example of that command:
Note that the quay container comes up on ports 8080 and 8443 for v3, instead of 80 and 443, as they did for v2. Therefore, we recommend remapping 8080 and 8443 into 80 and 443, respectively, as shown in this example:
# docker run --restart=always -p 443:8443 -p 80:8080 \ --sysctl net.core.somaxconn=4096 \ --privileged=true \ -v /mnt/quay/config:/conf/stack:Z \ -v /mnt/quay/storage:/datastorage:Z \ -d quay.io/redhat/quay:v3.0.5
- Bring the Red Hat Quay v3 container up on all the other nodes.
-
Monitor the
/upgradeprogressAPI endpoint until it reports done enough to move to the next step (the status reaches 99%). For example, viewhttps://myquay.example.com/upgradeprogressor use some other tool to query the API. - Once the background process is far enough along you have to schedule another maintenance window.
- During your scheduled maintenance, take the entire Red Hat Quay cluster down.
Edit the
config.yamlfile on each node and set the upgrade mode tocompleteas follows:V3_UPGRADE_MODE: complete
- Bring Red Hat Quay back up on one node to have it do a final check.
- Once the final check is done, bring Red Hat Quay v3 back up on all the other nodes.
- Start v3.0.z versions of quay-builder and clair to replace any instances of those containers you want to return to your cluster.
- Verify Quay is working, including pushes and pulls of containers compatible with Docker version 2, schema 2. This can include windows container images and images of different computer architectures (arm, ppc, etc.).
