Chapter 15. Updating OpenShift Container Storage
15.1. Overview of the OpenShift Container Storage update process
You can upgrade Red Hat OpenShift Container Storage and its components, either between minor releases like 4.6 and 4.7, or between batch updates like 4.7.0 and 4.7.1.
You need to upgrade the different parts of OpenShift Container Storage in a specific order.
- Update OpenShift Container Platform according to the Updating clusters documentation for OpenShift Container Platform.
Update OpenShift Container Storage.
- To prepare a disconnected environment for updates, see Operators guide to using Operator Lifecycle Manager on restricted networks to be able to update Openshift Container Storage as well as Local Storage Operator when in use.
Update the OpenShift Container Storage operator, using the appropriate process for your setup:
Update considerations
Review the following important considerations before you begin.
Red Hat recommends using the same version of Red Hat OpenShift Container Platform with Red Hat OpenShift Container Storage.
See the Interoperability Matrix for more information about supported combinations of OpenShift Container Platform and OpenShift Container Storage.
- The Local Storage Operator is fully supported only when the Local Storage Operator version matches the Red Hat OpenShift Container Platform version.
15.2. Preparing to update in a disconnected environment
When your Red Hat OpenShift Container Storage environment is not directly connected to the internet, some additional configuration is required to provide the Operator Lifecycle Manager (OLM) with alternatives to the default Operator Hub and image registries.
See the OpenShift Container Platform documentation for more general information: Updating an Operator catalog image.
To configure your cluster for disconnected update:
When these steps are complete, Continue with update as usual.
15.2.1. Adding mirror registry authentication details
Prerequisites
- Verify that your existing disconnected cluster uses OpenShift Container Platform 4.3 or higher.
-
Verify that you have an
oc client
version of 4.4 or higher. - Prepare a mirror host with a mirror registry. See Preparing your mirror host for details.
Procedure
-
Log in to the OpenShift Container Platform cluster using the
cluster-admin
role. Locate your
auth.json
file.This file is generated when you use podman or docker to log in to a registry. It is located in one of the following locations:
-
~/.docker/auth.json
-
/run/user/<UID>/containers/auth.json
-
/var/run/containers/<UID>/auth.json
-
Obtain your unique Red Hat registry pull secret and paste it into your
auth.json
file. It will look something like this.{ "auths": { "cloud.openshift.com": { "auth": "*****************", "email": "user@example.com" }, "quay.io": { "auth": "*****************", "email": "user@example.com" }, "registry.connect.redhat.com": { "auth": "*****************", "email": "user@example.com" }, "registry.redhat.io": { "auth": "*****************", "email": "user@example.com" } } }
Export environment variables with the appropriate details for your setup.
$ export AUTH_FILE="<location_of_auth.json>" $ export MIRROR_REGISTRY_DNS="<your_registry_url>:<port>"
Use
podman
to log in to the mirror registry and store the credentials in the${AUTH_FILE}
.$ podman login ${MIRROR_REGISTRY_DNS} --tls-verify=false --authfile ${AUTH_FILE}
This adds the mirror registry to the
auth.json
file.{ "auths": { "cloud.openshift.com": { "auth": "*****************", "email": "user@example.com" }, "quay.io": { "auth": "*****************", "email": "user@example.com" }, "registry.connect.redhat.com": { "auth": "*****************", "email": "user@example.com" }, "registry.redhat.io": { "auth": "*****************", "email": "user@example.com" }, "<mirror_registry>": { "auth": "*****************", } } }
15.2.2. Building and mirroring the Red Hat operator catalog
Follow this process on a host that has access to Red Hat registries to create a mirror of those registries.
Prerequisites
- Run these commands as a cluster administrator.
-
Be aware that mirroring the
redhat-operator
catalog can take hours to complete, and requires substantial available disk space on the mirror host.
Procedure
Build the catalog for
redhat-operators
.Set
--from
to theose-operator-registry
base image using the tag that matches the target OpenShift Container Platform cluster major and minor version.$ oc adm catalog build --appregistry-org redhat-operators \ --from=registry.redhat.io/openshift4/ose-operator-registry:v4.7 \ --to=${MIRROR_REGISTRY_DNS}/olm/redhat-operators:v2 \ --registry-config=${AUTH_FILE} \ --filter-by-os="linux/amd64" --insecure
Mirror the catalog for
redhat-operators
.This is a long operation and can take 1-5 hours. Make sure there is 100 GB available disk space on the mirror host.
$ oc adm catalog mirror ${MIRROR_REGISTRY_DNS}/olm/redhat-operators:v2 \ ${MIRROR_REGISTRY_DNS} --registry-config=${AUTH_FILE} --insecure
15.2.3. Creating Operator imageContentSourcePolicy
After the oc adm catalog mirror
command is completed, the imageContentSourcePolicy.yaml
file gets created. The output directory for this file is usually, ./[catalog image name]-manifests)
. Use this procedure to add any missing entries to the .yaml
file and apply them to cluster.
Procedure
Check the content of this file for the mirrors mapping shown as follows:
spec: repositoryDigestMirrors: - mirrors: - <your_registry>/ocs4 source: registry.redhat.io/ocs4 - mirrors: - <your_registry>/rhceph source: registry.redhat.io/rhceph - mirrors: - <your_registry>/openshift4 source: registry.redhat.io/openshift4 - mirrors: - <your_registry>/rhscl source: registry.redhat.io/rhscl
-
Add any missing entries to the end of the
imageContentSourcePolicy.yaml
file. Apply the imageContentSourcePolicy.yaml file to the cluster.
$ oc apply -f ./[output dir]/imageContentSourcePolicy.yaml
Once the Image Content Source Policy is updated, all the nodes (master, infra, and workers) in the cluster need to be updated and rebooted. This process is automatically handled through the Machine Config Pool operator and take up to 30 minutes although the exact elapsed time might vary based on the number of nodes in your OpenShift cluster. You can monitor the update process by using the
oc get mcp
command or theoc get node
command.
15.2.4. Updating redhat-operator CatalogSource
Procedure
Recreate a
CatalogSource
object that references the catalog image for Red Hat operators.NoteMake sure you have mirrored the correct catalog source with the correct version (that is,
v2
).Save the following in a
redhat-operator-catalogsource.yaml
file, remembering to replace <your_registry> with your mirror registry URL:apiVersion: operators.coreos.com/v1alpha1 kind: CatalogSource metadata: name: redhat-operators namespace: openshift-marketplace spec: sourceType: grpc icon: base64data: PHN2ZyBpZD0iTGF5ZXJfMSIgZGF0YS1uYW1lPSJMYXllciAxIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHZpZXdCb3g9IjAgMCAxOTIgMTQ1Ij48ZGVmcz48c3R5bGU+LmNscy0xe2ZpbGw6I2UwMDt9PC9zdHlsZT48L2RlZnM+PHRpdGxlPlJlZEhhdC1Mb2dvLUhhdC1Db2xvcjwvdGl0bGU+PHBhdGggZD0iTTE1Ny43Nyw2Mi42MWExNCwxNCwwLDAsMSwuMzEsMy40MmMwLDE0Ljg4LTE4LjEsMTcuNDYtMzAuNjEsMTcuNDZDNzguODMsODMuNDksNDIuNTMsNTMuMjYsNDIuNTMsNDRhNi40Myw2LjQzLDAsMCwxLC4yMi0xLjk0bC0zLjY2LDkuMDZhMTguNDUsMTguNDUsMCwwLDAtMS41MSw3LjMzYzAsMTguMTEsNDEsNDUuNDgsODcuNzQsNDUuNDgsMjAuNjksMCwzNi40My03Ljc2LDM2LjQzLTIxLjc3LDAtMS4wOCwwLTEuOTQtMS43My0xMC4xM1oiLz48cGF0aCBjbGFzcz0iY2xzLTEiIGQ9Ik0xMjcuNDcsODMuNDljMTIuNTEsMCwzMC42MS0yLjU4LDMwLjYxLTE3LjQ2YTE0LDE0LDAsMCwwLS4zMS0zLjQybC03LjQ1LTMyLjM2Yy0xLjcyLTcuMTItMy4yMy0xMC4zNS0xNS43My0xNi42QzEyNC44OSw4LjY5LDEwMy43Ni41LDk3LjUxLjUsOTEuNjkuNSw5MCw4LDgzLjA2LDhjLTYuNjgsMC0xMS42NC01LjYtMTcuODktNS42LTYsMC05LjkxLDQuMDktMTIuOTMsMTIuNSwwLDAtOC40MSwyMy43Mi05LjQ5LDI3LjE2QTYuNDMsNi40MywwLDAsMCw0Mi41Myw0NGMwLDkuMjIsMzYuMywzOS40NSw4NC45NCwzOS40NU0xNjAsNzIuMDdjMS43Myw4LjE5LDEuNzMsOS4wNSwxLjczLDEwLjEzLDAsMTQtMTUuNzQsMjEuNzctMzYuNDMsMjEuNzdDNzguNTQsMTA0LDM3LjU4LDc2LjYsMzcuNTgsNTguNDlhMTguNDUsMTguNDUsMCwwLDEsMS41MS03LjMzQzIyLjI3LDUyLC41LDU1LC41LDc0LjIyYzAsMzEuNDgsNzQuNTksNzAuMjgsMTMzLjY1LDcwLjI4LDQ1LjI4LDAsNTYuNy0yMC40OCw1Ni43LTM2LjY1LDAtMTIuNzItMTEtMjcuMTYtMzAuODMtMzUuNzgiLz48L3N2Zz4= mediatype: image/svg+xml image: <your_registry>/olm/redhat-operators:v2 displayName: Redhat Operators Catalog publisher: Red Hat
Create a catalogsource using the redhat-operator-catalogsource.yaml file:
$ oc apply -f redhat-operator-catalogsource.yaml
Verify that the new
redhat-operator
pod is running.$ oc get pod -n openshift-marketplace | grep redhat-operators
15.2.5. Continue to update
After your alternative catalog source is configured, you can continue to the appropriate update process:
15.3. Updating OpenShift Container Storage in internal mode
Use the following procedures to update your OpenShift Container Storage cluster deployed in internal mode.
15.3.1. Enabling automatic updates for OpenShift Container Storage operator in internal mode
Use this procedure to enable automatic update approval for updating OpenShift Container Storage operator in OpenShift Container Platform.
Prerequisites
- Under Persistent Storage in the Status card, confirm that the OCS Cluster and Data Resiliency has a green tick mark.
-
Under Object Service in the Status card, confirm that both the Object Service and Data Resiliency are in
Ready
state (green tick). - Update the OpenShift Container Platform cluster to the latest stable release of version 4.7.Y, see Updating Clusters.
Switch the Red Hat OpenShift Container Storage channel from
stable-4.6
tostable-4.7
. For details about channels, see OpenShift Container Storage upgrade channels and releases.NoteYou are required to switch channels only when you are updating minor versions (for example, updating from 4.6 to 4.7) and not when updating between batch updates of 4.7 (for example, updating from 4.7.0 to 4.7.1).
Ensure that all OpenShift Container Storage Pods, including the operator pods, are in
Running
state in theopenshift-storage namespace
.To view the state of the pods, click Workloads
Pods from the left pane of the OpenShift Web Console. Select openshift-storage from the Project drop down list. - Ensure that you have sufficient time to complete the Openshift Container Storage update process, as the update time varies depending on the number of OSDs that run in the cluster.
Procedure
- Log in to OpenShift Web Console.
-
Click Operators
Installed Operators -
Select the
openshift-storage
project. - Click the OpenShift Container Storage operator name.
- Click the Subscription tab and click the link under Approval.
- Select Automatic (default) and click Save.
Perform one of the following depending on the Upgrade Status:
Upgrade Status shows requires approval.
NoteUpgrade status shows requires approval if the new OpenShift Container Storage version is already detected in the channel, and approval strategy was changed from Manual to Automatic at the time of update.
- Click on the Install Plan link.
- On the InstallPlan Details page, click Preview Install Plan.
- Review the install plan and click Approve.
- Wait for the Status to change from Unknown to Created.
-
Click Operators
Installed Operators -
Select the
openshift-storage
project. - Wait for the Status to change to Up to date
Upgrade Status does not show requires approval:
- Wait for the update to initiate. This may take up to 20 minutes.
-
Click Operators
Installed Operators -
Select the
openshift-storage
project. - Wait for the Status to change to Up to date
Multicloud Object Gateway outage is expected for a short period of time during upgrade due to migration of NooBaa DB from MongoDB to PostgreSQL.
Verification steps
-
Click Overview
Persistent Storage tab and in the Status card confirm that the OCS Cluster and Data Resiliency has a green tick mark indicating it is healthy. -
Click Overview
Object Service tab and in the Status card confirm that both the Object Service and Data Resiliency are in Ready
state (green tick) indicating it is healthy. Click Operators
Installed Operators OpenShift Container Storage Operator. Under Storage Cluster, verify that the cluster service status is Ready
.NoteOnce updated from OpenShift Container Storage version 4.6 to 4.7, the
Version
field here will still display 4.6. This is because theocs-operator
does not update the string represented in this field.Ensure that all OpenShift Container Storage Pods, including the operator pods, are in
Running
state in theopenshift-storage namespace
.To view the state of the pods, click Workloads
Pods. Select openshift-storage from the Project drop down list. - If verification steps fail, contact Red Hat Support.
The flexible scaling feature is available only in the new deployments of Red Hat OpenShift Container Storage 4.7. Storage clusters upgraded to the 4.7 version do not support flexible scaling.
Additional Resources
If you face any issues while updating OpenShift Container Storage, see the Commonly required logs for troubleshooting section in the Troubleshooting guide.
15.3.2. Manually updating OpenShift Container Storage operator in internal mode
Use this procedure to update OpenShift Container Storage operator by providing manual approval to the install plan.
Prerequisites
- Under Persistent Storage in the Status card, confirm that the OCS Cluster and Data Resiliency has a green tick mark.
-
Under Object Service in the Status card, confirm that both the Object Service and Data Resiliency are in
Ready
state (green tick). - Update the OpenShift Container Platform cluster to the latest stable release of version 4.7.Y, see Updating Clusters.
Switch the Red Hat OpenShift Container Storage channel from
stable-4.6
tostable-4.7
. For details about channels, see OpenShift Container Storage upgrade channels and releases.NoteYou are required to switch channels only when you are updating minor versions (for example, updating from 4.6 to 4.7) and not when updating between batch updates of 4.7 (for example, updating from 4.7.0 to 4.7.1).
Ensure that all OpenShift Container Storage Pods, including the operator pods, are in
Running
state in theopenshift-storage namespace
.To view the state of the pods, click Workloads
Pods from the left pane of the OpenShift Web Console. Select openshift-storage from the Project drop down list. - Ensure that you have sufficient time to complete the Openshift Container Storage update process, as the update time varies depending on the number of OSDs that run in the cluster.
Procedure
- Log in to OpenShift Web Console.
-
Click Operators
Installed Operators -
Select the
openshift-storage
project. - Click the OpenShift Container Storage operator name.
- Click the Subscription tab and click the link under Approval.
- Select Manual and click Save.
- Wait for the Upgrade Status to change to Upgrading.
- If the Upgrade Status shows requires approval, click on requires approval.
- On the InstallPlan Details page, click Preview Install Plan.
- Review the install plan and click Approve.
- Wait for the Status to change from Unknown to Created.
-
Click Operators
Installed Operators -
Select the
openshift-storage
project. - Wait for the Status to change to Up to date
Multicloud Object Gateway outage is expected for a short period of time during upgrade due to migration of NooBaa DB from MongoDB to PostgreSQL.
Verification steps
-
Click Overview
Persistent Storage tab and in the Status card confirm that the OCS Cluster and Data Resiliency has a green tick mark indicating it is healthy. -
Click Overview
Object Service tab and in the Status card confirm that both the Object Service and Data Resiliency are in Ready
state (green tick) indicating it is healthy. Click Operators
Installed Operators OpenShift Container Storage Operator. Under Storage Cluster, verify that the cluster service status is Ready
.NoteOnce updated from OpenShift Container Storage version 4.6 to 4.7, the
Version
field here will still display 4.6. This is because theocs-operator
does not update the string represented in this field.Ensure that all OpenShift Container Storage Pods, including the operator pods, are in
Running
state in theopenshift-storage namespace
.To view the state of the pods, click Workloads
Pods from the left pane of the OpenShift Web Console. Select openshift-storage from the Project drop down list. - If verification steps fail, contact Red Hat Support.
Additional Resources
If you face any issues while updating OpenShift Container Storage, see the Commonly required logs for troubleshooting section in the Troubleshooting guide.