Updating OpenShift Container Storage
Instructions for cluster and storage administrators
Abstract
Making open source more inclusive
Red Hat is committed to replacing problematic language in our code, documentation, and web properties. We are beginning with these four terms: master, slave, blacklist, and whitelist. Because of the enormity of this endeavor, these changes will be implemented gradually over several upcoming releases. For more details, see our CTO Chris Wright’s message.
Providing feedback on Red Hat documentation
We appreciate your input on our documentation. Do let us know how we can make it better. To give feedback:
For simple comments on specific passages:
- Make sure you are viewing the documentation in the Multi-page HTML format. In addition, ensure you see the Feedback button in the upper right corner of the document.
- Use your mouse cursor to highlight the part of text that you want to comment on.
- Click the Add Feedback pop-up that appears below the highlighted text.
- Follow the displayed instructions.
For submitting more complex feedback, create a Bugzilla ticket:
- Go to the Bugzilla website.
- As the Component, use Documentation.
- Fill in the Description field with your suggestion for improvement. Include a link to the relevant part(s) of documentation.
- Click Submit Bug.
Chapter 1. Overview of the OpenShift Container Storage update process
You can upgrade Red Hat OpenShift Container Storage and its components, either between minor releases like 4.6 and 4.7, or between batch updates like 4.7.0 and 4.7.1.
You need to upgrade the different parts of OpenShift Container Storage in a specific order.
- Update OpenShift Container Platform according to the Updating clusters documentation for OpenShift Container Platform.
Update OpenShift Container Storage.
- To prepare a disconnected environment for updates, see Operators guide to using Operator Lifecycle Manager on restricted networks to be able to update Openshift Container Storage as well as Local Storage Operator when in use.
Update the OpenShift Container Storage operator, using the appropriate process for your setup:
If you use local storage:
Update the Local Storage operator.
See Checking for Local Storage Operator deployments if you are unsure.
Perform post-update configuration changes for clusters backed by local storage.
See Post-update configuration for clusters backed by local storage for details.
Update considerations
Review the following important considerations before you begin.
Red Hat recommends using the same version of Red Hat OpenShift Container Platform with Red Hat OpenShift Container Storage.
See the Interoperability Matrix for more information about supported combinations of OpenShift Container Platform and OpenShift Container Storage.
- The Local Storage Operator is fully supported only when the Local Storage Operator version matches the Red Hat OpenShift Container Platform version.
Chapter 2. OpenShift Container Storage upgrade channels and releases
In OpenShift Container Platform 4.1, Red Hat introduced the concept of channels for recommending the appropriate release versions for cluster upgrades. By controlling the pace of upgrades, these upgrade channels allow you to choose an upgrade strategy. As OpenShift Container Storage gets deployed as an Operator in OpenShift Container Platform, it follows the same strategy to control the pace of upgrades by shipping the fixes in multiple channels. Upgrade channels are tied to a minor version of OpenShift Container Storage. For example, OpenShift Container Storage 4.7 upgrade channels recommend upgrades from 4.6 to 4.7 and upgrades within 4.7. They do not recommend upgrades to 4.8 or later releases. This strategy ensures that administrators can explicitly decide to upgrade to the next minor version of OpenShift Container Storage.
Upgrade channels control only release selection and do not impact the version of the cluster that you install; the ocs-operator
decides the version of OpenShift Container Storage to be installed. Out of the box, it always installs the latest OpenShift Container Storage release maintaining the compatibility with OpenShift Container Platform. So on OpenShift Container Platform 4.7, OpenShift Container Storage 4.7 will be the latest version which can be installed.
OpenShift Container Storage upgrades are tied to the OpenShift Container Platform upgrade to ensure that compatibility and interoperability are maintained with the OpenShift Container Platform. For OpenShift Container Storage 4.7, OpenShift Container Platform 4.7 and 4.8 are supported. OpenShift Container Platform 4.8 is supported to maintain forward compatibility with OpenShift Container Platform. Keep the OpenShift Container Storage version the same as OpenShift Container Platform in order to get the benefit of all the features and enhancements in that release.
OpenShift Container Platform 4.7 offers the following upgrade channel:
stable-4.7
- stable-4.7 channel
- Once a new version is Generally Available, the stable channel corresponding to the minor version gets updated with the new image which can be used to upgrade. You can use the stable-4.7 channel to upgrade from OpenShift Container storage 4.6 and upgrades within 4.7.
Chapter 3. Updating OpenShift Container Storage in internal mode
Use the following procedures to update your OpenShift Container Storage cluster deployed in internal mode.
3.1. Enabling automatic updates for OpenShift Container Storage operator in internal mode
Use this procedure to enable automatic update approval for updating OpenShift Container Storage operator in OpenShift Container Platform.
Prerequisites
- Under Persistent Storage in the Status card, confirm that the OCS Cluster and Data Resiliency has a green tick mark.
-
Under Object Service in the Status card, confirm that both the Object Service and Data Resiliency are in
Ready
state (green tick). - Update the OpenShift Container Platform cluster to the latest stable release of version 4.7.Y, see Updating Clusters.
Switch the Red Hat OpenShift Container Storage channel from
stable-4.6
tostable-4.7
. For details about channels, see OpenShift Container Storage upgrade channels and releases.NoteYou are required to switch channels only when you are updating minor versions (for example, updating from 4.6 to 4.7) and not when updating between batch updates of 4.7 (for example, updating from 4.7.0 to 4.7.1).
Ensure that all OpenShift Container Storage Pods, including the operator pods, are in
Running
state in theopenshift-storage namespace
.To view the state of the pods, click Workloads → Pods from the left pane of the OpenShift Web Console. Select openshift-storage from the Project drop down list.
- Ensure that you have sufficient time to complete the Openshift Container Storage update process, as the update time varies depending on the number of OSDs that run in the cluster.
Procedure
- Log in to OpenShift Web Console.
- Click Operators → Installed Operators
-
Select the
openshift-storage
project. - Click the OpenShift Container Storage operator name.
- Click the Subscription tab and click the link under Approval.
- Select Automatic (default) and click Save.
Perform one of the following depending on the Upgrade Status:
Upgrade Status shows requires approval.
NoteUpgrade status shows requires approval if the new OpenShift Container Storage version is already detected in the channel, and approval strategy was changed from Manual to Automatic at the time of update.
- Click on the Install Plan link.
- On the InstallPlan Details page, click Preview Install Plan.
- Review the install plan and click Approve.
- Wait for the Status to change from Unknown to Created.
- Click Operators → Installed Operators
-
Select the
openshift-storage
project. - Wait for the Status to change to Up to date
Upgrade Status does not show requires approval:
- Wait for the update to initiate. This may take up to 20 minutes.
- Click Operators → Installed Operators
-
Select the
openshift-storage
project. - Wait for the Status to change to Up to date
Multicloud Object Gateway outage is expected for a short period of time during upgrade due to migration of NooBaa DB from MongoDB to PostgreSQL.
Verification steps
- Click Overview → Persistent Storage tab and in the Status card confirm that the OCS Cluster and Data Resiliency has a green tick mark indicating it is healthy.
-
Click Overview → Object Service tab and in the Status card confirm that both the Object Service and Data Resiliency are in
Ready
state (green tick) indicating it is healthy. Click Operators → Installed Operators → OpenShift Container Storage Operator. Under Storage Cluster, verify that the cluster service status is
Ready
.NoteOnce updated from OpenShift Container Storage version 4.6 to 4.7, the
Version
field here will still display 4.6. This is because theocs-operator
does not update the string represented in this field.Ensure that all OpenShift Container Storage Pods, including the operator pods, are in
Running
state in theopenshift-storage namespace
.To view the state of the pods, click Workloads → Pods. Select openshift-storage from the Project drop down list.
- If verification steps fail, contact Red Hat Support.
The flexible scaling feature is available only in the new deployments of Red Hat OpenShift Container Storage 4.7. Storage clusters upgraded to the 4.7 version do not support flexible scaling.
Additional Resources
If you face any issues while updating OpenShift Container Storage, see the Commonly required logs for troubleshooting section in the Troubleshooting guide.
3.2. Manually updating OpenShift Container Storage operator in internal mode
Use this procedure to update OpenShift Container Storage operator by providing manual approval to the install plan.
Prerequisites
- Under Persistent Storage in the Status card, confirm that the OCS Cluster and Data Resiliency has a green tick mark.
-
Under Object Service in the Status card, confirm that both the Object Service and Data Resiliency are in
Ready
state (green tick). - Update the OpenShift Container Platform cluster to the latest stable release of version 4.7.Y, see Updating Clusters.
Switch the Red Hat OpenShift Container Storage channel from
stable-4.6
tostable-4.7
. For details about channels, see OpenShift Container Storage upgrade channels and releases.NoteYou are required to switch channels only when you are updating minor versions (for example, updating from 4.6 to 4.7) and not when updating between batch updates of 4.7 (for example, updating from 4.7.0 to 4.7.1).
Ensure that all OpenShift Container Storage Pods, including the operator pods, are in
Running
state in theopenshift-storage namespace
.To view the state of the pods, click Workloads → Pods from the left pane of the OpenShift Web Console. Select openshift-storage from the Project drop down list.
- Ensure that you have sufficient time to complete the Openshift Container Storage update process, as the update time varies depending on the number of OSDs that run in the cluster.
Procedure
- Log in to OpenShift Web Console.
- Click Operators → Installed Operators
-
Select the
openshift-storage
project. - Click the OpenShift Container Storage operator name.
- Click the Subscription tab and click the link under Approval.
- Select Manual and click Save.
- Wait for the Upgrade Status to change to Upgrading.
- If the Upgrade Status shows requires approval, click on requires approval.
- On the InstallPlan Details page, click Preview Install Plan.
- Review the install plan and click Approve.
- Wait for the Status to change from Unknown to Created.
- Click Operators → Installed Operators
-
Select the
openshift-storage
project. - Wait for the Status to change to Up to date
Multicloud Object Gateway outage is expected for a short period of time during upgrade due to migration of NooBaa DB from MongoDB to PostgreSQL.
Verification steps
- Click Overview → Persistent Storage tab and in the Status card confirm that the OCS Cluster and Data Resiliency has a green tick mark indicating it is healthy.
-
Click Overview → Object Service tab and in the Status card confirm that both the Object Service and Data Resiliency are in
Ready
state (green tick) indicating it is healthy. Click Operators → Installed Operators → OpenShift Container Storage Operator. Under Storage Cluster, verify that the cluster service status is
Ready
.NoteOnce updated from OpenShift Container Storage version 4.6 to 4.7, the
Version
field here will still display 4.6. This is because theocs-operator
does not update the string represented in this field.Ensure that all OpenShift Container Storage Pods, including the operator pods, are in
Running
state in theopenshift-storage namespace
.To view the state of the pods, click Workloads → Pods from the left pane of the OpenShift Web Console. Select openshift-storage from the Project drop down list.
- If verification steps fail, contact Red Hat Support.
Additional Resources
If you face any issues while updating OpenShift Container Storage, see the Commonly required logs for troubleshooting section in the Troubleshooting guide.
Chapter 4. Updating OpenShift Container Storage in external mode
Use the following procedures to update your OpenShift Container Storage cluster deployed in external mode.
Upgrading Red Hat OpenShift Container Storage Operator does not upgrade the external Red Hat Ceph Storage cluster. It only upgrades the Red Hat OpenShift Container Storage Services running on the OpenShift Container Platform.
To upgrade the external Red Hat Ceph Storage cluster contact your Red Hat Ceph Storage administrator.
4.1. Enabling automatic updates for OpenShift Container Storage operator in external mode
Use this procedure to enable automatic update approval for updating OpenShift Container Storage operator in OpenShift Container Platform.
Updating OpenShift Container Storage will not update the external Red Hat Ceph Storage cluster.
Prerequisites
- Red Hat Ceph Storage version 4.2z1 or later is required for the external cluster. For more information, see this knowledge base article on Red Hat Ceph Storage releases and corresponding Ceph package versions.
- Update the OpenShift Container Platform cluster to the latest stable release of version 4.7.y, see Updating Clusters.
Switch the Red Hat OpenShift Container Storage channel from
stable-4.6
tostable-4.7
. For details about channels, see OpenShift Container Storage upgrade channels and releases.NoteYou are required to switch channels only when you are updating minor versions (for example, updating from 4.6 to 4.7) and not when updating between batch updates of 4.7 (for example, updating from 4.7.0 to 4.7.1).
Ensure that all OpenShift Container Storage Pods, including the operator pods, are in
Running
state in theopenshift-storage namespace
.To view the state of the pods, click Workloads → Pods from the left pane of the OpenShift Web Console. Select openshift-storage from the Project drop down list.
- Under Persistent Storage in the Status card, confirm that the OCS Cluster has a green tick mark.
-
Under Object Service in the Status card, confirm that both the Object Service and Data Resiliency are in
Ready
state (green tick). - Ensure that you have sufficient time to complete the Openshift Container Storage update process.
Procedure
- Log in to OpenShift Web Console.
- Click Operators → Installed Operators
-
Select the
openshift-storage
project. - Click on the OpenShift Container Storage operator name.
- Click the Subscription tab and click the link under Approval.
- Select Automatic (default) and click Save.
Perform one of the following depending on the Upgrade Status:
Upgrade Status shows requires approval.
NoteUpgrade status shows requires approval if the new OpenShift Container Storage version is already detected in the channel, and approval strategy was changed from Manual to Automatic at the time of update.
- Click the Install Plan link.
- On the InstallPlan Details page, click Preview Install Plan.
- Review the install plan and click Approve.
- Wait for the Status to change from Unknown to Created.
- Click Operators → Installed Operators
-
Select the
openshift-storage
project. - Wait for the Status to change to Up to date
Upgrade Status does not show requires approval:
- Wait for the update to initiate. This may take up to 20 minutes.
- Click Operators → Installed Operators
-
Select the
openshift-storage
project. - Wait for the Status to change to Up to date
Multicloud Object Gateway outage is expected for a short period of time during upgrade due to migration of NooBaa DB from MongoDB to PostgreSQL.
Verification steps
- Click Overview → Persistent Storage tab and in the Status card confirm that the OCS Cluster has a green tick mark indicating it is healthy.
-
Click Overview → Object Service tab and in the Status card confirm that both the Object Service and Data Resiliency are in
Ready
state (green tick) indicating it is healthy. Click Operators → Installed Operators → OpenShift Container Storage Operator. Under Storage Cluster, verify that the cluster service status in
Ready
.NoteOnce updated from OpenShift Container Storage version 4.6 to 4.7, the
Version
field here will still display 4.6. This is because theocs-operator
does not update the string represented in this field.Ensure that all OpenShift Container Storage Pods, including the operator pods, are in
Running
state in theopenshift-storage
namespace.To view the state of the pods, click Workloads → Pods from the left pane of the OpenShift Web Console. Select openshift-storage from the Project drop down list.
- If verification steps fail, contact Red Hat Support.
Additional Resources
If you face any issues while updating OpenShift Container Storage, see the Commonly required logs for troubleshooting section in the Troubleshooting guide.
4.2. Manually updating OpenShift Container Storage operator in external mode
Use this procedure to update OpenShift Container Storage operator by providing manual approval to the install plan.
Updating OpenShift Container Storage will not update the external Red Hat Ceph Storage cluster.
Prerequisites
- Red Hat Ceph Storage version 4.2z1 or later is required for the external cluster. For more information, see this knowledge base article on Red Hat Ceph Storage releases and corresponding Ceph package versions.
- Update the OpenShift Container Platform cluster to the latest stable release of version 4.7.Y, see Updating Clusters.
Switch the Red Hat OpenShift Container Storage channel from
stable-4.6
tostable-4.7
. For details about channels, see OpenShift Container Storage upgrade channels and releases.NoteYou are required to switch channels only when you are updating minor versions (for example, updating from 4.6 to 4.7) and not when updating between batch updates of 4.7 (for example, updating from 4.7.0 to 4.7.1).
Ensure that all OpenShift Container Storage Pods, including the operator pods, are in
Running
state in theopenshift-storage
namespace.To view the state of the pods, click Workloads → Pods from the left pane of the OpenShift Web Console. Select openshift-storage from the Project drop down list.
- Under Persistent Storage in the Status card, confirm that the OCS Cluster has a green tick mark.
-
Under Object Service in the Status card, confirm that both the Object Service and Data Resiliency are in
Ready
state (green tick). - Ensure that you have sufficient time to complete the Openshift Container Storage update process.
Procedure
- Log in to OpenShift Web Console.
- Click Operators → Installed Operators
-
Select the
openshift-storage
project. - Click the OpenShift Container Storage operator name.
- Click Subscription tab and click the link under Approval.
- Select Manual and click Save.
- Wait for the Upgrade Status to change to Upgrading.
- If the Upgrade Status shows requires approval, click on requires approval.
- On the InstallPlan Details page, click Preview Install Plan.
- Review the install plan and click Approve.
- Wait for the Status to change from Unknown to Created.
- Click Operators → Installed Operators
-
Select the
openshift-storage
project. - Wait for the Status to change to Up to date
Multicloud Object Gateway outage is expected for a short period of time during upgrade due to migration of NooBaa DB from MongoDB to PostgreSQL.
Verification steps
- Click Overview → Persistent Storage tab and in the Status card confirm that the OCS Cluster has a green tick mark indicating it is healthy.
-
Click Overview → Object Service tab and in the Status card confirm that both the Object Service and Data Resiliency are in
Ready
state (green tick) indicating it is healthy. -
Click Operators → Installed Operators → OpenShift Container Storage Operator. Under Storage Cluster, verify that the cluster service status in
Ready
. Ensure that all OpenShift Container Storage Pods, including the operator pods, are in
Running
state in theopenshift-storage
namespace.To view the state of the pods, click Workloads → Pods from the left pane of the OpenShift Web Console. Select openshift-storage from the Project drop down list.
NoteOnce updated from OpenShift Container Storage version 4.6 to 4.7, the
Version
field here will still display 4.6. This is because theocs-operator
does not update the string represented in this field.- If verification steps fail, contact Red Hat Support.
Additional Resources
If you face any issues while updating OpenShift Container Storage, see the Commonly required logs for troubleshooting section in the Troubleshooting guide.
Chapter 5. Adding annotation to the pre-existing backingstores
Adding the correct annotation to the pre-existing backingstores allows the backingstores backed by object gateways (RGWs) to report its actual and free size. The Multicloud Object Gateway (MCG) can retrieve and use this information.
Procedure
- Log in to the OpenShift Container Platform Web Console.
- Click Home → Search.
- Search for BackingStore in Resources and click on it.
-
Beside the
S3-compatible BackingStore
, click Action Menu (⋮) → Edit annotations. -
Add
rgw
for KEY. - Click Save.
Next steps
For clusters backed by local storage, additional configuration steps are required after an update to ensure that all features work as expected.
For more information, see Post-update configuration changes for clusters backed by local storage.
Chapter 6. Post-update configuration changes for clusters backed by local storage
In clusters backed by local storage, additional configuration steps are required after an update to ensure that all features work as expected. In Red Hat OpenShift Container Platform 4.6 and onward, the Local Storage Operator provides new custom resource types for managing local storage:
-
LocalVolumeDiscovery
-
LocalVolumeSet
These resource types are not automatically handled as part of an update from earlier versions and must be created manually.
If you had already created these resources after upgrading from 4.5 to 4.6, then you do not need to create them after upgrading from 4.6 to 4.7.
6.1. Creating a LocalVolumeDiscovery custom resource using the command line
Create a LocalVolumeDiscovery
custom resource to ensure that the device management user interface can discover the state of local devices and provide information about devices that are available on cluster nodes.
Prerequisites
- Administrative access to the OpenShift Container Platform cluster.
Procedure
Identify the namespace where OpenShift local storage operator is installed and assign it to
local_storage_project
variable:$ local_storage_project=$(oc get csv --all-namespaces | awk '{print $1}' | grep local)
For example:
$ local_storage_project=$(oc get csv --all-namespaces | awk '{print $1}' | grep local) echo $local_storage_project openshift-local-storage
Change into the project that has the Local Storage Operator installed.
$ oc project $local_storage_project
Define the
LocalVolumeDiscovery
custom resource.For example, define the following in a
local-volume-discovery.yaml
file.apiVersion: local.storage.openshift.io/v1alpha1 kind: LocalVolumeDiscovery metadata: name: auto-discover-devices spec: nodeSelector: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - worker1.example.com - worker2.example.com - worker3.example.com
Create the
LocalVolumeDiscovery
custom resource.$ oc create -f local-volume-discovery.yaml
Verification steps
- Log in to the OpenShift web console.
- Click Compute → Node and click on the name of the node.
- Click the Disks tab and check that you can see the devices available on that node.
6.2. Creating a LocalVolumeSet custom resource using the command line
Create a LocalVolumeSet
custom resource to automatically provision certain storage devices as persistent volumes based on criteria that you specify. Persistent volumes are created for any devices that match the deviceInclusionSpec
criteria on any node that matches the nodeSelector
criteria.
Prerequisites
- Administrative access to the OpenShift Container Platform cluster.
Procedure
Define a
LocalVolumeSet
custom resource in alocal-volume-set.yaml
file.apiVersion: local.storage.openshift.io/v1alpha1 kind: LocalVolumeSet metadata: name: localblock spec: nodeSelector: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - worker1.example.com - worker2.example.com - worker3.example.com storageClassName: localblock volumeMode: Block maxDeviceCount: 10 # optional, limit devices provisioned per node deviceInclusionSpec: deviceTypes: # list of types to allow - disk - part # omit this to use only whole devices deviceMechanicalProperty: - NonRotational minSize: 100Gi # optional, minimum size of device to allow maxSize: 100Ti # optional, maximum size of device to allow models: # (optional) list of models to allow - SAMSUNG - Crucial_CT525MX3 vendors: # (optional) list of device vendors to allow - ATA - ST2000LM
The above definition selects whole disks or partitions on specific models of non-rotational devices that are between 100 GB and 100 TB in size, provided by specific vendors, from the
worker1
,worker2
andworker3
nodes. Thelocalblock
storage class is created and persistent volumes are provisioned from discovered devices.ImportantSelect an appropriate value for
minSize
to ensure system partitions are not selected.Create the
LocalVolumeSet
.$ oc create -f local-volume-set.yaml
Verification steps
Use the following command to track provisioning of persistent volumes for devices that match the
deviceInclusionSpec
. It can take a few minutes to provision persistent volumes.$ oc describe localvolumeset localblock [...] Status: Conditions: Last Transition Time: 2020-11-17T05:03:32Z Message: DiskMaker: Available, LocalProvisioner: Available Status: True Type: DaemonSetsAvailable Last Transition Time: 2020-11-17T05:03:34Z Message: Operator reconciled successfully. Status: True Type: Available Observed Generation: 1 Total Provisioned Device Count: 4 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Discovered 2m30s (x4 localvolumeset- ip-10-0-147-124.us-east- NewDevice over 2m30s) symlink-controller 2.compute.internal - found possible matching disk, waiting 1m to claim Normal FoundMatch 89s (x4 localvolumeset- ip-10-0-147-124.us-east- ingDisk over 89s) symlink-controller 2.compute.internal - symlinking matching disk
Verify the state of the provisioned persistent volumes.
$ oc get pv ACCESS RECLAIM STORAGE NAME CAPACITY MODES POLICY STATUS CLASS AGE local-pv- 500Gi RWO Delete Available localblock 7m48s 3584969f local-pv- 500Gi RWO Delete Available localblock 7m48s 3aee84fa local-pv- 500Gi RWO Delete Available localblock 7m48s 644d09ac local-pv- 500Gi RWO Delete Available localblock 7m48s c73cee1
6.3. Adding annotations
Use this procedure to add annotations to storage cluster to enable replacing of failed storage devices through the user interface when you upgraded to OpenShift Container Storage 4.7 from a previous version.
Procedure
- Log in to OpenShift Container Platform Web Console.
- Click Home → Search.
- Search for StorageCluster in Resources and click on it.
-
Beside
ocs-storagecluster
, click Action menu (⋮) → Edit annotations. -
Add
cluster.ocs.openshift.io/local-devices
andtrue
for KEY and VALUE respectively. - Click Save.