This documentation is for a release that is no longer maintained.
You can select a different version or view all RHACS documentation.Chapter 1. Upgrading by using the Operator
Upgrades through the Red Hat Advanced Cluster Security for Kubernetes (RHACS) Operator are performed automatically or manually, depending on the Update approval option you chose at installation.
Follow these guidelines when upgrading:
- If the version for Central is earlier than 3.74, you must upgrade to 3.74 before upgrading to a 4.x version. For upgrading Central to version 3.74, see the upgrade documentation for version 3.74.
-
When upgrading Operator-based Central deployments from version 3.74, first ensure the Operator upgrade mode is set to
Manual. Then, upgrade the Operator to version 4.0 following the procedure in the upgrade documentation for version 4.0 and ensure that Central is online. After the upgrade to version 4.0 is complete, Red Hat recommends upgrading Central to the latest version for full functionality.
1.1. Preparing to upgrade Copy linkLink copied to clipboard!
Before you upgrade the Red Hat Advanced Cluster Security for Kubernetes (RHACS) version, complete the following steps:
- If you are upgrading from version 3.74, verify that you are running the latest patch release version of the RHACS Operator 3.74.
- Backup your existing Central database.
-
If the cluster you are upgrading contains the
SecuredClustercustom resource (CR), change the collection method toCORE_BPF. For more information, see "Changing the collection method".
1.1.1. Changing the collection method Copy linkLink copied to clipboard!
If the cluster that you are upgrading contains the SecuredCluster CR, you must ensure that the per node collection setting is set to CORE_BPF before you upgrade.
Procedure
- In the OpenShift Container Platform web console, go to the RHACS Operator page.
- In the top navigation menu, select Secured Cluster.
- Click the instance name, for example, stackrox-secured-cluster-services.
Use one of the following methods to change the setting:
-
In the Form view, under Per Node Settings
Collector Settings Collection, select CORE_BPF. -
Click YAML to open the YAML editor and locate the
spec.perNode.collector.collectionattribute. If the value isKernelModuleorEBPF, then change it toCORE_BPF.
-
In the Form view, under Per Node Settings
- Click Save.
1.2. Modifying the Central custom resource Copy linkLink copied to clipboard!
The Central DB service requires persistent storage. If you have not configured a default storage class for the Central cluster that is an SSD or is high performance, you must reinstall and configure the Central custom resource (CR) with the storage class for the Central DB persistent volume claim (PVC).
When changing the storageClassName, because of limitations with the Kubernetes storage layer, you cannot modify a claim’s storage class after it is bound; therefore, you must re-create the Central CR.
Skip this section if you have already configured a default storage class for Central.
Procedure
- Back up and uninstall the Central instance.
Reinstall RHACS and configure the Central custom resource as shown in the following example:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow -
Restore the Central database backup by using the
roxctlCLI.
1.3. Modifying Central custom resource for external database Copy linkLink copied to clipboard!
Prerequisites
You must have a database in your database instance that supports PostgreSQL 13 and a user with the following permissions:
- Connection rights to the database.
-
UsageandCreateon the schema. -
Select,Insert,Update, andDeleteon all tables in the schema. -
Usageon all sequences in the schema.
Procedure
Create a password secret in the deployed namespace by using the OpenShift Container Platform web console or the terminal.
-
On the OpenShift Container Platform web console, go to the Workloads
Secrets page. Create a Key/Value secret with the key passwordand the value as the path of a plain text file containing the password for the superuser of the provisioned database. Or, run the following command in your terminal:
oc create secret generic external-db-password \ --from-file=password=<password.txt>
$ oc create secret generic external-db-password \1 --from-file=password=<password.txt>2 Copy to Clipboard Copied! Toggle word wrap Toggle overflow
-
On the OpenShift Container Platform web console, go to the Workloads
- Go to the Red Hat Advanced Cluster Security for Kubernetes operator page in the OpenShift Container Platform web console. Select Central in the top navigation bar and select the instance you want to connect to the database.
- Go to the YAML editor view.
-
For
db.passwordSecret.namespecify the referenced secret that you created in earlier steps. For example,external-db-password. -
For
db.connectionStringspecify the connection string inkeyword=valueformat, for example,host=<host> port=5432 database=stackrox user=stackrox sslmode=verify-ca -
For
db.persistencedelete the entire block. If necessary, you can specify a Certificate Authority for Central to trust the database certificate by adding a TLS block under the top-level spec, as shown in the following example:
Update the central custom resource with the following configuration:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- You must not change the value of
IsEnabledtoEnabled.
- Click Save.
1.4. Changing the subscription channel Copy linkLink copied to clipboard!
You can change the update channel for the RHACS Operator by using the OpenShift Container Platform web console or by using the command line. For upgrading to RHACS 4.0 from RHACS 3.74, you must change the update channel.
You must change the subscription channel for all clusters where you installed the RHACS Operator, including Central and all secured clusters.
Prerequisites
- You must verify that you are using the latest RHACS 3.74 Operator and there are no pending manual Operator upgrades.
- You must verify that you backed up your Central database.
-
You have access to an OpenShift Container Platform cluster web console using an account with
cluster-adminpermissions.
1.4.1. Changing the subscription channel by using the web console Copy linkLink copied to clipboard!
Use the following instructions for changing the subscription channel by using the web console:
Procedure
-
In the Administrator perspective of the OpenShift Container Platform web console, go to Operators
Installed Operators. - Click the RHACS Operator.
- Click the Subscription tab.
- Click the name of the update channel under Update Channel.
- Select stable, then click Save.
For subscriptions with an Automatic approval strategy, the update begins automatically. Go back to the Operators
Installed Operators page to monitor the progress of the update. When complete, the status changes to Succeeded and Up to date. For subscriptions with a Manual approval strategy, you can manually approve the update from the Subscription tab.
1.4.2. Changing the subscription channel by using command line Copy linkLink copied to clipboard!
Use the following instructions for changing the subscription channel by using command line:
Procedure
Run the following command to change the subscription channel to
stable:oc -n rhacs-operator \ patch subscriptions.operators.coreos.com rhacs-operator \ --type=merge --patch='{ "spec": { "channel": "stable" }}'$ oc -n rhacs-operator \1 patch subscriptions.operators.coreos.com rhacs-operator \ --type=merge --patch='{ "spec": { "channel": "stable" }}'Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- If you use Kubernetes, enter
kubectlinstead ofoc.
During the update, the RHACS Operator provisions a new deployment called central-db and your data begins migrating. It takes around 30 minutes and happens only after you upgrade.
1.5. Rolling back an Operator upgrade Copy linkLink copied to clipboard!
To roll back an Operator upgrade, you must perform the steps described in one of the following sections. You can roll back an Operator upgrade by using the CLI or the OpenShift Container Platform web console.
If you are rolling back from RHACS 4.0, you can only roll back to the latest patch release version of RHACS 3.74.
1.5.1. Rolling back an Operator upgrade by using the CLI Copy linkLink copied to clipboard!
You can roll back the Operator version by using command-line interface (CLI) commands.
Procedure
Delete the Operator Lifecycle Manager (OLM) subscription and cluster service version (CSV):
ImportantIf you use Kubernetes, enter
kubectlinstead ofoc.To delete the OLM subscription, run the following command:
oc -n rhacs-operator delete subscription rhacs-operator
$ oc -n rhacs-operator delete subscription rhacs-operatorCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
subscription.operators.coreos.com "rhacs-operator" deleted
subscription.operators.coreos.com "rhacs-operator" deletedCopy to Clipboard Copied! Toggle word wrap Toggle overflow To delete the CSV, run the following command:
oc -n rhacs-operator delete csv -l operators.coreos.com/rhacs-operator.rhacs-operator
$ oc -n rhacs-operator delete csv -l operators.coreos.com/rhacs-operator.rhacs-operatorCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
clusterserviceversion.operators.coreos.com "rhacs-operator.v4.8.4" deleted
clusterserviceversion.operators.coreos.com "rhacs-operator.v4.8.4" deletedCopy to Clipboard Copied! Toggle word wrap Toggle overflow
Install the previous version of the RHACS Operator:
- In the OpenShift web console, click OperatorHub.
- Search for Advanced Cluster Security for Kubernetes.
Select and install the previous version of the Operator.
NoteInstalling the previous Operator version rolls back RHACS to the same version as the Operator.
1.5.2. Rolling back an Operator upgrade by using the web console Copy linkLink copied to clipboard!
You can roll back the Operator version by using the OpenShift Container Platform web console.
Prerequisites
-
You have access to an OpenShift Container Platform cluster web console using an account with
cluster-adminpermissions.
Procedure
-
In the OpenShift web console, click Operators
Installed Operators. - From the list of projects, select rhacs-operator.
Locate the Advanced Cluster Security for Kubernetes Operator:
Click the overflow menu
Uninstall Operator. The uninstall Operator dialog is displayed.
- Ensure that the Delete all operand instances for this operator checkbox is clear to avoid uninstallation of Red Hat Advanced Cluster Security for Kubernetes (RHACS).
- Click Uninstall.
Install the previous version of RHACS Operator:
- In the OpenShift web console, click OperatorHub.
- Search for Advanced Cluster Security for Kubernetes.
Select and install the previous version of the Operator.
NoteInstalling the previous Operator version rolls back RHACS to the same version as the Operator.
1.6. Troubleshooting Operator upgrade issues Copy linkLink copied to clipboard!
Follow these instructions to investigate and resolve upgrade-related issues for the RHACS Operator.
1.6.1. Central DB cannot be scheduled Copy linkLink copied to clipboard!
Follow the instructions here to troubleshoot a failing Central DB pod during an upgrade:
Check the status of the
central-dbpod:oc -n <namespace> get pod -l app=central-db
$ oc -n <namespace> get pod -l app=central-db1 Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- If you use Kubernetes, enter
kubectlinstead ofoc.
If the status of the pod is
Pending, use the describe command to get more details:oc -n <namespace> describe po/<central-db-pod-name>
$ oc -n <namespace> describe po/<central-db-pod-name>1 Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- If you use Kubernetes, enter
kubectlinstead ofoc.
You might see the
FailedSchedulingwarning message:Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 54s default-scheduler 0/7 nodes are available: 1 Insufficient memory, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }, 4 Insufficient cpu. preemption: 0/7 nodes are available: 3 Preemption is not helpful for scheduling, 4 No preemption victims found for incoming pod.Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 54s default-scheduler 0/7 nodes are available: 1 Insufficient memory, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }, 4 Insufficient cpu. preemption: 0/7 nodes are available: 3 Preemption is not helpful for scheduling, 4 No preemption victims found for incoming pod.Copy to Clipboard Copied! Toggle word wrap Toggle overflow This warning message suggests that the scheduled node had insufficient memory to accommodate the pod’s resource requirements. If you have a small environment, consider increasing resources on the nodes or adding a larger node that can support the database.
Otherwise, consider decreasing the resource requirements for the
central-dbpod in the custom resource undercentraldbresources. However, running central with fewer resources than the recommended minimum might lead to degraded performance for RHACS.
1.6.2. Central or Secured cluster fails to deploy Copy linkLink copied to clipboard!
When RHACS Operator has the following conditions, you must check the custom resource conditions to find the issue:
- If the Operator fails to deploy Central or Secured Cluster
- If the Operator fails to apply CR changes to actual resources
For Central, run the following command to check the conditions:
oc -n rhacs-operator describe centrals.platform.stackrox.io
$ oc -n rhacs-operator describe centrals.platform.stackrox.io1 Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- If you use Kubernetes, enter
kubectlinstead ofoc.
For Secured clusters, run the following command to check the conditions:
oc -n rhacs-operator describe securedclusters.platform.stackrox.io
$ oc -n rhacs-operator describe securedclusters.platform.stackrox.io1 Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- If you use Kubernetes, enter
kubectlinstead ofoc.
You can identify configuration errors from the conditions output:
Example output
Additionally, you can view RHACS pod logs to find more information about the issue. Run the following command to view the logs:
oc -n rhacs-operator logs deploy/rhacs-operator-controller-manager manager
oc -n rhacs-operator logs deploy/rhacs-operator-controller-manager manager
- 1
- If you use Kubernetes, enter
kubectlinstead ofoc.