Chapter 3. Manually upgrading using the roxctl CLI
You can upgrade to the latest version of Red Hat Advanced Cluster Security for Kubernetes (RHACS) from a supported older version.
-
You need to perform the manual upgrade procedure only if you used the
roxctl
CLI to install RHACS. - There are manual steps for each version upgrade that must be followed, for example, from version 3.74 to version 4.0, and from version 4.0 to version 4.1. Therefore, Red Hat recommends upgrading first from 3.74 to 4.0, then from 4.0 to 4.1, then 4.1 to 4.2, until the selected version is installed. For full functionality, Red Hat recommends upgrading to the most recent version.
To upgrade RHACS to the latest version, perform the following steps:
3.1. Backing up the Central database Copy linkLink copied to clipboard!
You can back up the Central database and use that backup for rolling back from a failed upgrade or data restoration in the case of an infrastructure disaster.
Prerequisites
-
You must have an API token with
read
permission for all resources of Red Hat Advanced Cluster Security for Kubernetes. The Analyst system role hasread
permissions for all resources. -
You have installed the
roxctl
CLI. -
You have configured the
ROX_API_TOKEN
and theROX_CENTRAL_ADDRESS
environment variables.
Procedure
Run the backup command:
roxctl -e "$ROX_CENTRAL_ADDRESS" central backup
$ roxctl -e "$ROX_CENTRAL_ADDRESS" central backup
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
3.2. Upgrading the roxctl CLI Copy linkLink copied to clipboard!
To upgrade the roxctl
CLI to the latest version you must uninstall the existing version of roxctl
CLI and then install the latest version of the roxctl
CLI.
3.2.1. Uninstalling the roxctl CLI Copy linkLink copied to clipboard!
You can uninstall the roxctl
CLI binary on Linux by using the following procedure.
Procedure
Find and delete the
roxctl
binary:ROXPATH=$(which roxctl) && rm -f $ROXPATH
$ ROXPATH=$(which roxctl) && rm -f $ROXPATH
1 Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- Depending on your environment, you might need administrator rights to delete the
roxctl
binary.
3.2.2. Installing the roxctl CLI on Linux Copy linkLink copied to clipboard!
You can install the roxctl
CLI binary on Linux by using the following procedure.
roxctl
CLI for Linux is available for amd64
, arm64
, ppc64le
, and s390x
architectures.
Procedure
Determine the
roxctl
architecture for the target operating system:arch="$(uname -m | sed "s/x86_64//")"; arch="${arch:+-$arch}"
$ arch="$(uname -m | sed "s/x86_64//")"; arch="${arch:+-$arch}"
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Download the
roxctl
CLI:curl -L -f -o roxctl "https://mirror.openshift.com/pub/rhacs/assets/4.6.9/bin/Linux/roxctl${arch}"
$ curl -L -f -o roxctl "https://mirror.openshift.com/pub/rhacs/assets/4.6.9/bin/Linux/roxctl${arch}"
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Make the
roxctl
binary executable:chmod +x roxctl
$ chmod +x roxctl
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Place the
roxctl
binary in a directory that is on yourPATH
:To check your
PATH
, execute the following command:echo $PATH
$ echo $PATH
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Verification
Verify the
roxctl
version you have installed:roxctl version
$ roxctl version
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
3.2.3. Installing the roxctl CLI on macOS Copy linkLink copied to clipboard!
You can install the roxctl
CLI binary on macOS by using the following procedure.
roxctl
CLI for macOS is available for amd64
and arm64
architectures.
Procedure
Determine the
roxctl
architecture for the target operating system:arch="$(uname -m | sed "s/x86_64//")"; arch="${arch:+-$arch}"
$ arch="$(uname -m | sed "s/x86_64//")"; arch="${arch:+-$arch}"
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Download the
roxctl
CLI:curl -L -f -o roxctl "https://mirror.openshift.com/pub/rhacs/assets/4.6.9/bin/Darwin/roxctl${arch}"
$ curl -L -f -o roxctl "https://mirror.openshift.com/pub/rhacs/assets/4.6.9/bin/Darwin/roxctl${arch}"
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Remove all extended attributes from the binary:
xattr -c roxctl
$ xattr -c roxctl
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Make the
roxctl
binary executable:chmod +x roxctl
$ chmod +x roxctl
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Place the
roxctl
binary in a directory that is on yourPATH
:To check your
PATH
, execute the following command:echo $PATH
$ echo $PATH
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Verification
Verify the
roxctl
version you have installed:roxctl version
$ roxctl version
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
3.2.4. Installing the roxctl CLI on Windows Copy linkLink copied to clipboard!
You can install the roxctl
CLI binary on Windows by using the following procedure.
roxctl
CLI for Windows is available for the amd64
architecture.
Procedure
Download the
roxctl
CLI:curl -f -O https://mirror.openshift.com/pub/rhacs/assets/4.6.9/bin/Windows/roxctl.exe
$ curl -f -O https://mirror.openshift.com/pub/rhacs/assets/4.6.9/bin/Windows/roxctl.exe
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Verification
Verify the
roxctl
version you have installed:roxctl version
$ roxctl version
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
3.3. Upgrading the Central cluster Copy linkLink copied to clipboard!
After you have created a backup of the Central database and generated the necessary resources by using the provisioning bundle, the next step is to upgrade the Central cluster. This process involves upgrading Central and Scanner.
3.3.1. Upgrading Central Copy linkLink copied to clipboard!
You can update Central to the latest version by downloading and deploying the updated images.
Procedure
Run the following command to update the Central image:
oc -n stackrox set image deploy/central central=registry.redhat.io/advanced-cluster-security/rhacs-main-rhel8:4.6.9
$ oc -n stackrox set image deploy/central central=registry.redhat.io/advanced-cluster-security/rhacs-main-rhel8:4.6.9
1 Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- If you use Kubernetes, enter
kubectl
instead ofoc
.
Run the following command to update the Config Controller image:
oc -n stackrox set image deploy/config-controller central=registry.redhat.io/advanced-cluster-security/rhacs-main-rhel8:4.6.9
$ oc -n stackrox set image deploy/config-controller central=registry.redhat.io/advanced-cluster-security/rhacs-main-rhel8:4.6.9
1 Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- If you use Kubernetes, enter
kubectl
instead ofoc
.
Verification
Verify that the new pods have deployed:
oc get deploy -n stackrox -o wide
$ oc get deploy -n stackrox -o wide
Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc get pod -n stackrox --watch
$ oc get pod -n stackrox --watch
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
3.3.1.1. Editing the GOMEMLIMIT environment variable for the Central deployment Copy linkLink copied to clipboard!
Upgrading to version 4.4 requires that you manually replace the GOMEMLIMIT
environment variable with the ROX_MEMLIMIT
environment variable. You must edit this variable for each deployment.
Procedure
Run the following command to edit the variable for the Central deployment:
oc -n stackrox edit deploy/central
$ oc -n stackrox edit deploy/central
1 Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- If you use Kubernetes, enter
kubectl
instead ofoc
.
-
Replace the
GOMEMLIMIT
variable withROX_MEMLIMIT
. - Save the file.
3.3.2. Upgrading Scanner Copy linkLink copied to clipboard!
You can update Scanner to the latest version by downloading and deploying the updated images.
If you are using Kubernetes, enter the kubectl
command instead of the oc
command.
Procedure
If you have created custom Scanner configurations, you must apply these changes before updating the Scanner configuration file:
To generate Scanner, run the following command:
roxctl -e "$ROX_CENTRAL_ADDRESS" scanner generate
$ roxctl -e "$ROX_CENTRAL_ADDRESS" scanner generate
Copy to Clipboard Copied! Toggle word wrap Toggle overflow To apply the TLS secrets YAML file, run the following command:
oc apply -f scanner-bundle/scanner/02-scanner-03-tls-secret.yaml
$ oc apply -f scanner-bundle/scanner/02-scanner-03-tls-secret.yaml
Copy to Clipboard Copied! Toggle word wrap Toggle overflow To apply the Scanner configuration YAML file, run the following command:
oc apply -f scanner-bundle/scanner/02-scanner-04-scanner-config.yaml
$ oc apply -f scanner-bundle/scanner/02-scanner-04-scanner-config.yaml
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
To update the Scanner image, run the following command:
oc -n stackrox set image deploy/scanner \ scanner=registry.redhat.io/advanced-cluster-security/rhacs-scanner-rhel8:4.6.9
$ oc -n stackrox set image deploy/scanner \ scanner=registry.redhat.io/advanced-cluster-security/rhacs-scanner-rhel8:4.6.9
Copy to Clipboard Copied! Toggle word wrap Toggle overflow To update the Scanner database image, run the following command:
oc -n stackrox set image deploy/scanner-db \ db=registry.redhat.io/advanced-cluster-security/rhacs-scanner-db-rhel8:4.6.9 \ init-db=registry.redhat.io/advanced-cluster-security/rhacs-scanner-db-rhel8:4.6.9
$ oc -n stackrox set image deploy/scanner-db \ db=registry.redhat.io/advanced-cluster-security/rhacs-scanner-db-rhel8:4.6.9 \ init-db=registry.redhat.io/advanced-cluster-security/rhacs-scanner-db-rhel8:4.6.9
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Verification
To verify that the new pods have been deployed, run the following commands:
oc get deploy -n stackrox -o wide
$ oc get deploy -n stackrox -o wide
Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc get pod -n stackrox --watch
$ oc get pod -n stackrox --watch
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
3.3.2.1. Editing the GOMEMLIMIT environment variable for the Scanner deployment Copy linkLink copied to clipboard!
Upgrading to version 4.4 requires that you manually replace the GOMEMLIMIT
environment variable with the ROX_MEMLIMIT
environment variable. You must edit this variable for each deployment.
Procedure
Run the following command to edit the variable for the Scanner deployment:
oc -n stackrox edit deploy/scanner
$ oc -n stackrox edit deploy/scanner
1 Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- If you use Kubernetes, enter
kubectl
instead ofoc
.
-
Replace the
GOMEMLIMIT
variable withROX_MEMLIMIT
. - Save the file.
3.3.3. Verifying the Central cluster upgrade Copy linkLink copied to clipboard!
After you have upgraded both Central and Scanner, verify that the Central cluster upgrade is complete.
Procedure
Check the Central logs by running the following command:
oc logs -n stackrox deploy/central -c central
$ oc logs -n stackrox deploy/central -c central
1 Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- If you use Kubernetes, enter
kubectl
instead ofoc
.
Sample output of a successful upgrade
3.4. Upgrading all secured clusters Copy linkLink copied to clipboard!
After upgrading Central services, you must upgrade all secured clusters.
If you are using automatic upgrades:
- Update all your secured clusters by using automatic upgrades.
- For information about troubleshooting problems with the automatic cluster upgrader, see Troubleshooting the cluster upgrader.
- Skip the instructions in this section and follow the instructions in the Verify upgrades and Revoking the API token sections.
If you are not using automatic upgrades, you must run the instructions in this section on all secured clusters including the Central cluster.
- To ensure optimal functionality, use the same RHACS version for your secured clusters and the cluster on which Central is installed.
To complete manual upgrades of each secured cluster running Sensor, Collector, and Admission controller, follow the instructions in this section.
3.4.1. Updating other images Copy linkLink copied to clipboard!
You must update the sensor, collector and compliance images on each secured cluster when not using automatic upgrades.
If you are using Kubernetes, use kubectl
instead of oc
for the commands listed in this procedure.
Procedure
Update the Sensor image:
oc -n stackrox set image deploy/sensor sensor=registry.redhat.io/advanced-cluster-security/rhacs-main-rhel8:4.6.9
$ oc -n stackrox set image deploy/sensor sensor=registry.redhat.io/advanced-cluster-security/rhacs-main-rhel8:4.6.9
1 Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- If you use Kubernetes, enter
kubectl
instead ofoc
.
Update the Compliance image:
oc -n stackrox set image ds/collector compliance=registry.redhat.io/advanced-cluster-security/rhacs-main-rhel8:4.6.9
$ oc -n stackrox set image ds/collector compliance=registry.redhat.io/advanced-cluster-security/rhacs-main-rhel8:4.6.9
1 Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- If you use Kubernetes, enter
kubectl
instead ofoc
.
Update the Collector image:
oc -n stackrox set image ds/collector collector=registry.redhat.io/advanced-cluster-security/rhacs-collector-rhel8:4.6.9
$ oc -n stackrox set image ds/collector collector=registry.redhat.io/advanced-cluster-security/rhacs-collector-rhel8:4.6.9
1 Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- If you use Kubernetes, enter
kubectl
instead ofoc
.
NoteIf you are using the collector slim image, run the following command instead:
oc -n stackrox set image ds/collector collector=registry.redhat.io/advanced-cluster-security/rhacs-collector-slim-rhel8:{rhacs-version}
$ oc -n stackrox set image ds/collector collector=registry.redhat.io/advanced-cluster-security/rhacs-collector-slim-rhel8:{rhacs-version}
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Update the admission control image:
oc -n stackrox set image deploy/admission-control admission-control=registry.redhat.io/advanced-cluster-security/rhacs-main-rhel8:4.6.9
$ oc -n stackrox set image deploy/admission-control admission-control=registry.redhat.io/advanced-cluster-security/rhacs-main-rhel8:4.6.9
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
If you have installed RHACS on Red Hat OpenShift by using the roxctl
CLI, you need to migrate the security context constraints (SCCs).
For more information, see "Migrating SCCs during the manual upgrade" in the "Additional resources" section.
3.4.2. Adding POD_NAMESPACE to sensor and admission-control deployments Copy linkLink copied to clipboard!
When upgrading to version 4.6 or later from a version earlier than 4.6, you must patch the sensor and admission-control deployments to set the POD_NAMESPACE
environment variable.
If you are using Kubernetes, use kubectl
instead of oc
for the commands listed in this procedure.
Procedure
Patch sensor to ensure
POD_NAMESPACE
is set by running the following command:[[ -z "$(oc -n stackrox get deployment sensor -o yaml | grep POD_NAMESPACE)" ]] && oc -n stackrox patch deployment sensor --type=json -p '[{"op":"add","path":"/spec/template/spec/containers/0/env/-","value":{"name":"POD_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}}}]'
$ [[ -z "$(oc -n stackrox get deployment sensor -o yaml | grep POD_NAMESPACE)" ]] && oc -n stackrox patch deployment sensor --type=json -p '[{"op":"add","path":"/spec/template/spec/containers/0/env/-","value":{"name":"POD_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}}}]'
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Patch admission-control to ensure
POD_NAMESPACE
is set by running the following command:[[ -z "$(oc -n stackrox get deployment admission-control -o yaml | grep POD_NAMESPACE)" ]] && oc -n stackrox patch deployment admission-control --type=json -p '[{"op":"add","path":"/spec/template/spec/containers/0/env/-","value":{"name":"POD_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}}}]'
$ [[ -z "$(oc -n stackrox get deployment admission-control -o yaml | grep POD_NAMESPACE)" ]] && oc -n stackrox patch deployment admission-control --type=json -p '[{"op":"add","path":"/spec/template/spec/containers/0/env/-","value":{"name":"POD_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}}}]'
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Next steps
3.4.3. Migrating SCCs during the manual upgrade Copy linkLink copied to clipboard!
By migrating the security context constraints (SCCs) during the manual upgrade by using roxctl
CLI, you can seamlessly transition the Red Hat Advanced Cluster Security for Kubernetes (RHACS) services to use the Red Hat OpenShift SCCs, ensuring compatibility and optimal security configurations across Central and all secured clusters.
Procedure
List all of the RHACS services that are deployed on Central and all secured clusters:
oc -n stackrox describe pods | grep 'openshift.io/scc\|^Name:'
$ oc -n stackrox describe pods | grep 'openshift.io/scc\|^Name:'
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
Copy to Clipboard Copied! Toggle word wrap Toggle overflow In this example, you can see that each pod has its own custom SCC, which is specified through the
openshift.io/scc
field.Add the required roles and role bindings to use the Red Hat OpenShift SCCs instead of the RHACS custom SCCs.
To add the required roles and role bindings to use the Red Hat OpenShift SCCs for the Central cluster, complete the following steps:
Create a file named
update-central.yaml
that defines the role and role binding resources by using the following content:Example 3.1. Example YAML file
Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- The type of Kubernetes resource, in this example,
Role
. - 2
- The name of the role resource.
- 3
- The namespace in which the role is created.
- 4
- Describes the permissions granted by the role resource.
- 5
- The type of Kubernetes resource, in this example,
RoleBinding
. - 6
- The name of the role binding resource.
- 7
- Specifies the role to bind in the same namespace.
- 8
- Specifies the subjects that are bound to the role.
Create the role and role binding resources specified in the
update-central.yaml
file by running the following command:oc -n stackrox create -f ./update-central.yaml
$ oc -n stackrox create -f ./update-central.yaml
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
To add the required roles and role bindings to use the Red Hat OpenShift SCCs for all secured clusters, complete the following steps:
Create a file named
upgrade-scs.yaml
that defines the role and role binding resources by using the following content:Example 3.2. Example YAML file
Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- The type of Kubernetes resource, in this example,
Role
. - 2
- The name of the role resource.
- 3
- The namespace in which the role is created.
- 4
- Describes the permissions granted by the role resource.
- 5
- The type of Kubernetes resource, in this example,
RoleBinding
. - 6
- The name of the role binding resource.
- 7
- Specifies the role to bind in the same namespace.
- 8
- Specifies the subjects that are bound to the role.
Create the role and role binding resources specified in the
upgrade-scs.yaml
file by running the following command:oc -n stackrox create -f ./update-scs.yaml
$ oc -n stackrox create -f ./update-scs.yaml
Copy to Clipboard Copied! Toggle word wrap Toggle overflow ImportantYou must run this command on each secured cluster to create the role and role bindings specified in the
upgrade-scs.yaml
file.
Delete the SCCs that are specific to RHACS:
To delete the SCCs that are specific to the Central cluster, run the following command:
oc delete scc/stackrox-central scc/stackrox-central-db scc/stackrox-scanner
$ oc delete scc/stackrox-central scc/stackrox-central-db scc/stackrox-scanner
Copy to Clipboard Copied! Toggle word wrap Toggle overflow To delete the SCCs that are specific to all secured clusters, run the following command:
oc delete scc/stackrox-admission-control scc/stackrox-collector scc/stackrox-sensor
$ oc delete scc/stackrox-admission-control scc/stackrox-collector scc/stackrox-sensor
Copy to Clipboard Copied! Toggle word wrap Toggle overflow ImportantYou must run this command on each secured cluster to delete the SCCs that are specific to each secured cluster.
Verification
Ensure that all the pods are using the correct SCCs by running the following command:
oc -n stackrox describe pods | grep 'openshift.io/scc\|^Name:'
$ oc -n stackrox describe pods | grep 'openshift.io/scc\|^Name:'
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Compare the output with the following table:
Expand Component Previous custom SCC New Red Hat OpenShift 4 SCC Central
stackrox-central
nonroot-v2
Central-db
stackrox-central-db
nonroot-v2
Scanner
stackrox-scanner
nonroot-v2
Scanner-db
stackrox-scanner
nonroot-v2
Admission Controller
stackrox-admission-control
restricted-v2
Collector
stackrox-collector
privileged
Sensor
stackrox-sensor
restricted-v2
3.4.3.1. Editing the GOMEMLIMIT environment variable for the Sensor deployment Copy linkLink copied to clipboard!
Upgrading to version 4.4 requires that you manually replace the GOMEMLIMIT
environment variable with the ROX_MEMLIMIT
environment variable. You must edit this variable for each deployment.
Procedure
Run the following command to edit the variable for the Sensor deployment:
oc -n stackrox edit deploy/sensor
$ oc -n stackrox edit deploy/sensor
1 Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- If you use Kubernetes, enter
kubectl
instead ofoc
.
-
Replace the
GOMEMLIMIT
variable withROX_MEMLIMIT
. - Save the file.
3.4.3.2. Editing the GOMEMLIMIT environment variable for the Collector deployment Copy linkLink copied to clipboard!
Upgrading to version 4.4 requires that you manually replace the GOMEMLIMIT
environment variable with the ROX_MEMLIMIT
environment variable. You must edit this variable for each deployment.
Procedure
Run the following command to edit the variable for the Collector deployment:
oc -n stackrox edit deploy/collector
$ oc -n stackrox edit deploy/collector
1 Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- If you use Kubernetes, enter
kubectl
instead ofoc
.
-
Replace the
GOMEMLIMIT
variable withROX_MEMLIMIT
. - Save the file.
3.4.3.3. Editing the GOMEMLIMIT environment variable for the Admission Controller deployment Copy linkLink copied to clipboard!
Upgrading to version 4.4 requires that you manually replace the GOMEMLIMIT
environment variable with the ROX_MEMLIMIT
environment variable. You must edit this variable for each deployment.
Procedure
Run the following command to edit the variable for the Admission Controller deployment:
oc -n stackrox edit deploy/admission-control
$ oc -n stackrox edit deploy/admission-control
1 Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- If you use Kubernetes, enter
kubectl
instead ofoc
.
-
Replace the
GOMEMLIMIT
variable withROX_MEMLIMIT
. - Save the file.
3.4.3.4. Verifying secured cluster upgrade Copy linkLink copied to clipboard!
After you have upgraded secured clusters, verify that the updated pods are working.
Procedure
Check that the new pods have deployed:
oc get deploy,ds -n stackrox -o wide
$ oc get deploy,ds -n stackrox -o wide
1 Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- If you use Kubernetes, enter
kubectl
instead ofoc
.
oc get pod -n stackrox --watch
$ oc get pod -n stackrox --watch
1 Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- If you use Kubernetes, enter
kubectl
instead ofoc
.
3.5. Enabling RHCOS node scanning with the StackRox Scanner Copy linkLink copied to clipboard!
If you use OpenShift Container Platform, you can enable scanning of Red Hat Enterprise Linux CoreOS (RHCOS) nodes for vulnerabilities by using Red Hat Advanced Cluster Security for Kubernetes (RHACS).
Prerequisites
- For scanning RHCOS node hosts of the secured cluster, you must have installed Secured Cluster services on OpenShift Container Platform 4.12 or later. For information about supported platforms and architecture, see the Red Hat Advanced Cluster Security for Kubernetes Support Matrix. For life cycle support information for RHACS, see the Red Hat Advanced Cluster Security for Kubernetes Support Policy.
- This procedure describes how to enable node scanning for the first time. If you are reconfiguring Red Hat Advanced Cluster Security for Kubernetes to use the StackRox Scanner instead of Scanner V4, follow the procedure in "Restoring RHCOS node scanning with the StackRox Scanner".
Procedure
Run one of the following commands to update the compliance container.
For a default compliance container with metrics disabled, run the following command:
oc -n stackrox patch daemonset/collector -p '{"spec":{"template":{"spec":{"containers":[{"name":"compliance","env":[{"name":"ROX_METRICS_PORT","value":"disabled"},{"name":"ROX_NODE_SCANNING_ENDPOINT","value":"127.0.0.1:8444"},{"name":"ROX_NODE_SCANNING_INTERVAL","value":"4h"},{"name":"ROX_NODE_SCANNING_INTERVAL_DEVIATION","value":"24m"},{"name":"ROX_NODE_SCANNING_MAX_INITIAL_WAIT","value":"5m"},{"name":"ROX_RHCOS_NODE_SCANNING","value":"true"},{"name":"ROX_CALL_NODE_INVENTORY_ENABLED","value":"true"}]}]}}}}'
$ oc -n stackrox patch daemonset/collector -p '{"spec":{"template":{"spec":{"containers":[{"name":"compliance","env":[{"name":"ROX_METRICS_PORT","value":"disabled"},{"name":"ROX_NODE_SCANNING_ENDPOINT","value":"127.0.0.1:8444"},{"name":"ROX_NODE_SCANNING_INTERVAL","value":"4h"},{"name":"ROX_NODE_SCANNING_INTERVAL_DEVIATION","value":"24m"},{"name":"ROX_NODE_SCANNING_MAX_INITIAL_WAIT","value":"5m"},{"name":"ROX_RHCOS_NODE_SCANNING","value":"true"},{"name":"ROX_CALL_NODE_INVENTORY_ENABLED","value":"true"}]}]}}}}'
Copy to Clipboard Copied! Toggle word wrap Toggle overflow For a compliance container with Prometheus metrics enabled, run the following command:
oc -n stackrox patch daemonset/collector -p '{"spec":{"template":{"spec":{"containers":[{"name":"compliance","env":[{"name":"ROX_METRICS_PORT","value":":9091"},{"name":"ROX_NODE_SCANNING_ENDPOINT","value":"127.0.0.1:8444"},{"name":"ROX_NODE_SCANNING_INTERVAL","value":"4h"},{"name":"ROX_NODE_SCANNING_INTERVAL_DEVIATION","value":"24m"},{"name":"ROX_NODE_SCANNING_MAX_INITIAL_WAIT","value":"5m"},{"name":"ROX_RHCOS_NODE_SCANNING","value":"true"},{"name":"ROX_CALL_NODE_INVENTORY_ENABLED","value":"true"}]}]}}}}'
$ oc -n stackrox patch daemonset/collector -p '{"spec":{"template":{"spec":{"containers":[{"name":"compliance","env":[{"name":"ROX_METRICS_PORT","value":":9091"},{"name":"ROX_NODE_SCANNING_ENDPOINT","value":"127.0.0.1:8444"},{"name":"ROX_NODE_SCANNING_INTERVAL","value":"4h"},{"name":"ROX_NODE_SCANNING_INTERVAL_DEVIATION","value":"24m"},{"name":"ROX_NODE_SCANNING_MAX_INITIAL_WAIT","value":"5m"},{"name":"ROX_RHCOS_NODE_SCANNING","value":"true"},{"name":"ROX_CALL_NODE_INVENTORY_ENABLED","value":"true"}]}]}}}}'
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Update the Collector DaemonSet (DS) by taking the following steps:
Add new volume mounts to Collector DS by running the following command:
oc -n stackrox patch daemonset/collector -p '{"spec":{"template":{"spec":{"volumes":[{"name":"tmp-volume","emptyDir":{}},{"name":"cache-volume","emptyDir":{"sizeLimit":"200Mi"}}]}}}}'
$ oc -n stackrox patch daemonset/collector -p '{"spec":{"template":{"spec":{"volumes":[{"name":"tmp-volume","emptyDir":{}},{"name":"cache-volume","emptyDir":{"sizeLimit":"200Mi"}}]}}}}'
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Add the new
NodeScanner
container by running the following command:oc -n stackrox patch daemonset/collector -p '{"spec":{"template":{"spec":{"containers":[{"command":["/scanner","--nodeinventory","--config=",""],"env":[{"name":"ROX_NODE_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"spec.nodeName"}}},{"name":"ROX_CLAIR_V4_SCANNING","value":"true"},{"name":"ROX_COMPLIANCE_OPERATOR_INTEGRATION","value":"true"},{"name":"ROX_CSV_EXPORT","value":"false"},{"name":"ROX_DECLARATIVE_CONFIGURATION","value":"false"},{"name":"ROX_INTEGRATIONS_AS_CONFIG","value":"false"},{"name":"ROX_NETPOL_FIELDS","value":"true"},{"name":"ROX_NETWORK_DETECTION_BASELINE_SIMULATION","value":"true"},{"name":"ROX_NETWORK_GRAPH_PATTERNFLY","value":"true"},{"name":"ROX_NODE_SCANNING_CACHE_TIME","value":"3h36m"},{"name":"ROX_NODE_SCANNING_INITIAL_BACKOFF","value":"30s"},{"name":"ROX_NODE_SCANNING_MAX_BACKOFF","value":"5m"},{"name":"ROX_PROCESSES_LISTENING_ON_PORT","value":"false"},{"name":"ROX_QUAY_ROBOT_ACCOUNTS","value":"true"},{"name":"ROX_ROXCTL_NETPOL_GENERATE","value":"true"},{"name":"ROX_SOURCED_AUTOGENERATED_INTEGRATIONS","value":"false"},{"name":"ROX_SYSLOG_EXTRA_FIELDS","value":"true"},{"name":"ROX_SYSTEM_HEALTH_PF","value":"false"},{"name":"ROX_VULN_MGMT_WORKLOAD_CVES","value":"false"}],"image":"registry.redhat.io/advanced-cluster-security/rhacs-scanner-slim-rhel8:4.6.9","imagePullPolicy":"IfNotPresent","name":"node-inventory","ports":[{"containerPort":8444,"name":"grpc","protocol":"TCP"}],"volumeMounts":[{"mountPath":"/host","name":"host-root-ro","readOnly":true},{"mountPath":"/tmp/","name":"tmp-volume"},{"mountPath":"/cache","name":"cache-volume"}]}]}}}}'
$ oc -n stackrox patch daemonset/collector -p '{"spec":{"template":{"spec":{"containers":[{"command":["/scanner","--nodeinventory","--config=",""],"env":[{"name":"ROX_NODE_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"spec.nodeName"}}},{"name":"ROX_CLAIR_V4_SCANNING","value":"true"},{"name":"ROX_COMPLIANCE_OPERATOR_INTEGRATION","value":"true"},{"name":"ROX_CSV_EXPORT","value":"false"},{"name":"ROX_DECLARATIVE_CONFIGURATION","value":"false"},{"name":"ROX_INTEGRATIONS_AS_CONFIG","value":"false"},{"name":"ROX_NETPOL_FIELDS","value":"true"},{"name":"ROX_NETWORK_DETECTION_BASELINE_SIMULATION","value":"true"},{"name":"ROX_NETWORK_GRAPH_PATTERNFLY","value":"true"},{"name":"ROX_NODE_SCANNING_CACHE_TIME","value":"3h36m"},{"name":"ROX_NODE_SCANNING_INITIAL_BACKOFF","value":"30s"},{"name":"ROX_NODE_SCANNING_MAX_BACKOFF","value":"5m"},{"name":"ROX_PROCESSES_LISTENING_ON_PORT","value":"false"},{"name":"ROX_QUAY_ROBOT_ACCOUNTS","value":"true"},{"name":"ROX_ROXCTL_NETPOL_GENERATE","value":"true"},{"name":"ROX_SOURCED_AUTOGENERATED_INTEGRATIONS","value":"false"},{"name":"ROX_SYSLOG_EXTRA_FIELDS","value":"true"},{"name":"ROX_SYSTEM_HEALTH_PF","value":"false"},{"name":"ROX_VULN_MGMT_WORKLOAD_CVES","value":"false"}],"image":"registry.redhat.io/advanced-cluster-security/rhacs-scanner-slim-rhel8:4.6.9","imagePullPolicy":"IfNotPresent","name":"node-inventory","ports":[{"containerPort":8444,"name":"grpc","protocol":"TCP"}],"volumeMounts":[{"mountPath":"/host","name":"host-root-ro","readOnly":true},{"mountPath":"/tmp/","name":"tmp-volume"},{"mountPath":"/cache","name":"cache-volume"}]}]}}}}'
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
3.6. Rolling back Central Copy linkLink copied to clipboard!
You can roll back to a previous version of Central if the upgrade to a new version is unsuccessful.
3.6.1. Rolling back Central normally Copy linkLink copied to clipboard!
You can roll back to a previous version of Central if upgrading Red Hat Advanced Cluster Security for Kubernetes fails.
Prerequisites
- Before you can perform a rollback, you must have free disk space available on your persistent storage. Red Hat Advanced Cluster Security for Kubernetes uses disk space to keep a copy of databases during the upgrade. If the disk space is not enough to store a copy and the upgrade fails, you might not be able to roll back to an earlier version.
Procedure
Run the following command to roll back to a previous version when an upgrade fails (before the Central service starts):
oc -n stackrox rollout undo deploy/central
$ oc -n stackrox rollout undo deploy/central
1 Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- If you use Kubernetes, enter
kubectl
instead ofoc
.
3.6.2. Rolling back Central forcefully Copy linkLink copied to clipboard!
You can use forced rollback to roll back to an earlier version of Central (after the Central service starts).
Using forced rollback to switch back to a previous version might result in loss of data and functionality.
Prerequisites
- Before you can perform a rollback, you must have free disk space available on your persistent storage. Red Hat Advanced Cluster Security for Kubernetes uses disk space to keep a copy of databases during the upgrade. If the disk space is not enough to store a copy and the upgrade fails, you will not be able to roll back to an earlier version.
Procedure
Run the following commands to perform a forced rollback:
To forcefully rollback to the previously installed version:
oc -n stackrox rollout undo deploy/central
$ oc -n stackrox rollout undo deploy/central
1 Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- If you use Kubernetes, enter
kubectl
instead ofoc
.
To forcefully rollback to a specific version:
Edit Central’s
ConfigMap
:oc -n stackrox edit configmap/central-config
$ oc -n stackrox edit configmap/central-config
1 Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- If you use Kubernetes, enter
kubectl
instead ofoc
.
Update the value of the
maintenance.forceRollbackVersion
key:Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- Specify the version that you want to roll back to.
Update the Central image version:
oc -n stackrox \ set image deploy/central central=registry.redhat.io/advanced-cluster-security/rhacs-main-rhel8:<x.x.x.x>
$ oc -n stackrox \
1 set image deploy/central central=registry.redhat.io/advanced-cluster-security/rhacs-main-rhel8:<x.x.x.x>
2 Copy to Clipboard Copied! Toggle word wrap Toggle overflow
3.7. Verifying upgrades Copy linkLink copied to clipboard!
The updated Sensors and Collectors continue to report the latest data from each secured cluster.
The last time Sensor contacted Central is visible in the RHACS portal.
Procedure
-
In the RHACS portal, go to Platform Configuration
System Health. - Check to ensure that Sensor Upgrade shows clusters up to date with Central.
3.8. Revoking the API token Copy linkLink copied to clipboard!
For security reasons, Red Hat recommends that you revoke the API token that you have used to complete Central database backup.
Prerequisites
- After the upgrade, you must reload the RHACS portal page and re-accept the certificate to continue using the RHACS portal.
Procedure
-
In the RHACS portal, go to Platform Configuration
Integrations. - Scroll down to the Authentication Tokens category, and click API Token.
- Select the checkbox in front of the token name that you want to revoke.
- Click Revoke.
- On the confirmation dialog box, click Confirm.
3.9. Troubleshooting the cluster upgrader Copy linkLink copied to clipboard!
If you encounter problems when using the legacy installation method for the secured cluster and enabling the automated updates, you can try troubleshooting the problem. The following errors can be found in the clusters view when the upgrader fails.
3.9.1. Upgrader is missing permissions Copy linkLink copied to clipboard!
Symptom
The following error is displayed in the cluster page:
Upgrader failed to execute PreflightStage of the roll-forward workflow: executing stage "Run preflight checks": preflight check "Kubernetes authorization" reported errors. This usually means that access is denied. Have you configured this Secured Cluster for automatically receiving upgrades?"
Upgrader failed to execute PreflightStage of the roll-forward workflow: executing stage "Run preflight checks": preflight check "Kubernetes authorization" reported errors. This usually means that access is denied. Have you configured this Secured Cluster for automatically receiving upgrades?"
Procedure
- Ensure that the bundle for the secured cluster was generated with future upgrades enabled before clicking Download YAML file and keys.
- If possible, remove that secured cluster and generate a new bundle making sure that future upgrades are enabled.
If you cannot re-create the cluster, you can take these actions:
-
Ensure that the service account
sensor-upgrader
exists in the same namespace as Sensor. -
Ensure that a ClusterRoleBinding exists (default name:
<namespace>:upgrade-sensors
) that grants thecluster-admin
ClusterRole to thesensor-upgrader
service account.
-
Ensure that the service account
3.9.2. Upgrader cannot start due to missing image Copy linkLink copied to clipboard!
Symptom
The following error is displayed in the cluster page:
"Upgrade initialization error: The upgrader pods have trouble pulling the new image: Error pulling image: (...) (<image_reference:tag>: not found)"
"Upgrade initialization error: The upgrader pods have trouble pulling the new image: Error pulling image: (...) (<image_reference:tag>: not found)"
Procedure
-
Ensure that the Secured Cluster can access the registry and pull the image
<image_reference:tag>
. - Ensure that the image pull secrets are configured correctly in the secured cluster.
3.9.3. Upgrader cannot start due to an unknown reason Copy linkLink copied to clipboard!
Symptom
The following error is displayed in the cluster page:
"Upgrade initialization error: Pod terminated: (Error)"
"Upgrade initialization error: Pod terminated: (Error)"
Procedure
- Ensure that the upgrader has enough permissions for accessing the cluster objects. For more information, see "Upgrader is missing permissions".
- Check the upgrader logs for more insights.
3.9.3.1. Obtaining upgrader logs Copy linkLink copied to clipboard!
The logs can be accessed by running the following command:
kubectl -n <namespace> logs deploy/sensor-upgrader
$ kubectl -n <namespace> logs deploy/sensor-upgrader
- 1
- For
<namespace>
, specify the namespace in which Sensor is running.
Usually, the upgrader deployment is only running in the cluster for a short time while doing the upgrades. It is removed later, so accessing its logs using the orchestrator CLI can require proper timing.