Upgrading

Procedure

1. Delete the OLM subscription by running the following command:

   • For OpenShift Container Platform, run the following command:

     $ oc -n rhacs-operator delete subscription rhacs-operator

     Copy to Clipboard Toggle word wrap

   • For Kubernetes, run the following command:

     $ kubectl -n rhacs-operator delete subscription rhacs-operator

     Copy to Clipboard Toggle word wrap

2. Delete the cluster service version (CSV) by running the following command:

   • For OpenShift Container Platform, run the following command:

     $ oc -n rhacs-operator delete csv -l operators.coreos.com/rhacs-operator.rhacs-operator

     Copy to Clipboard Toggle word wrap

   • For Kubernetes, run the following command:

     $ kubectl -n rhacs-operator delete csv -l operators.coreos.com/rhacs-operator.rhacs-operator

     Copy to Clipboard Toggle word wrap

3. Determine the previous version you want to roll back to by choosing one of the following options:

   • If the current Central instance is running, query the RHACS API to get the rollback version by running the following command:

     $ curl -k -s -u <user>:<password> https://<central hostname>/v1/centralhealth/upgradestatus | jq -r .upgradeStatus.forceRollbackTo

     Copy to Clipboard Toggle word wrap

   • If the current Central instance is not running, perform the following steps:

     Note

     This procedure can only be used for RHACS release 3.74 and earlier when the rocksdb database is installed.

     1. Ensure the Central deployment is scaled down by running the following command:

        • For OpenShift Container Platform, run the following command:

          $ oc scale -n <central namespace> –replicas=0 deploy/central

          Copy to Clipboard Toggle word wrap

        • For Kubernetes, run the following command:

          $ kubectl scale -n <central namespace> –replicas=0 deploy/central

          Copy to Clipboard Toggle word wrap

     2. Save the following pod spec as a YAML file:

        apiVersion: v1
        kind: Pod
        metadata:
          name: get-previous-db-version
        spec:
          containers:
          - name: get-previous-db-version
            image: registry.redhat.io/advanced-cluster-security/rhacs-main-rhel8:<rollback version>
            command:
            - sh
            args:
            - '-c'
            - "cat /var/lib/stackrox/.previous/migration_version.yaml | grep '^image:' | cut -f 2 -d : | tr -d ' '"
            volumeMounts:
            - name: stackrox-db
              mountPath: /var/lib/stackrox
          volumes:
          - name: stackrox-db
            persistentVolumeClaim:
              claimName: stackrox-db

        Copy to Clipboard Toggle word wrap

     3. Create a pod in your Central namespace by running the following command using the YAML file that you saved:

        • For OpenShift Container Platform, run the following command:

          $ oc create -n <central namespace> -f pod.yaml

          Copy to Clipboard Toggle word wrap

        • For Kubernetes, run the following command:

          $ kubectl create -n <central namespace> -f pod.yaml

          Copy to Clipboard Toggle word wrap

     4. After pod creation is complete, get the version by running the following command:

        • For OpenShift Container Platform, run the following command:

          $ oc logs -n <central namespace> get-previous-db-version

          Copy to Clipboard Toggle word wrap

        • For Kubernetes, run the following command:

          $ kubectl logs -n <central namespace> get-previous-db-version

          Copy to Clipboard Toggle word wrap

4. Edit the central-config.yaml ConfigMap to set the maintenance.forceRollBackVersion:<version> parameter by running the following command:

   • For OpenShift Container Platform, run the following command:

     $ oc get configmap -n <central namespace> central-config -o yaml | sed -e "s/forceRollbackVersion: none/forceRollbackVersion: <version>/" | oc -n <central namespace> apply -f -

     Copy to Clipboard Toggle word wrap

   • For Kubernetes, run the following command:

     $ kubectl get configmap -n <central namespace> central-config -o yaml | sed -e "s/forceRollbackVersion: none/forceRollbackVersion: <version>/" | kubectl -n <central namespace> apply -f -

     Copy to Clipboard Toggle word wrap

5. Set the image for the Central deployment using the version string shown in Step 3 as the image tag. For example, run the following command:

   • For OpenShift Container Platform, run the following command:

     $ oc set image -n <central namespace> deploy/central central=registry.redhat.io/advanced-cluster-security/rhacs-main-rhel8:<version>

     Copy to Clipboard Toggle word wrap

   • For Kubernetes, run the following command:

     $ kubectl set image -n <central namespace> deploy/central central=registry.redhat.io/advanced-cluster-security/rhacs-main-rhel8:<version>

     Copy to Clipboard Toggle word wrap

Verification

1. Ensure that the Central pod starts and has a ready status. If the pod crashes, check the logs to see if the backup was restored. A successful log message appears similar to the following example:

   Clone to Migrate ".previous", ""

   Copy to Clipboard Toggle word wrap
2. Reinstall the Operator on the rolled back channel. For example, 3.74.2 is installed on the rhacs-3.74 channel.

Procedure

1. Navigate to the Operators → Installed Operators page.
2. Locate the RHACS Operator and click on it.
3. On the Operator Details page, select Uninstall Operator from the Actions list. Following this action, the Operator stops running and no longer receives updates.

4. Determine the previous version you want to roll back to by choosing one of the following options:

   • If the current Central instance is running, you can query the RHACS API to get the rollback version by running the following command from a terminal window:

     $ curl -k -s -u <user>:<password> https://<central hostname>/v1/centralhealth/upgradestatus | jq -r .upgradeStatus.forceRollbackTo

     Copy to Clipboard Toggle word wrap

   • You can create a pod and extract the previous version by performing the following steps:

     Note

     This procedure can only be used for RHACS release 3.74 and earlier when the rocksdb database is installed.

     1. Navigate to Workloads → Deployments → central.
     2. Under Deployment details, click the down arrow next to the pod count to scale down the pod.

     3. Navigate to Workloads → Pods → Create Pod and paste the contents of the pod spec as shown in the following example into the editor:

        apiVersion: v1
        kind: Pod
        metadata:
          name: get-previous-db-version
        spec:
          containers:
          - name: get-previous-db-version
            image: registry.redhat.io/advanced-cluster-security/rhacs-main-rhel8:<rollback version>
            command:
            - sh
            args:
            - '-c'
            - "cat /var/lib/stackrox/.previous/migration_version.yaml | grep '^image:' | cut -f 2 -d : | tr -d ' '"
            volumeMounts:
            - name: stackrox-db
              mountPath: /var/lib/stackrox
          volumes:
          - name: stackrox-db
            persistentVolumeClaim:
              claimName: stackrox-db

        Copy to Clipboard Toggle word wrap
     4. Click Create.
     5. After the pod is created, click the Logs tab to get the version string.

5. Update the rollback configuration by performing the following steps:

   1. Navigate to Workloads → ConfigMaps → central-config and select Edit ConfigMap from the Actions list.
   2. Find the forceRollbackVersion line in the value of the central-config.yaml key.
   3. Replace none with 3.73.3, and then save the file.

6. Update Central to the earlier version by performing the following steps:

   1. Navigate to Workloads → Deployments → central and select Edit Deployment from the Actions list.
   2. Update the image name, and then save the changes.

Verification

1. Ensure that the Central pod starts and has a ready status. If the pod crashes, check the logs to see if the backup was restored. A successful log message appears similar to the following example:

   Clone to Migrate ".previous", ""

   Copy to Clipboard Toggle word wrap
2. Reinstall the Operator on the rolled back channel. For example, 3.74.2 is installed on the rhacs-3.74 channel.

1. Check the status of the central-db pod:

   $ oc -n <namespace> get pod -l app=central-db 

   1

   Copy to Clipboard Toggle word wrap

   1

   If you use Kubernetes, enter kubectl instead of oc.

2. If the status of the pod is Pending, use the describe command to get more details:

   $ oc -n <namespace> describe po/<central-db-pod-name> 

   1

   Copy to Clipboard Toggle word wrap

   1

   If you use Kubernetes, enter kubectl instead of oc.

3. You might see the FailedScheduling warning message:

   Type     Reason            Age   From               Message
   ----     ------            ----  ----               -------
   Warning  FailedScheduling  54s   default-scheduler  0/7 nodes are available: 1 Insufficient memory, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }, 4 Insufficient cpu. preemption: 0/7 nodes are available: 3 Preemption is not helpful for scheduling, 4 No preemption victims found for incoming pod.

   Copy to Clipboard Toggle word wrap

4. This warning message suggests that the scheduled node had insufficient memory to accommodate the pod’s resource requirements. If you have a small environment, consider increasing resources on the nodes or adding a larger node that can support the database.

   Otherwise, consider decreasing the resource requirements for the central-db pod in the custom resource under central → db → resources. However, running central with fewer resources than the recommended minimum might lead to degraded performance for RHACS.

Procedure

1. Download the latest version of the roxctl CLI:

   $ curl -O https://mirror.openshift.com/pub/rhacs/assets/4.2.5/bin/Linux/roxctl

   Copy to Clipboard Toggle word wrap

2. Make the roxctl binary executable:

   $ chmod +x roxctl

   Copy to Clipboard Toggle word wrap

3. Place the roxctl binary in a directory that is on your PATH:

   To check your PATH, execute the following command:

   $ echo $PATH

   Copy to Clipboard Toggle word wrap

Procedure

1. Download the latest version of the roxctl CLI:

   $ curl -O https://mirror.openshift.com/pub/rhacs/assets/4.2.5/bin/Darwin/roxctl

   Copy to Clipboard Toggle word wrap

2. Remove all extended attributes from the binary:

   $ xattr -c roxctl

   Copy to Clipboard Toggle word wrap

3. Make the roxctl binary executable:

   $ chmod +x roxctl

   Copy to Clipboard Toggle word wrap

4. Place the roxctl binary in a directory that is on your PATH:

   To check your PATH, execute the following command:

   $ echo $PATH

   Copy to Clipboard Toggle word wrap

Procedure

1. Update the Sensor image:

   $ oc -n stackrox set image deploy/sensor sensor=registry.redhat.io/advanced-cluster-security/rhacs-main-rhel8:4.2.5 

   1

   Copy to Clipboard Toggle word wrap

   1

   If you use Kubernetes, enter kubectl instead of oc.

2. Update the Compliance image:

   $ oc -n stackrox set image ds/collector compliance=registry.redhat.io/advanced-cluster-security/rhacs-main-rhel8:4.2.5 

   1

   Copy to Clipboard Toggle word wrap

   1

   If you use Kubernetes, enter kubectl instead of oc.

3. Update the Collector image:

   $ oc -n stackrox set image ds/collector collector=registry.redhat.io/advanced-cluster-security/rhacs-collector-rhel8:4.2.5 

   1

   Copy to Clipboard Toggle word wrap

   1

   If you use Kubernetes, enter kubectl instead of oc.

   Note

   If you are using the collector slim image, run the following command instead:

   $ oc -n stackrox set image ds/collector collector=registry.redhat.io/advanced-cluster-security/rhacs-collector-slim-rhel8:{rhacs-version}

   Copy to Clipboard Toggle word wrap

4. Update the admission control image:

   $ oc -n stackrox set image deploy/admission-control admission-control=registry.redhat.io/advanced-cluster-security/rhacs-main-rhel8:4.2.5

   Copy to Clipboard Toggle word wrap

Procedure

1. Check that the new pods have deployed:

   $ oc get deploy,ds -n stackrox -o wide 

   1

   Copy to Clipboard Toggle word wrap

   1

   If you use Kubernetes, enter kubectl instead of oc.

2. Enter the following command:

   $ oc get pod -n stackrox --watch 

   1

   Copy to Clipboard Toggle word wrap

   1

   If you use Kubernetes, enter kubectl instead of oc.

Procedure

1. Run one of the following commands to update the compliance container.

   • For a default compliance container with metrics disabled, run the following command:

     $ oc -n stackrox patch daemonset/collector -p '{"spec":{"template":{"spec":{"containers":[{"name":"compliance","env":[{"name":"ROX_METRICS_PORT","value":"disabled"},{"name":"ROX_NODE_SCANNING_ENDPOINT","value":"127.0.0.1:8444"},{"name":"ROX_NODE_SCANNING_INTERVAL","value":"4h"},{"name":"ROX_NODE_SCANNING_INTERVAL_DEVIATION","value":"24m"},{"name":"ROX_NODE_SCANNING_MAX_INITIAL_WAIT","value":"5m"},{"name":"ROX_RHCOS_NODE_SCANNING","value":"true"},{"name":"ROX_CALL_NODE_INVENTORY_ENABLED","value":"true"}]}]}}}}'

     Copy to Clipboard Toggle word wrap

   • For a compliance container with Prometheus metrics enabled, run the following command:

     $ oc -n stackrox patch daemonset/collector -p '{"spec":{"template":{"spec":{"containers":[{"name":"compliance","env":[{"name":"ROX_METRICS_PORT","value":":9091"},{"name":"ROX_NODE_SCANNING_ENDPOINT","value":"127.0.0.1:8444"},{"name":"ROX_NODE_SCANNING_INTERVAL","value":"4h"},{"name":"ROX_NODE_SCANNING_INTERVAL_DEVIATION","value":"24m"},{"name":"ROX_NODE_SCANNING_MAX_INITIAL_WAIT","value":"5m"},{"name":"ROX_RHCOS_NODE_SCANNING","value":"true"},{"name":"ROX_CALL_NODE_INVENTORY_ENABLED","value":"true"}]}]}}}}'

     Copy to Clipboard Toggle word wrap

2. Update the Collector DaemonSet (DS) by taking the following steps:

   1. Add new volume mounts to Collector DS by running the following command:

      $ oc -n stackrox patch daemonset/collector -p '{"spec":{"template":{"spec":{"volumes":[{"name":"tmp-volume","emptyDir":{}},{"name":"cache-volume","emptyDir":{"sizeLimit":"200Mi"}}]}}}}'

      Copy to Clipboard Toggle word wrap

   2. Add the new NodeScanner container by running the following command:

      $ oc -n stackrox patch daemonset/collector -p '{"spec":{"template":{"spec":{"containers":[{"command":["/scanner","--nodeinventory","--config=",""],"env":[{"name":"ROX_NODE_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"spec.nodeName"}}},{"name":"ROX_CLAIR_V4_SCANNING","value":"true"},{"name":"ROX_COMPLIANCE_OPERATOR_INTEGRATION","value":"true"},{"name":"ROX_CSV_EXPORT","value":"false"},{"name":"ROX_DECLARATIVE_CONFIGURATION","value":"false"},{"name":"ROX_INTEGRATIONS_AS_CONFIG","value":"false"},{"name":"ROX_NETPOL_FIELDS","value":"true"},{"name":"ROX_NETWORK_DETECTION_BASELINE_SIMULATION","value":"true"},{"name":"ROX_NETWORK_GRAPH_PATTERNFLY","value":"true"},{"name":"ROX_NODE_SCANNING_CACHE_TIME","value":"3h36m"},{"name":"ROX_NODE_SCANNING_INITIAL_BACKOFF","value":"30s"},{"name":"ROX_NODE_SCANNING_MAX_BACKOFF","value":"5m"},{"name":"ROX_PROCESSES_LISTENING_ON_PORT","value":"false"},{"name":"ROX_QUAY_ROBOT_ACCOUNTS","value":"true"},{"name":"ROX_ROXCTL_NETPOL_GENERATE","value":"true"},{"name":"ROX_SOURCED_AUTOGENERATED_INTEGRATIONS","value":"false"},{"name":"ROX_SYSLOG_EXTRA_FIELDS","value":"true"},{"name":"ROX_SYSTEM_HEALTH_PF","value":"false"},{"name":"ROX_VULN_MGMT_WORKLOAD_CVES","value":"false"}],"image":"registry.redhat.io/advanced-cluster-security/rhacs-scanner-slim-rhel8:4.2.5","imagePullPolicy":"IfNotPresent","name":"node-inventory","ports":[{"containerPort":8444,"name":"grpc","protocol":"TCP"}],"volumeMounts":[{"mountPath":"/host","name":"host-root-ro","readOnly":true},{"mountPath":"/tmp/","name":"tmp-volume"},{"mountPath":"/cache","name":"cache-volume"}]}]}}}}'

      Copy to Clipboard Toggle word wrap