Chapter 15. Monitoring RHACS
You can monitor Red Hat Advanced Cluster Security for Kubernetes (RHACS) by using the built-in monitoring for Red Hat OpenShift or by using custom Prometheus monitoring.
If you use RHACS with Red Hat OpenShift, OpenShift Container Platform includes a preconfigured, preinstalled, and self-updating monitoring stack that provides monitoring for core platform components. RHACS exposes metrics to Red Hat OpenShift monitoring via an encrypted and authenticated endpoint.
15.1. Monitoring with Red Hat OpenShift Copy linkLink copied to clipboard!
Monitoring with Red Hat OpenShift is enabled by default. No configuration is required for this default behavior.
If you have previously configured monitoring with the Prometheus Operator, consider removing your custom ServiceMonitor resources. RHACS ships with a pre-configured ServiceMonitor for Red Hat OpenShift monitoring. Multiple ServiceMonitors might result in duplicated scraping.
Monitoring with Red Hat OpenShift is not supported by Scanner. If you want to monitor Scanner, you must first disable the default Red Hat OpenShift monitoring. Then, configure custom Prometheus monitoring.
For more information on disabling Red Hat OpenShift monitoring, see "Disabling Red Hat OpenShift monitoring for Central services by using the RHACS Operator" or "Disabling Red Hat OpenShift monitoring for Central services by using Helm". For more information on configuring Prometheus, see "Monitoring with custom Prometheus".
15.2. Monitoring with custom Prometheus Copy linkLink copied to clipboard!
Prometheus is an open-source monitoring and alerting platform. You can use it to monitor health and availability of Central and Sensor components of RHACS. When you enable monitoring, RHACS creates a new monitoring service on port number 9090 and a network policy allowing inbound connections to that port.
This monitoring service exposes an endpoint that is not encrypted by TLS and has no authorization. Use this only when you do not want to use Red Hat OpenShift monitoring.
Before you can use custom Prometheus monitoring, if you have Red Hat OpenShift, you must disable the default monitoring. If you are using Kubernetes, you do not need to perform this step.
15.2.1. Disabling Red Hat OpenShift monitoring for Central services by using the RHACS Operator Copy linkLink copied to clipboard!
To disable the default monitoring by using the Operator, change the configuration of the Central custom resource as shown in the following example. For more information on configuration options, see "Central configuration options using the Operator" in the "Additional resources" section.
Procedure
-
On the OpenShift Container Platform web console, go to the Ecosystem
Installed Operators page. - Select the RHACS Operator from the list of installed Operators.
- Click on the Central tab.
- From the list of Central instances, click on a Central instance for which you want to enable monitoring.
Click on the YAML tab and update the YAML configuration as shown in the following example:
monitoring: openshift: enabled: falsemonitoring: openshift: enabled: falseCopy to Clipboard Copied! Toggle word wrap Toggle overflow
15.2.2. Disabling Red Hat OpenShift monitoring for Central services by using Helm Copy linkLink copied to clipboard!
To disable the default monitoring by using Helm, change the configuration options in the central-services Helm chart. For more information on configuration options, see the documents in the "Additional resources" section.
Procedure
Update the configuration file with the following value:
monitoring.openshift.enabled: false
monitoring.openshift.enabled: falseCopy to Clipboard Copied! Toggle word wrap Toggle overflow -
Run the
helm upgradecommand and specify the configuration files.
15.2.3. Monitoring Central services by using the RHACS Operator Copy linkLink copied to clipboard!
You can monitor Central services, Central and Scanner, by changing the configuration of the Central custom resource. For more information on configuration options, see "Central configuration options using the Operator" in the "Additional resources" section.
Procedure
-
On the OpenShift Container Platform web console, go to the Ecosystem
Installed Operators page. - Select the Red Hat Advanced Cluster Security for Kubernetes Operator from the list of installed Operators.
- Click on the Central tab.
- From the list of Central instances, click on a Central instance for which you want to enable monitoring for.
Click on the YAML tab and update the YAML configuration:
-
For monitoring Central, enable the
central.monitoring.exposeEndpointconfiguration option for theCentralcustom resource. -
For monitoring Scanner, enable the
scanner.monitoring.exposeEndpointconfiguration option for theCentralcustom resource.
-
For monitoring Central, enable the
- Click Save.
15.3. Monitoring Central services by using Helm Copy linkLink copied to clipboard!
You can monitor Central services, Central and Scanner, by changing the configuration options in the central-services Helm chart. For more information, see "Changing configuration options after deploying the central-services Helm chart" in the "Additional resources" section.
Procedure
Update the
values-public.yamlconfiguration file with the following values:central.exposeMonitoring: true scanner.exposeMonitoring: true
central.exposeMonitoring: true scanner.exposeMonitoring: trueCopy to Clipboard Copied! Toggle word wrap Toggle overflow -
Run the
helm upgradecommand and specify the configuration files.
15.3.1. Monitoring Central by using Prometheus service monitor Copy linkLink copied to clipboard!
If you are using the Prometheus Operator, you can use a service monitor to scrape the metrics from Red Hat Advanced Cluster Security for Kubernetes (RHACS).
- If you are not using the Prometheus operator, you must edit the Prometheus configuration files to receive the data from RHACS.
-
If you use Kubernetes, enter
kubectlinstead ofoc.
Procedure
Create a new
servicemonitor.yamlfile with the following content:Copy to Clipboard Copied! Toggle word wrap Toggle overflow where:
app.kubernetes.io/name:<stackrox-service>-
Specifies the name of the service to monitor. The label must match with the
Serviceresource that you want to monitor. For example,centralorscanner.
Apply the YAML to the cluster:
oc apply -f servicemonitor.yaml
$ oc apply -f servicemonitor.yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow
Verification
Run the following command to check the status of service monitor:
oc get servicemonitor --namespace stackrox
$ oc get servicemonitor --namespace stackroxCopy to Clipboard Copied! Toggle word wrap Toggle overflow
15.4. Custom Prometheus metrics Copy linkLink copied to clipboard!
You can enable some Red Hat Advanced Cluster Security for Kubernetes (RHACS) product metrics and scrape them with a dedicated Prometheus server in order to display them in a visualization application such as Perses or Grafana.
You can use a tool such as Perses to display charts based on these metrics. For example, you can create a Perses dashboard to use in the OpenShift Container Platform console that displays RHACS data.
15.4.1. Custom metric categories Copy linkLink copied to clipboard!
You can enable specific Prometheus metrics that you want to expose on the /metrics path of the central API endpoint.
The following table lists the available metric categories, their configurability, and default status:
| Metric category | Configurable | Default status |
|---|---|---|
| Image vulnerabilities | Yes | Disabled by default |
| Node vulnerabilities | Yes | Disabled by default |
| Policy violations | Yes | Disabled by default |
| Total policy numbers | No | Enabled by default |
| Cluster health | No | Enabled by default |
| Certificate expiry | No | Enabled by default |
Each category is configured with a data gathering period and a list of metrics. A metric is associated with a set of labels.
The /metrics path is accessible for API tokens with Administration resource view permissions. The value of the labels are subject to scoped access control.
15.4.2. Configure custom Prometheus metrics Copy linkLink copied to clipboard!
You can configure custom Prometheus metrics for Red Hat Advanced Cluster Security for Kubernetes (RHACS) by using the API or RHACS portal.
You can view the Prometheus metrics that are exposed on the API endpoint at the /metrics path. Use the /v1/config API to configure metrics with custom names and labels. You can enable or disable one or more of the predefined metrics in the System Configuration page.
Collecting and exposing metrics is resource intensive in large environments. The output size depends on the selected labels. For example, selecting only the Cluster and Severity labels for an image vulnerability metric produces up to five times the number of clusters in records. Adding labels such as Namespace increases the output proportionally. Choose labels carefully to avoid unnecessary load.
Scrape requests require permissions to view the Administration resources and are subject for the scoped access control.
15.4.2.1. Configuring custom Prometheus metrics by using the API Copy linkLink copied to clipboard!
You can configure custom Prometheus metrics for Red Hat Advanced Cluster Security for Kubernetes (RHACS) by using the API. The metrics configuration is a part of the private config section of the /v1/config service request or response payload.
Procedure
To get the current configuration, run the following command:
curl "$ROX_API_ENDPOINT/v1/config" -H "Authorization: Bearer $ROX_API_TOKEN" | jq
$ curl "$ROX_API_ENDPOINT/v1/config" -H "Authorization: Bearer $ROX_API_TOKEN" | jqCopy to Clipboard Copied! Toggle word wrap Toggle overflow The
jqcommand formats and displays the JSON response from the API in a readable way.Example output
{ "publicConfig": { ... } "privateConfig": { ... } }{ "publicConfig": { ... } "privateConfig": { ... } }Copy to Clipboard Copied! Toggle word wrap Toggle overflow To add a policy violation metric named
component_severity, which includes only component and severity labels, to the configuration, run the following command:Copy to Clipboard Copied! Toggle word wrap Toggle overflow The
jqcommand adds a newcomponent_severitymetric descriptor with the labelsComponentandSeverityto thepolicyViolationsconfiguration, then wraps the result in aconfigobject for the update request.
15.4.2.2. Configuring custom Prometheus metrics by using the RHACS portal Copy linkLink copied to clipboard!
You can configure custom Prometheus metrics for Red Hat Advanced Cluster Security for Kubernetes (RHACS) by using the RHACS portal.
Procedure
-
In the RHACS portal, click Platform Configuration
System Configuration. - Click Edit.
Scroll down to the Prometheus metrics configuration section, and do any of the following tasks:
To enable metrics gathering:
Enter a positive integer for the gathering period in minutes. You can adjust the number by using the up and down arrows in the spin button.
ImportantUse a larger value to reduce how often Central gathers metrics, as the process is resource intensive.
- Select one or more predefined metrics from the list of metrics.
-
To disable metrics gathering, enter
0.
- Click Save.
Verification
To verify that the predefined metric that you enabled is being exposed by Central, run the following command:
curl "$ROX_API_ENDPOINT/metrics" \ -H "Authorization: Bearer $ROX_API_TOKEN" \ -s | grep <metric_name> | head
$ curl "$ROX_API_ENDPOINT/metrics" \ -H "Authorization: Bearer $ROX_API_TOKEN" \ -s | grep <metric_name> | headCopy to Clipboard Copied! Toggle word wrap Toggle overflow where:
<metric_name>Specifies the predefined metric that you enabled. For example,
rox_central_image_vuln_deployment_severity.The
grepcommand filters the output for the predefined metric.The
headcommand limits the output to the first few matching lines for readability.
The output shows the predefined metric aggregated by cluster, namespace, and severity, indicating the number of vulnerabilities for each combination.
15.4.3. Setting up a Prometheus server scraping from Central service Copy linkLink copied to clipboard!
Before connecting Perses or other monitoring tools, configure Prometheus to scrape metrics from the Central service. For example, configure the machine access integration for the Kubernetes service account by creating a service account, performing role mapping, and assigning the necessary permissions. This ensures that Prometheus can securely access Central API data for monitoring and visualization purposes.
Procedure
Perform role mapping to ensure that the Prometheus server can access the Central API with a service account token:
- Create and assign a role with the appropriate permission set and scope.
Add a role mapping to the machine access configuration by using the following service account information:
typeKUBE_SERVICE_ACCOUNTkeysubvaluesystem:serviceaccount:stackrox:prometheus-server
Verify that the Prometheus custom resource (CR) contains the following information:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Create the additional scrape configuration secret by using the following content, for example:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
15.4.4. Setting up a Perses dashboard in OpenShift Container Platform to monitor RHACS Copy linkLink copied to clipboard!
You can configure an in-cluster Prometheus server to work with Red Hat Advanced Cluster Security for Kubernetes (RHACS). You can then use a tool such as Perses running in OpenShift Container Platform or externally to monitor RHACS.
To view a sample Perses configuration, navigate to the monitoring-examples repository in the stackrox organization on GitHub, select the main branch, and then open the perses directory.
Prerequisites
You have configured Prometheus to scrape metrics from the Central service.
For more information, see "Setting a Prometheus server scraping from Central service".
Procedure
Create the Perses data source by using the following content, for example:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow where:
<PROMETHEUS_SERVICE_HOSTNAME>Specifies the hostname of the Prometheus service in your cluster.
-
If Prometheus is installed by using the monitoring stack, the hostname might be, for example,
prometheus-operated.stackrox.svc.cluster.local. - If Prometheus in installed by using the Prometheus Operator or another installation, the hostname might be something different.
-
If Prometheus is installed by using the monitoring stack, the hostname might be, for example,
<PROMETHEUS_SERVICE_PORT>-
Specifies the port of the Prometheus service in your cluster. For example,
9090.
Create a Perses dashboard by using the following content, for example:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow This YAML creates a Perses dashboard in the
stackroxnamespace that shows a status panel for the total policy violations over the last 30 days, with variables forClusterandNamespace, that refreshes every minute.
15.4.5. Metric categories for custom metrics Copy linkLink copied to clipboard!
You can configure the Prometheus custom metrics by specifying a set of labels for each metric category. When using the API, you need to define the metric category for your custom metric.
The following are the supported metric categories:
-
imageVulnerabilities -
nodeVulnerabilities -
policyViolations
The following example shows the API request body for configuring a custom metric:
where:
metrics.imageVulnerabilities.descriptors.<metric_name>- Specifies the name of a custom metric descriptor.
metrics.imageVulnerabilities.descriptors.<metric name>.labels- Specifies the one or more labels separated by a comma used to aggregate and filter the metric data.
Each metric descriptor defines the labels that should be collected for that metric within the chosen category.
The following table lists the labels for image vulnerabilities, node vulnerabilities, and policy violations:
| Metric category | Labels |
|---|---|
| Image vulnerabilities |
|
| Node vulnerabilities |
|
| Policy violations |
|
15.4.6. System metrics and labels Copy linkLink copied to clipboard!
You can view additional system metrics that are exposed by default. These metrics are gathered once every hour.
The following table lists the metric names and their corresponding labels:
| Metric category | Metric name | Labels |
|---|---|---|
| Total policy numbers |
|
|
| Cluster health |
|
|
| Certificate expiry |
|
|
The Certificate expiry metric requires permission to view the integrations.