Chapter 15. Monitoring RHACS


You can monitor Red Hat Advanced Cluster Security for Kubernetes (RHACS) by using the built-in monitoring for Red Hat OpenShift or by using custom Prometheus monitoring.

If you use RHACS with Red Hat OpenShift, OpenShift Container Platform includes a preconfigured, preinstalled, and self-updating monitoring stack that provides monitoring for core platform components. RHACS exposes metrics to Red Hat OpenShift monitoring via an encrypted and authenticated endpoint.

15.1. Monitoring with Red Hat OpenShift

Monitoring with Red Hat OpenShift is enabled by default. No configuration is required for this default behavior.

Important

If you have previously configured monitoring with the Prometheus Operator, consider removing your custom ServiceMonitor resources. RHACS ships with a pre-configured ServiceMonitor for Red Hat OpenShift monitoring. Multiple ServiceMonitors might result in duplicated scraping.

Monitoring with Red Hat OpenShift is not supported by Scanner. If you want to monitor Scanner, you must first disable the default Red Hat OpenShift monitoring. Then, configure custom Prometheus monitoring.

For more information on disabling Red Hat OpenShift monitoring, see "Disabling Red Hat OpenShift monitoring for Central services by using the RHACS Operator" or "Disabling Red Hat OpenShift monitoring for Central services by using Helm". For more information on configuring Prometheus, see "Monitoring with custom Prometheus".

15.2. Monitoring with custom Prometheus

Prometheus is an open-source monitoring and alerting platform. You can use it to monitor health and availability of Central and Sensor components of RHACS. When you enable monitoring, RHACS creates a new monitoring service on port number 9090 and a network policy allowing inbound connections to that port.

Note

This monitoring service exposes an endpoint that is not encrypted by TLS and has no authorization. Use this only when you do not want to use Red Hat OpenShift monitoring.

Before you can use custom Prometheus monitoring, if you have Red Hat OpenShift, you must disable the default monitoring. If you are using Kubernetes, you do not need to perform this step.

To disable the default monitoring by using the Operator, change the configuration of the Central custom resource as shown in the following example. For more information on configuration options, see "Central configuration options using the Operator" in the "Additional resources" section.

Procedure

  1. On the OpenShift Container Platform web console, go to the Ecosystem Installed Operators page.
  2. Select the RHACS Operator from the list of installed Operators.
  3. Click on the Central tab.
  4. From the list of Central instances, click on a Central instance for which you want to enable monitoring.
  5. Click on the YAML tab and update the YAML configuration as shown in the following example:

    monitoring:
        openshift:
            enabled: false
    Copy to Clipboard Toggle word wrap

To disable the default monitoring by using Helm, change the configuration options in the central-services Helm chart. For more information on configuration options, see the documents in the "Additional resources" section.

Procedure

  1. Update the configuration file with the following value:

    monitoring.openshift.enabled: false
    Copy to Clipboard Toggle word wrap
  2. Run the helm upgrade command and specify the configuration files.

You can monitor Central services, Central and Scanner, by changing the configuration of the Central custom resource. For more information on configuration options, see "Central configuration options using the Operator" in the "Additional resources" section.

Procedure

  1. On the OpenShift Container Platform web console, go to the Ecosystem Installed Operators page.
  2. Select the Red Hat Advanced Cluster Security for Kubernetes Operator from the list of installed Operators.
  3. Click on the Central tab.
  4. From the list of Central instances, click on a Central instance for which you want to enable monitoring for.
  5. Click on the YAML tab and update the YAML configuration:

    • For monitoring Central, enable the central.monitoring.exposeEndpoint configuration option for the Central custom resource.
    • For monitoring Scanner, enable the scanner.monitoring.exposeEndpoint configuration option for the Central custom resource.
  6. Click Save.

15.3. Monitoring Central services by using Helm

You can monitor Central services, Central and Scanner, by changing the configuration options in the central-services Helm chart. For more information, see "Changing configuration options after deploying the central-services Helm chart" in the "Additional resources" section.

Procedure

  1. Update the values-public.yaml configuration file with the following values:

    central.exposeMonitoring: true
    scanner.exposeMonitoring: true
    Copy to Clipboard Toggle word wrap
  2. Run the helm upgrade command and specify the configuration files.

If you are using the Prometheus Operator, you can use a service monitor to scrape the metrics from Red Hat Advanced Cluster Security for Kubernetes (RHACS).

Important
  • If you are not using the Prometheus operator, you must edit the Prometheus configuration files to receive the data from RHACS.
  • If you use Kubernetes, enter kubectl instead of oc.

Procedure

  1. Create a new servicemonitor.yaml file with the following content:

    apiVersion: monitoring.coreos.com/v1
    kind: ServiceMonitor
    metadata:
      name: prometheus-stackrox
      namespace: stackrox
    spec:
      endpoints:
        - interval: 30s
          port: monitoring
          scheme: http
      selector:
        matchLabels:
          app.kubernetes.io/name: <stackrox-service>
    Copy to Clipboard Toggle word wrap

    where:

    app.kubernetes.io/name:<stackrox-service>
    Specifies the name of the service to monitor. The label must match with the Service resource that you want to monitor. For example, central or scanner.
  2. Apply the YAML to the cluster:

    $ oc apply -f servicemonitor.yaml
    Copy to Clipboard Toggle word wrap

Verification

  • Run the following command to check the status of service monitor:

    $ oc get servicemonitor --namespace stackrox
    Copy to Clipboard Toggle word wrap

15.4. Custom Prometheus metrics

You can enable some Red Hat Advanced Cluster Security for Kubernetes (RHACS) product metrics and scrape them with a dedicated Prometheus server in order to display them in a visualization application such as Perses or Grafana.

You can use a tool such as Perses to display charts based on these metrics. For example, you can create a Perses dashboard to use in the OpenShift Container Platform console that displays RHACS data.

15.4.1. Custom metric categories

You can enable specific Prometheus metrics that you want to expose on the /metrics path of the central API endpoint.

The following table lists the available metric categories, their configurability, and default status:

Expand
Table 15.1. Metric category and configuration status
Metric categoryConfigurableDefault status

Image vulnerabilities

Yes

Disabled by default

Node vulnerabilities

Yes

Disabled by default

Policy violations

Yes

Disabled by default

Total policy numbers

No

Enabled by default

Cluster health

No

Enabled by default

Certificate expiry

No

Enabled by default

Each category is configured with a data gathering period and a list of metrics. A metric is associated with a set of labels.

Note

The /metrics path is accessible for API tokens with Administration resource view permissions. The value of the labels are subject to scoped access control.

15.4.2. Configure custom Prometheus metrics

You can configure custom Prometheus metrics for Red Hat Advanced Cluster Security for Kubernetes (RHACS) by using the API or RHACS portal.

You can view the Prometheus metrics that are exposed on the API endpoint at the /metrics path. Use the /v1/config API to configure metrics with custom names and labels. You can enable or disable one or more of the predefined metrics in the System Configuration page.

Collecting and exposing metrics is resource intensive in large environments. The output size depends on the selected labels. For example, selecting only the Cluster and Severity labels for an image vulnerability metric produces up to five times the number of clusters in records. Adding labels such as Namespace increases the output proportionally. Choose labels carefully to avoid unnecessary load.

Note

Scrape requests require permissions to view the Administration resources and are subject for the scoped access control.

You can configure custom Prometheus metrics for Red Hat Advanced Cluster Security for Kubernetes (RHACS) by using the API. The metrics configuration is a part of the private config section of the /v1/config service request or response payload.

Procedure

  1. To get the current configuration, run the following command:

    $ curl "$ROX_API_ENDPOINT/v1/config" -H "Authorization: Bearer $ROX_API_TOKEN" | jq
    Copy to Clipboard Toggle word wrap

    The jq command formats and displays the JSON response from the API in a readable way.

    Example output

    {
      "publicConfig": { ... }
      "privateConfig": { ... }
    }
    Copy to Clipboard Toggle word wrap

  2. To add a policy violation metric named component_severity, which includes only component and severity labels, to the configuration, run the following command:

    $ curl "$ROX_API_ENDPOINT/v1/config" -H "Authorization: Bearer $ROX_API_TOKEN" | \
      jq '.privateConfig.metrics.policyViolations.descriptors += {
            component_severity: { labels: [ "Component", "Severity" ] }
          } |
          { config: . }' | \
      curl -X PUT "$ROX_API_ENDPOINT/v1/config" -H "Authorization: Bearer $ROX_API_TOKEN" --data-binary @-
    Copy to Clipboard Toggle word wrap

    The jq command adds a new component_severity metric descriptor with the labels Component and Severity to the policyViolations configuration, then wraps the result in a config object for the update request.

You can configure custom Prometheus metrics for Red Hat Advanced Cluster Security for Kubernetes (RHACS) by using the RHACS portal.

Procedure

  1. In the RHACS portal, click Platform Configuration System Configuration.
  2. Click Edit.
  3. Scroll down to the Prometheus metrics configuration section, and do any of the following tasks:

    • To enable metrics gathering:

      1. Enter a positive integer for the gathering period in minutes. You can adjust the number by using the up and down arrows in the spin button.

        Important

        Use a larger value to reduce how often Central gathers metrics, as the process is resource intensive.

      2. Select one or more predefined metrics from the list of metrics.
    • To disable metrics gathering, enter 0.
  4. Click Save.

Verification

  • To verify that the predefined metric that you enabled is being exposed by Central, run the following command:

    $ curl "$ROX_API_ENDPOINT/metrics" \
      -H "Authorization: Bearer $ROX_API_TOKEN" \
      -s | grep <metric_name> | head
    Copy to Clipboard Toggle word wrap

    where:

    <metric_name>

    Specifies the predefined metric that you enabled. For example, rox_central_image_vuln_deployment_severity.

    The grep command filters the output for the predefined metric.

    The head command limits the output to the first few matching lines for readability.

    The output shows the predefined metric aggregated by cluster, namespace, and severity, indicating the number of vulnerabilities for each combination.

Before connecting Perses or other monitoring tools, configure Prometheus to scrape metrics from the Central service. For example, configure the machine access integration for the Kubernetes service account by creating a service account, performing role mapping, and assigning the necessary permissions. This ensures that Prometheus can securely access Central API data for monitoring and visualization purposes.

Procedure

  1. Perform role mapping to ensure that the Prometheus server can access the Central API with a service account token:

    1. Create and assign a role with the appropriate permission set and scope.
    2. Add a role mapping to the machine access configuration by using the following service account information:

      type

      KUBE_SERVICE_ACCOUNT

      key

      sub

      value

      system:serviceaccount:stackrox:prometheus-server

  2. Verify that the Prometheus custom resource (CR) contains the following information:

    # ...
      serviceAccountName: prometheus-server
      volumes:
      - name: prometheus-token
        projected:
          sources:
          - serviceAccountToken:
              path: token
              expirationSeconds: 3600
      volumeMounts:
      - name: prometheus-token
        mountPath: /var/run/secrets/tokens
        readOnly: true
      additionalScrapeConfigs:
        name: prometheus-additional-scrape-configs
        key: prometheus-additional.yaml
    # ...
    Copy to Clipboard Toggle word wrap
  3. Create the additional scrape configuration secret by using the following content, for example:

    apiVersion: v1
    kind: Secret
    metadata:
      name: prometheus-additional-scrape-configs
      namespace: stackrox
    stringData:
      prometheus-additional.yaml: |
        - job_name: 'central-metrics'
          scheme: https
          metrics_path: /metrics
          bearer_token_file: /var/run/secrets/tokens/token
          tls_config:
            insecure_skip_verify: true
          static_configs:
          - targets: ['central.stackrox.svc.cluster.local:443']
    Copy to Clipboard Toggle word wrap

You can configure an in-cluster Prometheus server to work with Red Hat Advanced Cluster Security for Kubernetes (RHACS). You can then use a tool such as Perses running in OpenShift Container Platform or externally to monitor RHACS.

To view a sample Perses configuration, navigate to the monitoring-examples repository in the stackrox organization on GitHub, select the main branch, and then open the perses directory.

Prerequisites

  1. You have configured Prometheus to scrape metrics from the Central service.

    For more information, see "Setting a Prometheus server scraping from Central service".

Procedure

  1. Create the Perses data source by using the following content, for example:

    apiVersion: perses.dev/v1alpha1
    kind: PersesDatasource
    metadata:
      name: rhacs-datasource
      namespace: stackrox
    spec:
      client:
        tls:
          caCert:
            certPath: /ca/service-ca.crt
            type: file
          enable: true
      config:
        default: true
        display:
          name: RHACS Prometheus Datasource
        plugin:
          kind: PrometheusDatasource
          spec:
            proxy:
              kind: HTTPProxy
              spec:
                url: 'http://<PROMETHEUS_SERVICE_HOSTNAME>:<PROMETHEUS_SERVICE_PORT>'
    Copy to Clipboard Toggle word wrap

    where:

    <PROMETHEUS_SERVICE_HOSTNAME>

    Specifies the hostname of the Prometheus service in your cluster.

    • If Prometheus is installed by using the monitoring stack, the hostname might be, for example, prometheus-operated.stackrox.svc.cluster.local.
    • If Prometheus in installed by using the Prometheus Operator or another installation, the hostname might be something different.
    <PROMETHEUS_SERVICE_PORT>
    Specifies the port of the Prometheus service in your cluster. For example, 9090.
  2. Create a Perses dashboard by using the following content, for example:

    apiVersion: perses.dev/v1alpha1
    kind: PersesDashboard
    metadata:
      name: sample-stackrox-dashboard-mini
      namespace: stackrox
    spec:
      display:
        name: Advanced Cluster Security / Sample
      panels:
        "total_policy_violations":
          kind: "Panel"
          spec:
            display:
              name: "Total policy violations"
            plugin:
              kind: "StatChart"
              spec:
                calculation: "last"
            queries:
              - kind: "TimeSeriesQuery"
                spec:
                  plugin:
                    kind: "PrometheusTimeSeriesQuery"
                    spec:
                      query: sum(rox_central_policy_violation_namespace_severity{Cluster=~'$Cluster',Namespace=~'$Namespace'})
    
      layouts:
        - kind: Grid
          spec:
            display:
              title: Policies
              collapse:
                open: true
            items:
              - content:
                  $ref: "#/spec/panels/total_policy_violations"
                "x": 0
                "y": 0
                "width": 6
                "height": 3
    
      variables:
        - kind: ListVariable
          spec:
            name: Cluster
            allowMultiple: true
            allowAllValue: true
            plugin:
              kind: PrometheusLabelValuesVariable
              spec:
                labelName: Cluster
    
        - kind: ListVariable
          spec:
            name: Namespace
            allowMultiple: true
            allowAllValue: true
            plugin:
              kind: PrometheusLabelValuesVariable
              spec:
                labelName: Namespace
      duration: 30d
      refreshInterval: 1m
    Copy to Clipboard Toggle word wrap

    This YAML creates a Perses dashboard in the stackrox namespace that shows a status panel for the total policy violations over the last 30 days, with variables for Cluster and Namespace, that refreshes every minute.

15.4.5. Metric categories for custom metrics

You can configure the Prometheus custom metrics by specifying a set of labels for each metric category. When using the API, you need to define the metric category for your custom metric.

The following are the supported metric categories:

  • imageVulnerabilities
  • nodeVulnerabilities
  • policyViolations

The following example shows the API request body for configuring a custom metric:

{
# ...

  "metrics": {
    "imageVulnerabilities": {
      "gatheringPeriodMinutes": 60,
      "descriptors": {
        "<metric_name>": {
          "labels": [ "<Labels>" ]
        }
      }
    }
  }

# ...
}
Copy to Clipboard Toggle word wrap

where:

metrics.imageVulnerabilities.descriptors.<metric_name>
Specifies the name of a custom metric descriptor.
metrics.imageVulnerabilities.descriptors.<metric name>.labels
Specifies the one or more labels separated by a comma used to aggregate and filter the metric data.

Each metric descriptor defines the labels that should be collected for that metric within the chosen category.

The following table lists the labels for image vulnerabilities, node vulnerabilities, and policy violations:

Expand
Table 15.2. Metric categories and labels
Metric categoryLabels

Image vulnerabilities

Cluster, Namespace, Deployment, IsPlatformWorkload, ImageID, ImageRegistry, ImageRemote, ImageTag, Component, ComponentVersion, OperatingSystem, CVE, CVSS, Severity, EPSSPercentile, EPSSProbability, IsFixable

Node vulnerabilities

Cluster, Node, Kernel, OperatingSystem, OSImage, Component, ComponentVersion, CVE, CVSS, Severity, EPSSPercentile, EPSSProbability, IsFixable, IsSnoozed

Policy violations

Cluster, Namespace, Resource, Deployment, IsDeploymentActive, IsPlatformComponent, Policy, Categories, Severity, Action, Message, Stage, State, Entity, EntityName

15.4.6. System metrics and labels

You can view additional system metrics that are exposed by default. These metrics are gathered once every hour.

The following table lists the metric names and their corresponding labels:

Expand
Table 15.3. Metric names and labels
Metric categoryMetric nameLabels

Total policy numbers

rox_central_cfg_total_policies

Enabled

Cluster health

rox_central_health_cluster_info

Cluster, Type, Status, Upgradability

Certificate expiry

rox_central_cert_exp_hours

Component

Note

The Certificate expiry metric requires permission to view the integrations.

Back to top
Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Theme

© 2025 Red Hat