Chapter 12. Network observability health rules

Maintaining reliable and secure network connectivity is critical for cluster administrators and security teams. Unresolved network issues can result in the following consequences:

Early detection of these issues allows for resolution before service level objectives (SLOs) are affected.

The Network Observability Operator provides automated health monitoring through the following features:

Health rules monitor network flow metrics and trigger alerts when defined thresholds are exceeded. For example, the PacketDropsByKernel rule reports an alert when kernel packet drop rates exceed defined levels.

Monitoring network health involves the following phases:

The PrometheusRule resource in the netobserv namespace can be viewed by running the following command:

$ oc get prometheusrules -n netobserv -o yaml

The Network Observability Operator includes a rule-based system to detect network anomalies and infrastructure failures. By converting configurations into alerting rules, the Operator provides automated monitoring and troubleshooting through the OpenShift Container Platform web console.

The FlowCollector health rule API maps to the Prometheus Operator to generate PrometheusRule objects. Use these base Prometheus Query Language (PromQL) patterns and metadata configurations to create custom health rules for network observability.

The PrometheusRule resource in the netobserv namespace can be viewed by running the following command:

$ oc get prometheusrules -n netobserv -o yaml

sum(rate(netobserv_workload_ingress_bytes_total{SrcK8S_Namespace="openshift-ingress"}[30m])) by (DstK8S_Namespace)

expr: |-
  (100 *
    (
      (sum(rate(netobserv_workload_ingress_bytes_total{SrcK8S_Namespace="openshift-ingress"}[30m])) by (DstK8S_Namespace) > 1000)
      - sum(rate(netobserv_workload_ingress_bytes_total{SrcK8S_Namespace="openshift-ingress"}[30m] offset 1d)) by (DstK8S_Namespace)
    )
    / sum(rate(netobserv_workload_ingress_bytes_total{SrcK8S_Namespace="openshift-ingress"}[30m] offset 1d)) by (DstK8S_Namespace))
  > 100

apiVersion: monitoring.openshift.io/v1
kind: AlertingRule
metadata:
  name: netobserv-alerts
  namespace: openshift-monitoring
spec:
  groups:
  - name: NetObservAlerts
    rules:
    - alert: NetObservIncomingBandwidth
      annotations:
        netobserv_io_network_health: '{"namespaceLabels":["DstK8S_Namespace"],"threshold":"100","unit":"%","upperBound":"500"}'
        message: |-
          Surge of incoming traffic detected: current traffic to {{ $labels.DstK8S_Namespace }} increased by more than 100% since yesterday.
        summary: "Surge in incoming traffic"
      expr: |-
        # ... (PromQL expression)
      for: 1m
      labels:
        app: netobserv
        netobserv: "true"
        severity: warning

Create custom health rules by using Prometheus Query Language (PromQL) to define an AlertingRule resource. These rules trigger alerts based on specific network metrics, such as traffic surges.

Recording rules pre-compute complex Prometheus Query Language (PromQL) expressions and save the results as new time series. Unlike alerting rules, recording rules do not monitor thresholds.

Using recording rules provides the following advantages:

The following table compares rule modes based on the expected outcome:

Custom recording rules that contribute to the Network Health dashboard must meet specific metadata requirements.

Create custom recording rules to pre-compute metrics for the Network Health dashboard. Recording rules require specific annotations and labels to integrate with the Network Observability Operator.