Chapter 8. Using metrics with dashboards and alerts

Procedure

1. In the web console Observe Dashboards, select the Netobserv dashboard.

2. View network traffic metrics in the following categories, with each having the subset per node, namespace, source, and destination:

   • Byte rates
   • Packet drops
   • DNS
   • RTT
3. Select the Netobserv/Health dashboard.

4. View metrics about the health of the Operator in the following categories, with each having the subset per node, namespace, source, and destination.

   • Flows
   • Flows Overhead
   • Flow rates
   • Agents
   • Processor
   • Operator

Procedure

1. Create a YAML file by clicking the import icon, +.

2. Add an alerting rule configuration to the YAML file. In the YAML sample that follows, an alert is created for when the cluster ingress traffic reaches a given threshold of 10 MBps per destination workload.

   apiVersion: monitoring.openshift.io/v1
   kind: AlertingRule
   metadata:
     name: netobserv-alerts
     namespace: openshift-monitoring
   spec:
     groups:
     - name: NetObservAlerts
       rules:
       - alert: NetObservIncomingBandwidth
         annotations:
           message: |-
             {{ $labels.job }}: incoming traffic exceeding 10 MBps for 30s on {{ $labels.DstK8S_OwnerType }} {{ $labels.DstK8S_OwnerName }} ({{ $labels.DstK8S_Namespace }}).
           summary: "High incoming traffic."
         expr: sum(rate(netobserv_workload_ingress_bytes_total     {SrcK8S_Namespace="openshift-ingress"}[1m])) by (job, DstK8S_Namespace, DstK8S_OwnerName, DstK8S_OwnerType) > 10000000      1
         for: 30s
         labels:
           severity: warning

   1

   The netobserv_workload_ingress_bytes_total metric is enabled by default in spec.processor.metrics.includeList.

3. Click Create to apply the configuration file to the cluster.

Procedure

1. In the web console, navigate to Operators Installed Operators.
2. In the Provided APIs heading for the NetObserv Operator, select FlowMetric.
3. In the Project: dropdown list, select the project of the Network Observability Operator instance.
4. Click Create FlowMetric.

5. Configure the FlowMetric resource, similar to the following sample configurations:

   Example 8.1. Generate a metric that tracks ingress bytes received from cluster external sources

   apiVersion: flows.netobserv.io/v1alpha1
   kind: FlowMetric
   metadata:
     name: flowmetric-cluster-external-ingress-traffic
     namespace: netobserv                              1
   spec:
     metricName: cluster_external_ingress_bytes_total  2
     type: Counter                                     3
     valueField: Bytes
     direction: Ingress                                4
     labels: [DstK8S_HostName,DstK8S_Namespace,DstK8S_OwnerName,DstK8S_OwnerType] 5
     filters:                                          6
     - field: SrcSubnetLabel
       matchType: Absence

   1

   The FlowMetric resources need to be created in the namespace defined in the FlowCollector spec.namespace, which is netobserv by default.

   2

   The name of the Prometheus metric, which in the web console appears with the prefix netobserv-<metricName>.

   3

   The type specifies the type of metric. The Counter type is useful for counting bytes or packets.

   4

   The direction of traffic to capture. If not specified, both ingress and egress are captured, which can lead to duplicated counts.

   5

   Labels define what the metrics look like and the relationship between the different entities and also define the metrics cardinality. For example, SrcK8S_Name is a high cardinality metric.

   6

   Refines results based on the listed criteria. In this example, selecting only the cluster external traffic is done by matching only flows where SrcSubnetLabel is absent. This assumes the subnet labels feature is enabled (via spec.processor.subnetLabels), which is done by default.

   Verification

   1. Once the pods refresh, navigate to Observe Metrics.
   2. In the Expression field, type the metric name to view the corresponding result. You can also enter an expression, such as topk(5, sum(rate(netobserv_cluster_external_ingress_bytes_total{DstK8S_Namespace="my-namespace"}[2m])) by (DstK8S_HostName, DstK8S_OwnerName, DstK8S_OwnerType))

   Example 8.2. Show RTT latency for cluster external ingress traffic

   apiVersion: flows.netobserv.io/v1alpha1
   kind: FlowMetric
   metadata:
     name: flowmetric-cluster-external-ingress-rtt
     namespace: netobserv    1
   spec:
     metricName: cluster_external_ingress_rtt_seconds
     type: Histogram                 2
     valueField: TimeFlowRttNs
     direction: Ingress
     labels: [DstK8S_HostName,DstK8S_Namespace,DstK8S_OwnerName,DstK8S_OwnerType]
     filters:
     - field: SrcSubnetLabel
       matchType: Absence
     - field: TimeFlowRttNs
       matchType: Presence
     divider: "1000000000"      3
     buckets: [".001", ".005", ".01", ".02", ".03", ".04", ".05", ".075", ".1", ".25", "1"]  4

   1

   The FlowMetric resources need to be created in the namespace defined in the FlowCollector spec.namespace, which is netobserv by default.

   2

   The type specifies the type of metric. The Histogram type is useful for a latency value (TimeFlowRttNs).

   3

   Since the Round-trip time (RTT) is provided as nanos in flows, use a divider of 1 billion to convert into seconds, which is standard in Prometheus guidelines.

   4

   The custom buckets specify precision on RTT, with optimal precision ranging between 5ms and 250ms.

   Verification

   1. Once the pods refresh, navigate to Observe Metrics.
   2. In the Expression field, you can type the metric name to view the corresponding result.

Procedure

1. In the web console, navigate to Operators Installed Operators.
2. In the Provided APIs heading for the NetObserv Operator, select FlowMetric.
3. In the Project: dropdown list, select the project of the Network Observability Operator instance.
4. Click Create FlowMetric.
5. Configure the FlowMetric resource, similar to the following sample configurations:

Procedure

1. In the web console, navigate to Operators Installed Operators.
2. In the Provided APIs heading for the NetObserv Operator, select FlowMetric.
3. In the Project dropdown list, select the project of the Network Observability Operator instance.
4. Click Create FlowMetric.

5. Create FlowMetric resources to add the following configurations:

   Configuration counting flows per destination host and resource, with TCP flags

   apiVersion: flows.netobserv.io/v1alpha1
   kind: FlowMetric
   metadata:
     name: flows-with-flags-per-destination
   spec:
     metricName: flows_with_flags_per_destination_total
     type: Counter
     labels: [SrcSubnetLabel,DstSubnetLabel,DstK8S_Name,DstK8S_Type,DstK8S_HostName,DstK8S_Namespace,Flags]

   Configuration counting flows per source host and resource, with TCP flags

   apiVersion: flows.netobserv.io/v1alpha1
   kind: FlowMetric
   metadata:
     name: flows-with-flags-per-source
   spec:
     metricName: flows_with_flags_per_source_total
     type: Counter
     labels: [DstSubnetLabel,SrcSubnetLabel,SrcK8S_Name,SrcK8S_Type,SrcK8S_HostName,SrcK8S_Namespace,Flags]

6. Deploy the following AlertingRule resource to alert for SYN flooding:

   AlertingRule for SYN flooding

   apiVersion: monitoring.openshift.io/v1
   kind: AlertingRule
   metadata:
     name: netobserv-syn-alerts
     namespace: openshift-monitoring
   # ...
     spec:
     groups:
     - name: NetObservSYNAlerts
       rules:
       - alert: NetObserv-SYNFlood-in
         annotations:
           message: |-
             {{ $labels.job }}: incoming SYN-flood attack suspected to Host={{ $labels.DstK8S_HostName}}, Namespace={{ $labels.DstK8S_Namespace }}, Resource={{ $labels.DstK8S_Name }}. This is characterized by a high volume of SYN-only flows with different source IPs and/or ports.
           summary: "Incoming SYN-flood"
         expr: sum(rate(netobserv_flows_with_flags_per_destination_total{Flags="2"}[1m])) by (job, DstK8S_HostName, DstK8S_Namespace, DstK8S_Name) > 300      1
         for: 15s
         labels:
           severity: warning
           app: netobserv
       - alert: NetObserv-SYNFlood-out
         annotations:
           message: |-
             {{ $labels.job }}: outgoing SYN-flood attack suspected from Host={{ $labels.SrcK8S_HostName}}, Namespace={{ $labels.SrcK8S_Namespace }}, Resource={{ $labels.SrcK8S_Name }}. This is characterized by a high volume of SYN-only flows with different source IPs and/or ports.
           summary: "Outgoing SYN-flood"
         expr: sum(rate(netobserv_flows_with_flags_per_source_total{Flags="2"}[1m])) by (job, SrcK8S_HostName, SrcK8S_Namespace, SrcK8S_Name) > 300       2
         for: 15s
         labels:
           severity: warning
           app: netobserv
   # ...

   1 2

   In this example, the threshold for the alert is 300; however, you can adapt this value empirically. A threshold that is too low might produce false-positives, and if it’s too high it might miss actual attacks.

Verification

1. In the web console, click Manage Columns in the Network Traffic table view and click TCP flags.
2. In the Network Traffic table view, filter on TCP protocol SYN TCPFlag. A large number of flows with the same byteSize indicates a SYN flood.
3. Go to Observe Alerting and select the Alerting Rules tab.
4. Filter on netobserv-synflood-in alert. The alert should fire when SYN flooding occurs.