Chapter 12. Using metrics with dashboards and alerts

Procedure

1. In the web console Observe Dashboards, select the Netobserv dashboard.

2. View network traffic metrics in the following categories, with each having the subset per node, namespace, source, and destination:

   • Byte rates
   • Packet drops
   • DNS
   • RTT
3. Select the Netobserv/Health dashboard.

4. View metrics about the health of the Operator in the following categories, with each having the subset per node, namespace, source, and destination:

   • Flows
   • Flows Overhead
   • Flow rates
   • Agents
   • Processor

   • Operator

     Infrastructure and Application metrics are shown in a split-view for namespace and workloads.

Procedure

1. Create a YAML file by clicking the import icon, +.

2. Add an alerting rule configuration to the YAML file. In the YAML sample that follows, an alert is created for when the cluster ingress traffic reaches a given threshold of 10 MBps per destination workload.

   apiVersion: monitoring.openshift.io/v1
   kind: AlertingRule
   metadata:
     name: netobserv-alerts
     namespace: openshift-monitoring
   spec:
     groups:
     - name: NetObservAlerts
       rules:
       - alert: NetObservIncomingBandwidth
         annotations:
           message: |-
             {{ $labels.job }}: incoming traffic exceeding 10 MBps for 30s on {{ $labels.DstK8S_OwnerType }} {{ $labels.DstK8S_OwnerName }} ({{ $labels.DstK8S_Namespace }}).
           summary: "High incoming traffic."
         expr: sum(rate(netobserv_workload_ingress_bytes_total     {SrcK8S_Namespace="openshift-ingress"}[1m])) by (job, DstK8S_Namespace, DstK8S_OwnerName, DstK8S_OwnerType) > 10000000      

   1

   
         for: 30s
         labels:
           severity: warning

   1

   The netobserv_workload_ingress_bytes_total metric is enabled by default in spec.processor.metrics.includeList.

3. Click Create to apply the configuration file to the cluster.

Procedure

1. In the web console, navigate to Ecosystem Installed Operators.
2. In the Provided APIs heading for the NetObserv Operator, select FlowMetric.
3. In the Project: dropdown list, select the project of the Network Observability Operator instance.
4. Click Create FlowMetric.
5. Configure the FlowMetric resource. See "Custom metrics configuration examples".

Verification

1. Once the pods refresh, navigate to Observe Metrics.
2. In the Expression field, type the metric name to view the corresponding result. You can also enter an expression, such as topk(5, sum(rate(netobserv_cluster_external_ingress_bytes_total{DstK8S_Namespace="my-namespace"}[2m])) by (DstK8S_HostName, DstK8S_OwnerName, DstK8S_OwnerType))

Procedure

1. In the web console, navigate to Ecosystem Installed Operators.
2. In the Provided APIs heading for the NetObserv Operator, select FlowMetric.
3. In the Project dropdown list, select the project of the Network Observability Operator instance.
4. Click Create FlowMetric.

5. Create FlowMetric resources to add the following configurations:

   Configuration counting network policy events per policy name and namespace

   apiVersion: flows.netobserv.io/v1alpha1
   kind: FlowMetric
   metadata:
     name: network-policy-events
     namespace: netobserv
   spec:
     metricName: network_policy_events_total
     type: Counter
     labels: [NetworkEvents>Type, NetworkEvents>Namespace, NetworkEvents>Name, NetworkEvents>Action, NetworkEvents>Direction]       

   1

   
     filters:
     - field: NetworkEvents>Feature
       value: acl
     flatten: [NetworkEvents]       

   2

   
     remap:                         

   3

   
       "NetworkEvents>Type": type
       "NetworkEvents>Namespace": namespace
       "NetworkEvents>Name": name
       "NetworkEvents>Direction": direction

   1

   These labels represent the nested fields for Network Events from the Traffic flows table. Each network event has a specific type, namespace, name, action, and direction. You can alternatively specify the Interfaces if NetworkEvents is unavailable in your OpenShift Container Platform version.

   2

   Optional: You can choose to represent a field that contains a list of items as distinct items.

   3

   Optional: You can rename the fields in Prometheus.

Verification

1. In the web console, navigate to Observe Dashboards and scroll down to see the Network Policy tab.
2. You should begin seeing metrics filter in based on the metric you created along with the network policy specifications.

Procedure

1. In the web console, navigate to Ecosystem Installed Operators.
2. In the Provided APIs heading for the NetObserv Operator, select FlowMetric.
3. In the Project: dropdown list, select the project of the Network Observability Operator instance.
4. Click Create FlowMetric.
5. Configure the FlowMetric resource. See "Flowmetric chart configuration examples".

Verification

1. Once the pods refresh, navigate to Observe Dashboards.

2. Search for the NetObserv / Main dashboard. View two panels under the NetObserv / Main dashboard, or optionally a dashboard name that you create:

   • A textual single statistic showing the global external ingress rate summed across all dimensions
   • A timeseries graph showing the same metric per destination workload

Procedure

1. In the web console, navigate to Ecosystem Installed Operators.
2. In the Provided APIs heading for the NetObserv Operator, select FlowMetric.
3. In the Project dropdown list, select the project of the Network Observability Operator instance.
4. Click Create FlowMetric.

5. Create FlowMetric resources to add the following configurations:

   Configuration counting flows per destination host and resource, with TCP flags

   apiVersion: flows.netobserv.io/v1alpha1
   kind: FlowMetric
   metadata:
     name: flows-with-flags-per-destination
   spec:
     metricName: flows_with_flags_per_destination_total
     type: Counter
     labels: [SrcSubnetLabel,DstSubnetLabel,DstK8S_Name,DstK8S_Type,DstK8S_HostName,DstK8S_Namespace,Flags]

   Configuration counting flows per source host and resource, with TCP flags

   apiVersion: flows.netobserv.io/v1alpha1
   kind: FlowMetric
   metadata:
     name: flows-with-flags-per-source
   spec:
     metricName: flows_with_flags_per_source_total
     type: Counter
     labels: [DstSubnetLabel,SrcSubnetLabel,SrcK8S_Name,SrcK8S_Type,SrcK8S_HostName,SrcK8S_Namespace,Flags]

6. Deploy the following AlertingRule resource to alert for SYN flooding:

   AlertingRule for SYN flooding

   apiVersion: monitoring.openshift.io/v1
   kind: AlertingRule
   metadata:
     name: netobserv-syn-alerts
     namespace: openshift-monitoring
   # ...
     spec:
     groups:
     - name: NetObservSYNAlerts
       rules:
       - alert: NetObserv-SYNFlood-in
         annotations:
           message: |-
             {{ $labels.job }}: incoming SYN-flood attack suspected to Host={{ $labels.DstK8S_HostName}}, Namespace={{ $labels.DstK8S_Namespace }}, Resource={{ $labels.DstK8S_Name }}. This is characterized by a high volume of SYN-only flows with different source IPs and/or ports.
           summary: "Incoming SYN-flood"
         expr: sum(rate(netobserv_flows_with_flags_per_destination_total{Flags="2"}[1m])) by (job, DstK8S_HostName, DstK8S_Namespace, DstK8S_Name) > 300      

   1

   
         for: 15s
         labels:
           severity: warning
           app: netobserv
       - alert: NetObserv-SYNFlood-out
         annotations:
           message: |-
             {{ $labels.job }}: outgoing SYN-flood attack suspected from Host={{ $labels.SrcK8S_HostName}}, Namespace={{ $labels.SrcK8S_Namespace }}, Resource={{ $labels.SrcK8S_Name }}. This is characterized by a high volume of SYN-only flows with different source IPs and/or ports.
           summary: "Outgoing SYN-flood"
         expr: sum(rate(netobserv_flows_with_flags_per_source_total{Flags="2"}[1m])) by (job, SrcK8S_HostName, SrcK8S_Namespace, SrcK8S_Name) > 300       

   2

   
         for: 15s
         labels:
           severity: warning
           app: netobserv
   # ...

   1 2

   In this example, the threshold for the alert is 300; however, you can adapt this value empirically. A threshold that is too low might produce false-positives, and if it’s too high it might miss actual attacks.

Verification

1. In the web console, click Manage Columns in the Network Traffic table view and click TCP flags.
2. In the Network Traffic table view, filter on TCP protocol SYN TCPFlag. A large number of flows with the same byteSize indicates a SYN flood.
3. Go to Observe Alerting and select the Alerting Rules tab.
4. Filter on netobserv-synflood-in alert. The alert should fire when SYN flooding occurs.