Chapter 10. Optimizing flowlogs-pipeline resource consumption in large clusters

Procedure

1. In the OpenShift Container Platform web console, navigate to Operators Installed Operators.
2. Select the Network Observability Operator, and then click the FlowCollector tab.
3. Click the cluster instance, and then click the YAML tab.

4. In the spec.processor section, set the informerCacheProxy configuration to enable the shared cache by using the following example:

   apiVersion: flows.netobserv.io/v1beta2
   kind: FlowCollector
   metadata:
     name: cluster
   spec:
     processor:
       informerCacheProxy:
         enabled: true
         replicas: 2

   where:

   spec.processor.informerCacheProxy.enabled

   Set to true to enable the informer cache proxy. The default value is false.

   spec.processor.informerCacheProxy.replicas

   Specifies the number of replicas. For high availability in production environments, the minimum is 2. The Operator enforces this requirement with leader election to prevent a single point of failure.

5. Click Save.

Verification

1. Navigate to Workloads Pods.
2. Ensure that the pods beginning with flowlogs-pipeline-informers are in the Running state.
3. Verify that one pod shows as the active leader by checking the pod logs for leader election messages.
4. Confirm that the flowlogs-pipeline pods remain Running and show decreased CPU and memory use in the Metrics tab.

Procedure

1. Check the status of the shared cache pods by running the following command:

   $ oc get pods -n netobserv -l app=flowlogs-pipeline-informers

   Example output

   NAME                             READY   STATUS    RESTARTS   AGE
   flowlogs-pipeline-informers-7d9f8b5c4d-abc12   1/1     Running   0          5m
   flowlogs-pipeline-informers-7d9f8b5c4d-xyz34   1/1     Running   0          5m

   Ensure all pods show Running status. If pods are in CrashLoopBackOff or Error status, proceed to examine the logs.

2. Identify the current leader pod:

   $ oc logs -n netobserv -l app=flowlogs-pipeline-informers --tail=50 | grep -i "leader"

   Look for log entries indicating leader election status. Ensure that only one pod reports itself as the active leader.

3. View detailed logs from the leader pod to diagnose issues:

   $ oc logs -n netobserv <flowlogs_pipeline_informers_pod_name>

   Common issues to look for:

   • API server connection failures: Error messages indicating inability to connect to the Kubernetes API server suggest network or RBAC permission issues.
   • gRPC communication errors: Messages about failed connections on port 9402 indicate flowlogs-pipeline instances cannot reach the shared cache.
   • Leader election conflicts: If multiple pods claim to be the leader, there might be a configuration issue with leader election.

4. Verify that flowlogs-pipeline instances are receiving cache updates:

   $ oc logs -n netobserv -l app=flowlogs-pipeline --tail=100 | grep -i "k8scache"

   flowlogs-pipeline pods show successful connection messages to the shared cache on port 9402.

5. Check the shared cache service:

   $ oc get svc -n netobserv flowlogs-pipeline-k8scache

   Verify that the service exists and is routing to the correct pods.

6. If the shared cache is not functioning correctly, check for RBAC permission issues:

   $ oc describe clusterrolebinding netobserv-informers

   The ClusterRoleBinding grants the flowlogs-pipeline-informers ServiceAccount the necessary permissions to watch Kubernetes resources.

Verification

1. After you resolve issues, complete the following verification steps:

   • Confirm all shared cache pods are running and healthy.
   • Verify that one pod shows as the active leader in the logs.
   • Check that flowlogs-pipeline pods successfully connect to the shared cache.
   • Monitor flow enrichment in the network observability console to ensure metadata is displayed correctly on network flows.