Chapter 11. Troubleshooting Network Observability

Procedure

1. Navigate to the directory where you want to store the must-gather data.

2. Run the following command to collect cluster-wide must-gather resources:

   $ oc adm must-gather
    --image-stream=openshift/must-gather \
    --image=quay.io/netobserv/must-gather

Procedure

1. Check if the

   spec.consolePlugin.register

   field is set to

   true

   by running the following command:

   $ oc -n netobserv get flowcollector cluster -o yaml

   Example output

   apiVersion: flows.netobserv.io/v1alpha1
   kind: FlowCollector
   metadata:
     name: cluster
   spec:
     consolePlugin:
       register: false

2. Optional: Add the

   netobserv-plugin

   plugin by manually editing the Console Operator config:

   $ oc edit console.operator.openshift.io cluster

   Example output

   ...
   spec:
     plugins:
     - netobserv-plugin
   ...

3. Optional: Set the

   spec.consolePlugin.register

   field to

   true

   by running the following command:

   $ oc -n netobserv edit flowcollector cluster -o yaml

   Example output

   apiVersion: flows.netobserv.io/v1alpha1
   kind: FlowCollector
   metadata:
     name: cluster
   spec:
     consolePlugin:
       register: true

4. Ensure the status of console pods is

   running

   by running the following command:

   $ oc get pods -n openshift-console -l app=console

5. Restart the console pods by running the following command:

   $ oc delete pods -n openshift-console -l app=console

6. Clear your browser cache and history.

7. Check the status of Network Observability plugin pods by running the following command:

   $ oc get pods -n netobserv -l app=netobserv-plugin

   Example output

   NAME                                READY   STATUS    RESTARTS   AGE
   netobserv-plugin-68c7bbb9bb-b69q6   1/1     Running   0          21s

8. Check the logs of the Network Observability plugin pods by running the following command:

   $ oc logs -n netobserv -l app=netobserv-plugin

   Example output

   time="2022-12-13T12:06:49Z" level=info msg="Starting netobserv-console-plugin [build version: , build date: 2022-10-21 15:15] at log level info" module=main
   time="2022-12-13T12:06:49Z" level=info msg="listening on https://:9001" module=server

Procedure

1. Delete the flow-pipeline pods to restart them by running the following command:

   $ oc delete pods -n netobserv -l app=flowlogs-pipeline-transformer

Procedure

1. Remove the

   interfaces: [ 'br-int', 'br-ex' ]

   field. This allows the agent to fetch information from all the interfaces. Alternatively, you can specify the Layer-3 interface for example,

   eth0

   . Run the following command:

   $ oc edit -n netobserv flowcollector.yaml -o yaml

   Example output

   apiVersion: flows.netobserv.io/v1alpha1
   kind: FlowCollector
   metadata:
     name: cluster
   spec:
     agent:
       type: EBPF
       ebpf:
         interfaces: [ 'br-int', 'br-ex' ] 

   1

   1

   Specifies the network interfaces.

Procedure

1. In the web console, navigate to Operators Installed Operators
2. Click Network Observability and then select Subscription.

3. From the Actions menu, click Edit Subscription.

   1. Alternatively, you can use the CLI to open the YAML configuration for the

      Subscription

      object by running the following command:

      $ oc edit subscription netobserv-operator -n openshift-netobserv-operator

4. Edit the

   Subscription

   object to add the

   config.resources.limits.memory

   specification and set the value to account for your memory requirements. See the Additional resources for more information about resource considerations:

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: netobserv-operator
     namespace: openshift-netobserv-operator
   spec:
     channel: stable
     config:
       resources:
         limits:
           memory: 800Mi     

   1

   
         requests:
           cpu: 100m
           memory: 100Mi
     installPlanApproval: Automatic
     name: netobserv-operator
     source: redhat-operators
     sourceNamespace: openshift-marketplace
     startingCSV: <network_observability_operator_latest_version> 

   2

   1

   For example, you can increase the memory limit to 800Mi.

   2

   This value should not be edited, but note that it changes depending on the most current release of the Operator.

Procedure

1. Navigate to Operators Installed Operators, viewing All projects from the Project drop-down menu.
2. In the Provided APIs list, select the Network Observability Operator.

3. Click the Flow Collector then the YAML view tab.

   1. If you are using the Loki Operator, check that the

      spec.loki.batchSize

      value does not exceed 98 MiB.
   2. If you are using a Loki installation method that is different from the Red Hat Loki Operator, such as Grafana Loki, verify that the

      grpc_server_max_recv_msg_size

      Grafana Loki server setting is higher than the

      FlowCollector

      resource

      spec.loki.batchSize

      value. If it is not, you must either increase the

      grpc_server_max_recv_msg_size

      value, or decrease the

      spec.loki.batchSize

      value so that it is lower than the limit.
4. Click Save if you edited the FlowCollector.

Procedure

1. Navigate to Operators Installed Operators, viewing All projects from the Project dropdown.
2. Look for Loki Operator, and select the LokiStack tab.

3. Create or edit an existing LokiStack instance using the YAML view to add the

   perStreamRateLimit

   and

   perStreamRateLimitBurst

   specifications:

   apiVersion: loki.grafana.com/v1
   kind: LokiStack
   metadata:
     name: loki
     namespace: netobserv
   spec:
     limits:
       global:
         ingestion:
           perStreamRateLimit: 6        

   1

   
           perStreamRateLimitBurst: 30  

   2

   
     tenants:
       mode: openshift-network
     managementState: Managed

   1

   The default value for perStreamRateLimit is 3.

   2

   The default value for perStreamRateLimitBurst is 15.

4. Click Save.