Chapter 16. Troubleshooting Network Observability

Procedure

1. Navigate to the directory where you want to store the must-gather data.

2. Run the following command to collect cluster-wide must-gather resources:

   $ oc adm must-gather
    --image-stream=openshift/must-gather \
    --image=quay.io/netobserv/must-gather

Procedure

1. Check if the spec.consolePlugin.register field is set to true by running the following command:

   $ oc -n netobserv get flowcollector cluster -o yaml

   Example output

   apiVersion: flows.netobserv.io/v1alpha1
   kind: FlowCollector
   metadata:
     name: cluster
   spec:
     consolePlugin:
       register: false

2. Optional: Add the netobserv-plugin plugin by manually editing the Console Operator config:

   $ oc edit console.operator.openshift.io cluster

   Example output

   ...
   spec:
     plugins:
     - netobserv-plugin
   ...

3. Optional: Set the spec.consolePlugin.register field to true by running the following command:

   $ oc -n netobserv edit flowcollector cluster -o yaml

   Example output

   apiVersion: flows.netobserv.io/v1alpha1
   kind: FlowCollector
   metadata:
     name: cluster
   spec:
     consolePlugin:
       register: true

4. Ensure the status of console pods is running by running the following command:

   $ oc get pods -n openshift-console -l app=console

5. Restart the console pods by running the following command:

   $ oc delete pods -n openshift-console -l app=console

6. Clear your browser cache and history.

7. Check the status of Network Observability plugin pods by running the following command:

   $ oc get pods -n netobserv -l app=netobserv-plugin

   Example output

   NAME                                READY   STATUS    RESTARTS   AGE
   netobserv-plugin-68c7bbb9bb-b69q6   1/1     Running   0          21s

8. Check the logs of the Network Observability plugin pods by running the following command:

   $ oc logs -n netobserv -l app=netobserv-plugin

   Example output

   time="2022-12-13T12:06:49Z" level=info msg="Starting netobserv-console-plugin [build version: , build date: 2022-10-21 15:15] at log level info" module=main
   time="2022-12-13T12:06:49Z" level=info msg="listening on https://:9001" module=server

Procedure

1. Delete the flow-pipeline pods to restart them by running the following command:

   $ oc delete pods -n netobserv -l app=flowlogs-pipeline-transformer

Procedure

1. Remove the interfaces: [ 'br-int', 'br-ex' ] field. This allows the agent to fetch information from all the interfaces. Alternatively, you can specify the Layer-3 interface for example, eth0. Run the following command:

   $ oc edit -n netobserv flowcollector.yaml -o yaml

   Example output

   apiVersion: flows.netobserv.io/v1alpha1
   kind: FlowCollector
   metadata:
     name: cluster
   spec:
     agent:
       type: EBPF
       ebpf:
         interfaces: [ 'br-int', 'br-ex' ] 1

   1

   Specifies the network interfaces.

Procedure

1. In the web console, navigate to Operators Installed Operators
2. Click Network Observability and then select Subscription.

3. From the Actions menu, click Edit Subscription.

   1. Alternatively, you can use the CLI to open the YAML configuration for the Subscription object by running the following command:

      $ oc edit subscription netobserv-operator -n openshift-netobserv-operator

4. Edit the Subscription object to add the config.resources.limits.memory specification and set the value to account for your memory requirements. See the Additional resources for more information about resource considerations:

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: netobserv-operator
     namespace: openshift-netobserv-operator
   spec:
     channel: stable
     config:
       resources:
         limits:
           memory: 800Mi     1
         requests:
           cpu: 100m
           memory: 100Mi
     installPlanApproval: Automatic
     name: netobserv-operator
     source: redhat-operators
     sourceNamespace: openshift-marketplace
     startingCSV: <network_observability_operator_latest_version> 2

   1

   For example, you can increase the memory limit to 800Mi.

   2

   This value should not be edited, but note that it changes depending on the most current release of the Operator.

Procedure

1. Navigate to Operators Installed Operators, viewing All projects from the Project drop-down menu.
2. In the Provided APIs list, select the Network Observability Operator.

3. Click the Flow Collector then the YAML view tab.

   1. If you are using the Loki Operator, check that the spec.loki.batchSize value does not exceed 98 MiB.
   2. If you are using a Loki installation method that is different from the Red Hat Loki Operator, such as Grafana Loki, verify that the grpc_server_max_recv_msg_size Grafana Loki server setting is higher than the FlowCollector resource spec.loki.batchSize value. If it is not, you must either increase the grpc_server_max_recv_msg_size value, or decrease the spec.loki.batchSize value so that it is lower than the limit.
4. Click Save if you edited the FlowCollector.

Procedure

1. Navigate to Operators Installed Operators, viewing All projects from the Project dropdown.
2. Look for Loki Operator, and select the LokiStack tab.

3. Create or edit an existing LokiStack instance using the YAML view to add the perStreamRateLimit and perStreamRateLimitBurst specifications:

   apiVersion: loki.grafana.com/v1
   kind: LokiStack
   metadata:
     name: loki
     namespace: netobserv
   spec:
     limits:
       global:
         ingestion:
           perStreamRateLimit: 6        1
           perStreamRateLimitBurst: 30  2
     tenants:
       mode: openshift-network
     managementState: Managed

   1

   The default value for perStreamRateLimit is 3.

   2

   The default value for perStreamRateLimitBurst is 15.

4. Click Save.