Chapter 10. Troubleshooting

Procedure

1. Set the relevant log level in the OpenTelemetryCollector custom resource (CR):

     config: |
       service:
         telemetry:
           logs:
             level: debug 1

   1

   Collector’s log level. Supported values include info, warn, error, or debug. Defaults to info.

2. Use the oc logs command or the web console to retrieve the logs.

Procedure

1. Enable the telemetry service by adding the following lines in the OpenTelemetryCollector custom resource (CR):

   # ...
     config: |
       service:
         telemetry:
           metrics:
             address: ":8888" 1
   # ...

   1

   The address at which the internal collector metrics are exposed. Defaults to :8888.

2. Retrieve the metrics by running the following command, which uses the port-forwarding Collector pod:

   $ oc port-forward <collector_pod>

3. In the OpenTelemetryCollector CR, set the enableMetrics field to true to scrape internal metrics:

   apiVersion: opentelemetry.io/v1alpha1
   kind: OpenTelemetryCollector
   spec:
   # ...
     mode: deployment
     observability:
       metrics:
         enableMetrics: true
   # ...

   Depending on the deployment mode of the OpenTelemetry Collector, the internal metrics are scraped by using PodMonitors or ServiceMonitors.

   Note

   Alternatively, if you do not set the enableMetrics field to true, you can access the metrics endpoint at http://localhost:8888/metrics.

4. On the Observe page in the web console, enable User Workload Monitoring to visualize the scraped metrics.

   Note

   Not all processors expose the required metrics.

5. In the web console, go to Observe Dashboards and select the OpenTelemetry Collector dashboard from the drop-down list to view it.

   Tip

   You can filter the visualized data such as spans or metrics by the Collector instance, namespace, or OpenTelemetry components such as processors, receivers, or exporters.

Procedure

1. Configure the OpenTelemetryCollector custom resource as follows:

     config: |
       exporters:
         debug:
           verbosity: detailed
       service:
         pipelines:
           traces:
             exporters: [debug]
           metrics:
             exporters: [debug]
           logs:
             exporters: [debug]

2. Use the oc logs command or the web console to export the logs to the standard output.

Procedure

1. In the OpenShift Container Platform web console, go to Observe Network Traffic Topology.
2. Select Namespace to filter the workloads by the namespace in which your OpenTelemetry Collector is deployed.
3. Use the network traffic visuals to troubleshoot possible issues. See "Observing the network traffic from the Topology view" for more details.