Chapter 6. Troubleshooting the Red Hat build of OpenTelemetry


The OpenTelemetry Collector offers multiple ways to measure its health as well as investigate data ingestion issues.

6.1. Getting the OpenTelemetry Collector logs

You can get the logs for the OpenTelemetry Collector as follows.

Procedure

  1. Set the relevant log level in the OpenTelemetryCollector custom resource (CR):

      config: |
        service:
          telemetry:
            logs:
              level: debug 1
    1
    Collector’s log level. Supported values include info, warn, error, or debug. Defaults to info.
  2. Use the oc logs command or the web console to retrieve the logs.

6.2. Exposing the metrics

The OpenTelemetry Collector exposes the metrics about the data volumes it has processed. The following metrics are for spans, although similar metrics are exposed for metrics and logs signals:

otelcol_receiver_accepted_spans
The number of spans successfully pushed into the pipeline.
otelcol_receiver_refused_spans
The number of spans that could not be pushed into the pipeline.
otelcol_exporter_sent_spans
The number of spans successfully sent to the destination.
otelcol_exporter_enqueue_failed_spans
The number of spans failed to be added to the sending queue.

The operator creates a <cr_name>-collector-monitoring telemetry service that you can use to scrape the metrics endpoint.

Procedure

  1. Enable the telemetry service by adding the following lines in the OpenTelemetryCollector custom resource:

      config: |
        service:
          telemetry:
            metrics:
              address: ":8888" 1
    1
    The address at which the internal collector metrics are exposed. Defaults to :8888.
  1. Retrieve the metrics by running the following command, which uses the port-forwarding Collector pod:

    $ oc port-forward <collector_pod>
  2. Access the metrics endpoint at http://localhost:8888/metrics.

6.3. Debug exporter

You can configure the debug exporter to export the collected data to the standard output.

Procedure

  1. Configure the OpenTelemetryCollector custom resource as follows:

      config: |
        exporters:
          debug:
            verbosity: detailed
        service:
          pipelines:
            traces:
              exporters: [debug]
            metrics:
              exporters: [debug]
            logs:
              exporters: [debug]
  2. Use the oc logs command or the web console to export the logs to the standard output.
Red Hat logoGithubRedditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

© 2024 Red Hat, Inc.