Este conteúdo não está disponível no idioma selecionado.

Chapter 10. Troubleshooting


The OpenTelemetry Collector offers multiple ways to measure its health as well as investigate data ingestion issues.

10.1. Collecting diagnostic data from the command line

When submitting a support case, it is helpful to include diagnostic information about your cluster to Red Hat Support. You can use the oc adm must-gather tool to gather diagnostic data for resources of various types, such as OpenTelemetryCollector, Instrumentation, and the created resources like Deployment, Pod, or ConfigMap. The oc adm must-gather tool creates a new pod that collects this data.

Procedure

  • From the directory where you want to save the collected data, run the oc adm must-gather command to collect the data:

    $ oc adm must-gather --image=ghcr.io/open-telemetry/opentelemetry-operator/must-gather -- \
    /usr/bin/must-gather --operator-namespace <operator_namespace> 1
    1
    The default namespace where the Operator is installed is openshift-opentelemetry-operator.

Verification

  • Verify that the new directory is created and contains the collected data.

10.2. Getting the OpenTelemetry Collector logs

You can get the logs for the OpenTelemetry Collector as follows.

Procedure

  1. Set the relevant log level in the OpenTelemetryCollector custom resource (CR):

      config: |
        service:
          telemetry:
            logs:
              level: debug 1
    1
    Collector’s log level. Supported values include info, warn, error, or debug. Defaults to info.
  2. Use the oc logs command or the web console to retrieve the logs.

10.3. Exposing the metrics

The OpenTelemetry Collector exposes the metrics about the data volumes it has processed. The following metrics are for spans, although similar metrics are exposed for metrics and logs signals:

otelcol_receiver_accepted_spans
The number of spans successfully pushed into the pipeline.
otelcol_receiver_refused_spans
The number of spans that could not be pushed into the pipeline.
otelcol_exporter_sent_spans
The number of spans successfully sent to the destination.
otelcol_exporter_enqueue_failed_spans
The number of spans failed to be added to the sending queue.

The Operator creates a <cr_name>-collector-monitoring telemetry service that you can use to scrape the metrics endpoint.

Procedure

  1. Enable the telemetry service by adding the following lines in the OpenTelemetryCollector custom resource (CR):

    # ...
      config: |
        service:
          telemetry:
            metrics:
              address: ":8888" 1
    # ...
    1
    The address at which the internal collector metrics are exposed. Defaults to :8888.
  2. Retrieve the metrics by running the following command, which uses the port-forwarding Collector pod:

    $ oc port-forward <collector_pod>
  3. In the OpenTelemetryCollector CR, set the enableMetrics field to true to scrape internal metrics:

    apiVersion: opentelemetry.io/v1alpha1
    kind: OpenTelemetryCollector
    spec:
    # ...
      mode: deployment
      observability:
        metrics:
          enableMetrics: true
    # ...

    Depending on the deployment mode of the OpenTelemetry Collector, the internal metrics are scraped by using PodMonitors or ServiceMonitors.

    Note

    Alternatively, if you do not set the enableMetrics field to true, you can access the metrics endpoint at http://localhost:8888/metrics.

  4. On the Observe page in the web console, enable User Workload Monitoring to visualize the scraped metrics.

    Note

    Not all processors expose the required metrics.

  5. In the web console, go to Observe Dashboards and select the OpenTelemetry Collector dashboard from the drop-down list to view it.

    Tip

    You can filter the visualized data such as spans or metrics by the Collector instance, namespace, or OpenTelemetry components such as processors, receivers, or exporters.

10.4. Debug exporter

You can configure the debug exporter to export the collected data to the standard output.

Procedure

  1. Configure the OpenTelemetryCollector custom resource as follows:

      config: |
        exporters:
          debug:
            verbosity: detailed
        service:
          pipelines:
            traces:
              exporters: [debug]
            metrics:
              exporters: [debug]
            logs:
              exporters: [debug]
  2. Use the oc logs command or the web console to export the logs to the standard output.

10.5. Using the Network Observability Operator for troubleshooting

You can debug the traffic between your observability components by visualizing it with the Network Observability Operator.

Prerequisites

  • You have installed the Network Observability Operator as explained in "Installing the Network Observability Operator".

Procedure

  1. In the OpenShift Container Platform web console, go to Observe Network Traffic Topology.
  2. Select Namespace to filter the workloads by the namespace in which your OpenTelemetry Collector is deployed.
  3. Use the network traffic visuals to troubleshoot possible issues. See "Observing the network traffic from the Topology view" for more details.
Red Hat logoGithubRedditYoutubeTwitter

Aprender

Experimente, compre e venda

Comunidades

Sobre a documentação da Red Hat

Ajudamos os usuários da Red Hat a inovar e atingir seus objetivos com nossos produtos e serviços com conteúdo em que podem confiar.

Tornando o open source mais inclusivo

A Red Hat está comprometida em substituir a linguagem problemática em nosso código, documentação e propriedades da web. Para mais detalhes veja oBlog da Red Hat.

Sobre a Red Hat

Fornecemos soluções robustas que facilitam o trabalho das empresas em plataformas e ambientes, desde o data center principal até a borda da rede.

© 2024 Red Hat, Inc.