Chapter 8. Analyzing outliers and errors with exemplars

Enable exemplar storage in your monitoring system.

For Prometheus, this is a preview feature that you need to enable.

Scrape the metrics using the OpenMetricsText1.0.0 protocol, which is not enabled by default in Prometheus.

If you are using PodMonitors or similar in a Kubernetes environment, this can be achieved by adding it to the spec of the custom resource:

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  ...
spec:
  scrapeProtocols:
    - OpenMetricsText1.0.0

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  ...
spec:
  scrapeProtocols:
    - OpenMetricsText1.0.0

Configure your metrics datasource where to link to for traces.

When using Grafana and Prometheus, this would be setting up a exemplarTraceIdDestinations for the Prometheus datasource, which then points to your tracing datasource that is provided by tools like Jaeger or Tempo.

Enable exemplars in your dashboards.

Enable the Exemplars toggle in each query on each dashboard where you want to show exemplars. When set up correctly, you will notice little dots or stars in your dashboards that you can click on to view the traces.

Scrape the metrics with a command similar to the following and search for those metrics that have an exemplar set:

curl -s http://localhost:9000/metrics \
-H 'Accept: application/openmetrics-text; version=1.0.0; charset=utf-8' \
| grep "#.*trace_id"

$ curl -s http://localhost:9000/metrics \
-H 'Accept: application/openmetrics-text; version=1.0.0; charset=utf-8' \
| grep "#.*trace_id"

This should result in an output similar to the following. Note the additional # after which the span and trace IDs are added:

http_server_requests_seconds_count {...} ... # {span_id="...",trace_id="..."} ...

http_server_requests_seconds_count {...} ... # {span_id="...",trace_id="..."} ...