Chapter 5. Sending traces and metrics to the OpenTelemetry Collector

Procedure

1. Create a project for an OpenTelemetry Collector instance.

   apiVersion: project.openshift.io/v1
   kind: Project
   metadata:
     name: observability

2. Create a service account.

   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: otel-collector-sidecar
     namespace: observability

3. Grant the permissions to the service account for the k8sattributes and resourcedetection processors.

   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRole
   metadata:
     name: otel-collector
   rules:
   - apiGroups: ["", "config.openshift.io"]
     resources: ["pods", "namespaces", "infrastructures", "infrastructures/status"]
     verbs: ["get", "watch", "list"]
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRoleBinding
   metadata:
     name: otel-collector
   subjects:
   - kind: ServiceAccount
     name: otel-collector-sidecar
     namespace: observability
   roleRef:
     kind: ClusterRole
     name: otel-collector
     apiGroup: rbac.authorization.k8s.io

4. Deploy the OpenTelemetry Collector as a sidecar.

   apiVersion: opentelemetry.io/v1beta1
   kind: OpenTelemetryCollector
   metadata:
     name: otel
     namespace: observability
   spec:
     serviceAccount: otel-collector-sidecar
     mode: sidecar
     config:
       serviceAccount: otel-collector-sidecar
       receivers:
         otlp:
           protocols:
             grpc: {}
             http: {}
       processors:
         batch: {}
         memory_limiter:
           check_interval: 1s
           limit_percentage: 50
           spike_limit_percentage: 30
         resourcedetection:
           detectors: [openshift]
           timeout: 2s
       exporters:
         otlp:
           endpoint: "tempo-<example>-gateway:8090" 1
           tls:
             insecure: true
       service:
         pipelines:
           traces:
             receivers: [otlp]
             processors: [memory_limiter, resourcedetection, batch]
             exporters: [otlp]

   1

   This points to the Gateway of the TempoStack instance deployed by using the <example> Tempo Operator.

5. Create your deployment using the otel-collector-sidecar service account.
6. Add the sidecar.opentelemetry.io/inject: "true" annotation to your Deployment object. This will inject all the needed environment variables to send data from your workloads to the OpenTelemetry Collector instance.

Procedure

1. Create a project for an OpenTelemetry Collector instance.

   apiVersion: project.openshift.io/v1
   kind: Project
   metadata:
     name: observability

2. Create a service account.

   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: otel-collector-deployment
     namespace: observability

3. Grant the permissions to the service account for the k8sattributes and resourcedetection processors.

   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRole
   metadata:
     name: otel-collector
   rules:
   - apiGroups: ["", "config.openshift.io"]
     resources: ["pods", "namespaces", "infrastructures", "infrastructures/status"]
     verbs: ["get", "watch", "list"]
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRoleBinding
   metadata:
     name: otel-collector
   subjects:
   - kind: ServiceAccount
     name: otel-collector-deployment
     namespace: observability
   roleRef:
     kind: ClusterRole
     name: otel-collector
     apiGroup: rbac.authorization.k8s.io

4. Deploy the OpenTelemetry Collector instance with the OpenTelemetryCollector custom resource.

   apiVersion: opentelemetry.io/v1beta1
   kind: OpenTelemetryCollector
   metadata:
     name: otel
     namespace: observability
   spec:
     mode: deployment
     serviceAccount: otel-collector-deployment
     config:
       receivers:
         jaeger:
           protocols:
             grpc: {}
             thrift_binary: {}
             thrift_compact: {}
             thrift_http: {}
         opencensus:
         otlp:
           protocols:
             grpc: {}
             http: {}
         zipkin: {}
       processors:
         batch: {}
         k8sattributes: {}
         memory_limiter:
           check_interval: 1s
           limit_percentage: 50
           spike_limit_percentage: 30
         resourcedetection:
           detectors: [openshift]
       exporters:
         otlp:
           endpoint: "tempo-<example>-distributor:4317" 1
           tls:
             insecure: true
       service:
         pipelines:
           traces:
             receivers: [jaeger, opencensus, otlp, zipkin]
             processors: [memory_limiter, k8sattributes, resourcedetection, batch]
             exporters: [otlp]

   1

   This points to the Gateway of the TempoStack instance deployed by using the <example> Tempo Operator.

5. Set the environment variables in the container with your instrumented application.

   Name	Description	Default value
   OTEL_SERVICE_NAME	Sets the value of the service.name resource attribute.	""
   OTEL_EXPORTER_OTLP_ENDPOINT	Base endpoint URL for any signal type with an optionally specified port number.	https://localhost:4317
   OTEL_EXPORTER_OTLP_CERTIFICATE	Path to the certificate file for the TLS credentials of the gRPC client.	https://localhost:4317
   OTEL_TRACES_SAMPLER	Sampler to be used for traces.	parentbased_always_on
   OTEL_EXPORTER_OTLP_PROTOCOL	Transport protocol for the OTLP exporter.	grpc
   OTEL_EXPORTER_OTLP_TIMEOUT	Maximum time interval for the OTLP exporter to wait for each batch export.	10s
   OTEL_EXPORTER_OTLP_INSECURE	Disables client transport security for gRPC requests. An HTTPS schema overrides it.	False