Chapter 4. Configuring the instrumentation

The Red Hat build of OpenTelemetry Operator uses a custom resource definition (CRD) file that defines the configuration of the instrumentation.

4.1. OpenTelemetry instrumentation configuration options

The Red Hat build of OpenTelemetry can inject and configure the OpenTelemetry auto-instrumentation libraries into your workloads. Currently, the project supports injection of the instrumentation libraries from Go, Java, Node.js, Python, .NET, and the Apache HTTP Server (httpd).

Auto-instrumentation in OpenTelemetry refers to the capability where the framework automatically instruments an application without manual code changes. This enables developers and administrators to get observability into their applications with minimal effort and changes to the existing codebase.

Important

The Red Hat build of OpenTelemetry Operator only supports the injection mechanism of the instrumentation libraries but does not support instrumentation libraries or upstream images. Customers can build their own instrumentation images or use community images.

4.1.1. Instrumentation options

Instrumentation options are specified in an Instrumentation custom resource (CR).

Sample Instrumentation CR

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: java-instrumentation
spec:
  env:
    - name: OTEL_EXPORTER_OTLP_TIMEOUT
      value: "20"
  exporter:
    endpoint: http://production-collector.observability.svc.cluster.local:4317
  propagators:
    - w3c
  sampler:
    type: parentbased_traceidratio
    argument: "0.25"
  java:
    env:
    - name: OTEL_JAVAAGENT_DEBUG
      value: "true"

Table 4.1. Parameters used by the Operator to define the Instrumentation
Parameter	Description	Values
`env`	Common environment variables to define across all the instrumentations.
`exporter`	Exporter configuration.
`propagators`	Propagators defines inter-process context propagation configuration.	`tracecontext`, `baggage`, `b3`, `b3multi`, `jaeger`, `ottrace`, `none`
`resource`	Resource attributes configuration.
`sampler`	Sampling configuration.
`apacheHttpd`	Configuration for the Apache HTTP Server instrumentation.
`dotnet`	Configuration for the .NET instrumentation.
`go`	Configuration for the Go instrumentation.
`java`	Configuration for the Java instrumentation.
`nodejs`	Configuration for the Node.js instrumentation.
`python`	Configuration for the Python instrumentation.

Table 4.2. Default protocol for auto-instrumentation
Auto-instrumentation	Default protocol
Java 1.x	`otlp/grpc`
Java 2.x	`otlp/http`
Python	`otlp/http`
.NET	`otlp/http`
Go	`otlp/http`
Apache HTTP Server	`otlp/grpc`

4.1.2. Configuration of the OpenTelemetry SDK variables

You can use the instrumentation.opentelemetry.io/inject-sdk annotation in the OpenTelemetry Collector custom resource to instruct the Red Hat build of OpenTelemetry Operator to inject some of the following OpenTelemetry SDK environment variables, depending on the Instrumentation CR, into your pod:

OTEL_SERVICE_NAME
OTEL_TRACES_SAMPLER
OTEL_TRACES_SAMPLER_ARG
OTEL_PROPAGATORS
OTEL_RESOURCE_ATTRIBUTES
OTEL_EXPORTER_OTLP_ENDPOINT
OTEL_EXPORTER_OTLP_CERTIFICATE
OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE
OTEL_EXPORTER_OTLP_CLIENT_KEY

Table 4.3. Values for the instrumentation.opentelemetry.io/inject-sdk annotation
Value	Description
`"true"`	Injects the `Instrumentation` resource with the default name from the current namespace.
`"false"`	Injects no `Instrumentation` resource.
`"<instrumentation_name>"`	Specifies the name of the `Instrumentation` resource to inject from the current namespace.
`"<namespace>/<instrumentation_name>"`	Specifies the name of the `Instrumentation` resource to inject from another namespace.

4.1.3. Exporter configuration

Although the Instrumentation custom resource supports setting up one or more exporters per signal, auto-instrumentation configures only the OTLP Exporter. So you must configure the endpoint to point to the OTLP Receiver on the Collector.

Sample exporter TLS CA configuration using a config map

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
# ...
spec
# ...
  exporter:
    endpoint: https://production-collector.observability.svc.cluster.local:4317  1
    tls:
      configMapName: ca-bundle  2
      ca_file: service-ca.crt 3
# ...

1: Specifies the OTLP endpoint using the HTTPS scheme and TLS.
2: Specifies the name of the config map. The config map must already exist in the namespace of the pod injecting the auto-instrumentation.
3: Points to the CA certificate in the config map or the absolute path to the certificate if the certificate is already present in the workload file system.

Sample exporter mTLS configuration using a Secret

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
# ...
spec
# ...
  exporter:
    endpoint: https://production-collector.observability.svc.cluster.local:4317  1
    tls:
      secretName: serving-certs 2
      ca_file: service-ca.crt 3
      cert_file: tls.crt 4
      key_file: tls.key 5
# ...

1: Specifies the OTLP endpoint using the HTTPS scheme and TLS.
2: Specifies the name of the Secret for the ca_file, cert_file, and key_file values. The Secret must already exist in the namespace of the pod injecting the auto-instrumentation.
3: Points to the CA certificate in the Secret or the absolute path to the certificate if the certificate is already present in the workload file system.
4: Points to the client certificate in the Secret or the absolute path to the certificate if the certificate is already present in the workload file system.
5: Points to the client key in the Secret or the absolute path to a key if the key is already present in the workload file system.

Note

You can provide the CA certificate in a config map or Secret. If you provide it in both, the config map takes higher precedence than the Secret.

Example configuration for CA bundle injection by using a config map and Instrumentation CR

apiVersion: v1
kind: ConfigMap
metadata:
  name: otelcol-cabundle
  namespace: tutorial-application
  annotations:
    service.beta.openshift.io/inject-cabundle: "true"
# ...
---
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: my-instrumentation
spec:
  exporter:
    endpoint: https://simplest-collector.tracing-system.svc.cluster.local:4317
    tls:
      configMapName: otelcol-cabundle
      ca: service-ca.crt
# ...

4.1.4. Configuration of the Apache HTTP Server auto-instrumentation

Important

The Apache HTTP Server auto-instrumentation is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

Table 4.4. Parameters for the .spec.apacheHttpd field
Name	Description	Default
`attrs`	Attributes specific to the Apache HTTP Server.
`configPath`	Location of the Apache HTTP Server configuration.	`/usr/local/apache2/conf`
`env`	Environment variables specific to the Apache HTTP Server.
`image`	Container image with the Apache SDK and auto-instrumentation.
`resourceRequirements`	The compute resource requirements.
`version`	Apache HTTP Server version.	2.4

The PodSpec annotation to enable injection

instrumentation.opentelemetry.io/inject-apache-httpd: "true"

4.1.5. Configuration of the .NET auto-instrumentation

Important

The .NET auto-instrumentation is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

Important

By default, this feature injects unsupported, upstream instrumentation libraries.

Name	Description
`env`	Environment variables specific to .NET.
`image`	Container image with the .NET SDK and auto-instrumentation.
`resourceRequirements`	The compute resource requirements.

For the .NET auto-instrumentation, the required OTEL_EXPORTER_OTLP_ENDPOINT environment variable must be set if the endpoint of the exporters is set to 4317. The .NET autoinstrumentation uses http/proto by default, and the telemetry data must be set to the 4318 port.

The PodSpec annotation to enable injection

instrumentation.opentelemetry.io/inject-dotnet: "true"

4.1.6. Configuration of the Go auto-instrumentation

Important

The Go auto-instrumentation is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

Important

By default, this feature injects unsupported, upstream instrumentation libraries.

Name	Description
`env`	Environment variables specific to Go.
`image`	Container image with the Go SDK and auto-instrumentation.
`resourceRequirements`	The compute resource requirements.

The PodSpec annotation to enable injection

instrumentation.opentelemetry.io/inject-go: "true"

Additional permissions required for the Go auto-instrumentation in the OpenShift cluster

apiVersion: security.openshift.io/v1
kind: SecurityContextConstraints
metadata:
  name: otel-go-instrumentation-scc
allowHostDirVolumePlugin: true
allowPrivilegeEscalation: true
allowPrivilegedContainer: true
allowedCapabilities:
- "SYS_PTRACE"
fsGroup:
  type: RunAsAny
runAsUser:
  type: RunAsAny
seLinuxContext:
  type: RunAsAny
seccompProfiles:
- '*'
supplementalGroups:
  type: RunAsAny

Tip

The CLI command for applying the permissions for the Go auto-instrumentation in the OpenShift cluster is as follows:

$ oc adm policy add-scc-to-user otel-go-instrumentation-scc -z <service_account>

4.1.7. Configuration of the Java auto-instrumentation

Important

The Java auto-instrumentation is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

Important

By default, this feature injects unsupported, upstream instrumentation libraries.

Name	Description
`env`	Environment variables specific to Java.
`image`	Container image with the Java SDK and auto-instrumentation.
`resourceRequirements`	The compute resource requirements.

The PodSpec annotation to enable injection

instrumentation.opentelemetry.io/inject-java: "true"

4.1.8. Configuration of the Node.js auto-instrumentation

Important

The Node.js auto-instrumentation is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

Important

By default, this feature injects unsupported, upstream instrumentation libraries.

Name	Description
`env`	Environment variables specific to Node.js.
`image`	Container image with the Node.js SDK and auto-instrumentation.
`resourceRequirements`	The compute resource requirements.

The PodSpec annotations to enable injection

instrumentation.opentelemetry.io/inject-nodejs: "true"
instrumentation.opentelemetry.io/otel-go-auto-target-exe: "/path/to/container/executable"

The instrumentation.opentelemetry.io/otel-go-auto-target-exe annotation sets the value for the required OTEL_GO_AUTO_TARGET_EXE environment variable.

4.1.9. Configuration of the Python auto-instrumentation

Important

The Python auto-instrumentation is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

Important

By default, this feature injects unsupported, upstream instrumentation libraries.

Name	Description
`env`	Environment variables specific to Python.
`image`	Container image with the Python SDK and auto-instrumentation.
`resourceRequirements`	The compute resource requirements.

For Python auto-instrumentation, the OTEL_EXPORTER_OTLP_ENDPOINT environment variable must be set if the endpoint of the exporters is set to 4317. Python auto-instrumentation uses http/proto by default, and the telemetry data must be set to the 4318 port.

The PodSpec annotation to enable injection

instrumentation.opentelemetry.io/inject-python: "true"

4.1.10. Multi-container pods

The instrumentation is run on the first container that is available by default according to the pod specification. In some cases, you can also specify target containers for injection.

Pod annotation

instrumentation.opentelemetry.io/container-names: "<container_1>,<container_2>"

Note

The Go auto-instrumentation does not support multi-container auto-instrumentation injection.

4.1.11. Multi-container pods with multiple instrumentations

Injecting instrumentation for an application language to one or more containers in a multi-container pod requires the following annotation:

instrumentation.opentelemetry.io/<application_language>-container-names: "<container_1>,<container_2>" 1

1: You can inject instrumentation for only one language per container. For the list of supported <application_language> values, see the following table.

Table 4.5. Supported values for the <application_language>
Language	Value for `<application_language>`
ApacheHTTPD	`apache`
DotNet	`dotnet`
Java	`java`
NGINX	`inject-nginx`
NodeJS	`nodejs`
Python	`python`
SDK	`sdk`

4.1.12. Using the instrumentation CR with Service Mesh

When using the instrumentation custom resource (CR) with Red Hat OpenShift Service Mesh, you must use the b3multi propagator.

Chapter 4. Configuring the instrumentation

4.1. OpenTelemetry instrumentation configuration options

4.1.1. Instrumentation options

4.1.2. Configuration of the OpenTelemetry SDK variables

4.1.3. Exporter configuration

4.1.4. Configuration of the Apache HTTP Server auto-instrumentation

4.1.5. Configuration of the .NET auto-instrumentation

4.1.6. Configuration of the Go auto-instrumentation

4.1.7. Configuration of the Java auto-instrumentation

4.1.8. Configuration of the Node.js auto-instrumentation

4.1.9. Configuration of the Python auto-instrumentation

4.1.10. Multi-container pods

4.1.11. Multi-container pods with multiple instrumentations

4.1.12. Using the instrumentation CR with Service Mesh

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

Making open source more inclusive

About Red Hat

Red Hat legal and privacy links

Red Hat legal and privacy links