Ce contenu n'est pas disponible dans la langue sélectionnée.
Chapter 3. Configuring the Collector
3.1. Receivers
Receivers get data into the Collector. A receiver can be push or pull based. Generally, a receiver accepts data in a specified format, translates it into the internal format, and passes it to processors and exporters defined in the applicable pipelines. By default, no receivers are configured. One or more receivers must be configured. Receivers may support one or more data sources.
3.1.1. OTLP Receiver
The OTLP Receiver ingests traces, metrics, and logs by using the OpenTelemetry Protocol (OTLP). The OTLP Receiver ingests traces and metrics using the OpenTelemetry protocol (OTLP).
OpenTelemetry Collector custom resource with an enabled OTLP Receiver
# ... config: | receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 1 tls: 2 ca_file: ca.pem cert_file: cert.pem key_file: key.pem client_ca_file: client.pem 3 reload_interval: 1h 4 http: endpoint: 0.0.0.0:4318 5 tls: 6 service: pipelines: traces: receivers: [otlp] metrics: receivers: [otlp] # ...
- 1
- The OTLP gRPC endpoint. If omitted, the default
0.0.0.0:4317
is used. - 2
- The server-side TLS configuration. Defines paths to TLS certificates. If omitted, the TLS is disabled.
- 3
- The path to the TLS certificate at which the server verifies a client certificate. This sets the value of
ClientCAs
andClientAuth
toRequireAndVerifyClientCert
in theTLSConfig
. For more information, see theConfig
of the Golang TLS package. - 4
- Specifies the time interval at which the certificate is reloaded. If the value is not set, the certificate is never reloaded. The
reload_interval
field accepts a string containing valid units of time such asns
,us
(orµs
),ms
,s
,m
,h
. - 5
- The OTLP HTTP endpoint. The default value is
0.0.0.0:4318
. - 6
- The server-side TLS configuration. For more information, see the
grpc
protocol configuration section.
3.1.2. Jaeger Receiver
The Jaeger Receiver ingests traces in the Jaeger formats.
OpenTelemetry Collector custom resource with an enabled Jaeger Receiver
# ... config: | receivers: jaeger: protocols: grpc: endpoint: 0.0.0.0:14250 1 thrift_http: endpoint: 0.0.0.0:14268 2 thrift_compact: endpoint: 0.0.0.0:6831 3 thrift_binary: endpoint: 0.0.0.0:6832 4 tls: 5 service: pipelines: traces: receivers: [jaeger] # ...
- 1
- The Jaeger gRPC endpoint. If omitted, the default
0.0.0.0:14250
is used. - 2
- The Jaeger Thrift HTTP endpoint. If omitted, the default
0.0.0.0:14268
is used. - 3
- The Jaeger Thrift Compact endpoint. If omitted, the default
0.0.0.0:6831
is used. - 4
- The Jaeger Thrift Binary endpoint. If omitted, the default
0.0.0.0:6832
is used. - 5
- The server-side TLS configuration. See the OTLP Receiver configuration section for more details.
3.1.3. Host Metrics Receiver
The Host Metrics Receiver ingests metrics in the OTLP format.
The Host Metrics Receiver is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
OpenTelemetry Collector custom resource with an enabled Host Metrics Receiver
apiVersion: v1 kind: ServiceAccount metadata: name: otel-hostfs-daemonset namespace: <namespace> # ... --- apiVersion: security.openshift.io/v1 kind: SecurityContextConstraints allowHostDirVolumePlugin: true allowHostIPC: false allowHostNetwork: false allowHostPID: true allowHostPorts: false allowPrivilegeEscalation: true allowPrivilegedContainer: true allowedCapabilities: null defaultAddCapabilities: - SYS_ADMIN fsGroup: type: RunAsAny groups: [] metadata: name: otel-hostmetrics readOnlyRootFilesystem: true runAsUser: type: RunAsAny seLinuxContext: type: RunAsAny supplementalGroups: type: RunAsAny users: - system:serviceaccount:<namespace>:otel-hostfs-daemonset volumes: - configMap - emptyDir - hostPath - projected # ... --- apiVersion: opentelemetry.io/v1alpha1 kind: OpenTelemetryCollector metadata: name: otel namespace: <namespace> spec: serviceAccount: otel-hostfs-daemonset mode: daemonset volumeMounts: - mountPath: /hostfs name: host readOnly: true volumes: - hostPath: path: / name: host config: | receivers: hostmetrics: collection_interval: 10s 1 initial_delay: 1s 2 root_path: / 3 scrapers: 4 cpu: memory: disk: service: pipelines: metrics: receivers: [hostmetrics] # ...
- 1
- Sets the time interval for host metrics collection. If omitted, the default value is
1m
. - 2
- Sets the initial time delay for host metrics collection. If omitted, the default value is
1s
. - 3
- Configures the
root_path
so that the Host Metrics Receiver knows where the root filesystem is. If running multiple instances of the Host Metrics Receiver, set the sameroot_path
value for each instance. - 4
- Lists the enabled host metrics scrapers. Available scrapers are
cpu
,disk
,load
,filesystem
,memory
,network
,paging
,processes
, andprocess
.
3.1.4. Kubernetes Objects Receiver
The Kubernetes Objects Receiver pulls or watches objects to be collected from the Kubernetes API server. This receiver watches primarily Kubernetes events, but it can collect any type of Kubernetes objects. This receiver gathers telemetry for the cluster as a whole, so only one instance of this receiver suffices for collecting all the data.
The Kubernetes Objects Receiver is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
OpenTelemetry Collector custom resource with an enabled Kubernetes Objects Receiver
apiVersion: v1 kind: ServiceAccount metadata: name: otel-k8sobj namespace: <namespace> # ... --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: otel-k8sobj namespace: <namespace> rules: - apiGroups: - "" resources: - events - pods verbs: - get - list - watch - apiGroups: - "events.k8s.io" resources: - events verbs: - watch - list # ... --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: otel-k8sobj subjects: - kind: ServiceAccount name: otel-k8sobj namespace: <namespace> roleRef: kind: ClusterRole name: otel-k8sobj apiGroup: rbac.authorization.k8s.io # ... --- apiVersion: opentelemetry.io/v1alpha1 kind: OpenTelemetryCollector metadata: name: otel-k8s-obj namespace: <namespace> spec: serviceAccount: otel-k8sobj image: ghcr.io/os-observability/redhat-opentelemetry-collector/redhat-opentelemetry-collector:main mode: deployment config: | receivers: k8sobjects: auth_type: serviceAccount objects: - name: pods 1 mode: pull 2 interval: 30s 3 label_selector: 4 field_selector: 5 namespaces: [<namespace>,...] 6 - name: events mode: watch exporters: debug: service: pipelines: logs: receivers: [k8sobjects] exporters: [debug] # ...
- 1
- The Resource name that this receiver observes: for example,
pods
,deployments
, orevents
. - 2
- The observation mode that this receiver uses:
pull
orwatch
. - 3
- Only applicable to the pull mode. The request interval for pulling an object. If omitted, the default value is
1h
. - 4
- The label selector to define targets.
- 5
- The field selector to filter targets.
- 6
- The list of namespaces to collect events from. If omitted, the default value is
all
.
3.1.5. Kubelet Stats Receiver
The Kubelet Stats Receiver extracts metrics related to nodes, pods, containers, and volumes from the kubelet’s API server. These metrics are then channeled through the metrics-processing pipeline for additional analysis.
The Kubelet Stats Receiver is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
OpenTelemetry Collector custom resource with an enabled Kubelet Stats Receiver
# ...
config: |
receivers:
kubeletstats:
collection_interval: 20s
auth_type: "serviceAccount"
endpoint: "https://${env:K8S_NODE_NAME}:10250"
insecure_skip_verify: true
service:
pipelines:
metrics:
receivers: [kubeletstats]
env:
- name: K8S_NODE_NAME 1
valueFrom:
fieldRef:
fieldPath: spec.nodeName
# ...
- 1
- Sets the
K8S_NODE_NAME
to authenticate to the API.
The Kubelet Stats Receiver requires additional permissions for the service account used for running the OpenTelemetry Collector.
Permissions required by the service account
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: otel-collector
rules:
- apiGroups: ['']
resources: ['nodes/stats']
verbs: ['get', 'watch', 'list']
- apiGroups: [""]
resources: ["nodes/proxy"] 1
verbs: ["get"]
# ...
- 1
- The permissions required when using the
extra_metadata_labels
orrequest_utilization
orlimit_utilization
metrics.
3.1.6. Prometheus Receiver
The Prometheus Receiver scrapes the metrics endpoints.
The Prometheus Receiver is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
OpenTelemetry Collector custom resource with an enabled Prometheus Receiver
# ... config: | receivers: prometheus: config: scrape_configs: 1 - job_name: 'my-app' 2 scrape_interval: 5s 3 static_configs: - targets: ['my-app.example.svc.cluster.local:8888'] 4 service: pipelines: metrics: receivers: [prometheus] # ...
- 1
- Scrapes configurations using the Prometheus format.
- 2
- The Prometheus job name.
- 3
- The lnterval for scraping the metrics data. Accepts time units. The default value is
1m
. - 4
- The targets at which the metrics are exposed. This example scrapes the metrics from a
my-app
application in theexample
project.
3.1.7. Zipkin Receiver
The Zipkin Receiver ingests traces in the Zipkin v1 and v2 formats.
OpenTelemetry Collector custom resource with the enabled Zipkin Receiver
# ... config: | receivers: zipkin: endpoint: 0.0.0.0:9411 1 tls: 2 service: pipelines: traces: receivers: [zipkin] # ...
3.1.8. Kafka Receiver
The Kafka Receiver receives traces, metrics, and logs from Kafka in the OTLP format.
The Kafka Receiver is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
OpenTelemetry Collector custom resource with the enabled Kafka Receiver
# ... config: | receivers: kafka: brokers: ["localhost:9092"] 1 protocol_version: 2.0.0 2 topic: otlp_spans 3 auth: plain_text: 4 username: example password: example tls: 5 ca_file: ca.pem cert_file: cert.pem key_file: key.pem insecure: false 6 server_name_override: kafka.example.corp 7 service: pipelines: traces: receivers: [kafka] # ...
- 1
- The list of Kafka brokers. The default is
localhost:9092
. - 2
- The Kafka protocol version. For example,
2.0.0
. This is a required field. - 3
- The name of the Kafka topic to read from. The default is
otlp_spans
. - 4
- The plain text authentication configuration. If omitted, plain text authentication is disabled.
- 5
- The client-side TLS configuration. Defines paths to the TLS certificates. If omitted, TLS authentication is disabled.
- 6
- Disables verifying the server’s certificate chain and host name. The default is
false
. - 7
- ServerName indicates the name of the server requested by the client to support virtual hosting.
3.1.9. Kubernetes Cluster Receiver
The Kubernetes Cluster Receiver gathers cluster metrics and entity events from the Kubernetes API server. It uses the Kubernetes API to receive information about updates. Authentication for this receiver is only supported through service accounts.
The Kubernetes Cluster Receiver is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
OpenTelemetry Collector custom resource with the enabled Kubernetes Cluster Receiver
# ... receivers: k8s_cluster: distribution: openshift collection_interval: 10s exporters: debug: service: pipelines: metrics: receivers: [k8s_cluster] exporters: [debug] logs/entity_events: receivers: [k8s_cluster] exporters: [debug] # ...
This receiver requires a configured service account, RBAC rules for the cluster role, and the cluster role binding that binds the RBAC with the service account.
ServiceAccount
object
apiVersion: v1 kind: ServiceAccount metadata: labels: app: otelcontribcol name: otelcontribcol # ...
RBAC rules for the ClusterRole
object
apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: otelcontribcol labels: app: otelcontribcol rules: - apiGroups: - quota.openshift.io resources: - clusterresourcequotas verbs: - get - list - watch - apiGroups: - "" resources: - events - namespaces - namespaces/status - nodes - nodes/spec - pods - pods/status - replicationcontrollers - replicationcontrollers/status - resourcequotas - services verbs: - get - list - watch - apiGroups: - apps resources: - daemonsets - deployments - replicasets - statefulsets verbs: - get - list - watch - apiGroups: - extensions resources: - daemonsets - deployments - replicasets verbs: - get - list - watch - apiGroups: - batch resources: - jobs - cronjobs verbs: - get - list - watch - apiGroups: - autoscaling resources: - horizontalpodautoscalers verbs: - get - list - watch # ...
ClusterRoleBinding
object
apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: otelcontribcol labels: app: otelcontribcol roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: otelcontribcol subjects: - kind: ServiceAccount name: otelcontribcol namespace: default # ...
3.1.10. OpenCensus Receiver
The OpenCensus Receiver provides backwards compatibility with the OpenCensus project for easier migration of instrumented codebases. It receives metrics and traces in the OpenCensus format via gRPC or HTTP and Json.
OpenTelemetry Collector custom resource with the enabled OpenCensus Receiver
# ... config: | receivers: opencensus: endpoint: 0.0.0.0:9411 1 tls: 2 cors_allowed_origins: 3 - https://*.<example>.com service: pipelines: traces: receivers: [opencensus] # ...
- 1
- The OpenCensus endpoint. If omitted, the default is
0.0.0.0:55678
. - 2
- The server-side TLS configuration. See the OTLP Receiver configuration section for more details.
- 3
- You can also use the HTTP JSON endpoint to optionally configure CORS, which is enabled by specifying a list of allowed CORS origins in this field. Wildcards with
*
are accepted under thecors_allowed_origins
. To match any origin, enter only*
.
3.1.11. Filelog Receiver
The Filelog Receiver tails and parses logs from files.
The Filelog Receiver is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
OpenTelemetry Collector custom resource with the enabled Filelog Receiver that tails a text file
# ... receivers: filelog: include: [ /simple.log ] 1 operators: 2 - type: regex_parser regex: '^(?P<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}) (?P<sev>[A-Z]*) (?P<msg>.*)$' timestamp: parse_from: attributes.time layout: '%Y-%m-%d %H:%M:%S' severity: parse_from: attributes.sev # ...
3.1.12. Journald Receiver
The Journald Receiver parses journald events from the systemd journal and sends them as logs.
The Journald Receiver is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
OpenTelemetry Collector custom resource with the enabled Journald Receiver
apiVersion: v1 kind: Namespace metadata: name: otel-journald labels: security.openshift.io/scc.podSecurityLabelSync: "false" pod-security.kubernetes.io/enforce: "privileged" pod-security.kubernetes.io/audit: "privileged" pod-security.kubernetes.io/warn: "privileged" # ... --- apiVersion: v1 kind: ServiceAccount metadata: name: privileged-sa namespace: otel-journald # ... --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: otel-journald-binding roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:openshift:scc:privileged subjects: - kind: ServiceAccount name: privileged-sa namespace: otel-journald # ... --- apiVersion: opentelemetry.io/v1alpha1 kind: OpenTelemetryCollector metadata: name: otel-journald-logs namespace: otel-journald spec: mode: daemonset serviceAccount: privileged-sa securityContext: allowPrivilegeEscalation: false capabilities: drop: - CHOWN - DAC_OVERRIDE - FOWNER - FSETID - KILL - NET_BIND_SERVICE - SETGID - SETPCAP - SETUID readOnlyRootFilesystem: true seLinuxOptions: type: spc_t seccompProfile: type: RuntimeDefault config: | receivers: journald: files: /var/log/journal/*/* priority: info 1 units: 2 - kubelet - crio - init.scope - dnsmasq all: true 3 retry_on_failure: enabled: true 4 initial_interval: 1s 5 max_interval: 30s 6 max_elapsed_time: 5m 7 processors: exporters: debug: verbosity: detailed service: pipelines: logs: receivers: [journald] exporters: [debug] volumeMounts: - name: journal-logs mountPath: /var/log/journal/ readOnly: true volumes: - name: journal-logs hostPath: path: /var/log/journal tolerations: - key: node-role.kubernetes.io/master operator: Exists effect: NoSchedule # ...
- 1
- Filters output by message priorities or priority ranges. The default value is
info
. - 2
- Lists the units to read entries from. If empty, entries are read from all units.
- 3
- Includes very long logs and logs with unprintable characters. The default value is
false
. - 4
- If set to
true
, the receiver pauses reading a file and attempts to resend the current batch of logs when encountering an error from downstream components. The default value isfalse
. - 5
- The time interval to wait after the first failure before retrying. The default value is
1s
. The units arems
,s
,m
,h
. - 6
- The upper bound for the retry backoff interval. When this value is reached, the time interval between consecutive retry attempts remains constant at this value. The default value is
30s
. The supported units arems
,s
,m
,h
. - 7
- The maximum time interval, including retry attempts, for attempting to send a logs batch to a downstream consumer. When this value is reached, the data are discarded. If the set value is
0
, retrying never stops. The default value is5m
. The supported units arems
,s
,m
,h
.
3.1.13. Kubernetes Events Receiver
The Kubernetes Events Receiver collects events from the Kubernetes API server. The collected events are converted into logs.
The Kubernetes Events Receiver is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
OpenShift Container Platform permissions required for the Kubernetes Events Receiver
apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: otel-collector labels: app: otel-collector rules: - apiGroups: - "" resources: - events - namespaces - namespaces/status - nodes - nodes/spec - pods - pods/status - replicationcontrollers - replicationcontrollers/status - resourcequotas - services verbs: - get - list - watch - apiGroups: - apps resources: - daemonsets - deployments - replicasets - statefulsets verbs: - get - list - watch - apiGroups: - extensions resources: - daemonsets - deployments - replicasets verbs: - get - list - watch - apiGroups: - batch resources: - jobs - cronjobs verbs: - get - list - watch - apiGroups: - autoscaling resources: - horizontalpodautoscalers verbs: - get - list - watch # ...
OpenTelemetry Collector custom resource with the enabled Kubernetes Event Receiver
# ... serviceAccount: otel-collector 1 config: | receivers: k8s_events: namespaces: [project1, project2] 2 service: pipelines: logs: receivers: [k8s_events] # ...
3.1.14. Additional resources
3.2. Processors
Processors process the data between it is received and exported. Processors are optional. By default, no processors are enabled. Processors must be enabled for every data source. Not all processors support all data sources. Depending on the data source, multiple processors might be enabled. Note that the order of processors matters.
3.2.1. Batch Processor
The Batch Processor batches traces and metrics to reduce the number of outgoing connections needed to transfer the telemetry information.
Example of the OpenTelemetry Collector custom resource when using the Batch Processor
# ... config: | processor: batch: timeout: 5s send_batch_max_size: 10000 service: pipelines: traces: processors: [batch] metrics: processors: [batch] # ...
Parameter | Description | Default |
---|---|---|
| Sends the batch after a specific time duration and irrespective of the batch size. |
|
| Sends the batch of telemetry data after the specified number of spans or metrics. |
|
|
The maximum allowable size of the batch. Must be equal or greater than the |
|
|
When activated, a batcher instance is created for each unique set of values found in the |
|
|
When the |
|
3.2.2. Memory Limiter Processor
The Memory Limiter Processor periodically checks the Collector’s memory usage and pauses data processing when the soft memory limit is reached. This processor supports traces, metrics, and logs. The preceding component, which is typically a receiver, is expected to retry sending the same data and may apply a backpressure to the incoming data. When memory usage exceeds the hard limit, the Memory Limiter Processor forces garbage collection to run.
Example of the OpenTelemetry Collector custom resource when using the Memory Limiter Processor
# ... config: | processor: memory_limiter: check_interval: 1s limit_mib: 4000 spike_limit_mib: 800 service: pipelines: traces: processors: [batch] metrics: processors: [batch] # ...
Parameter | Description | Default |
---|---|---|
|
Time between memory usage measurements. The optimal value is |
|
| The hard limit, which is the maximum amount of memory in MiB allocated on the heap. Typically, the total memory usage of the OpenTelemetry Collector is about 50 MiB greater than this value. |
|
|
Spike limit, which is the maximum expected spike of memory usage in MiB. The optimal value is approximately 20% of |
20% of |
|
Same as the |
|
|
Same as the |
|
3.2.3. Resource Detection Processor
The Resource Detection Processor identifies host resource details in alignment with OpenTelemetry’s resource semantic standards. Using the detected information, this processor can add or replace the resource values in telemetry data. This processor supports traces and metrics. You can use this processor with multiple detectors such as the Docket metadata detector or the OTEL_RESOURCE_ATTRIBUTES
environment variable detector.
The Resource Detection Processor is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
OpenShift Container Platform permissions required for the Resource Detection Processor
kind: ClusterRole metadata: name: otel-collector rules: - apiGroups: ["config.openshift.io"] resources: ["infrastructures", "infrastructures/status"] verbs: ["get", "watch", "list"] # ...
OpenTelemetry Collector using the Resource Detection Processor
# ... config: | processor: resourcedetection: detectors: [openshift] override: true service: pipelines: traces: processors: [resourcedetection] metrics: processors: [resourcedetection] # ...
OpenTelemetry Collector using the Resource Detection Processor with an environment variable detector
# ...
config: |
processors:
resourcedetection/env:
detectors: [env] 1
timeout: 2s
override: false
# ...
- 1
- Specifies which detector to use. In this example, the environment detector is specified.
3.2.4. Attributes Processor
The Attributes Processor can modify attributes of a span, log, or metric. You can configure this processor to filter and match input data and include or exclude such data for specific actions.
The Attributes Processor is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
This processor operates on a list of actions, executing them in the order specified in the configuration. The following actions are supported:
- Insert
- Inserts a new attribute into the input data when the specified key does not already exist.
- Update
- Updates an attribute in the input data if the key already exists.
- Upsert
- Combines the insert and update actions: Inserts a new attribute if the key does not exist yet. Updates the attribute if the key already exists.
- Delete
- Removes an attribute from the input data.
- Hash
- Hashes an existing attribute value as SHA1.
- Extract
-
Extracts values by using a regular expression rule from the input key to the target keys defined in the rule. If a target key already exists, it is overridden similarly to the Span Processor’s
to_attributes
setting with the existing attribute as the source. - Convert
- Converts an existing attribute to a specified type.
OpenTelemetry Collector using the Attributes Processor
# ... config: | processors: attributes/example: actions: - key: db.table action: delete - key: redacted_span value: true action: upsert - key: copy_key from_attribute: key_original action: update - key: account_id value: 2245 action: insert - key: account_password action: delete - key: account_email action: hash - key: http.status_code action: convert converted_type: int # ...
3.2.5. Resource Processor
The Resource Processor applies changes to the resource attributes. This processor supports traces, metrics, and logs.
The Resource Processor is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
OpenTelemetry Collector using the Resource Detection Processor
# ... config: | processor: attributes: - key: cloud.availability_zone value: "zone-1" action: upsert - key: k8s.cluster.name from_attribute: k8s-cluster action: insert - key: redundant-attribute action: delete # ...
Attributes represent the actions that are applied to the resource attributes, such as delete the attribute, insert the attribute, or upsert the attribute.
3.2.6. Span Processor
The Span Processor modifies the span name based on its attributes or extracts the span attributes from the span name. This processor can also change the span status and include or exclude spans. This processor supports traces.
Span renaming requires specifying attributes for the new name by using the from_attributes
configuration.
The Span Processor is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
OpenTelemetry Collector using the Span Processor for renaming a span
# ... config: | processor: span: name: from_attributes: [<key1>, <key2>, ...] 1 separator: <value> 2 # ...
You can use this processor to extract attributes from the span name.
OpenTelemetry Collector using the Span Processor for extracting attributes from a span name
# ...
config: |
processor:
span/to_attributes:
name:
to_attributes:
rules:
- ^\/api\/v1\/document\/(?P<documentId>.*)\/update$ 1
# ...
- 1
- This rule defines how the extraction is to be executed. You can define more rules: for example, in this case, if the regular expression matches the name, a
documentID
attibute is created. In this example, if the input span name is/api/v1/document/12345678/update
, this results in the/api/v1/document/{documentId}/update
output span name, and a new"documentId"="12345678"
attribute is added to the span.
You can have the span status modified.
OpenTelemetry Collector using the Span Processor for status change
# ... config: | processor: span/set_status: status: code: Error description: "<error_description>" # ...
3.2.7. Kubernetes Attributes Processor
The Kubernetes Attributes Processor enables automatic configuration of spans, metrics, and log resource attributes by using the Kubernetes metadata. This processor supports traces, metrics, and logs. This processor automatically identifies the Kubernetes resources, extracts the metadata from them, and incorporates this extracted metadata as resource attributes into relevant spans, metrics, and logs. It utilizes the Kubernetes API to discover all pods operating within a cluster, maintaining records of their IP addresses, pod UIDs, and other relevant metadata.
The Kubernetes Attributes Processor is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
Minimum OpenShift Container Platform permissions required for the Kubernetes Attributes Processor
kind: ClusterRole metadata: name: otel-collector rules: - apiGroups: [''] resources: ['pods', 'namespaces'] verbs: ['get', 'watch', 'list'] # ...
OpenTelemetry Collector using the Kubernetes Attributes Processor
# ... config: | processors: k8sattributes: filter: node_from_env_var: KUBE_NODE_NAME # ...
3.2.8. Filter Processor
The Filter Processor leverages the OpenTelemetry Transformation Language to establish criteria for discarding telemetry data. If any of these conditions are satisfied, the telemetry data are discarded. You can combine the conditions by using the logical OR operator. This processor supports traces, metrics, and logs.
The Filter Processor is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
OpenTelemetry Collector custom resource with an enabled OTLP Exporter
# ... config: | processors: filter/ottl: error_mode: ignore 1 traces: span: - 'attributes["container.name"] == "app_container_1"' 2 - 'resource.attributes["host.name"] == "localhost"' 3 # ...
- 1
- Defines the error mode. When set to
ignore
, ignores errors returned by conditions. When set topropagate
, returns the error up the pipeline. An error causes the payload to be dropped from the Collector. - 2
- Filters the spans that have the
container.name == app_container_1
attribute. - 3
- Filters the spans that have the
host.name == localhost
resource attribute.
3.2.9. Routing Processor
The Routing Processor routes logs, metrics, or traces to specific exporters. This processor can read a header from an incoming gRPC or plain HTTP request or read a resource attribute, and then direct the trace information to relevant exporters according to the read value.
The Routing Processor is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
OpenTelemetry Collector custom resource with an enabled OTLP Exporter
# ... config: | processors: routing: from_attribute: X-Tenant 1 default_exporters: 2 - jaeger table: 3 - value: acme exporters: [jaeger/acme] exporters: jaeger: endpoint: localhost:14250 jaeger/acme: endpoint: localhost:24250 # ...
Optionally, you can create an attribute_source
configuration, which defines where to look for the attribute that you specify in the from_attribute
field. The supported values are context
for searching the context including the HTTP headers, and resource
for searching the resource attributes.
3.2.10. Cumulative-to-Delta Processor
The Cumulative-to-Delta Processor processor converts monotonic, cumulative-sum, and histogram metrics to monotonic delta metrics.
You can filter metrics by using the include:
or exclude:
fields and specifying the strict
or regexp
metric name matching.
This processor does not convert non-monotonic sums and exponential histograms.
The Cumulative-to-Delta Processor is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
Example of an OpenTelemetry Collector custom resource with an enabled Cumulative-to-Delta Processor
# ... config: | processors: cumulativetodelta: include: 1 match_type: strict 2 metrics: 3 - <metric_1_name> - <metric_2_name> exclude: 4 match_type: regexp metrics: - "<regular_expression_for_metric_names>" # ...
- 1
- Optional: Configures which metrics to include. When omitted, all metrics, except for those listed in the
exclude
field, are converted to delta metrics. - 2
- Defines a value provided in the
metrics
field as astrict
exact match orregexp
regular expression. - 3
- Lists the metric names, which are exact matches or matches for regular expressions, of the metrics to be converted to delta metrics. If a metric matches both the
include
andexclude
filters, theexclude
filter takes precedence. - 4
- Optional: Configures which metrics to exclude. When omitted, no metrics are excluded from conversion to delta metrics.
3.2.11. Group-by-Attributes Processor
The Group-by-Attributes Processor groups all spans, log records, and metric datapoints that share the same attributes by reassigning them to a Resource that matches those attributes.
The Group-by-Attributes Processor is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
At minimum, configuring this processor involves specifying an array of attribute keys to be used to group spans, log records, or metric datapoints together, as in the following example:
# ... processors: groupbyattrs: keys: 1 - <key1> 2 - <key2> # ...
- 1
- Specifies attribute keys to group by.
- 2
- If a processed span, log record, or metric datapoint contains at least one of the specified attribute keys, it is reassigned to a Resource that shares the same attribute values; and if no such Resource exists, a new one is created. If none of the specified attribute keys is present in the processed span, log record, or metric datapoint, then it remains associated with its current Resource. Multiple instances of the same Resource are consolidated.
3.2.12. Transform Processor
The Transform Processor enables modification of telemetry data according to specified rules and in the OpenTelemetry Transformation Language (OTTL). For each signal type, the processor processes a series of conditions and statements associated with a specific OTTL Context type and then executes them in sequence on incoming telemetry data as specified in the configuration. Each condition and statement can access and modify telemetry data by using various functions, allowing conditions to dictate if a function is to be executed.
All statements are written in the OTTL. You can configure multiple context statements for different signals, traces, metrics, and logs. The value of the context
type specifies which OTTL Context the processor must use when interpreting the associated statements.
The Transform Processor is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
Configuration summary
# ... config: | processors: transform: error_mode: ignore 1 <trace|metric|log>_statements: 2 - context: <string> 3 conditions: 4 - <string> - <string> statements: 5 - <string> - <string> - <string> - context: <string> statements: - <string> - <string> - <string> # ...
Configuration example
# ... config: | transform: error_mode: ignore trace_statements: 1 - context: resource statements: - keep_keys(attributes, ["service.name", "service.namespace", "cloud.region", "process.command_line"]) 2 - replace_pattern(attributes["process.command_line"], "password\\=[^\\s]*(\\s?)", "password=***") 3 - limit(attributes, 100, []) - truncate_all(attributes, 4096) - context: span 4 statements: - set(status.code, 1) where attributes["http.path"] == "/health" - set(name, attributes["http.route"]) - replace_match(attributes["http.target"], "/user/*/list/*", "/user/{userId}/list/{listId}") - limit(attributes, 100, []) - truncate_all(attributes, 4096) # ...
Signal Statement | Valid Contexts |
---|---|
|
|
|
|
|
|
Value | Description |
---|---|
| Ignores and logs errors returned by statements and then continues to the next statement. |
| Ignores and doesn’t log errors returned by statements and then continues to the next statement. |
| Returns errors up the pipeline and drops the payload. Implicit default. |
3.2.13. Additional resources
3.3. Exporters
Exporters send data to one or more back ends or destinations. An exporter can be push or pull based. By default, no exporters are configured. One or more exporters must be configured. Exporters can support one or more data sources. Exporters might be used with their default settings, but many exporters require configuration to specify at least the destination and security settings.
3.3.1. OTLP Exporter
The OTLP gRPC Exporter exports traces and metrics by using the OpenTelemetry protocol (OTLP).
OpenTelemetry Collector custom resource with an enabled OTLP Exporter
# ... config: | exporters: otlp: endpoint: tempo-ingester:4317 1 tls: 2 ca_file: ca.pem cert_file: cert.pem key_file: key.pem insecure: false 3 insecure_skip_verify: false # 4 reload_interval: 1h 5 server_name_override: <name> 6 headers: 7 X-Scope-OrgID: "dev" service: pipelines: traces: exporters: [otlp] metrics: exporters: [otlp] # ...
- 1
- The OTLP gRPC endpoint. If the
https://
scheme is used, then client transport security is enabled and overrides theinsecure
setting in thetls
. - 2
- The client-side TLS configuration. Defines paths to TLS certificates.
- 3
- Disables client transport security when set to
true
. The default value isfalse
by default. - 4
- Skips verifying the certificate when set to
true
. The default value isfalse
. - 5
- Specifies the time interval at which the certificate is reloaded. If the value is not set, the certificate is never reloaded. The
reload_interval
accepts a string containing valid units of time such asns
,us
(orµs
),ms
,s
,m
,h
. - 6
- Overrides the virtual host name of authority such as the authority header field in requests. You can use this for testing.
- 7
- Headers are sent for every request performed during an established connection.
3.3.2. OTLP HTTP Exporter
The OTLP HTTP Exporter exports traces and metrics by using the OpenTelemetry protocol (OTLP).
OpenTelemetry Collector custom resource with an enabled OTLP Exporter
# ... config: | exporters: otlphttp: endpoint: http://tempo-ingester:4318 1 tls: 2 headers: 3 X-Scope-OrgID: "dev" disable_keep_alives: false 4 service: pipelines: traces: exporters: [otlphttp] metrics: exporters: [otlphttp] # ...
- 1
- The OTLP HTTP endpoint. If the
https://
scheme is used, then client transport security is enabled and overrides theinsecure
setting in thetls
. - 2
- The client side TLS configuration. Defines paths to TLS certificates.
- 3
- Headers are sent in every HTTP request.
- 4
- If true, disables HTTP keep-alives. It will only use the connection to the server for a single HTTP request.
3.3.3. Debug Exporter
The Debug Exporter prints traces and metrics to the standard output.
OpenTelemetry Collector custom resource with an enabled Debug Exporter
# ... config: | exporters: debug: verbosity: detailed 1 sampling_initial: 5 2 sampling_thereafter: 200 3 use_internal_logger: true 4 service: pipelines: traces: exporters: [debug] metrics: exporters: [debug] # ...
- 1
- Verbosity of the debug export:
detailed
,normal
, orbasic
. When set todetailed
, pipeline data are verbosely logged. Defaults tonormal
. - 2
- Initial number of messages logged per second. The default value is
2
messages per second. - 3
- Sampling rate after the initial number of messages, the value in
sampling_initial
, has been logged. Disabled by default with the default1
value. Sampling is enabled with values greater than1
. For more information, see the page for the sampler function in thezapcore
package on the Go Project’s website. - 4
- When set to
true
, enables output from the Collector’s internal logger for the exporter.
3.3.4. Load Balancing Exporter
The Load Balancing Exporter consistently exports spans, metrics, and logs according to the routing_key
configuration.
The Load Balancing Exporter is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
OpenTelemetry Collector custom resource with an enabled Load Balancing Exporter
# ... config: | exporters: loadbalancing: routing_key: "service" 1 protocol: otlp: 2 timeout: 1s resolver: 3 static: 4 hostnames: - backend-1:4317 - backend-2:4317 dns: 5 hostname: otelcol-headless.observability.svc.cluster.local k8s: 6 service: lb-svc.kube-public ports: - 15317 - 16317 # ...
- 1
- The
routing_key: service
exports spans for the same service name to the same Collector instance to provide accurate aggregation. Therouting_key: traceID
exports spans based on theirtraceID
. The implicit default istraceID
based routing. - 2
- The OTLP is the only supported load-balancing protocol. All options of the OTLP exporter are supported.
- 3
- You can configure only one resolver.
- 4
- The static resolver distributes the load across the listed endpoints.
- 5
- You can use the DNS resolver only with a Kubernetes headless service.
- 6
- The Kubernetes resolver is recommended.
3.3.5. Prometheus Exporter
The Prometheus Exporter exports metrics in the Prometheus or OpenMetrics formats.
The Prometheus Exporter is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
OpenTelemetry Collector custom resource with an enabled Prometheus Exporter
# ... config: | exporters: prometheus: endpoint: 0.0.0.0:8889 1 tls: 2 ca_file: ca.pem cert_file: cert.pem key_file: key.pem namespace: prefix 3 const_labels: 4 label1: value1 enable_open_metrics: true 5 resource_to_telemetry_conversion: 6 enabled: true metric_expiration: 180m 7 add_metric_suffixes: false 8 service: pipelines: metrics: exporters: [prometheus] # ...
- 1
- The network endpoint where the metrics are exposed. The Red Hat build of OpenTelemetry Operator automatically exposes the port specified in the
endpoint
field to the<instance_name>-collector
service. - 2
- The server-side TLS configuration. Defines paths to TLS certificates.
- 3
- If set, exports metrics under the provided value.
- 4
- Key-value pair labels that are applied for every exported metric.
- 5
- If
true
, metrics are exported by using the OpenMetrics format. Exemplars are only exported in the OpenMetrics format and only for histogram and monotonic sum metrics such ascounter
. Disabled by default. - 6
- If
enabled
istrue
, all the resource attributes are converted to metric labels. Disabled by default. - 7
- Defines how long metrics are exposed without updates. The default is
5m
. - 8
- Adds the metrics types and units suffixes. Must be disabled if the monitor tab in the Jaeger console is enabled. The default is
true
.
When the spec.observability.metrics.enableMetrics
field in the OpenTelemetryCollector
custom resource (CR) is set to true
, the OpenTelemetryCollector
CR automatically creates a Prometheus ServiceMonitor
or PodMonitor
CR to enable Prometheus to scrape your metrics.
3.3.6. Prometheus Remote Write Exporter
The Prometheus Remote Write Exporter exports metrics to compatible back ends.
The Prometheus Remote Write Exporter is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
OpenTelemetry Collector custom resource with an enabled Prometheus Remote Write Exporter
# ... config: | exporters: prometheusremotewrite: endpoint: "https://my-prometheus:7900/api/v1/push" 1 tls: 2 ca_file: ca.pem cert_file: cert.pem key_file: key.pem target_info: true 3 export_created_metric: true 4 max_batch_size_bytes: 3000000 5 service: pipelines: metrics: exporters: [prometheusremotewrite] # ...
- 1
- Endpoint for sending the metrics.
- 2
- Server-side TLS configuration. Defines paths to TLS certificates.
- 3
- When set to
true
, creates atarget_info
metric for each resource metric. - 4
- When set to
true
, exports a_created
metric for the Summary, Histogram, and Monotonic Sum metric points. - 5
- Maximum size of the batch of samples that is sent to the remote write endpoint. Exceeding this value results in batch splitting. The default value is
3000000
, which is approximately 2.861 megabytes.
- This exporter drops non-cumulative monotonic, histogram, and summary OTLP metrics.
-
You must enable the
--web.enable-remote-write-receiver
feature flag on the remote Prometheus instance. Without it, pushing the metrics to the instance using this exporter fails.
3.3.7. Kafka Exporter
The Kafka Exporter exports logs, metrics, and traces to Kafka. This exporter uses a synchronous producer that blocks and does not batch messages. You must use it with batch and queued retry processors for higher throughput and resiliency.
The Kafka Exporter is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
OpenTelemetry Collector custom resource with an enabled Kafka Exporter
# ... config: | exporters: kafka: brokers: ["localhost:9092"] 1 protocol_version: 2.0.0 2 topic: otlp_spans 3 auth: plain_text: 4 username: example password: example tls: 5 ca_file: ca.pem cert_file: cert.pem key_file: key.pem insecure: false 6 server_name_override: kafka.example.corp 7 service: pipelines: traces: exporters: [kafka] # ...
- 1
- The list of Kafka brokers. The default is
localhost:9092
. - 2
- The Kafka protocol version. For example,
2.0.0
. This is a required field. - 3
- The name of the Kafka topic to read from. The following are the defaults:
otlp_spans
for traces,otlp_metrics
for metrics,otlp_logs
for logs. - 4
- The plain text authentication configuration. If omitted, plain text authentication is disabled.
- 5
- The client-side TLS configuration. Defines paths to the TLS certificates. If omitted, TLS authentication is disabled.
- 6
- Disables verifying the server’s certificate chain and host name. The default is
false
. - 7
- ServerName indicates the name of the server requested by the client to support virtual hosting.
3.3.8. Additional resources
3.4. Connectors
A connector connects two pipelines. It consumes data as an exporter at the end of one pipeline and emits data as a receiver at the start of another pipeline. It can consume and emit data of the same or different data type. It can generate and emit data to summarize the consumed data, or it can merely replicate or route data.
3.4.1. Routing Connector
The Routing Connector routes logs, metrics, and traces to specified pipelines according to resource attributes and their routing conditions, which are written as OpenTelemetry Transformation Language (OTTL) statements.
The Routing Connector is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
OpenTelemetry Collector custom resource with an enabled Routing Connector
config: | connectors: routing: table: 1 - statement: route() where attributes["X-Tenant"] == "dev" 2 pipelines: [traces/dev] 3 - statement: route() where attributes["X-Tenant"] == "prod" pipelines: [traces/prod] default_pipelines: [traces/dev] 4 error_mode: ignore 5 match_once: false 6 service: pipelines: traces/in: receivers: [otlp] exporters: [routing] traces/dev: receivers: [routing] exporters: [otlp/dev] traces/prod: receivers: [routing] exporters: [otlp/prod]
- 1
- Connector routing table.
- 2
- Routing conditions written as OTTL statements.
- 3
- Destination pipelines for routing the matching telemetry data.
- 4
- Destination pipelines for routing the telemetry data for which no routing condition is satisfied.
- 5
- Error-handling mode: The
propagate
value is for logging an error and dropping the payload. Theignore
value is for ignoring the condition and attempting to match with the next one. Thesilent
value is the same asignore
but without logging the error. The default ispropagate
. - 6
- When set to
true
, the payload is routed only to the first pipeline whose routing condition is met. The default isfalse
.
3.4.2. Forward Connector
The Forward Connector merges two pipelines of the same type.
The Forward Connector is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
OpenTelemetry Collector custom resource with an enabled Forward Connector
# ... receivers: otlp: protocols: grpc: jaeger: protocols: grpc: processors: batch: exporters: otlp: endpoint: tempo-simplest-distributor:4317 tls: insecure: true connectors: forward: service: pipelines: traces/regiona: receivers: [otlp] processors: [] exporters: [forward] traces/regionb: receivers: [jaeger] processors: [] exporters: [forward] traces: receivers: [forward] processors: [batch] exporters: [otlp] # ...
3.4.3. Spanmetrics Connector
The Spanmetrics Connector aggregates Request, Error, and Duration (R.E.D) OpenTelemetry metrics from span data.
The Spanmetrics Connector is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
OpenTelemetry Collector custom resource with an enabled Spanmetrics Connector
# ...
config: |
connectors:
spanmetrics:
metrics_flush_interval: 15s 1
service:
pipelines:
traces:
exporters: [spanmetrics]
metrics:
receivers: [spanmetrics]
# ...
- 1
- Defines the flush interval of the generated metrics. Defaults to
15s
.
3.4.4. Additional resources
3.5. Extensions
Extensions add capabilities to the Collector. For example, authentication can be added to the receivers and exporters automatically.
3.5.1. BearerTokenAuth Extension
The BearerTokenAuth Extension is an authenticator for receivers and exporters that are based on the HTTP and the gRPC protocol. You can use the OpenTelemetry Collector custom resource to configure client authentication and server authentication for the BearerTokenAuth Extension on the receiver and exporter side. This extension supports traces, metrics, and logs.
The BearerTokenAuth Extension is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
OpenTelemetry Collector custom resource with client and server authentication configured for the BearerTokenAuth Extension
# ... config: | extensions: bearertokenauth: scheme: "Bearer" 1 token: "<token>" 2 filename: "<token_file>" 3 receivers: otlp: protocols: http: auth: authenticator: bearertokenauth 4 exporters: otlp: auth: authenticator: bearertokenauth 5 service: extensions: [bearertokenauth] pipelines: traces: receivers: [otlp] exporters: [otlp] # ...
- 1
- You can configure the BearerTokenAuth Extension to send a custom
scheme
. The default isBearer
. - 2
- You can add the BearerTokenAuth Extension token as metadata to identify a message.
- 3
- Path to a file that contains an authorization token that is transmitted with every message.
- 4
- You can assign the authenticator configuration to an OTLP Receiver.
- 5
- You can assign the authenticator configuration to an OTLP Exporter.
3.5.2. OAuth2Client Extension
The OAuth2Client Extension is an authenticator for exporters that are based on the HTTP and the gRPC protocol. Client authentication for the OAuth2Client Extension is configured in a separate section in the OpenTelemetry Collector custom resource. This extension supports traces, metrics, and logs.
The OAuth2Client Extension is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
OpenTelemetry Collector custom resource with client authentication configured for the OAuth2Client Extension
# ... config: | extensions: oauth2client: client_id: <client_id> 1 client_secret: <client_secret> 2 endpoint_params: 3 audience: <audience> token_url: https://example.com/oauth2/default/v1/token 4 scopes: ["api.metrics"] 5 # tls settings for the token client tls: 6 insecure: true 7 ca_file: /var/lib/mycert.pem 8 cert_file: <cert_file> 9 key_file: <key_file> 10 timeout: 2s 11 receivers: otlp: protocols: http: {} exporters: otlp: auth: authenticator: oauth2client 12 service: extensions: [oauth2client] pipelines: traces: receivers: [otlp] exporters: [otlp] # ...
- 1
- Client identifier, which is provided by the identity provider.
- 2
- Confidential key used to authenticate the client to the identity provider.
- 3
- Further metadata, in the key-value pair format, which is transferred during authentication. For example,
audience
specifies the intended audience for the access token, indicating the recipient of the token. - 4
- The URL of the OAuth2 token endpoint, where the Collector requests access tokens.
- 5
- The scopes define the specific permissions or access levels requested by the client.
- 6
- The Transport Layer Security (TLS) settings for the token client, which is used to establish a secure connection when requesting tokens.
- 7
- When set to
true
, configures the Collector to use an insecure or non-verified TLS connection to call the configured token endpoint. - 8
- The path to a Certificate Authority (CA) file that is used to verify the server’s certificate during the TLS handshake.
- 9
- The path to the client certificate file that the client must use to authenticate itself to the OAuth2 server if required.
- 10
- The path to the client’s private key file that is used with the client certificate if needed for authentication.
- 11
- Sets a timeout for the token client’s request.
- 12
- You can assign the authenticator configuration to an OTLP exporter.
3.5.3. File Storage Extension
The File Storage Extension supports traces, metrics, and logs. This extension can persist the state to the local file system. This extension persists the sending queue for the OpenTelemetry Protocol (OTLP) exporters that are based on the HTTP and the gRPC protocols. This extension requires the read and write access to a directory. This extension can use a default directory, but the default directory must already exist.
The File Storage Extension is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
OpenTelemetry Collector custom resource with a configured File Storage Extension that persists an OTLP sending queue
# ... config: | extensions: file_storage/all_settings: directory: /var/lib/otelcol/mydir 1 timeout: 1s 2 compaction: on_start: true 3 directory: /tmp/ 4 max_transaction_size: 65_536 5 fsync: false 6 exporters: otlp: sending_queue: storage: file_storage/all_settings 7 service: extensions: [file_storage/all_settings] 8 pipelines: traces: receivers: [otlp] exporters: [otlp] # ...
- 1
- Specifies the directory in which the telemetry data is stored.
- 2
- Specifies the timeout time interval for opening the stored files.
- 3
- Starts compaction when the Collector starts. If omitted, the default is
false
. - 4
- Specifies the directory in which the compactor stores the telemetry data.
- 5
- Defines the maximum size of the compaction transaction. To ignore the transaction size, set to zero. If omitted, the default is
65536
bytes. - 6
- When set, forces the database to perform an
fsync
call after each write operation. This helps to ensure database integrity if there is an interruption to the database process, but at the cost of performance. - 7
- Buffers the OTLP Exporter data on the local file system.
- 8
- Starts the File Storage Extension by the Collector.
3.5.4. OIDC Auth Extension
The OIDC Auth Extension authenticates incoming requests to receivers by using the OpenID Connect (OIDC) protocol. It validates the ID token in the authorization header against the issuer and updates the authentication context of the incoming request.
The OIDC Auth Extension is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
OpenTelemetry Collector custom resource with the configured OIDC Auth Extension
# ... config: | extensions: oidc: attribute: authorization 1 issuer_url: https://example.com/auth/realms/opentelemetry 2 issuer_ca_path: /var/run/tls/issuer.pem 3 audience: otel-collector 4 username_claim: email 5 receivers: otlp: protocols: grpc: auth: authenticator: oidc exporters: otlp: endpoint: <endpoint> service: extensions: [oidc] pipelines: traces: receivers: [otlp] exporters: [otlp] # ...
3.5.5. Jaeger Remote Sampling Extension
The Jaeger Remote Sampling Extension enables serving sampling strategies after Jaeger’s remote sampling API. You can configure this extension to proxy requests to a backing remote sampling server such as a Jaeger collector down the pipeline or to a static JSON file from the local file system.
The Jaeger Remote Sampling Extension is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
OpenTelemetry Collector custom resource with a configured Jaeger Remote Sampling Extension
# ... config: | extensions: jaegerremotesampling: source: reload_interval: 30s 1 remote: endpoint: jaeger-collector:14250 2 file: /etc/otelcol/sampling_strategies.json 3 receivers: otlp: protocols: http: {} exporters: otlp: service: extensions: [jaegerremotesampling] pipelines: traces: receivers: [otlp] exporters: [otlp] # ...
Example of a Jaeger Remote Sampling strategy file
{ "service_strategies": [ { "service": "foo", "type": "probabilistic", "param": 0.8, "operation_strategies": [ { "operation": "op1", "type": "probabilistic", "param": 0.2 }, { "operation": "op2", "type": "probabilistic", "param": 0.4 } ] }, { "service": "bar", "type": "ratelimiting", "param": 5 } ], "default_strategy": { "type": "probabilistic", "param": 0.5, "operation_strategies": [ { "operation": "/health", "type": "probabilistic", "param": 0.0 }, { "operation": "/metrics", "type": "probabilistic", "param": 0.0 } ] } }
3.5.6. Performance Profiler Extension
The Performance Profiler Extension enables the Go net/http/pprof
endpoint. Developers use this extension to collect performance profiles and investigate issues with the service.
The Performance Profiler Extension is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
OpenTelemetry Collector custom resource with the configured Performance Profiler Extension
# ... config: | extensions: pprof: endpoint: localhost:1777 1 block_profile_fraction: 0 2 mutex_profile_fraction: 0 3 save_to_file: test.pprof 4 receivers: otlp: protocols: http: {} exporters: otlp: service: extensions: [pprof] pipelines: traces: receivers: [otlp] exporters: [otlp] # ...
- 1
- The endpoint at which this extension listens. Use
localhost:
to make it available only locally or":"
to make it available on all network interfaces. The default value islocalhost:1777
. - 2
- Sets a fraction of blocking events to be profiled. To disable profiling, set this to
0
or a negative integer. See the documentation for theruntime
package. The default value is0
. - 3
- Set a fraction of mutex contention events to be profiled. To disable profiling, set this to
0
or a negative integer. See the documentation for theruntime
package. The default value is0
. - 4
- The name of the file in which the CPU profile is to be saved. Profiling starts when the Collector starts. Profiling is saved to the file when the Collector is terminated.
3.5.7. Health Check Extension
The Health Check Extension provides an HTTP URL for checking the status of the OpenTelemetry Collector. You can use this extension as a liveness and readiness probe on OpenShift.
The Health Check Extension is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
OpenTelemetry Collector custom resource with the configured Health Check Extension
# ... config: | extensions: health_check: endpoint: "0.0.0.0:13133" 1 tls: 2 ca_file: "/path/to/ca.crt" cert_file: "/path/to/cert.crt" key_file: "/path/to/key.key" path: "/health/status" 3 check_collector_pipeline: 4 enabled: true 5 interval: "5m" 6 exporter_failure_threshold: 5 7 receivers: otlp: protocols: http: {} exporters: otlp: service: extensions: [health_check] pipelines: traces: receivers: [otlp] exporters: [otlp] # ...
- 1
- The target IP address for publishing the health check status. The default is
0.0.0.0:13133
. - 2
- The TLS server-side configuration. Defines paths to TLS certificates. If omitted, the TLS is disabled.
- 3
- The path for the health check server. The default is
/
. - 4
- Settings for the Collector pipeline health check.
- 5
- Enables the Collector pipeline health check. The default is
false
. - 6
- The time interval for checking the number of failures. The default is
5m
. - 7
- The threshold of multiple failures until which a container is still marked as healthy. The default is
5
.
3.5.8. Memory Ballast Extension
The Memory Ballast Extension enables applications to configure memory ballast for the process.
The Memory Ballast Extension is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
OpenTelemetry Collector custom resource with the configured Memory Ballast Extension
# ... config: | extensions: memory_ballast: size_mib: 64 1 size_in_percentage: 20 2 receivers: otlp: protocols: http: {} exporters: otlp: service: extensions: [memory_ballast] pipelines: traces: receivers: [otlp] exporters: [otlp] # ...
3.5.9. zPages Extension
The zPages Extension provides an HTTP endpoint that serves live data for debugging instrumented components in real time. You can use this extension for in-process diagnostics and insights into traces and metrics without relying on an external backend. With this extension, you can monitor and troubleshoot the behavior of the OpenTelemetry Collector and related components by watching the diagnostic information at the provided endpoint.
The zPages Extension is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
OpenTelemetry Collector custom resource with the configured zPages Extension
# ...
config: |
extensions:
zpages:
endpoint: "localhost:55679" 1
receivers:
otlp:
protocols:
http: {}
exporters:
debug:
service:
extensions: [zpages]
pipelines:
traces:
receivers: [otlp]
exporters: [debug]
# ...
- 1
- Specifies the HTTP endpoint for serving the zPages extension. The default is
localhost:55679
.
Accessing the HTTP endpoint requires port-forwarding because the Red Hat build of OpenTelemetry Operator does not expose this route.
You can enable port-forwarding by running the following oc
command:
$ oc port-forward pod/$(oc get pod -l app.kubernetes.io/name=instance-collector -o=jsonpath='{.items[0].metadata.name}') 55679
The Collector provides the following zPages for diagnostics:
- ServiceZ
-
Shows an overview of the Collector services and links to the following zPages: PipelineZ, ExtensionZ, and FeatureZ. This page also displays information about the build version and runtime. An example of this page’s URL is
http://localhost:55679/debug/servicez
. - PipelineZ
-
Shows detailed information about the active pipelines in the Collector. This page displays the pipeline type, whether data are modified, and the associated receivers, processors, and exporters for each pipeline. An example of this page’s URL is
http://localhost:55679/debug/pipelinez
. - ExtensionZ
-
Shows the currently active extensions in the Collector. An example of this page’s URL is
http://localhost:55679/debug/extensionz
. - FeatureZ
-
Shows the feature gates enabled in the Collector along with their status and description. An example of this page’s URL is
http://localhost:55679/debug/featurez
. - TraceZ
-
Shows spans categorized by latency. Available time ranges include 0 µs, 10 µs, 100 µs, 1 ms, 10 ms, 100 ms, 1 s, 10 s, 1 m. This page also allows for quick inspection of error samples. An example of this page’s URL is
http://localhost:55679/debug/tracez
.
3.5.10. Additional resources
3.6. Target Allocator
The Target Allocator is an optional component of the OpenTelemetry Operator that shards scrape targets across the deployed fleet of OpenTelemetry Collector instances. The Target Allocator integrates with the Prometheus PodMonitor
and ServiceMonitor
custom resources (CR). When the Target Allocator is enabled, the OpenTelemetry Operator adds the http_sd_config
field to the enabled prometheus
receiver that connects to the Target Allocator service.
The Target Allocator is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
Example OpenTelemetryCollector CR with the enabled Target Allocator
apiVersion: opentelemetry.io/v1alpha1 kind: OpenTelemetryCollector metadata: name: otel namespace: observability spec: mode: statefulset 1 targetAllocator: enabled: true 2 serviceAccount: 3 prometheusCR: enabled: true 4 scrapeInterval: 10s serviceMonitorSelector: 5 name: app1 podMonitorSelector: 6 name: app2 config: | receivers: prometheus: 7 config: scrape_configs: [] processors: exporters: debug: {} service: pipelines: metrics: receivers: [prometheus] processors: [] exporters: [debug] # ...
- 1
- When the Target Allocator is enabled, the deployment mode must be set to
statefulset
. - 2
- Enables the Target Allocator. Defaults to
false
. - 3
- The service account name of the Target Allocator deployment. The service account needs to have RBAC to get the
ServiceMonitor
,PodMonitor
custom resources, and other objects from the cluster to properly set labels on scraped metrics. The default service name is<collector_name>-targetallocator
. - 4
- Enables integration with the Prometheus
PodMonitor
andServiceMonitor
custom resources. - 5
- Label selector for the Prometheus
ServiceMonitor
custom resources. When left empty, enables all service monitors. - 6
- Label selector for the Prometheus
PodMonitor
custom resources. When left empty, enables all pod monitors. - 7
- Prometheus receiver with the minimal, empty
scrape_config: []
configuration option.
The Target Allocator deployment uses the Kubernetes API to get relevant objects from the cluster, so it requires a custom RBAC configuration.
RBAC configuration for the Target Allocator service account
apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: otel-targetallocator rules: - apiGroups: [""] resources: - services - pods verbs: ["get", "list", "watch"] - apiGroups: ["monitoring.coreos.com"] resources: - servicemonitors - podmonitors verbs: ["get", "list", "watch"] - apiGroups: ["discovery.k8s.io"] resources: - endpointslices verbs: ["get", "list", "watch"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: otel-targetallocator roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: otel-targetallocator subjects: - kind: ServiceAccount name: otel-targetallocator 1 namespace: observability 2 # ...