Red Hat build of OpenTelemetry

Procedure

1. Install the Red Hat build of OpenTelemetry Operator:

   1. Go to Operators → OperatorHub and search for

      Red Hat build of OpenTelemetry Operator

      .

   2. Select the Red Hat build of OpenTelemetry Operator that is provided by Red Hat → Install → Install → View Operator.

      Important

      This installs the Operator with the default presets:

      • Update channel → stable
      • Installation mode → All namespaces on the cluster
      • Installed Namespace → openshift-operators
      • Update approval → Automatic
   3. In the Details tab of the installed Operator page, under ClusterServiceVersion details, verify that the installation Status is Succeeded.
2. Create a project of your choice for the OpenTelemetry Collector instance that you will create in the next step by going to Home → Projects → Create Project.

3. Create an OpenTelemetry Collector instance.

   1. Go to Operators → Installed Operators.
   2. Select OpenTelemetry Collector → Create OpenTelemetry Collector → YAML view.

   3. In the YAML view, customize the

      OpenTelemetryCollector

      custom resource (CR) with the OTLP, Jaeger, Zipkin receivers and the debug exporter.

      apiVersion: opentelemetry.io/v1alpha1
      kind: OpenTelemetryCollector
      metadata:
        name: otel
        namespace: <project_of_opentelemetry_collector_instance>
      spec:
        mode: deployment
        config: |
          receivers:
            otlp:
              protocols:
                grpc:
                http:
            jaeger:
              protocols:
                grpc:
                thrift_binary:
                thrift_compact:
                thrift_http:
            zipkin:
          processors:
            batch:
            memory_limiter:
              check_interval: 1s
              limit_percentage: 50
              spike_limit_percentage: 30
          exporters:
            debug:
          service:
            pipelines:
              traces:
                receivers: [otlp,jaeger,zipkin]
                processors: [memory_limiter,batch]
                exporters: [debug]

   4. Select Create.

Verification

1. Use the Project: dropdown list to select the project of the OpenTelemetry Collector instance.
2. Go to Operators → Installed Operators to verify that the Status of the OpenTelemetry Collector instance is Condition: Ready.
3. Go to Workloads → Pods to verify that all the component pods of the OpenTelemetry Collector instance are running.

Procedure

1. Install the Red Hat build of OpenTelemetry Operator:

   1. Create a project for the Red Hat build of OpenTelemetry Operator by running the following command:

      $ oc apply -f - << EOF
      apiVersion: project.openshift.io/v1
      kind: Project
      metadata:
        labels:
          kubernetes.io/metadata.name: openshift-opentelemetry-operator
          openshift.io/cluster-monitoring: "true"
        name: openshift-opentelemetry-operator
      EOF

   2. Create an Operator group by running the following command:

      $ oc apply -f - << EOF
      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        name: openshift-opentelemetry-operator
        namespace: openshift-opentelemetry-operator
      spec:
        upgradeStrategy: Default
      EOF

   3. Create a subscription by running the following command:

      $ oc apply -f - << EOF
      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        name: opentelemetry-product
        namespace: openshift-opentelemetry-operator
      spec:
        channel: stable
        installPlanApproval: Automatic
        name: opentelemetry-product
        source: redhat-operators
        sourceNamespace: openshift-marketplace
      EOF

   4. Check the Operator status by running the following command:

      $ oc get csv -n openshift-opentelemetry-operator

2. Create a project of your choice for the OpenTelemetry Collector instance that you will create in a subsequent step:

   • To create a project without metadata, run the following command:

     $ oc new-project <project_of_opentelemetry_collector_instance>

   • To create a project with metadata, run the following command:

     $ oc apply -f - << EOF
     apiVersion: project.openshift.io/v1
     kind: Project
     metadata:
       name: <project_of_opentelemetry_collector_instance>
     EOF

3. Create an OpenTelemetry Collector instance in the project that you created for it.

   Note

   You can create multiple OpenTelemetry Collector instances in separate projects on the same cluster.

   1. Customize the

      OpenTelemetry Collector

      custom resource (CR) with the OTLP, Jaeger, and Zipkin receivers and the debug exporter:

      apiVersion: opentelemetry.io/v1alpha1
      kind: OpenTelemetryCollector
      metadata:
        name: otel
        namespace: <project_of_opentelemetry_collector_instance>
      spec:
        mode: deployment
        config: |
          receivers:
            otlp:
              protocols:
                grpc:
                http:
            jaeger:
              protocols:
                grpc:
                thrift_binary:
                thrift_compact:
                thrift_http:
            zipkin:
          processors:
            batch:
            memory_limiter:
              check_interval: 1s
              limit_percentage: 50
              spike_limit_percentage: 30
          exporters:
            debug:
          service:
            pipelines:
              traces:
                receivers: [otlp,jaeger,zipkin]
                processors: [memory_limiter,batch]
                exporters: [debug]

   2. Apply the customized CR by running the following command:

      $ oc apply -f - << EOF
      <OpenTelemetryCollector_custom_resource>
      EOF

Verification

1. Verify that the

   status.phase

   of the OpenTelemetry Collector pod is

   Running

   and the

   conditions

   are

   type: Ready

   by running the following command:

   $ oc get pod -l app.kubernetes.io/managed-by=opentelemetry-operator,app.kubernetes.io/instance=<namespace>.<instance_name> -o yaml

2. Get the OpenTelemetry Collector service by running the following command:

   $ oc get service -l app.kubernetes.io/managed-by=opentelemetry-operator,app.kubernetes.io/instance=<namespace>.<instance_name>

Procedure

1. Mount the following certificates in the OpenTelemetry Collector instance, skipping already mounted certificates.

   1. An Issuer to generate the certificates by using the cert-manager Operator for Red Hat OpenShift.

      apiVersion: cert-manager.io/v1
      kind: Issuer
      metadata:
        name: selfsigned-issuer
      spec:
        selfSigned: {}

   2. A self-signed certificate.

      apiVersion: cert-manager.io/v1
      kind: Certificate
      metadata:
        name: ca
      spec:
        isCA: true
        commonName: ca
        subject:
          organizations:
            - Organization # <your_organization_name>
          organizationalUnits:
            - Widgets
        secretName: ca-secret
        privateKey:
          algorithm: ECDSA
          size: 256
        issuerRef:
          name: selfsigned-issuer
          kind: Issuer
          group: cert-manager.io

   3. A CA issuer.

      apiVersion: cert-manager.io/v1
      kind: Issuer
      metadata:
        name: test-ca-issuer
      spec:
        ca:
          secretName: ca-secret

   4. The client and server certificates.

      apiVersion: cert-manager.io/v1
      kind: Certificate
      metadata:
        name: server
      spec:
        secretName: server-tls
        isCA: false
        usages:
          - server auth
          - client auth
        dnsNames:
        - "otel.observability.svc.cluster.local" 

      1

      
        issuerRef:
          name: ca-issuer
      ---
      apiVersion: cert-manager.io/v1
      kind: Certificate
      metadata:
        name: client
      spec:
        secretName: client-tls
        isCA: false
        usages:
          - server auth
          - client auth
        dnsNames:
        - "otel.observability.svc.cluster.local" 

      2

      
        issuerRef:
          name: ca-issuer

      1

      List of exact DNS names to be mapped to a solver in the server OpenTelemetry Collector instance.

      2

      List of exact DNS names to be mapped to a solver in the client OpenTelemetry Collector instance.

2. Create a service account for the OpenTelemetry Collector instance.

   Example ServiceAccount

   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: otel-collector-deployment

3. Create a cluster role for the service account.

   Example ClusterRole

   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRole
   metadata:
     name: otel-collector
   rules:
     

   1

   2

   
   - apiGroups: ["", "config.openshift.io"]
     resources: ["pods", "namespaces", "infrastructures", "infrastructures/status"]
     verbs: ["get", "watch", "list"]

   1

   The k8sattributesprocessor requires permissions for pods and namespace resources.

   2

   The resourcedetectionprocessor requires permissions for infrastructures and status.

4. Bind the cluster role to the service account.

   Example ClusterRoleBinding

   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRoleBinding
   metadata:
     name: otel-collector
   subjects:
   - kind: ServiceAccount
     name: otel-collector-deployment
     namespace: otel-collector-<example>
   roleRef:
     kind: ClusterRole
     name: otel-collector
     apiGroup: rbac.authorization.k8s.io

5. Create the YAML file to define the

   OpenTelemetryCollector

   custom resource (CR) in the edge clusters.

   Example OpenTelemetryCollector custom resource for the edge clusters

   apiVersion: opentelemetry.io/v1alpha1
   kind: OpenTelemetryCollector
   metadata:
     name: otel
     namespace: otel-collector-<example>
   spec:
     mode: daemonset
     serviceAccount: otel-collector-deployment
     config: |
       receivers:
         jaeger:
           protocols:
             grpc:
             thrift_binary:
             thrift_compact:
             thrift_http:
         opencensus:
         otlp:
           protocols:
             grpc:
             http:
         zipkin:
       processors:
         batch:
         k8sattributes:
         memory_limiter:
           check_interval: 1s
           limit_percentage: 50
           spike_limit_percentage: 30
         resourcedetection:
           detectors: [openshift]
       exporters:
         otlphttp:
           endpoint: https://observability-cluster.com:443 

   1

   
           tls:
             insecure: false
             cert_file: /certs/server.crt
             key_file: /certs/server.key
             ca_file: /certs/ca.crt
       service:
         pipelines:
           traces:
             receivers: [jaeger, opencensus, otlp, zipkin]
             processors: [memory_limiter, k8sattributes, resourcedetection, batch]
             exporters: [otlp]
     volumes:
       - name: otel-certs
         secret:
           name: otel-certs
     volumeMounts:
       - name: otel-certs
         mountPath: /certs

   1

   The Collector exporter is configured to export OTLP HTTP and points to the OpenTelemetry Collector from the central cluster.

6. Create the YAML file to define the

   OpenTelemetryCollector

   custom resource (CR) in the central cluster.

   Example OpenTelemetryCollector custom resource for the central cluster

   apiVersion: opentelemetry.io/v1alpha1
   kind: OpenTelemetryCollector
   metadata:
     name: otlp-receiver
     namespace: observability
   spec:
     mode: "deployment"
     ingress:
       type: route
       route:
         termination: "passthrough"
     config: |
       receivers:
         otlp:
           protocols:
             http:
               tls: 

   1

   
                 cert_file: /certs/server.crt
                 key_file: /certs/server.key
                 client_ca_file: /certs/ca.crt
       exporters:
         logging:
         otlp:
           endpoint: "tempo-<simplest>-distributor:4317" 

   2

   
           tls:
             insecure: true
       service:
         pipelines:
           traces:
             receivers: [otlp]
             processors: []
             exporters: [otlp]
     volumes:
       - name: otel-certs
         secret:
           name: otel-certs
     volumeMounts:
       - name: otel-certs
         mountPath: /certs

   1

   The Collector receiver requires the certificates listed in the first step.

   2

   The Collector exporter is configured to export OTLP and points to the Tempo distributor endpoint, which in this example is "tempo-simplest-distributor:4317" and already created.

Procedure

1. Create a service account for the OpenTelemetry Collector.

   Example ServiceAccount

   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: otel-collector-deployment

2. Create a cluster role for the service account.

   Example ClusterRole

   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRole
   metadata:
     name: otel-collector
   rules:
     

   1

   2

   
   - apiGroups: ["", "config.openshift.io"]
     resources: ["pods", "namespaces", "infrastructures", "infrastructures/status"]
     verbs: ["get", "watch", "list"]

   1

   The k8sattributesprocessor requires permissions for pods and namespaces resources.

   2

   The resourcedetectionprocessor requires permissions for infrastructures and status.

3. Bind the cluster role to the service account.

   Example ClusterRoleBinding

   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRoleBinding
   metadata:
     name: otel-collector
   subjects:
   - kind: ServiceAccount
     name: otel-collector-deployment
     namespace: otel-collector-example
   roleRef:
     kind: ClusterRole
     name: otel-collector
     apiGroup: rbac.authorization.k8s.io

4. Create the YAML file to define the

   OpenTelemetryCollector

   custom resource (CR).

   Example OpenTelemetryCollector

   apiVersion: opentelemetry.io/v1alpha1
   kind: OpenTelemetryCollector
   metadata:
     name: otel
   spec:
     mode: deployment
     serviceAccount: otel-collector-deployment
     config: |
       receivers:
         jaeger:
           protocols:
             grpc:
             thrift_binary:
             thrift_compact:
             thrift_http:
         opencensus:
         otlp:
           protocols:
             grpc:
             http:
         zipkin:
       processors:
         batch:
         k8sattributes:
         memory_limiter:
           check_interval: 1s
           limit_percentage: 50
           spike_limit_percentage: 30
         resourcedetection:
           detectors: [openshift]
       exporters:
         otlp:
           endpoint: "tempo-simplest-distributor:4317" 

   1

   
           tls:
             insecure: true
       service:
         pipelines:
           traces:
             receivers: [jaeger, opencensus, otlp, zipkin] 

   2

   
             processors: [memory_limiter, k8sattributes, resourcedetection, batch]
             exporters: [otlp]

   1

   The Collector exporter is configured to export OTLP and points to the Tempo distributor endpoint, "tempo-simplest-distributor:4317" in this example, which is already created.

   2

   The Collector is configured with a receiver for Jaeger traces, OpenCensus traces over the OpenCensus protocol, Zipkin traces over the Zipkin protocol, and OTLP traces over the GRPC protocol.

Procedure

1. Set the relevant log level in the

   OpenTelemetryCollector

   custom resource (CR):

     config: |
       service:
         telemetry:
           logs:
             level: debug 

   1

   1

   Collector’s log level. Supported values include info, warn, error, or debug. Defaults to info.

2. Use the

   oc logs

   command or the web console to retrieve the logs.

Procedure

1. Enable the telemetry service by adding the following lines in the

   OpenTelemetryCollector

   custom resource:

     config: |
       service:
         telemetry:
           metrics:
             address: ":8888" 

   1

   1

   The address at which the internal collector metrics are exposed. Defaults to :8888.

1. Retrieve the metrics by running the following command, which uses the port-forwarding Collector pod:

   $ oc port-forward <collector_pod>

2. Access the metrics endpoint at

   http://localhost:8888/metrics

   .

Procedure

1. Configure the

   OpenTelemetryCollector

   custom resource as follows:

     config: |
       exporters:
         debug:
           verbosity: detailed
       service:
         pipelines:
           traces:
             exporters: [debug]
           metrics:
             exporters: [debug]
           logs:
             exporters: [debug]

2. Use the

   oc logs

   command or the web console to export the logs to the standard output.

Procedure

1. Configure the OpenTelemetry Collector as a sidecar.

   apiVersion: opentelemetry.io/v1alpha1
   kind: OpenTelemetryCollector
   metadata:
     name: otel
     namespace: <otel-collector-namespace>
   spec:
     mode: sidecar
     config: |
       receivers:
         jaeger:
           protocols:
             grpc:
             thrift_binary:
             thrift_compact:
             thrift_http:
       processors:
         batch:
         memory_limiter:
           check_interval: 1s
           limit_percentage: 50
           spike_limit_percentage: 30
         resourcedetection:
           detectors: [openshift]
           timeout: 2s
       exporters:
         otlp:
           endpoint: "tempo-<example>-gateway:8090" 

   1

   
           tls:
             insecure: true
       service:
         pipelines:
           traces:
             receivers: [jaeger]
             processors: [memory_limiter, resourcedetection, batch]
             exporters: [otlp]

   1

   This endpoint points to the Gateway of a TempoStack instance deployed by using the <example> Tempo Operator.

2. Create a service account for running your application.

   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: otel-collector-sidecar

3. Create a cluster role for the permissions needed by some processors.

   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRole
   metadata:
     name: otel-collector-sidecar
   rules:
     

   1

   
   - apiGroups: ["config.openshift.io"]
     resources: ["infrastructures", "infrastructures/status"]
     verbs: ["get", "watch", "list"]

   1

   The resourcedetectionprocessor requires permissions for infrastructures and infrastructures/status.

4. Create a

   ClusterRoleBinding

   to set the permissions for the service account.

   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRoleBinding
   metadata:
     name: otel-collector-sidecar
   subjects:
   - kind: ServiceAccount
     name: otel-collector-deployment
     namespace: otel-collector-example
   roleRef:
     kind: ClusterRole
     name: otel-collector
     apiGroup: rbac.authorization.k8s.io

5. Deploy the OpenTelemetry Collector as a sidecar.
6. Remove the injected Jaeger Agent from your application by removing the

   "sidecar.jaegertracing.io/inject": "true"

   annotation from your

   Deployment

   object.
7. Enable automatic injection of the OpenTelemetry sidecar by adding the

   sidecar.opentelemetry.io/inject: "true"

   annotation to the

   .spec.template.metadata.annotations

   field of your

   Deployment

   object.
8. Use the created service account for the deployment of your application to allow the processors to get the correct information and add it to your traces.

Procedure

1. Configure OpenTelemetry Collector deployment.

2. Create the project where the OpenTelemetry Collector will be deployed.

   apiVersion: project.openshift.io/v1
   kind: Project
   metadata:
     name: observability

3. Create a service account for running the OpenTelemetry Collector instance.

   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: otel-collector-deployment
     namespace: observability

4. Create a cluster role for setting the required permissions for the processors.

   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRole
   metadata:
     name: otel-collector
   rules:
     

   1

   2

   
   - apiGroups: ["", "config.openshift.io"]
     resources: ["pods", "namespaces", "infrastructures", "infrastructures/status"]
     verbs: ["get", "watch", "list"]

   1

   Permissions for the pods and namespaces resources are required for the k8sattributesprocessor.

   2

   Permissions for infrastructures and infrastructures/status are required for resourcedetectionprocessor.

5. Create a ClusterRoleBinding to set the permissions for the service account.

   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRoleBinding
   metadata:
     name: otel-collector
   subjects:
   - kind: ServiceAccount
     name: otel-collector-deployment
     namespace: observability
   roleRef:
     kind: ClusterRole
     name: otel-collector
     apiGroup: rbac.authorization.k8s.io

6. Create the OpenTelemetry Collector instance.

   Note

   This collector will export traces to a TempoStack instance. You must create your TempoStack instance by using the Red Hat Tempo Operator and place here the correct endpoint.

   apiVersion: opentelemetry.io/v1alpha1
   kind: OpenTelemetryCollector
   metadata:
     name: otel
     namespace: observability
   spec:
     mode: deployment
     serviceAccount: otel-collector-deployment
     config: |
       receivers:
         jaeger:
           protocols:
             grpc:
             thrift_binary:
             thrift_compact:
             thrift_http:
       processors:
         batch:
         k8sattributes:
         memory_limiter:
           check_interval: 1s
           limit_percentage: 50
           spike_limit_percentage: 30
         resourcedetection:
           detectors: [openshift]
       exporters:
         otlp:
           endpoint: "tempo-example-gateway:8090"
           tls:
             insecure: true
       service:
         pipelines:
           traces:
             receivers: [jaeger]
             processors: [memory_limiter, k8sattributes, resourcedetection, batch]
             exporters: [otlp]

7. Point your tracing endpoint to the OpenTelemetry Operator.

8. If you are exporting your traces directly from your application to Jaeger, change the API endpoint from the Jaeger endpoint to the OpenTelemetry Collector endpoint.

   Example of exporting traces by using the jaegerexporter with Golang

   exp, err := jaeger.New(jaeger.WithCollectorEndpoint(jaeger.WithEndpoint(url))) 

   1

   1

   The URL points to the OpenTelemetry Collector API endpoint.

Procedure

1. Go to Operators → Installed Operators → Red Hat build of OpenTelemetry Operator → OpenTelemetryInstrumentation or OpenTelemetryCollector.
2. To remove the relevant instance, select → Delete …​ → Delete.
3. Optional: Remove the Red Hat build of OpenTelemetry Operator.

Procedure

1. Get the name of the OpenTelemetry Collector instance by running the following command:

   $ oc get deployments -n <project_of_opentelemetry_instance>

2. Remove the OpenTelemetry Collector instance by running the following command:

   $ oc delete opentelemetrycollectors <opentelemetry_instance_name> -n <project_of_opentelemetry_instance>

3. Optional: Remove the Red Hat build of OpenTelemetry Operator.