Chapter 9. Gathering the observability data from multiple clusters

Procedure

1. Mount the following certificates in the OpenTelemetry Collector instance, skipping already mounted certificates.

   1. An Issuer to generate the certificates by using the cert-manager Operator for Red Hat OpenShift.

      apiVersion: cert-manager.io/v1
      kind: Issuer
      metadata:
        name: selfsigned-issuer
      spec:
        selfSigned: {}

   2. A self-signed certificate.

      apiVersion: cert-manager.io/v1
      kind: Certificate
      metadata:
        name: ca
      spec:
        isCA: true
        commonName: ca
        subject:
          organizations:
            - Organization # <your_organization_name>
          organizationalUnits:
            - Widgets
        secretName: ca-secret
        privateKey:
          algorithm: ECDSA
          size: 256
        issuerRef:
          name: selfsigned-issuer
          kind: Issuer
          group: cert-manager.io

   3. A CA issuer.

      apiVersion: cert-manager.io/v1
      kind: Issuer
      metadata:
        name: test-ca-issuer
      spec:
        ca:
          secretName: ca-secret

   4. The client and server certificates.

      apiVersion: cert-manager.io/v1
      kind: Certificate
      metadata:
        name: server
      spec:
        secretName: server-tls
        isCA: false
        usages:
          - server auth
          - client auth
        dnsNames:
        - "otel.observability.svc.cluster.local" 1
        issuerRef:
          name: ca-issuer
      ---
      apiVersion: cert-manager.io/v1
      kind: Certificate
      metadata:
        name: client
      spec:
        secretName: client-tls
        isCA: false
        usages:
          - server auth
          - client auth
        dnsNames:
        - "otel.observability.svc.cluster.local" 2
        issuerRef:
          name: ca-issuer

      1

      List of exact DNS names to be mapped to a solver in the server OpenTelemetry Collector instance.

      2

      List of exact DNS names to be mapped to a solver in the client OpenTelemetry Collector instance.

2. Create a service account for the OpenTelemetry Collector instance.

   Example ServiceAccount

   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: otel-collector-deployment

3. Create a cluster role for the service account.

   Example ClusterRole

   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRole
   metadata:
     name: otel-collector
   rules:
     1
     2
   - apiGroups: ["", "config.openshift.io"]
     resources: ["pods", "namespaces", "infrastructures", "infrastructures/status"]
     verbs: ["get", "watch", "list"]

   1

   The k8sattributesprocessor requires permissions for pods and namespace resources.

   2

   The resourcedetectionprocessor requires permissions for infrastructures and status.

4. Bind the cluster role to the service account.

   Example ClusterRoleBinding

   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRoleBinding
   metadata:
     name: otel-collector
   subjects:
   - kind: ServiceAccount
     name: otel-collector-deployment
     namespace: otel-collector-<example>
   roleRef:
     kind: ClusterRole
     name: otel-collector
     apiGroup: rbac.authorization.k8s.io

5. Create the YAML file to define the OpenTelemetryCollector custom resource (CR) in the edge clusters.

   Example OpenTelemetryCollector custom resource for the edge clusters

   apiVersion: opentelemetry.io/v1alpha1
   kind: OpenTelemetryCollector
   metadata:
     name: otel
     namespace: otel-collector-<example>
   spec:
     mode: daemonset
     serviceAccount: otel-collector-deployment
     config: |
       receivers:
         jaeger:
           protocols:
             grpc: {}
             thrift_binary: {}
             thrift_compact: {}
             thrift_http: {}
         opencensus:
         otlp:
           protocols:
             grpc: {}
             http: {}
         zipkin: {}
       processors:
         batch: {}
         k8sattributes: {}
         memory_limiter:
           check_interval: 1s
           limit_percentage: 50
           spike_limit_percentage: 30
         resourcedetection:
           detectors: [openshift]
       exporters:
         otlphttp:
           endpoint: https://observability-cluster.com:443 1
           tls:
             insecure: false
             cert_file: /certs/server.crt
             key_file: /certs/server.key
             ca_file: /certs/ca.crt
       service:
         pipelines:
           traces:
             receivers: [jaeger, opencensus, otlp, zipkin]
             processors: [memory_limiter, k8sattributes, resourcedetection, batch]
             exporters: [otlp]
     volumes:
       - name: otel-certs
         secret:
           name: otel-certs
     volumeMounts:
       - name: otel-certs
         mountPath: /certs

   1

   The Collector exporter is configured to export OTLP HTTP and points to the OpenTelemetry Collector from the central cluster.

6. Create the YAML file to define the OpenTelemetryCollector custom resource (CR) in the central cluster.

   Example OpenTelemetryCollector custom resource for the central cluster

   apiVersion: opentelemetry.io/v1alpha1
   kind: OpenTelemetryCollector
   metadata:
     name: otlp-receiver
     namespace: observability
   spec:
     mode: "deployment"
     ingress:
       type: route
       route:
         termination: "passthrough"
     config: |
       receivers:
         otlp:
           protocols:
             http:
               tls: 1
                 cert_file: /certs/server.crt
                 key_file: /certs/server.key
                 client_ca_file: /certs/ca.crt
       exporters:
         logging: {}
         otlp:
           endpoint: "tempo-<simplest>-distributor:4317" 2
           tls:
             insecure: true
       service:
         pipelines:
           traces:
             receivers: [otlp]
             processors: []
             exporters: [otlp]
     volumes:
       - name: otel-certs
         secret:
           name: otel-certs
     volumeMounts:
       - name: otel-certs
         mountPath: /certs

   1

   The Collector receiver requires the certificates listed in the first step.

   2

   The Collector exporter is configured to export OTLP and points to the Tempo distributor endpoint, which in this example is "tempo-simplest-distributor:4317" and already created.