Search

Chapter 9. Gathering the observability data from multiple clusters

download PDF

For a multicluster configuration, you can create one OpenTelemetry Collector instance in each one of the remote clusters and then forward all the telemetry data to one OpenTelemetry Collector instance.

Prerequisites

  • The Red Hat build of OpenTelemetry Operator is installed.
  • The Tempo Operator is installed.
  • A TempoStack instance is deployed on the cluster.
  • The following mounted certificates: Issuer, self-signed certificate, CA issuer, client and server certificates. To create any of these certificates, see step 1.

Procedure

  1. Mount the following certificates in the OpenTelemetry Collector instance, skipping already mounted certificates.

    1. An Issuer to generate the certificates by using the cert-manager Operator for Red Hat OpenShift.

      apiVersion: cert-manager.io/v1
      kind: Issuer
      metadata:
        name: selfsigned-issuer
      spec:
        selfSigned: {}
    2. A self-signed certificate.

      apiVersion: cert-manager.io/v1
      kind: Certificate
      metadata:
        name: ca
      spec:
        isCA: true
        commonName: ca
        subject:
          organizations:
            - Organization # <your_organization_name>
          organizationalUnits:
            - Widgets
        secretName: ca-secret
        privateKey:
          algorithm: ECDSA
          size: 256
        issuerRef:
          name: selfsigned-issuer
          kind: Issuer
          group: cert-manager.io
    3. A CA issuer.

      apiVersion: cert-manager.io/v1
      kind: Issuer
      metadata:
        name: test-ca-issuer
      spec:
        ca:
          secretName: ca-secret
    4. The client and server certificates.

      apiVersion: cert-manager.io/v1
      kind: Certificate
      metadata:
        name: server
      spec:
        secretName: server-tls
        isCA: false
        usages:
          - server auth
          - client auth
        dnsNames:
        - "otel.observability.svc.cluster.local" 1
        issuerRef:
          name: ca-issuer
      ---
      apiVersion: cert-manager.io/v1
      kind: Certificate
      metadata:
        name: client
      spec:
        secretName: client-tls
        isCA: false
        usages:
          - server auth
          - client auth
        dnsNames:
        - "otel.observability.svc.cluster.local" 2
        issuerRef:
          name: ca-issuer
      1
      List of exact DNS names to be mapped to a solver in the server OpenTelemetry Collector instance.
      2
      List of exact DNS names to be mapped to a solver in the client OpenTelemetry Collector instance.
  2. Create a service account for the OpenTelemetry Collector instance.

    Example ServiceAccount

    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: otel-collector-deployment

  3. Create a cluster role for the service account.

    Example ClusterRole

    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
      name: otel-collector
    rules:
      1
      2
    - apiGroups: ["", "config.openshift.io"]
      resources: ["pods", "namespaces", "infrastructures", "infrastructures/status"]
      verbs: ["get", "watch", "list"]

    1
    The k8sattributesprocessor requires permissions for pods and namespace resources.
    2
    The resourcedetectionprocessor requires permissions for infrastructures and status.
  4. Bind the cluster role to the service account.

    Example ClusterRoleBinding

    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      name: otel-collector
    subjects:
    - kind: ServiceAccount
      name: otel-collector-deployment
      namespace: otel-collector-<example>
    roleRef:
      kind: ClusterRole
      name: otel-collector
      apiGroup: rbac.authorization.k8s.io

  5. Create the YAML file to define the OpenTelemetryCollector custom resource (CR) in the edge clusters.

    Example OpenTelemetryCollector custom resource for the edge clusters

    apiVersion: opentelemetry.io/v1alpha1
    kind: OpenTelemetryCollector
    metadata:
      name: otel
      namespace: otel-collector-<example>
    spec:
      mode: daemonset
      serviceAccount: otel-collector-deployment
      config: |
        receivers:
          jaeger:
            protocols:
              grpc: {}
              thrift_binary: {}
              thrift_compact: {}
              thrift_http: {}
          opencensus:
          otlp:
            protocols:
              grpc: {}
              http: {}
          zipkin: {}
        processors:
          batch: {}
          k8sattributes: {}
          memory_limiter:
            check_interval: 1s
            limit_percentage: 50
            spike_limit_percentage: 30
          resourcedetection:
            detectors: [openshift]
        exporters:
          otlphttp:
            endpoint: https://observability-cluster.com:443 1
            tls:
              insecure: false
              cert_file: /certs/server.crt
              key_file: /certs/server.key
              ca_file: /certs/ca.crt
        service:
          pipelines:
            traces:
              receivers: [jaeger, opencensus, otlp, zipkin]
              processors: [memory_limiter, k8sattributes, resourcedetection, batch]
              exporters: [otlp]
      volumes:
        - name: otel-certs
          secret:
            name: otel-certs
      volumeMounts:
        - name: otel-certs
          mountPath: /certs

    1
    The Collector exporter is configured to export OTLP HTTP and points to the OpenTelemetry Collector from the central cluster.
  6. Create the YAML file to define the OpenTelemetryCollector custom resource (CR) in the central cluster.

    Example OpenTelemetryCollector custom resource for the central cluster

    apiVersion: opentelemetry.io/v1alpha1
    kind: OpenTelemetryCollector
    metadata:
      name: otlp-receiver
      namespace: observability
    spec:
      mode: "deployment"
      ingress:
        type: route
        route:
          termination: "passthrough"
      config: |
        receivers:
          otlp:
            protocols:
              http:
                tls: 1
                  cert_file: /certs/server.crt
                  key_file: /certs/server.key
                  client_ca_file: /certs/ca.crt
        exporters:
          logging: {}
          otlp:
            endpoint: "tempo-<simplest>-distributor:4317" 2
            tls:
              insecure: true
        service:
          pipelines:
            traces:
              receivers: [otlp]
              processors: []
              exporters: [otlp]
      volumes:
        - name: otel-certs
          secret:
            name: otel-certs
      volumeMounts:
        - name: otel-certs
          mountPath: /certs

    1
    The Collector receiver requires the certificates listed in the first step.
    2
    The Collector exporter is configured to export OTLP and points to the Tempo distributor endpoint, which in this example is "tempo-simplest-distributor:4317" and already created.
Red Hat logoGithubRedditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

© 2024 Red Hat, Inc.