Red Hat build of OpenTelemetry

流程

1. 安装红帽构建的 OpenTelemetry Operator：

   1. 进入 Operators → OperatorHub，搜索 红帽构建的 OpenTelemetry Operator。

   2. 选择 Red Hat build of OpenTelemetry Operator，provided by Red Hat → Install → Install → View Operator.

      重要

      这会使用默认预设置来安装 Operator：

      • Update channel → stable
      • Installation mode → All namespaces on the cluster
      • Installed Namespace → openshift-operators
      • Update approval → Automatic
   3. 在安装的 Operator 页面的 Details 选项卡中，在 ClusterServiceVersion details 下验证安装 Status 是否为 Succeeded。
2. 通过转至 Home → Projects → Create Project，为您在下一步中创建的 OpenTelemetry Collector 实例创建一个项目。

3. 创建 OpenTelemetry Collector 实例。

   1. 进入 Operators → Installed Operators。
   2. 选择 OpenTelemetry Collector → Create OpenTelemetry Collector → YAML view。

   3. 在 YAML 视图中，使用 OTLP、Jaeger、Zipkin receivers 和 debug exporter 自定义 OpenTelemetryCollector 自定义资源(CR)。

      apiVersion: opentelemetry.io/v1alpha1
      kind: OpenTelemetryCollector
      metadata:
        name: otel
        namespace: <project_of_opentelemetry_collector_instance>
      spec:
        mode: deployment
        config: |
          receivers:
            otlp:
              protocols:
                grpc:
                http:
            jaeger:
              protocols:
                grpc:
                thrift_binary:
                thrift_compact:
                thrift_http:
            zipkin:
          processors:
            batch:
            memory_limiter:
              check_interval: 1s
              limit_percentage: 50
              spike_limit_percentage: 30
          exporters:
            debug:
          service:
            pipelines:
              traces:
                receivers: [otlp,jaeger,zipkin]
                processors: [memory_limiter,batch]
                exporters: [debug]

   4. 选择 Create。

验证

1. 使用 Project: 下拉列表选择 OpenTelemetry Collector 实例的项目。
2. 进入 Operators → Installed Operators，以验证 OpenTelemetry Collector 实例的 Status 是否为 Condition: Ready。
3. 进入 Workloads → Pods，以验证 OpenTelemetry Collector 实例的所有组件 pod 都在运行。

流程

1. 安装红帽构建的 OpenTelemetry Operator：

   1. 运行以下命令，为红帽构建的 OpenTelemetry Operator 创建项目：

      $ oc apply -f - << EOF
      apiVersion: project.openshift.io/v1
      kind: Project
      metadata:
        labels:
          kubernetes.io/metadata.name: openshift-opentelemetry-operator
          openshift.io/cluster-monitoring: "true"
        name: openshift-opentelemetry-operator
      EOF

   2. 运行以下命令来创建 Operator 组：

      $ oc apply -f - << EOF
      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        name: openshift-opentelemetry-operator
        namespace: openshift-opentelemetry-operator
      spec:
        upgradeStrategy: Default
      EOF

   3. 运行以下命令来创建订阅：

      $ oc apply -f - << EOF
      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        name: opentelemetry-product
        namespace: openshift-opentelemetry-operator
      spec:
        channel: stable
        installPlanApproval: Automatic
        name: opentelemetry-product
        source: redhat-operators
        sourceNamespace: openshift-marketplace
      EOF

   4. 运行以下命令检查 Operator 状态：

      $ oc get csv -n openshift-opentelemetry-operator

2. 为您要在后续步骤中创建的 OpenTelemetry Collector 实例创建一个您选择的项目：

   • 要创建没有元数据的项目，请运行以下命令：

     $ oc new-project <project_of_opentelemetry_collector_instance>

   • 要使用元数据创建项目，请运行以下命令：

     $ oc apply -f - << EOF
     apiVersion: project.openshift.io/v1
     kind: Project
     metadata:
       name: <project_of_opentelemetry_collector_instance>
     EOF

3. 在为您创建的项目中创建一个 OpenTelemetry Collector 实例。

   注意

   您可以在同一集群中的独立项目中创建多个 OpenTelemetry Collector 实例。

   1. 使用 OTLP、Jaeger 和 Zipkin receivers 和 debug exporter 自定义 OpenTelemetry Collector 自定义资源 (CR)：

      apiVersion: opentelemetry.io/v1alpha1
      kind: OpenTelemetryCollector
      metadata:
        name: otel
        namespace: <project_of_opentelemetry_collector_instance>
      spec:
        mode: deployment
        config: |
          receivers:
            otlp:
              protocols:
                grpc:
                http:
            jaeger:
              protocols:
                grpc:
                thrift_binary:
                thrift_compact:
                thrift_http:
            zipkin:
          processors:
            batch:
            memory_limiter:
              check_interval: 1s
              limit_percentage: 50
              spike_limit_percentage: 30
          exporters:
            debug:
          service:
            pipelines:
              traces:
                receivers: [otlp,jaeger,zipkin]
                processors: [memory_limiter,batch]
                exporters: [debug]

   2. 运行以下命令来应用自定义 CR：

      $ oc apply -f - << EOF
      <OpenTelemetryCollector_custom_resource>
      EOF

验证

1. 运行以下命令，验证 OpenTelemetry Collector pod 的 status.phase 是否为 Running，条件为 type: Ready ：

   $ oc get pod -l app.kubernetes.io/managed-by=opentelemetry-operator,app.kubernetes.io/instance=<namespace>.<instance_name> -o yaml

2. 运行以下命令来获取 OpenTelemetry Collector 服务：

   $ oc get service -l app.kubernetes.io/managed-by=opentelemetry-operator,app.kubernetes.io/instance=<namespace>.<instance_name>

流程

1. 在 OpenTelemetry Collector 实例中挂载以下证书，跳过已挂载的证书。

   1. 使用 cert-manager Operator for Red Hat OpenShift 生成这些证书的签发者。

      apiVersion: cert-manager.io/v1
      kind: Issuer
      metadata:
        name: selfsigned-issuer
      spec:
        selfSigned: {}

   2. 一个自签名证书。

      apiVersion: cert-manager.io/v1
      kind: Certificate
      metadata:
        name: ca
      spec:
        isCA: true
        commonName: ca
        subject:
          organizations:
            - Organization # <your_organization_name>
          organizationalUnits:
            - Widgets
        secretName: ca-secret
        privateKey:
          algorithm: ECDSA
          size: 256
        issuerRef:
          name: selfsigned-issuer
          kind: Issuer
          group: cert-manager.io

   3. 一个 CA 签发者。

      apiVersion: cert-manager.io/v1
      kind: Issuer
      metadata:
        name: test-ca-issuer
      spec:
        ca:
          secretName: ca-secret

   4. 客户端和服务器证书。

      apiVersion: cert-manager.io/v1
      kind: Certificate
      metadata:
        name: server
      spec:
        secretName: server-tls
        isCA: false
        usages:
          - server auth
          - client auth
        dnsNames:
        - "otel.observability.svc.cluster.local" 1
        issuerRef:
          name: ca-issuer
      ---
      apiVersion: cert-manager.io/v1
      kind: Certificate
      metadata:
        name: client
      spec:
        secretName: client-tls
        isCA: false
        usages:
          - server auth
          - client auth
        dnsNames:
        - "otel.observability.svc.cluster.local" 2
        issuerRef:
          name: ca-issuer

      1

      在 server OpenTelemetry Collector 实例中映射到 solver 的确切 DNS 名称列表。

      2

      在客户端 OpenTelemetry Collector 实例中映射到 solver 的确切 DNS 名称列表。

2. 为 OpenTelemetry Collector 实例创建服务帐户。

   ServiceAccount 示例

   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: otel-collector-deployment

3. 为服务帐户创建集群角色。

   ClusterRole 示例

   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRole
   metadata:
     name: otel-collector
   rules:
     1
     2
   - apiGroups: ["", "config.openshift.io"]
     resources: ["pods", "namespaces", "infrastructures", "infrastructures/status"]
     verbs: ["get", "watch", "list"]

   1

   k8sattributesprocessor 需要 pod 和命名空间资源的权限。

   2

   resourcedetectionprocessor 需要基础架构和状态的权限。

4. 将集群角色绑定到服务帐户。

   ClusterRoleBinding 示例

   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRoleBinding
   metadata:
     name: otel-collector
   subjects:
   - kind: ServiceAccount
     name: otel-collector-deployment
     namespace: otel-collector-<example>
   roleRef:
     kind: ClusterRole
     name: otel-collector
     apiGroup: rbac.authorization.k8s.io

5. 创建 YAML 文件，在边缘集群中定义 OpenTelemetryCollector 自定义资源 (CR)。

   边缘集群的 OpenTelemetryCollector 自定义资源示例

   apiVersion: opentelemetry.io/v1alpha1
   kind: OpenTelemetryCollector
   metadata:
     name: otel
     namespace: otel-collector-<example>
   spec:
     mode: daemonset
     serviceAccount: otel-collector-deployment
     config: |
       receivers:
         jaeger:
           protocols:
             grpc:
             thrift_binary:
             thrift_compact:
             thrift_http:
         opencensus:
         otlp:
           protocols:
             grpc:
             http:
         zipkin:
       processors:
         batch:
         k8sattributes:
         memory_limiter:
           check_interval: 1s
           limit_percentage: 50
           spike_limit_percentage: 30
         resourcedetection:
           detectors: [openshift]
       exporters:
         otlphttp:
           endpoint: https://observability-cluster.com:443 1
           tls:
             insecure: false
             cert_file: /certs/server.crt
             key_file: /certs/server.key
             ca_file: /certs/ca.crt
       service:
         pipelines:
           traces:
             receivers: [jaeger, opencensus, otlp, zipkin]
             processors: [memory_limiter, k8sattributes, resourcedetection, batch]
             exporters: [otlp]
     volumes:
       - name: otel-certs
         secret:
           name: otel-certs
     volumeMounts:
       - name: otel-certs
         mountPath: /certs

   1

   Collector exporter 配置为导出 OTLP HTTP，并指向来自中央集群的 OpenTelemetry Collector。

6. 创建 YAML 文件，在中央集群中定义 OpenTelemetryCollector 自定义资源 (CR)。

   Central 集群的 OpenTelemetryCollector 自定义资源示例

   apiVersion: opentelemetry.io/v1alpha1
   kind: OpenTelemetryCollector
   metadata:
     name: otlp-receiver
     namespace: observability
   spec:
     mode: "deployment"
     ingress:
       type: route
       route:
         termination: "passthrough"
     config: |
       receivers:
         otlp:
           protocols:
             http:
               tls: 1
                 cert_file: /certs/server.crt
                 key_file: /certs/server.key
                 client_ca_file: /certs/ca.crt
       exporters:
         logging:
         otlp:
           endpoint: "tempo-<simplest>-distributor:4317" 2
           tls:
             insecure: true
       service:
         pipelines:
           traces:
             receivers: [otlp]
             processors: []
             exporters: [otlp]
     volumes:
       - name: otel-certs
         secret:
           name: otel-certs
     volumeMounts:
       - name: otel-certs
         mountPath: /certs

   1

   Collector 接收器需要第一步中列出的证书。

   2

   Collector exporter 配置为导出 OTLP 并指向 Tempo 经销商端点，本例中为 "tempo-simplest-distributor:4317" 并已创建。

流程

1. 为 OpenTelemetry Collector 创建服务帐户。

   ServiceAccount 示例

   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: otel-collector-deployment

2. 为服务帐户创建集群角色。

   ClusterRole 示例

   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRole
   metadata:
     name: otel-collector
   rules:
     1
     2
   - apiGroups: ["", "config.openshift.io"]
     resources: ["pods", "namespaces", "infrastructures", "infrastructures/status"]
     verbs: ["get", "watch", "list"]

   1

   k8sattributesprocessor 需要 pod 和命名空间资源的权限。

   2

   resourcedetectionprocessor 需要基础架构和状态的权限。

3. 将集群角色绑定到服务帐户。

   ClusterRoleBinding 示例

   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRoleBinding
   metadata:
     name: otel-collector
   subjects:
   - kind: ServiceAccount
     name: otel-collector-deployment
     namespace: otel-collector-example
   roleRef:
     kind: ClusterRole
     name: otel-collector
     apiGroup: rbac.authorization.k8s.io

4. 创建 YAML 文件以定义 OpenTelemetryCollector 自定义资源(CR)。

   OpenTelemetryCollector 示例

   apiVersion: opentelemetry.io/v1alpha1
   kind: OpenTelemetryCollector
   metadata:
     name: otel
   spec:
     mode: deployment
     serviceAccount: otel-collector-deployment
     config: |
       receivers:
         jaeger:
           protocols:
             grpc:
             thrift_binary:
             thrift_compact:
             thrift_http:
         opencensus:
         otlp:
           protocols:
             grpc:
             http:
         zipkin:
       processors:
         batch:
         k8sattributes:
         memory_limiter:
           check_interval: 1s
           limit_percentage: 50
           spike_limit_percentage: 30
         resourcedetection:
           detectors: [openshift]
       exporters:
         otlp:
           endpoint: "tempo-simplest-distributor:4317" 1
           tls:
             insecure: true
       service:
         pipelines:
           traces:
             receivers: [jaeger, opencensus, otlp, zipkin] 2
             processors: [memory_limiter, k8sattributes, resourcedetection, batch]
             exporters: [otlp]

   1

   Collector exporter 配置为导出 OTLP 并指向 Tempo 经销商端点 "tempo-simplest-distributor:4317" （在这个示例中已创建）。

   2

   Collector 配置了 Jaeger trace 的接收器，OpenCensus trace over the OpenCensus 协议, Zipkin trace over the Zipkin protocol, 和 OTLP trace over the GRPC 协议。

流程

1. 在 OpenTelemetryCollector 自定义资源(CR) 中设置相关的日志级别：

     config: |
       service:
         telemetry:
           logs:
             level: debug 1

   1

   收集器的日志级别。支持的值包括 info、warn、error 或 debug。默认为 info。

2. 使用 oc logs 命令或 Web 控制台来检索日志。

流程

1. 通过在 OpenTelemetryCollector 自定义资源中添加以下行来启用 telemetry 服务：

     config: |
       service:
         telemetry:
           metrics:
             address: ":8888" 1

   1

   公开内部收集器指标的地址。默认值为 :8888。

1. 运行以下命令来检索指标，该命令使用端口转发 Collector pod：

   $ oc port-forward <collector_pod>

2. 访问位于 http://localhost:8888/metrics 的指标端点。

流程

1. 配置 OpenTelemetryCollector 自定义资源，如下所示：

     config: |
       exporters:
         debug:
           verbosity: detailed
       service:
         pipelines:
           traces:
             exporters: [debug]
           metrics:
             exporters: [debug]
           logs:
             exporters: [debug]

2. 使用 oc logs 命令或 Web 控制台将日志导出到标准输出。

流程

1. 将 OpenTelemetry Collector 配置为 sidecar。

   apiVersion: opentelemetry.io/v1alpha1
   kind: OpenTelemetryCollector
   metadata:
     name: otel
     namespace: <otel-collector-namespace>
   spec:
     mode: sidecar
     config: |
       receivers:
         jaeger:
           protocols:
             grpc:
             thrift_binary:
             thrift_compact:
             thrift_http:
       processors:
         batch:
         memory_limiter:
           check_interval: 1s
           limit_percentage: 50
           spike_limit_percentage: 30
         resourcedetection:
           detectors: [openshift]
           timeout: 2s
       exporters:
         otlp:
           endpoint: "tempo-<example>-gateway:8090" 1
           tls:
             insecure: true
       service:
         pipelines:
           traces:
             receivers: [jaeger]
             processors: [memory_limiter, resourcedetection, batch]
             exporters: [otlp]

   1

   此端点指向使用 Tempo Operator 部署的 <example> TempoStack 实例的网关。

2. 创建用于运行应用程序的服务帐户。

   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: otel-collector-sidecar

3. 为某些处理器所需的权限创建集群角色。

   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRole
   metadata:
     name: otel-collector-sidecar
   rules:
     1
   - apiGroups: ["config.openshift.io"]
     resources: ["infrastructures", "infrastructures/status"]
     verbs: ["get", "watch", "list"]

   1

   resourcedetectionprocessor 需要基础架构和基础架构/状态的权限。

4. 创建 ClusterRoleBinding 来为服务帐户设置权限。

   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRoleBinding
   metadata:
     name: otel-collector-sidecar
   subjects:
   - kind: ServiceAccount
     name: otel-collector-deployment
     namespace: otel-collector-example
   roleRef:
     kind: ClusterRole
     name: otel-collector
     apiGroup: rbac.authorization.k8s.io

5. 将 OpenTelemetry Collector 部署为 sidecar。
6. 通过从 Deployment 对象中删除 "sidecar.jaegertracing.io/inject": "true" 注解，从应用程序中删除注入的 Jaeger Agent。
7. 通过将 sidecar.opentelemetry.io/inject: "true" 注解添加到 Deployment 对象的 .spec.template.metadata.annotations 字段来启用 OpenTelemetry sidecar 自动注入。
8. 使用为应用程序部署创建的服务帐户，以允许处理器获取正确的信息并将其添加到您的追踪中。

流程

1. 配置 OpenTelemetry Collector 部署。

2. 创建部署 OpenTelemetry Collector 的项目。

   apiVersion: project.openshift.io/v1
   kind: Project
   metadata:
     name: observability

3. 创建用于运行 OpenTelemetry Collector 实例的服务帐户。

   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: otel-collector-deployment
     namespace: observability

4. 创建集群角色，以设置处理器所需的权限。

   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRole
   metadata:
     name: otel-collector
   rules:
     1
     2
   - apiGroups: ["", "config.openshift.io"]
     resources: ["pods", "namespaces", "infrastructures", "infrastructures/status"]
     verbs: ["get", "watch", "list"]

   1

   k8sattributesprocessor 需要 pods 和 namespaces 资源的权限。

   2

   resourcedetectionprocessor 需要 infrastructures 和 infrastructures/status 的权限。

5. 创建 ClusterRoleBinding 来为服务帐户设置权限。

   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRoleBinding
   metadata:
     name: otel-collector
   subjects:
   - kind: ServiceAccount
     name: otel-collector-deployment
     namespace: observability
   roleRef:
     kind: ClusterRole
     name: otel-collector
     apiGroup: rbac.authorization.k8s.io

6. 创建 OpenTelemetry Collector 实例。

   注意

   此收集器会将 trace 导出至 TempoStack 实例。您必须使用 Red Hat Tempo Operator 创建 TempoStack 实例，并放在正确的端点中。

   apiVersion: opentelemetry.io/v1alpha1
   kind: OpenTelemetryCollector
   metadata:
     name: otel
     namespace: observability
   spec:
     mode: deployment
     serviceAccount: otel-collector-deployment
     config: |
       receivers:
         jaeger:
           protocols:
             grpc:
             thrift_binary:
             thrift_compact:
             thrift_http:
       processors:
         batch:
         k8sattributes:
         memory_limiter:
           check_interval: 1s
           limit_percentage: 50
           spike_limit_percentage: 30
         resourcedetection:
           detectors: [openshift]
       exporters:
         otlp:
           endpoint: "tempo-example-gateway:8090"
           tls:
             insecure: true
       service:
         pipelines:
           traces:
             receivers: [jaeger]
             processors: [memory_limiter, k8sattributes, resourcedetection, batch]
             exporters: [otlp]

7. 将追踪端点指向 OpenTelemetry Operator。

8. 如果您要将 trace 直接从应用程序导出到 Jaeger，请将 API 端点从 Jaeger 端点改为 OpenTelemetry Collector 端点。

   使用带有 Golang 的 jaegerexporter 导出 trace 的示例

   exp, err := jaeger.New(jaeger.WithCollectorEndpoint(jaeger.WithEndpoint(url))) 1

   1

   URL 指向 OpenTelemetry Collector API 端点。

流程

1. 进入 Operators → Installed Operators → Red Hat build of OpenTelemetry Operator → OpenTelemetryInstrumentation 或 OpenTelemetryCollector。
2. 要删除相关实例，请选择 → Delete …​ → Delete。
3. 可选：删除红帽构建的 OpenTelemetry Operator。

流程

1. 运行以下命令，获取 OpenTelemetry Collector 实例的名称：

   $ oc get deployments -n <project_of_opentelemetry_instance>

2. 运行以下命令来删除 OpenTelemetry Collector 实例：

   $ oc delete opentelemetrycollectors <opentelemetry_instance_name> -n <project_of_opentelemetry_instance>

3. 可选：删除红帽构建的 OpenTelemetry Operator。