Red Hat Summit Red Hat SummitSupportConsoleDevelopersStart a trialContact

Chapter 3. Using MicroShift Observability

MicroShift Observability collects and transmits system data for monitoring and analysis. The data includes performance and usage metrics, and error reporting.

You can install MicroShift Observability at any time, including during the initial MicroShift installation.

Procedure

  1. Install the microshift-observability RPM by entering the following command: 

    $ sudo dnf install microshift-observability
    Copy to Clipboard Toggle word wrap

  2. Enable the microshift-observability system service by entering the following command: 

    $ sudo systemctl enable microshift-observability
    Copy to Clipboard Toggle word wrap

  3. Start the microshift-observability system service by entering the following command: 

    $ sudo systemctl start microshift-observability
    Copy to Clipboard Toggle word wrap

  4. Restart MicroShift after the initial installation. 

    $ sudo systemctl restart microshift-observability
    Copy to Clipboard Toggle word wrap

The installation is successful if there is no output after you start the microshift-observability RPM.

3.2. Configuring MicroShift Observability
Copy link

You must configure MicroShift Observability after it is installed by specifying a valid endpoint. If an endpoint is not specified, MicroShift Observability does not start.

You can specify any OpenTelemetry Protocol (OTLP)-compatible endpoint for each configuration before starting MicroShift.

Procedure

  1. Update the /etc/microshift/observability/opentelemetry-collector.yaml file to specify an OTLP-compatible endpoint with the following information. The endpoint must link to an IP address or host name, and port number of an OTLP service.

    OTLP-compatible endpoint configuration

    # ...
exporters:
  otlp:
    sending_queue:
      storage: file_storage
    endpoint: ${env:OTEL_BACKEND}:4317
    1 
    
    tls:
      insecure: true
# ...
service:
# ...
  telemetry:
    metrics:
      readers:
        - periodic:
            exporter:
              otlp:
                protocol: http/protobuf
                endpoint: http://${env:OTEL_BACKEND}:4318
    2 
    
# ...
    Copy to Clipboard Toggle word wrap

    1
    Replace ${env:OTEL_BACKEND} with the IP address or hostname of the remote back end. This IP address resolves to the local node’s hostname. An unreachable endpoint is reported in the MicroShift service logs.
    2
    Replace ${env:OTEL_BACKEND} with the IP address or hostname of the remote back end. This IP address resolves to the local node’s hostname. An unreachable endpoint is reported in the MicroShift service logs.

  2. Each time that you update the opentelemetry-collector.yaml file, you must restart MicroShift Observability to apply the updates.

    Restart MicroShift Observability by entering the following command: 

    $ sudo systemctl restart microshift-observability
    Copy to Clipboard Toggle word wrap

The amount and complexity of the data depends on predefined configurations. These configurations determine the number of data sources and the amount of collected data that is transmitted. These configurations are defined as small, medium, and large (default).

The opentelemetry-collector.yaml file includes specific parameters that are used to collect data for monitoring the system resources. All warnings for cluster events are included in the collected data. MicroShift Observability collects and transmits data for the following resources:

  • CPU, memory, disk, and network metrics of containers, pods, and nodes
  • Kubernetes events
  • Host CPU, memory, disk, and network metrics
  • System journals for certain MicroShift services, and dependencies
  • Metrics exposed by pods that have the prometheus.io/scrape: true annotation

Replace the values of the exporters.otlp.endpoint and services.telemetry.metrics.readers[0].endpoint fields with the IP address or host name of the remote back end. This IP address resolves to the local node’s host name. Any unreachable endpoint is reported in the MicroShift observability service logs.

3.4. Selecting a small configuration
Copy link

You can configure MicroShift Observability to collect the smallest amount of performance and resource information from various sources by updating the YAML file.

Procedure

  1. Select a small configuration by adding the following information to the /etc/microshift/observability/opentelemetry-collector.yaml file: 

     receivers:
  kubeletstats:
    auth_type: tls
    ca_file: /var/lib/microshift/certs/ca-bundle/client-ca.crt
    key_file: /var/lib/microshift/certs/admin-kubeconfig-signer/openshift-observability-client/client.key
    cert_file: /var/lib/microshift/certs/admin-kubeconfig-signer/openshift-observability-client/client.crt
    insecure_skip_verify: true
    collection_interval: 10s
    endpoint: "${env:K8S_NODE_NAME}:10250"
    node: ${env:K8S_NODE_NAME}
    k8s_api_config:
      auth_type: kubeConfig
  k8s_events:
    auth_type: kubeConfig
processors:
  batch:
  resourcedetection/system:
    detectors: [ "system" ]
    system:
      hostname_sources: [ "os" ]
exporters:
  otlp:
    sending_queue:
      storage: file_storage
    endpoint: ${env:OTEL_BACKEND}:4317
    1 
    
    tls:
      insecure: true
extensions:
  file_storage:
    directory: /var/lib/microshift-observability
service:
  extensions: [ file_storage ]
  pipelines:
    metrics/kubeletstats:
      receivers: [ kubeletstats ]
      processors: [ batch ]
      exporters: [ otlp ]
    logs/kube_events:
      receivers: [ k8s_events ]
      processors: [ resourcedetection/system, batch ]
      exporters: [ otlp ]
  telemetry:
    metrics:
      readers:
        - periodic:
            exporter:
              otlp:
                protocol: http/protobuf
                endpoint: http://${env:OTEL_BACKEND}:4318
    2
    Copy to Clipboard Toggle word wrap
    1
    Replace the variable ${env:OTEL_BACKEND} with the IP address or hostname of the remote back end. This IP address resolves to the local node’s hostname. Any unreachable endpoint is reported in the MicroShift service logs.
    2
    Replace the variable ${env:OTEL_BACKEND} with the IP address or hostname of the remote back end. This IP address resolves to the local node’s hostname. Any unreachable endpoint is reported in the MicroShift service logs.

  2. Restart MicroShift Observability to complete the configuration selection by entering the following command: 

    $ sudo systemctl restart microshift-observability
    Copy to Clipboard Toggle word wrap

3.5. Selecting a medium configuration
Copy link

You can configure MicroShift Observability to collect performance and resource information from various sources by updating the YAML file.

Procedure

  1. Select a medium configuration by adding the following information to the /etc/microshift/observability/opentelemetry-collector.yaml file: 

     receivers:
  kubeletstats:
    auth_type: tls
    ca_file: /var/lib/microshift/certs/ca-bundle/client-ca.crt
    key_file: /var/lib/microshift/certs/admin-kubeconfig-signer/openshift-observability-client/client.key
    cert_file: /var/lib/microshift/certs/admin-kubeconfig-signer/openshift-observability-client/client.crt
    insecure_skip_verify: true
    collection_interval: 10s
    endpoint: "${env:K8S_NODE_NAME}:10250"
    node: ${env:K8S_NODE_NAME}
    k8s_api_config:
      auth_type: kubeConfig
  k8s_events:
    auth_type: kubeConfig
  journald:
    units:
      - microshift
    priority: info
processors:
  batch:
  resourcedetection/system:
    detectors: [ "system" ]
    system:
      hostname_sources: [ "os" ]
exporters:
  otlp:
    sending_queue:
      storage: file_storage
    endpoint: ${env:OTEL_BACKEND}:4317
    1 
    
    tls:
      insecure: true
extensions:
  file_storage:
    directory: /var/lib/microshift-observability
service:
  extensions: [ file_storage ]
  pipelines:
    metrics/kubeletstats:
      receivers: [ kubeletstats ]
      processors: [ batch ]
      exporters: [ otlp ]
    logs/kube_events:
      receivers: [ k8s_events ]
      processors: [ resourcedetection/system, batch ]
      exporters: [ otlp ]
    logs/journald:
      receivers: [ journald ]
      processors: [ resourcedetection/system ]
      exporters: [ otlp ]
  telemetry:
    metrics:
      readers:
        - periodic:
            exporter:
              otlp:
                protocol: http/protobuf
                endpoint: http://${env:OTEL_BACKEND}:4318
    2
    Copy to Clipboard Toggle word wrap
    1
    Replace the variable ${env:OTEL_BACKEND} with the IP address or hostname of the remote back end. This IP address resolves to the local node’s hostname. Any unreachable endpoint is reported in the microshift-observability service logs.
    2
    Replace the variable ${env:OTEL_BACKEND} with the IP address or hostname of the remote back end. This IP address resolves to the local node’s hostname. Any unreachable endpoint is reported in the microshift-observability service logs.

  2. Restart MicroShift Observability to complete the configuration selection by entering the following command: 

    $ sudo systemctl restart microshift-observability
    Copy to Clipboard Toggle word wrap

3.6. Selecting a large configuration
Copy link

You can configure MicroShift Observability to collect the maximum amount of performance and resource information, from the maximum number of sources, by updating the YAML file.

Procedure

  1. Select a large configuration by adding the following information to the /etc/microshift/observability/opentelemetry-collector.yaml file. Large is the default configuration. 

    receivers:
  kubeletstats:
    auth_type: tls
    ca_file: /var/lib/microshift/certs/ca-bundle/client-ca.crt
    key_file: /var/lib/microshift/certs/admin-kubeconfig-signer/openshift-observability-client/client.key
    cert_file: /var/lib/microshift/certs/admin-kubeconfig-signer/openshift-observability-client/client.crt
    insecure_skip_verify: true
    collection_interval: 10s
    endpoint: "${env:K8S_NODE_NAME}:10250"
    node: ${env:K8S_NODE_NAME}
    k8s_api_config:
      auth_type: kubeConfig
  k8s_events:
    auth_type: kubeConfig
  hostmetrics:
    root_path: /
    collection_interval: 10s
    scrapers:
      cpu:
      memory:
      network:
      disk:
      filesystem:
  journald:
    units:
      - microshift
      - microshift-observability
      - microshift-etcd
      - crio
      - openvswitch.service
      - ovsdb-server.service
      - ovs-vswitchd.service
    priority: info
  prometheus:
    config:
      scrape_configs:
        - job_name: k8s
          scrape_interval: 10s
          kubernetes_sd_configs:
            - kubeconfig_file: /var/lib/microshift/resources/observability-client/kubeconfig
              role: pod
          relabel_configs:
              # Only scrape Pods with annotation "prometheus.io/scrape": "true"
            - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
              action: keep
              regex: true
              # Use value of "prometheus.io/path" annotation for scraping
            - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
              action: replace
              target_label: __metrics_path__
              regex: (.+)
              # Use value of "prometheus.io/port" annotation for scraping
            - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
              action: replace
              regex: ([^:]+)(?::\d+)?;(\d+)
              replacement: $1:$2
              target_label: __address__
processors:
  batch:
  resourcedetection/system:
    detectors: [ "system" ]
    system:
      hostname_sources: [ "os" ]
exporters:
  otlp:
    sending_queue:
      storage: file_storage
    endpoint: ${env:OTEL_BACKEND}:4317
    1 
    
    tls:
      insecure: true
extensions:
  file_storage:
    directory: /var/lib/microshift-observability
service:
  extensions: [ file_storage ]
  pipelines:
    metrics/kubeletstats:
      receivers: [ kubeletstats ]
      processors: [ batch ]
      exporters: [ otlp ]
    metrics/hostmetrics:
      receivers: [ hostmetrics ]
      processors: [ resourcedetection/system, batch ]
      exporters: [ otlp ]
    logs/kube_events:
      receivers: [ k8s_events ]
      processors: [ resourcedetection/system, batch ]
      exporters: [ otlp ]
    logs/host:
      receivers: [ hostmetrics ]
      processors: [ resourcedetection/system ]
      exporters: [ otlp ]
    logs/journald:
      receivers: [ journald ]
      processors: [ resourcedetection/system ]
      exporters: [ otlp ]
    metrics/pods:
      receivers: [ prometheus ]
      processors: [ batch ]
      exporters: [ otlp ]
  telemetry:
    metrics:
      readers:
        - periodic:
            exporter:
              otlp:
                protocol: http/protobuf
                endpoint: http://${env:OTEL_BACKEND}:4318
    2
    Copy to Clipboard Toggle word wrap
    1
    Replace the variable ${env:OTEL_BACKEND} with the IP address or hostname of the remote back end. This IP address resolves to the local node’s hostname. Any unreachable endpoint is reported in the microshift-observability service logs.
    2
    Replace the variable ${env:OTEL_BACKEND} with the IP address or hostname of the remote back end. This IP address resolves to the local node’s hostname. Any unreachable endpoint is reported in the microshift-observability service logs.

  2. Restart MicroShift Observability to complete the configuration selection by entering the following command: 

    $ sudo systemctl restart microshift-observability
    Copy to Clipboard Toggle word wrap

3.7. Verifying the MicroShift Observability state
Copy link

After MicroShift Observability starts, you can verify the state by using a systemd service. The MicroShift Observability service logs are available as journald logs.

Procedure

  1. Check the MicroShift Observability status by entering the following command: 

    $ sudo systemctl status microshift-observability
    Copy to Clipboard Toggle word wrap

  2. Check the MicroShift Observability logs by entering the following command: 

    $ sudo journalctl -u microshift-observability
    Copy to Clipboard Toggle word wrap
Back to top
Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Theme

© 2025 Red Hat