Search

Chapter 5. Observability UI plugins

download PDF

5.1. Observability UI plugins overview

Important

The Cluster Observability Operator is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

You can use the Cluster Observability Operator (COO) to install and manage UI plugins to enhance the observability capabilities of the OpenShift Container Platform web console. The plugins extend the default functionality, providing new UI features for monitoring, troubleshooting, distributed tracing, and cluster logging.

<dashboards_observability-ui-plugins-overview><title>Dashboards</title>

The dashboard UI plugin supports enhanced dashboards in the OpenShift Container Platform web console at Observe Dashboards. You can add other Prometheus data sources from the cluster to the default dashboards, in addition to the in-cluster data source. This results in a unified observability experience across different data sources.

For more information, see the dashboard UI plugin page.

</dashboards_observability-ui-plugins-overview>
<troubleshooting_observability-ui-plugins-overview><title>Troubleshooting</title>

The troubleshooting panel UI plugin for OpenShift Container Platform version 4.16+ provides observability signal correlation, powered by the open source Korrel8r project. You can use the troubleshooting panel available from the Observe Alerting page to easily correlate metrics, logs, alerts, netflows, and additional observability signals and resources, across different data stores. Users of OpenShift Container Platform version 4.17+ can also access the troubleshooting UI panel from the Application Launcher app launcher .

The output of Korrel8r is displayed as an interactive node graph. When you click on a node, you are automatically redirected to the corresponding web console page with the specific information for that node, for example, metric, log, or pod.

For more information, see the troubleshooting UI plugin page.

</troubleshooting_observability-ui-plugins-overview>
<distributed-tracing_observability-ui-plugins-overview><title>Distributed tracing</title>

The distributed tracing UI plugin adds tracing-related features to the web console on the Observe Traces page. You can follow requests through the front end and into the backend of microservices, helping you identify code errors and performance bottlenecks in distributed systems. You can select a supported TempoStack or TempoMonolithic multi-tenant instance running in the cluster and set a time range and query to view the trace data.

For more information, see the distributed tracing UI plugin page.

</distributed-tracing_observability-ui-plugins-overview>
<cluster-logging_observability-ui-plugins-overview><title>Cluster logging</title>

The logging UI plugin surfaces logging data in the web console on the Observe Logs page. You can specify filters, queries, time ranges and refresh rates. The results displayed a list of collapsed logs, which can then be expanded to show more detailed information for each log.

For more information, see the logging UI plugin page.

</cluster-logging_observability-ui-plugins-overview>

5.2. Dashboard UI plugin

Important

The Cluster Observability Operator is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

The dashboard UI plugin supports enhanced dashboards in the OpenShift web console at Observe Dashboards . You can add other Prometheus datasources from the cluster to the default dashboards, in addition to the in-cluster datasource. This results in a unified observability experience across different data sources.

The plugin searches for datasources from ConfigMap resources in the openshift-config-managed namespace, that have the label console.openshift.io/dashboard-datasource: 'true'.

5.2.1. Installing the Cluster Observability Operator dashboard UI plugin

Prerequisites

  • You have access to the cluster as a user with the cluster-admin cluster role.
  • You have logged in to the OpenShift Container Platform web console.
  • You have installed the Cluster Observability Operator.

Procedure

  1. In the OpenShift Container Platform web console, click Operators Installed Operators and select Cluster Observability Operator.
  2. Choose the UI Plugin tab (at the far right of the tab list) and press Create UIPlugin.
  3. Select YAML view, enter the following content, and then press Create:

    apiVersion: observability.openshift.io/v1alpha1
    kind: UIPlugin
    metadata:
      name: dashboards
    spec:
      type: Dashboards

5.2.2. Configuring a dashboard

The dashboard UI plugin searches for datasources from ConfigMap resources in the openshift-config-managed namespace, that have the label console.openshift.io/dashboard-datasource: 'true'. The ConfigMap resource must define a datasource type and an in-cluster service where the data can be fetched.

The examples in the following section are taken from https://github.com/openshift/console-dashboards-plugin.

Prerequisites

  • You have access to the cluster as a user with the cluster-admin cluster role.
  • You have logged in to the OpenShift Container Platform web console.
  • You have installed the Cluster Observability Operator.
  • You have installed the dashboard UI plugin.

Procedure

  1. Create a ConfigMap resource in the openshift-config-managed namespace, with the label console.openshift.io/dashboard-datasource: 'true'. The example below is from prometheus-datasource-example.yaml

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: cluster-prometheus-proxy
      namespace: openshift-config-managed
      labels:
        console.openshift.io/dashboard-datasource: "true"
    data:
      "dashboard-datasource.yaml": |-
        kind: "Datasource"
        metadata:
          name: "cluster-prometheus-proxy"
          project: "openshift-config-managed"
        spec:
          plugin:
            kind: "prometheus"
            spec:
              direct_url: "https://prometheus-k8s.openshift-monitoring.svc.cluster.local:9091"
  2. Configure a custom dashboard that connects to the datasource. The YAML for a sample dashboard is available at prometheus-dashboard-example.yaml. An excerpt from that file is shown below for demonstration purposes:

    Example 5.1. Extract from example dashboard, taken from prometheus-dashboard-example.yaml

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: dashboard-example
      namespace: openshift-config-managed
      labels:
        console.openshift.io/dashboard: "true"
    data:
      k8s-resources-workloads-namespace.json: |-
        {
            "annotations": {
                "list": [
    
                ]
            },
            "editable": true,
            "gnetId": null,
            "graphTooltip": 0,
            "hideControls": false,
            "links": [
    
            ],
            "refresh": "10s",
            "rows": [
                {
                    "collapse": false,
                    "height": "250px",
                    "panels": [
                        {
                            "aliasColors": {
    
                            },
                            "bars": false,
                            "dashLength": 10,
                            "dashes": false,
                            "datasource": {
                                "name": "cluster-prometheus-proxy",
                                "type": "prometheus"
                            },
                            "fill": 10,
                            "id": 1,
                            "interval": "1m",
                            "legend": {
                                "alignAsTable": true,
                                "avg": false,
                                "current": false,
                                "max": false,
                                "min": false,
                                "rightSide": true,
                                "show": true,
                                "total": false,
                                "values": false
                            },
                            "lines": true,
                            "linewidth": 0,
                            "links": [
    
                            ],
                            "nullPointMode": "null as zero",
                            "percentage": false,
                            "pointradius": 5,
                            "points": false,
                            "renderer": "flot",
                            "seriesOverrides": [
                                {
                                    "alias": "quota - requests",
                                    "color": "#F2495C",
                                    "dashes": true,
                                    "fill": 0,
                                    "hiddenSeries": true,
                                    "hideTooltip": true,
                                    "legend": true,
                                    "linewidth": 2,
                                    "stack": false
                                },
                                {
                                    "alias": "quota - limits",
                                    "color": "#FF9830",
                                    "dashes": true,
                                    "fill": 0,
                                    "hiddenSeries": true,
                                    "hideTooltip": true,
                                    "legend": true,
                                    "linewidth": 2,
                                    "stack": false
                                }
                            ],
                            "spaceLength": 10,
                            "span": 12,
                            "stack": false,
                            "steppedLine": false,
                            "targets": [
                                {
                                    "expr": "sum(  node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster=\"$cluster\", namespace=\"$namespace\"}* on(namespace,pod)  group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}) by (workload, workload_type)",
                                    "format": "time_series",
                                    "intervalFactor": 2,
                                    "legendFormat": "{{workload}} - {{workload_type}}",
                                    "legendLink": null,
                                    "step": 10
                                },
                                {
                                    "expr": "scalar(kube_resourcequota{cluster=\"$cluster\", namespace=\"$namespace\", type=\"hard\",resource=\"requests.cpu\"})",
                                    "format": "time_series",
                                    "intervalFactor": 2,
                                    "legendFormat": "quota - requests",
                                    "legendLink": null,
                                    "step": 10
                                },
                                {
                                    "expr": "scalar(kube_resourcequota{cluster=\"$cluster\", namespace=\"$namespace\", type=\"hard\",resource=\"limits.cpu\"})",
                                    "format": "time_series",
                                    "intervalFactor": 2,
                                    "legendFormat": "quota - limits",
                                    "legendLink": null,
                                    "step": 10
                                }
                            ],
                            "thresholds": [
    
                            ],
                            "timeFrom": null,
                            "timeShift": null,
                            "title": "CPU Usage",
                            "tooltip": {
                                "shared": false,
                                "sort": 2,
                                "value_type": "individual"
                            },
                            "type": "graph",
                            "xaxis": {
                                "buckets": null,
                                "mode": "time",
                                "name": null,
                                "show": true,
                                "values": [
    
                                ]
                            },
    ...
  3. Click Observe Dashboards and the custom dashboard is available with the title ** DASHBOARD EXAMPLE **, based on the configuration in prometheus-dashboard-example.yaml.

    coo custom dashboard

    You can set the namespace, time range and refresh interval for the dashboard in the UI.

5.2.3. Additional resources

5.3. Distributed tracing UI plugin

Important

The Cluster Observability Operator is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

The distributed tracing UI plugin adds tracing-related features to the Administrator perspective of the OpenShift web console at Observe Traces. You can follow requests through the front end and into the backend of microservices, helping you identify code errors and performance bottlenecks in distributed systems.

5.3.1. Installing the Cluster Observability Operator distributed tracing UI plugin

Prerequisites

  • You have access to the cluster as a user with the cluster-admin cluster role.
  • You have logged in to the OpenShift Container Platform web console.
  • You have installed the Cluster Observability Operator

Procedure

  1. In the OpenShift Container Platform web console, click Operators Installed Operators and select Cluster Observability Operator
  2. Choose the UI Plugin tab (at the far right of the tab list) and press Create UIPlugin
  3. Select YAML view, enter the following content, and then press Create:

    apiVersion: observability.openshift.io/v1alpha1
    kind: UIPlugin
    metadata:
      name: distributed-tracing
    spec:
      type: DistributedTracing

5.3.2. Using the Cluster Observability Operator distributed tracing UI plugin

Prerequisites

  • You have access to the cluster as a user with the cluster-admin cluster role.
  • You have logged in to the OpenShift Container Platform web console.
  • You have installed the Cluster Observability Operator.
  • You have installed the Cluster Observability Operator distributed tracing UI plugin.
  • You have a TempoStack or TempoMonolithic multi-tenant instance in the cluster.

Procedure

  1. In the Administrator perspective of the OpenShift Container Platform web console, click Observe Traces.
  2. Select a TempoStack or TempoMonolithic multi-tenant instance and set a time range and query for the traces to be loaded.

    The traces are displayed on a scatter-plot showing the trace start time, duration, and number of spans. Underneath the scatter plot, there is a list of traces showing information such as the Trace Name, number of Spans, and Duration.

  3. Click on a trace name link.

    The trace detail page for the selected trace contains a Gantt Chart of all of the spans within the trace. Select a span to show a breakdown of the configured attributes.

5.4. Troubleshooting UI plugin

Important

The Cluster Observability Operator is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

The troubleshooting UI plugin for OpenShift Container Platform version 4.16+ provides observability signal correlation, powered by the open source Korrel8r project. With the troubleshooting panel that is available under Observe Alerting, you can easily correlate metrics, logs, alerts, netflows, and additional observability signals and resources, across different data stores. Users of OpenShift Container Platform version 4.17+ can also access the troubleshooting UI panel from the Application Launcher app launcher .

When you install the troubleshooting UI plugin, a Korrel8r service named korrel8r is deployed in the same namespace, and it is able to locate related observability signals and Kubernetes resources from its correlation engine.

The output of Korrel8r is displayed in the form of an interactive node graph in the OpenShift Container Platform web console. Nodes in the graph represent a type of resource or signal, while edges represent relationships. When you click on a node, you are automatically redirected to the corresponding web console page with the specific information for that node, for example, metric, log, pod.

5.4.1. Installing the Cluster Observability Operator Troubleshooting UI plugin

Prerequisites

  • You have access to the cluster as a user with the cluster-admin cluster role.
  • You have logged in to the OpenShift Container Platform web console.
  • You have installed the Cluster Observability Operator

Procedure

  1. In the OpenShift Container Platform web console, click Operators Installed Operators and select Cluster Observability Operator
  2. Choose the UI Plugin tab (at the far right of the tab list) and press Create UIPlugin
  3. Select YAML view, enter the following content, and then press Create:

    apiVersion: observability.openshift.io/v1alpha1
    kind: UIPlugin
    metadata:
      name: troubleshooting-panel
    spec:
      type: TroubleshootingPanel

5.4.2. Using the Cluster Observability Operator troubleshooting UI plugin

Prerequisites

  • You have access to the OpenShift Container Platform cluster as a user with the cluster-admin cluster role. If your cluster version is 4.17+, you can access the troubleshooting UI panel from the Application Launcher app launcher .
  • You have logged in to the OpenShift Container Platform web console.
  • You have installed OpenShift Container Platform Logging, if you want to visualize correlated logs.
  • You have installed OpenShift Container Platform Network Observability, if you want to visualize correlated netflows.
  • You have installed the Cluster Observability Operator.
  • You have installed the Cluster Observability Operator troubleshooting UI plugin.

    Note

    The troubleshooting panel relies on the observability signal stores installed in your cluster. Kuberenetes resources, alerts and metrics are always available by default in an OpenShift Container Platform cluster. Other signal types require optional components to be installed:

    • Logs: Red Hat Openshift Logging (collection) and Loki Operator provided by Red Hat (store)
    • Network events: Network observability provided by Red Hat (collection) and Loki Operator provided by Red Hat (store)

    Procedure

    1. In the admin perspective of the web console, navigate to Observe Alerting and then select an alert. If the alert has correlated items, a Troubleshooting Panel link will appear above the chart on the alert detail page.

      Troubleshooting Panel link

      Click on the Troubleshooting Panel link to display the panel.

    2. The panel consists of query details and a topology graph of the query results. The selected alert is converted into a Korrel8r query string and sent to the korrel8r service. The results are displayed as a graph network connecting the returned signals and resources. This is a neighbourhood graph, starting at the current resource and including related objects up to 3 steps away from the starting point. Clicking on nodes in the graph takes you to the corresponding web console pages for those resouces.
    3. You can use the troubleshooting panel to find resources relating to the chosen alert.

      Note

      Clicking on a node may sometimes show fewer results than indicated on the graph. This is a known issue that will be addressed in a future release.

      Troubleshooting panel
      1. Alert (1): This node is the starting point in the graph and represents the KubeContainerWaiting alert displayed in the web console.
      2. Pod (1): This node indicates that there is a single Pod resource associated with this alert. Clicking on this node will open a console search showing the related pod directly.
      3. Event (2): There are two Kuberenetes events associated with the pod. Click this node to see the events.
      4. Logs (74): This pod has 74 lines of logs, which you can access by clicking on this node.
      5. Metrics (105): There are many metrics associated with the pod.
      6. Network (6): There are network events, meaning the pod has communicated over the network. The remaining nodes in the graph represent the Service, Deployment and DaemonSet resources that the pod has communicated with.
      7. Focus: Clicking this button updates the graph. By default, the graph itself does not change when you click on nodes in the graph. Instead, the main web console page changes, and you can then navigate to other resources using links on the page, while the troubleshooting panel itself stays open and unchanged. To force an update to the graph in the troubleshooting panel, click Focus. This draws a new graph, using the current resource in the web console as the starting point.
      8. Show Query: Clicking this button enables some experimental features:

        Experimental features
        1. Hide Query hides the experimental features.
        2. The query that identifies the starting point for the graph. The query language, part of the Korrel8r correlation engine used to create the graphs, is experimental and may change in future. The query is updated by the Focus button to correspond to the resources in the main web console window.
        3. Neighbourhood depth is used to display a smaller or larger neighbourhood.

          Note

          Setting a large value in a large cluster might cause the query to fail, if the number of results is too big.

        4. Goal class results in a goal directed search instead of a neighbourhood search. A goal directed search shows all paths from the starting point to the goal class, which indicates a type of resource or signal. The format of the goal class is experimental and may change. Currently, the following goals are valid:

          • k8s:RESOURCE[VERSION.[GROUP]] identifying a kind of kuberenetes resource. For example k8s:Pod or k8s:Deployment.apps.v1.
          • alert:alert representing any alert.
          • metric:metric representing any metric.
          • netflow:network representing any network observability network event.
          • log:LOG_TYPE representing stored logs, where LOG_TYPE must be one of application, infrastructure or audit.

5.4.3. Creating the example alert

To trigger an alert as a starting point to use in the troubleshooting UI panel, you can deploy a container that is deliberately misconfigured.

Procedure

  1. Use the following YAML, either from the command line or in the web console, to create a broken deployment in a system namespace:

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: bad-deployment
      namespace: default 1
    spec:
      selector:
        matchLabels:
          app: bad-deployment
      template:
        metadata:
          labels:
            app: bad-deployment
        spec:
          containers: 2
          - name: bad-deployment
            image: quay.io/openshift-logging/vector:5.8
    1
    The deployment must be in a system namespace (such as default) to cause the desired alerts.
    2
    This container deliberately tries to start a vector server with no configuration file. The server logs a few messages, and then exits with an error. Alternatively, you can deploy any container you like that is badly configured, causing it to trigger an alert.
  2. View the alerts:

    1. Go to Observe Alerting and click clear all filters. View the Pending alerts.

      Important

      Alerts first appear in the Pending state. They do not start Firing until the container has been crashing for some time. By viewing Pending alerts, you do not have to wait as long to see them occur.

    2. Choose one of the KubeContainerWaiting, KubePodCrashLooping, or KubePodNotReady alerts and open the troubleshooting panel by clicking on the link. Alternatively, if the panel is already open, click the "Focus" button to update the graph.

5.5. Logging UI plugin

Important

The Cluster Observability Operator is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

The logging UI plugin surfaces logging data in the OpenShift Container Platform web console on the Observe Logs page. You can specify filters, queries, time ranges and refresh rates, with the results displayed as a list of collapsed logs, which can then be expanded to show more detailed information for each log.

When you have also deployed the Troubleshooting UI plugin on OpenShift Container Platform version 4.16+, it connects to the Korrel8r service and adds direct links from the Administration perspective, from the Observe Logs page, to the Observe Metrics page with a correlated PromQL query. It also adds a See Related Logs link from the Administration perspective alerting detail page, at Observe Alerting, to the Observe Logs page with a correlated filter set selected.

The features of the plugin are categorized as:

dev-console
Adds the logging view to the Developer perspective.
alerts
Merges the web console alerts with log-based alerts defined in the Loki ruler. Adds a log-based metrics chart in the alert detail view.
dev-alerts
Merges the web console alerts with log-based alerts defined in the Loki ruler. Adds a log-based metrics chart in the alert detail view for the Developer perspective.

For Cluster Observability Operator (COO) versions, the support for these features in OpenShift Container Platform versions is shown in the following table:

COO versionOCP versionsFeatures

0.3.0+

4.12

dev-console

0.3.0+

4.13

dev-console, alerts

0.3.0+

4.14+

dev-console, alerts, dev-alerts

5.5.1. Installing the Cluster Observability Operator logging UI plugin

Prerequisites

  • You have access to the cluster as a user with the cluster-admin role.
  • You have logged in to the OpenShift Container Platform web console.
  • You have installed the Cluster Observability Operator.
  • You have a LokiStack instance in you cluster.

Procedure

  1. In the OpenShift Container Platform web console, click Operators Installed Operators and select Cluster Observability Operator.
  2. Choose the UI Plugin tab (at the far right of the tab list) and click Create UIPlugin.
  3. Select YAML view, enter the following content, and then click Create:

    apiVersion: observability.openshift.io/v1alpha1
    kind: UIPlugin
    metadata:
      name: logging
    spec:
      type: Logging
      logging:
        lokiStack:
          name: logging-loki
        logsLimit: 50
        timeout: 30s
Red Hat logoGithubRedditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

© 2024 Red Hat, Inc.