UI plugins for Red Hat OpenShift Cluster Observability Operator
Installing and managing UI plugins for the Cluster Observability Operator.
Abstract
Chapter 1. Observability UI plugins overview Copy linkLink copied to clipboard!
You can use the Cluster Observability Operator (COO) to install and manage UI plugins to enhance the observability capabilities of the OpenShift Container Platform web console. The plugins extend the default functionality, providing new UI features for troubleshooting, distributed tracing, and cluster logging.
1.1. Monitoring Copy linkLink copied to clipboard!
The monitoring UI plugin adds monitoring related UI features to the OpenShift Container Platform web console, for the Advance Cluster Management (ACM) perspective and for incident detection.
- ACM: The monitoring plugin in Cluster Observability Operator (COO) allows it to function in Red Hat Advanced Cluster Management (RHACM) environments, providing ACM with the same monitoring capabilities as OpenShift Container Platform.
- Incident Detection: The incident detection feature groups alerts into incidents to help you identify the root causes of alert bursts instead of being overwhelmed by individual alerts. It presents a timeline of incidents, color-coded by severity, and you can drill down into the individual alerts within an incident. The system also categorizes alerts by affected component to help you focus on the most critical areas first.
For more information, see the monitoring UI plugin page.
1.2. Cluster logging Copy linkLink copied to clipboard!
The logging UI plugin surfaces logging data in the web console on the Observe → Logs page. You can specify filters, queries, time ranges and refresh rates. The results displayed a list of collapsed logs, which can then be expanded to show more detailed information for each log.
For more information, see the logging UI plugin page.
1.3. Troubleshooting Copy linkLink copied to clipboard!
The troubleshooting panel UI plugin for OpenShift Container Platform version 4.16+ provides observability signal correlation, powered by the open source Korrel8r project. You can use the troubleshooting panel available from the Observe → Alerting page to easily correlate metrics, logs, alerts, netflows, and additional observability signals and resources, across different data stores. Users of OpenShift Container Platform version 4.17+ can also access the troubleshooting UI panel from the Application Launcher
.
The output of Korrel8r is displayed as an interactive node graph. When you click on a node, you are automatically redirected to the corresponding web console page with the specific information for that node, for example, metric, log, or pod.
For more information, see the troubleshooting UI plugin page.
1.4. Distributed tracing Copy linkLink copied to clipboard!
The distributed tracing UI plugin adds tracing-related features to the web console on the Observe → Traces page. You can follow requests through the front end and into the backend of microservices, helping you identify code errors and performance bottlenecks in distributed systems. You can select a supported TempoStack or TempoMonolithic multi-tenant instance running in the cluster and set a time range and query to view the trace data.
For more information, see the distributed tracing UI plugin page.
Chapter 2. Monitoring UI plugin Copy linkLink copied to clipboard!
The monitoring UI plugin adds monitoring features to the Administrator perspective of the OpenShift Container Platform web console.
Starting with OpenShift Container Platform 4.19, the perspectives in the web console have unified. The Developer perspective is no longer enabled by default.
All users can interact with all OpenShift Container Platform web console features. However, if you are not the cluster owner, you might need to request permission to certain features from the cluster owner.
You can still enable the Developer perspective. On the Getting Started pane in the web console, you can take a tour of the console, find information on setting up your cluster, view a quick start for enabling the Developer perspective, and follow links to explore new features and capabilities.
- RHACM: The monitoring plugin in Cluster Observability Operator (COO) allows it to function in Red Hat Advanced Cluster Management (RHACM) environments, providing RHACM with the same alerting capabilities as OpenShift Container Platform. You can configure the plugin to fetch alerts from the RHACM Alertmanager backend. This enables seamless integration and user experience by aligning RHACM and OpenShift Container Platform monitoring workflows.
Incident detection: The incident detection feature groups related alerts into incidents, to help you identify the root causes of alert bursts, instead of being overwhelmed by individual alerts. It presents a timeline of incidents, color-coded by severity, and you can drill down into the individual alerts within an incident. The system also categorizes alerts by affected component, grouped by severity. This helps you focus on the most critical areas first.
The incident detection feature is available in the OpenShift Container Platform web console by clicking on Observe → Alerting and choosing the Incidents tab.
2.1. Installing the Cluster Observability Operator monitoring UI plugin Copy linkLink copied to clipboard!
The monitoring UI plugin adds monitoring related UI features to the OpenShift Container Platform web console, for the Advance Cluster Management (ACM) perspective and for incident detection.
Prerequisites
-
You have access to the cluster as a user with the
cluster-admincluster role. - You have logged in to the OpenShift Container Platform web console.
- You have installed the Cluster Observability Operator
Procedure
- In the OpenShift Container Platform web console, click Operators → Installed Operators and select Cluster Observability Operator
- Choose the UI Plugin tab (at the far right of the tab list) and press Create UIPlugin
Select YAML view, enter the following content, and then press Create:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
2.2. Cluster Observability Operator incident detection overview Copy linkLink copied to clipboard!
Clusters can generate significant volumes of monitoring data, making it hard for you to distinguish critical signals from noise. Single incidents can trigger a cascade of alerts, and this results in extended time to detect and resolve issues.
The Cluster Observability Operator incident detection feature groups related alerts into incidents. These incidents are then visualized as timelines that are color-coded by severity. Alerts are mapped to specific components, grouped by severity, helping you to identify root causes by focusing on high impact components first. You can then drill down from the incident timelines to individual alerts to determine how to fix the underlying issue.
Cluster Observability Operator incident detection transforms the alert storm into clear steps for faster understanding and resolution of the incidents that occur on your clusters.
2.3. Using Cluster Observability Operator incident detection Copy linkLink copied to clipboard!
Prerequisites
-
You have access to the cluster as a user with the
cluster-admincluster role. - You have logged in to the OpenShift Container Platform web console.
- You have installed the Cluster Observability Operator.
- You have installed the Cluster Observability Operator monitoring UI plugin with incident detection enabled.
If you have installed the Cluster Observability Operator programmatically, make sure that the namespace (by default openshift-cluster-observability-operator) has the label openshift.io/cluster-monitoring=true. If not add it by running the comand:
oc label namespace openshift-cluster-observability-operator openshift.io/cluster-monitoring=true
$ oc label namespace openshift-cluster-observability-operator openshift.io/cluster-monitoring=true
Procedure
- In the Administrator perspective of the web console, click on Observe → Alerting and choose the Incidents tab.
The Incidents Timeline UI shows the grouping of alerts into incidents. The color coding of the lines in the graph corresponds to the severity of the incident. By default, a seven day timeline is presented.
NoteIt will take at least five minutes to process the correlations and to see the timeline, after you enable incident detection.
Status changes are recorded at five-minute intervals. You can see the last update time next to the timeline.
The analysis and grouping into incidents is performed only for alerts that are firing after you have enabled this feature. Alerts that have been resolved before feature enablement are not included.
Zoom in to a 1-day view by clicking on the drop-down to specify the duration.
NoteFor performance reasons, the UI only loads the displayed data. The start time reflects the first data point rendered in the current timeline, not the absolute start time of the alert. For example, if you zoom to 1-day and an alert started two days ago, the start time will be shifted to twenty four hours before.
By clicking on an incident, you can see the timeline of alerts that are part of that incident, in the Alerts Timeline UI.
In the list of alerts that follows, alerts are mapped to specific components, which are grouped by severity.
Click to expand a component in the list. The underlying alerts related to that component are displayed.
- Click the link for an alert, to see detailed information about the associated alert rule.
Chapter 3. Logging UI plugin Copy linkLink copied to clipboard!
The logging UI plugin surfaces logging data in the OpenShift Container Platform web console on the Observe → Logs page. You can specify filters, queries, time ranges and refresh rates, with the results displayed as a list of collapsed logs, which can then be expanded to show more detailed information for each log.
If you also deploy the Troubleshooting UI plugin on OpenShift Container Platform version 4.16+, it connects to the Korrel8r service and adds direct links to the web console, from the Observe → Logs page, to the Observe → Metrics page with a correlated PromQL query. The plugin also adds a See Related Logs link from the web console alerting detail page, at Observe → Alerting, to the Observe → Logs page with a correlated filter set selected.
The features of the plugin are categorized as:
- dev-console
- Adds the logging view to web console.
- alerts
- Merges the web console alerts with log-based alerts defined in the Loki ruler. Adds a log-based metrics chart in the alert detail view.
- dev-alerts
- Merges the web console alerts with log-based alerts defined in the Loki ruler. Adds a log-based metrics chart in the alert detail view for the web console.
For Cluster Observability Operator (COO) versions, the support for these features in OpenShift Container Platform versions is shown in the following table:
| COO version | OCP versions | Features |
|---|---|---|
| 0.3.0+ | 4.12 |
|
| 0.3.0+ | 4.13 |
|
| 0.3.0+ | 4.14+ |
|
3.1. Installing the Cluster Observability Operator logging UI plugin Copy linkLink copied to clipboard!
Prerequisites
-
You have access to the cluster as a user with the
cluster-adminrole. - You have logged in to the OpenShift Container Platform web console.
- You have installed the Cluster Observability Operator.
-
You have a
LokiStackinstance in your cluster.
Procedure
- In the OpenShift Container Platform web console, click Operators → Installed Operators and select Cluster Observability Operator.
- Choose the UI Plugin tab (at the far right of the tab list) and click Create UIPlugin.
Select YAML view, enter the following content, and then click Create:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
schemais one ofotel,viaq, orselect. The default isviaqif no value is specified. When you chooseselect, you can select the mode in the UI when you run a query.
NoteThese are the known issues for the logging UI plugin - for more information, see OU-587.
-
The
schemafeature is only supported in OpenShift Container Platform 4.15 and later. In earlier versions of Red Hat OpenShift Cluster Observability Operator, the logging UI plugin will only use theviaqattribute, ignoring any other values that might be set. -
Non-administrator users cannot query logs using the
otelattribute with logging for Red Hat OpenShift versions 5.8 to 6.2. This issue will be fixed in a future logging release. (LOG-6589) -
In logging for Red Hat OpenShift version 5.9, the
severity_textOtel attribute is not set.
Chapter 4. Distributed tracing UI plugin Copy linkLink copied to clipboard!
The distributed tracing UI plugin adds tracing-related features to the OpenShift Container Platform web console at Observe → Traces. You can follow requests through the front end and into the backend of microservices, helping you identify code errors and performance bottlenecks in distributed systems.
4.1. Installing the Cluster Observability Operator distributed tracing UI plugin Copy linkLink copied to clipboard!
Prerequisites
-
You have access to the cluster as a user with the
cluster-admincluster role. - You have logged in to the OpenShift Container Platform web console.
- You have installed the Cluster Observability Operator
Procedure
- In the OpenShift Container Platform web console, click Operators → Installed Operators and select Cluster Observability Operator
- Choose the UI Plugin tab (at the far right of the tab list) and press Create UIPlugin
Select YAML view, enter the following content, and then press Create:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
4.2. Using the Cluster Observability Operator distributed tracing UI plugin Copy linkLink copied to clipboard!
Visualizing traces in the distributed tracing UI in the OpenShift Container Platform web console provides a user-friendly way of working with traces.
Prerequisites
-
You have access to the cluster as a user with the
cluster-admincluster role. - You have logged in to the OpenShift Container Platform web console.
- You have installed the Cluster Observability Operator.
- You have installed the Cluster Observability Operator distributed tracing UI plugin.
-
You have a
TempoStackorTempoMonolithicmulti-tenant instance in the cluster.
Procedure
- In the OpenShift Container Platform web console, click Observe → Traces.
- Select a Tempo instance. If selecting a multi-tenant instance, also select a tenant.
- Select a Time range and Limit traces.
Use the two Filter drop-down menus to filter by an attribute:
- Service Name
- Span Name
- Namespace
- Status
- Span Duration
- Trace Duration
Custom attributes
TipFor more advanced filtering, click the Show query link and edit the TraceQL query.
As a result of your filter selections, a scatter plot visualizes traces with trace start time, duration, and number of spans.
Also, the traces are listed under the scatter plot, including trace details such as the Trace Name, number of Spans, and Duration.
The back end searches both the object storage and ingester caches in parallel until the requested number of traces is found. The search results are a random and unordered sample of traces matching the search parameters.
- Select a trace to view a Gantt chart of its spans.
- In the Gantt chart, select a span to view its configured attributes.
Chapter 5. Troubleshooting panel UI plugin Copy linkLink copied to clipboard!
The troubleshooting panel UI plugin provides observability signal correlation. With the troubleshooting panel, which is available from the Application Launcher
, you can easily navigate to resources, metrics, logs, alerts, netflows, and other observability signals that are correlated to the current contents of the console screen.
The panel displays an interactive node graph in the OpenShift Container Platform web console. Nodes in the graph represent a type of resource or signal, while edges represent relationships. When you click on a node, you are automatically redirected to the corresponding web console page.
5.1. Installing the Cluster Observability Operator Troubleshooting UI plugin Copy linkLink copied to clipboard!
Prerequisites
-
You have access to an OpenShift Container Platform cluster version 4.19+ as a user with the
cluster-admincluster role. - You have logged in to the OpenShift Container Platform web console.
- You have installed the Cluster Observability Operator
Procedure
- In the OpenShift Container Platform web console, click Operators → Installed Operators and select Cluster Observability Operator
- Choose the UI Plugin tab (at the far right of the tab list) and press Create UIPlugin
Select YAML view, enter the following content, and then press Create:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
5.2. Using the Cluster Observability Operator troubleshooting panel UI plugin Copy linkLink copied to clipboard!
The Troubleshooting Panel is a side-panel that shows a graph of resources and signals related to the information displayed in the main console (in this example the apiserver Deployment). You can use the troubleshooting panel on most console pages such as resources, alerts, metrics, logs, traces, network observability and others.
Prerequisites
-
You have access to the OpenShift Container Platform cluster as a user with the
cluster-admincluster role. - You have logged in to the OpenShift Container Platform web console.
- You have installed the Cluster Observability Operator.
- You have installed the Cluster Observability Operator troubleshooting UI plugin.
The troubleshooting panel can display additional signals if you have installed the following Red Hat components in their recommended namespace:
Logs:
-
OpenShift Container Platform Logging(collection) -
OpenShift Container Platform Loki Operator(store)
-
Network events:
-
OpenShift Container Platform Network observability(collection) -
OpenShift Container Platform Loki Operator(store)
-
-
Traces:
OpenShift Container Platform Distributed Tracing(collection and store)
Users with ClusterRole/view (for example kube:admin) will get the most complete results. Other users will see more restricted graphs, showing only data that they have permission to see.
Procedure
Open the Application Launcher
and click the Signal Correlation menu item.
Click Focus to show a correlation graph starting from the resources or signals in the main console display. This example shows the
apiserverDeployment.- Click Focus to re-calculate the graph from the main console at any time.
- Click the refresh icon to update the graph that is currently displayed.
- The root (top) node represents the resource in the main console window.
- First-degree neighbors are directly related to the initial resource.
- Second-degree neighbors are indirectly related via first-degree neighbors.
Clicking on a node opens the data in the main console. For example if you click on the Log node, the following panel appears:
5.2.1. Advanced controls for the Cluster Observability Operator troubleshooting panel UI plugin Copy linkLink copied to clipboard!
The Advanced button opens additional controls over the correlation search. These are experimental and might change in a future release. They provide access to additional features of the Korrel8r upstream project that powers the correlation search.
The following list explains the additional controls under the Advanced button:
- Recent The search is limited to recent results, by default in the last day.
- Range Allows searching an explicit date/time range in the past.
- Distance Searches up to the given distance from the starting point. Each correlation rule (edge in the graph) is a "step", and the distance is the maximum number of steps.
- Goal Class Instead of a distance, you can select a goal, which is a resource or signal type. This searches for paths from the start to the goal type, and the path length is not limited. For more information about correlation classes, see the Korrel8r project documentation.
- Query The Korrel8r query representing the starting point for the search. Click Focus to reset the query to reflect the current console page. Click Sync to re-run the current query and update the graph. For more information about correlation queries, see the Korrel8r project documentation.