Cluster Observability Operator

Procedure

1. In the OpenShift Container Platform web console, click Operators → Installed Operators and select Cluster Observability Operator.
2. Choose the UI Plugin tab (at the far right of the tab list) and press Create UIPlugin.

3. Select YAML view, enter the following content, and then press Create:

   apiVersion: observability.openshift.io/v1alpha1
   kind: UIPlugin
   metadata:
     name: dashboards
   spec:
     type: Dashboards

Procedure

1. Create a ConfigMap resource in the openshift-config-managed namespace, with the label console.openshift.io/dashboard-datasource: 'true'. The example below is from prometheus-datasource-example.yaml

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-prometheus-proxy
     namespace: openshift-config-managed
     labels:
       console.openshift.io/dashboard-datasource: "true"
   data:
     "dashboard-datasource.yaml": |-
       kind: "Datasource"
       metadata:
         name: "cluster-prometheus-proxy"
         project: "openshift-config-managed"
       spec:
         plugin:
           kind: "prometheus"
           spec:
             direct_url: "https://prometheus-k8s.openshift-monitoring.svc.cluster.local:9091"

2. Configure a custom dashboard that connects to the datasource. The YAML for a sample dashboard is available at prometheus-dashboard-example.yaml. An excerpt from that file is shown below for demonstration purposes:

   Example 5.1. Extract from example dashboard, taken from prometheus-dashboard-example.yaml

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: dashboard-example
     namespace: openshift-config-managed
     labels:
       console.openshift.io/dashboard: "true"
   data:
     k8s-resources-workloads-namespace.json: |-
       {
           "annotations": {
               "list": [
   
               ]
           },
           "editable": true,
           "gnetId": null,
           "graphTooltip": 0,
           "hideControls": false,
           "links": [
   
           ],
           "refresh": "10s",
           "rows": [
               {
                   "collapse": false,
                   "height": "250px",
                   "panels": [
                       {
                           "aliasColors": {
   
                           },
                           "bars": false,
                           "dashLength": 10,
                           "dashes": false,
                           "datasource": {
                               "name": "cluster-prometheus-proxy",
                               "type": "prometheus"
                           },
                           "fill": 10,
                           "id": 1,
                           "interval": "1m",
                           "legend": {
                               "alignAsTable": true,
                               "avg": false,
                               "current": false,
                               "max": false,
                               "min": false,
                               "rightSide": true,
                               "show": true,
                               "total": false,
                               "values": false
                           },
                           "lines": true,
                           "linewidth": 0,
                           "links": [
   
                           ],
                           "nullPointMode": "null as zero",
                           "percentage": false,
                           "pointradius": 5,
                           "points": false,
                           "renderer": "flot",
                           "seriesOverrides": [
                               {
                                   "alias": "quota - requests",
                                   "color": "#F2495C",
                                   "dashes": true,
                                   "fill": 0,
                                   "hiddenSeries": true,
                                   "hideTooltip": true,
                                   "legend": true,
                                   "linewidth": 2,
                                   "stack": false
                               },
                               {
                                   "alias": "quota - limits",
                                   "color": "#FF9830",
                                   "dashes": true,
                                   "fill": 0,
                                   "hiddenSeries": true,
                                   "hideTooltip": true,
                                   "legend": true,
                                   "linewidth": 2,
                                   "stack": false
                               }
                           ],
                           "spaceLength": 10,
                           "span": 12,
                           "stack": false,
                           "steppedLine": false,
                           "targets": [
                               {
                                   "expr": "sum(  node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster=\"$cluster\", namespace=\"$namespace\"}* on(namespace,pod)  group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=\"$cluster\", namespace=\"$namespace\", workload_type=\"$type\"}) by (workload, workload_type)",
                                   "format": "time_series",
                                   "intervalFactor": 2,
                                   "legendFormat": "{{workload}} - {{workload_type}}",
                                   "legendLink": null,
                                   "step": 10
                               },
                               {
                                   "expr": "scalar(kube_resourcequota{cluster=\"$cluster\", namespace=\"$namespace\", type=\"hard\",resource=\"requests.cpu\"})",
                                   "format": "time_series",
                                   "intervalFactor": 2,
                                   "legendFormat": "quota - requests",
                                   "legendLink": null,
                                   "step": 10
                               },
                               {
                                   "expr": "scalar(kube_resourcequota{cluster=\"$cluster\", namespace=\"$namespace\", type=\"hard\",resource=\"limits.cpu\"})",
                                   "format": "time_series",
                                   "intervalFactor": 2,
                                   "legendFormat": "quota - limits",
                                   "legendLink": null,
                                   "step": 10
                               }
                           ],
                           "thresholds": [
   
                           ],
                           "timeFrom": null,
                           "timeShift": null,
                           "title": "CPU Usage",
                           "tooltip": {
                               "shared": false,
                               "sort": 2,
                               "value_type": "individual"
                           },
                           "type": "graph",
                           "xaxis": {
                               "buckets": null,
                               "mode": "time",
                               "name": null,
                               "show": true,
                               "values": [
   
                               ]
                           },
   ...

3. Click Observe → Dashboards and the custom dashboard is available with the title ** DASHBOARD EXAMPLE **, based on the configuration in prometheus-dashboard-example.yaml.

   

   You can set the namespace, time range and refresh interval for the dashboard in the UI.

Procedure

1. In the OpenShift Container Platform web console, click Operators → Installed Operators and select Cluster Observability Operator
2. Choose the UI Plugin tab (at the far right of the tab list) and press Create UIPlugin

3. Select YAML view, enter the following content, and then press Create:

   apiVersion: observability.openshift.io/v1alpha1
   kind: UIPlugin
   metadata:
     name: distributed-tracing
   spec:
     type: DistributedTracing

Procedure

1. In the Administrator perspective of the OpenShift Container Platform web console, click Observe → Traces.

2. Select a TempoStack or TempoMonolithic multi-tenant instance and set a time range and query for the traces to be loaded.

   The traces are displayed on a scatter-plot showing the trace start time, duration, and number of spans. Underneath the scatter plot, there is a list of traces showing information such as the Trace Name, number of Spans, and Duration.

3. Click on a trace name link.

   The trace detail page for the selected trace contains a Gantt Chart of all of the spans within the trace. Select a span to show a breakdown of the configured attributes.

Procedure

1. In the OpenShift Container Platform web console, click Operators → Installed Operators and select Cluster Observability Operator
2. Choose the UI Plugin tab (at the far right of the tab list) and press Create UIPlugin

3. Select YAML view, enter the following content, and then press Create:

   apiVersion: observability.openshift.io/v1alpha1
   kind: UIPlugin
   metadata:
     name: troubleshooting-panel
   spec:
     type: TroubleshootingPanel

Procedure

1. Use the following YAML, either from the command line or in the web console, to create a broken deployment in a system namespace:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: bad-deployment
     namespace: default 1
   spec:
     selector:
       matchLabels:
         app: bad-deployment
     template:
       metadata:
         labels:
           app: bad-deployment
       spec:
         containers: 2
         - name: bad-deployment
           image: quay.io/openshift-logging/vector:5.8

   1

   The deployment must be in a system namespace (such as default) to cause the desired alerts.

   2

   This container deliberately tries to start a vector server with no configuration file. The server logs a few messages, and then exits with an error. Alternatively, you can deploy any container you like that is badly configured, causing it to trigger an alert.

2. View the alerts:

   1. Go to Observe → Alerting and click clear all filters. View the Pending alerts.

      Important

      Alerts first appear in the Pending state. They do not start Firing until the container has been crashing for some time. By viewing Pending alerts, you do not have to wait as long to see them occur.

   2. Choose one of the KubeContainerWaiting, KubePodCrashLooping, or KubePodNotReady alerts and open the troubleshooting panel by clicking on the link. Alternatively, if the panel is already open, click the "Focus" button to update the graph.

Procedure

1. In the OpenShift Container Platform web console, click Operators → Installed Operators and select Cluster Observability Operator.
2. Choose the UI Plugin tab (at the far right of the tab list) and click Create UIPlugin.

3. Select YAML view, enter the following content, and then click Create:

   apiVersion: observability.openshift.io/v1alpha1
   kind: UIPlugin
   metadata:
     name: logging
   spec:
     type: Logging
     logging:
       lokiStack:
         name: logging-loki
       logsLimit: 50
       timeout: 30s