Chapter 16. Cluster Observability Operator
16.1. Cluster Observability Operator release notes
The Cluster Observability Operator is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
The Cluster Observability Operator (COO) is an optional OpenShift Container Platform Operator that enables administrators to create standalone monitoring stacks that are independently configurable for use by different services and users.
The COO complements the built-in monitoring capabilities of OpenShift Container Platform. You can deploy it in parallel with the default platform and user workload monitoring stacks managed by the Cluster Monitoring Operator (CMO).
These release notes track the development of the Cluster Observability Operator in OpenShift Container Platform.
16.1.1. Cluster Observability Operator 0.1.1
This release updates the Cluster Observability Operator to support installing the Operator in restricted networks or disconnected environments.
16.1.2. Cluster Observability Operator 0.1
This release makes a Technology Preview version of the Cluster Observability Operator available on OperatorHub.
16.2. Cluster Observability Operator overview
The Cluster Observability Operator is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
The Cluster Observability Operator (COO) is an optional component of the OpenShift Container Platform. You can deploy it to create standalone monitoring stacks that are independently configurable for use by different services and users.
The COO deploys the following monitoring components:
- Prometheus
- Thanos Querier (optional)
- Alertmanager (optional)
The COO components function independently of the default in-cluster monitoring stack, which is deployed and managed by the Cluster Monitoring Operator (CMO). Monitoring stacks deployed by the two Operators do not conflict. You can use a COO monitoring stack in addition to the default platform monitoring components deployed by the CMO.
16.2.1. Understanding the Cluster Observability Operator
A default monitoring stack created by the Cluster Observability Operator (COO) includes a highly available Prometheus instance capable of sending metrics to an external endpoint by using remote write.
Each COO stack also includes an optional Thanos Querier component, which you can use to query a highly available Prometheus instance from a central location, and an optional Alertmanager component, which you can use to set up alert configurations for different services.
16.2.1.1. Advantages of using the Cluster Observability Operator
The MonitoringStack
CRD used by the COO offers an opinionated default monitoring configuration for COO-deployed monitoring components, but you can customize it to suit more complex requirements.
Deploying a COO-managed monitoring stack can help meet monitoring needs that are difficult or impossible to address by using the core platform monitoring stack deployed by the Cluster Monitoring Operator (CMO). A monitoring stack deployed using COO has the following advantages over core platform and user workload monitoring:
- Extendability
- Users can add more metrics to a COO-deployed monitoring stack, which is not possible with core platform monitoring without losing support. In addition, COO-managed stacks can receive certain cluster-specific metrics from core platform monitoring by using federation.
- Multi-tenancy support
- The COO can create a monitoring stack per user namespace. You can also deploy multiple stacks per namespace or a single stack for multiple namespaces. For example, cluster administrators, SRE teams, and development teams can all deploy their own monitoring stacks on a single cluster, rather than having to use a single shared stack of monitoring components. Users on different teams can then independently configure features such as separate alerts, alert routing, and alert receivers for their applications and services.
- Scalability
- You can create COO-managed monitoring stacks as needed. Multiple monitoring stacks can run on a single cluster, which can facilitate the monitoring of very large clusters by using manual sharding. This ability addresses cases where the number of metrics exceeds the monitoring capabilities of a single Prometheus instance.
- Flexibility
- Deploying the COO with Operator Lifecycle Manager (OLM) decouples COO releases from OpenShift Container Platform release cycles. This method of deployment enables faster release iterations and the ability to respond rapidly to changing requirements and issues. Additionally, by deploying a COO-managed monitoring stack, users can manage alerting rules independently of OpenShift Container Platform release cycles.
- Highly customizable
- The COO can delegate ownership of single configurable fields in custom resources to users by using Server-Side Apply (SSA), which enhances customization.
Additional resources
16.3. Installing the Cluster Observability Operator
The Cluster Observability Operator is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
As a cluster administrator, you can install the Cluster Observability Operator (COO) from OperatorHub by using the OpenShift Container Platform web console or CLI. OperatorHub is a user interface that works in conjunction with Operator Lifecycle Manager (OLM), which installs and manages Operators on a cluster.
To install the COO using OperatorHub, follow the procedure described in Adding Operators to a cluster.
16.3.1. Uninstalling the Cluster Observability Operator using the web console
If you have installed the Cluster Observability Operator (COO) by using OperatorHub, you can uninstall it in the OpenShift Container Platform web console.
Prerequisites
-
You have access to the cluster as a user with the
cluster-admin
cluster role. - You have logged in to the OpenShift Container Platform web console.
Procedure
-
Go to Operators
Installed Operators. - Locate the Cluster Observability Operator entry in the list.
- Click for this entry and select Uninstall Operator.
16.4. Configuring the Cluster Observability Operator to monitor a service
The Cluster Observability Operator is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
You can monitor metrics for a service by configuring monitoring stacks managed by the Cluster Observability Operator (COO).
To test monitoring a service, follow these steps:
- Deploy a sample service that defines a service endpoint.
-
Create a
ServiceMonitor
object that specifies how the service is to be monitored by the COO. -
Create a
MonitoringStack
object to discover theServiceMonitor
object.
16.4.1. Deploying a sample service for Cluster Observability Operator
This configuration deploys a sample service named prometheus-coo-example-app
in the user-defined ns1-coo
project. The service exposes the custom version
metric.
Prerequisites
-
You have access to the cluster as a user with the
cluster-admin
cluster role or as a user with administrative permissions for the namespace.
Procedure
Create a YAML file named
prometheus-coo-example-app.yaml
that contains the following configuration details for a namespace, deployment, and service:apiVersion: v1 kind: Namespace metadata: name: ns1-coo --- apiVersion: apps/v1 kind: Deployment metadata: labels: app: prometheus-coo-example-app name: prometheus-coo-example-app namespace: ns1-coo spec: replicas: 1 selector: matchLabels: app: prometheus-coo-example-app template: metadata: labels: app: prometheus-coo-example-app spec: containers: - image: ghcr.io/rhobs/prometheus-example-app:0.4.2 imagePullPolicy: IfNotPresent name: prometheus-coo-example-app --- apiVersion: v1 kind: Service metadata: labels: app: prometheus-coo-example-app name: prometheus-coo-example-app namespace: ns1-coo spec: ports: - port: 8080 protocol: TCP targetPort: 8080 name: web selector: app: prometheus-coo-example-app type: ClusterIP
- Save the file.
Apply the configuration to the cluster by running the following command:
$ oc apply -f prometheus-coo-example-app.yaml
Verify that the pod is running by running the following command and observing the output:
$ oc -n -ns1-coo get pod
Example output
NAME READY STATUS RESTARTS AGE prometheus-coo-example-app-0927545cb7-anskj 1/1 Running 0 81m
16.4.2. Specifying how a service is monitored by Cluster Observability Operator
To use the metrics exposed by the sample service you created in the "Deploying a sample service for Cluster Observability Operator" section, you must configure monitoring components to scrape metrics from the /metrics
endpoint.
You can create this configuration by using a ServiceMonitor
object that specifies how the service is to be monitored, or a PodMonitor
object that specifies how a pod is to be monitored. The ServiceMonitor
object requires a Service
object. The PodMonitor
object does not, which enables the MonitoringStack
object to scrape metrics directly from the metrics endpoint exposed by a pod.
This procedure shows how to create a ServiceMonitor
object for a sample service named prometheus-coo-example-app
in the ns1-coo
namespace.
Prerequisites
-
You have access to the cluster as a user with the
cluster-admin
cluster role or as a user with administrative permissions for the namespace. - You have installed the Cluster Observability Operator.
You have deployed the
prometheus-coo-example-app
sample service in thens1-coo
namespace.NoteThe
prometheus-coo-example-app
sample service does not support TLS authentication.
Procedure
Create a YAML file named
example-coo-app-service-monitor.yaml
that contains the followingServiceMonitor
object configuration details:apiVersion: monitoring.rhobs/v1alpha1 kind: ServiceMonitor metadata: labels: k8s-app: prometheus-coo-example-monitor name: prometheus-coo-example-monitor namespace: ns1-coo spec: endpoints: - interval: 30s port: web scheme: http selector: matchLabels: app: prometheus-coo-example-app
This configuration defines a
ServiceMonitor
object that theMonitoringStack
object will reference to scrape the metrics data exposed by theprometheus-coo-example-app
sample service.Apply the configuration to the cluster by running the following command:
$ oc apply -f example-app-service-monitor.yaml
Verify that the
ServiceMonitor
resource is created by running the following command and observing the output:$ oc -n ns1-coo get servicemonitor
Example output
NAME AGE prometheus-coo-example-monitor 81m
16.4.3. Creating a MonitoringStack object for the Cluster Observability Operator
To scrape the metrics data exposed by the target prometheus-coo-example-app
service, create a MonitoringStack
object that references the ServiceMonitor
object you created in the "Specifying how a service is monitored for Cluster Observability Operator" section. This MonitoringStack
object can then discover the service and scrape the exposed metrics data from it.
Prerequisites
-
You have access to the cluster as a user with the
cluster-admin
cluster role or as a user with administrative permissions for the namespace. - You have installed the Cluster Observability Operator.
-
You have deployed the
prometheus-coo-example-app
sample service in thens1-coo
namespace. -
You have created a
ServiceMonitor
object namedprometheus-coo-example-monitor
in thens1-coo
namespace.
Procedure
-
Create a YAML file for the
MonitoringStack
object configuration. For this example, name the fileexample-coo-monitoring-stack.yaml
. Add the following
MonitoringStack
object configuration details:Example
MonitoringStack
objectapiVersion: monitoring.rhobs/v1alpha1 kind: MonitoringStack metadata: name: example-coo-monitoring-stack namespace: ns1-coo spec: logLevel: debug retention: 1d resourceSelector: matchLabels: k8s-app: prometheus-coo-example-monitor
Apply the
MonitoringStack
object by running the following command:$ oc apply -f example-coo-monitoring-stack.yaml
Verify that the
MonitoringStack
object is available by running the following command and inspecting the output:$ oc -n ns1-coo get monitoringstack
Example output
NAME AGE example-coo-monitoring-stack 81m