Chapter 1. About OpenShift Container Platform monitoring

• CoreDNS
• etcd
• HAProxy
• Image registry
• Kubelets
• Kubernetes API server
• Kubernetes controller manager
• Kubernetes scheduler
• OpenShift API server
• OpenShift Controller Manager
• Operator Lifecycle Manager (OLM)

• Metrics provided through service endpoints in user-defined projects.
• Pods running in user-defined projects.

• Topology spread constraints might be in place to control pod placement.
• Hard anti-affinity rules are in place for Prometheus, Alertmanager, and other monitoring components to ensure that multiple pods for these components are always spread across different nodes and are therefore always highly available.

• Alertmanager
• kube-state-metrics
• monitoring-plugin
• node-exporter
• openshift-state-metrics
• Prometheus
• Metrics Server
• Prometheus Operator and its admission webhook service
• Telemeter Client
• Thanos Querier

• Alertmanager
• Prometheus
• Thanos Ruler

• If cluster administrators require only alert, telemetry, and console metrics and do not require other metrics to be available.
• If a cluster increases in size, and the increased size of the default metrics data collected now requires a significant increase in CPU and memory resources.

• full: Prometheus collects metrics data exposed by all platform components. This setting is the default.
• minimal: Prometheus collects only the metrics data required for platform alerts, recording rules, telemetry, and console dashboards.

• Core platform monitoring: 15 days
• Monitoring for user-defined projects: 24 hours

• The size-based retention policy applies to all data block directories in the /prometheus directory, including persistent blocks, write-ahead log (WAL) data, and m-mapped chunks.
• Data in the /wal and /head_chunks directories counts toward the retention size limit, but Prometheus never purges data from these directories based on size- or time-based retention policies. Thus, if you set a retention size limit lower than the maximum size set for the /wal and /head_chunks directories, you have configured the system not to retain any data blocks in the /prometheus data directories.
• The size-based retention policy is applied only when Prometheus cuts a new data block, which occurs every two hours after the WAL contains at least three hours of data.
• If you do not explicitly define values for either retention or retentionSize, retention time defaults to 15 days for core platform monitoring and 24 hours for user-defined project monitoring. Retention size is not set.
• If you define values for both retention and retentionSize, both values apply. If any data blocks exceed the defined retention time or the defined size limit, Prometheus purges these data blocks.
• If you define a value for retentionSize and do not define retention, only the retentionSize value applies.
• If you do not define a value for retentionSize and only define a value for retention, only the retention value applies.
• If you set the retentionSize or retention value to 0, the default settings apply. The default settings set retention time to 15 days for core platform monitoring and 24 hours for user-defined project monitoring. By default, retention size is not set.

• Limit the number of samples that can be accepted per target scrape in user-defined projects
• Limit the number of scraped labels, the length of label names, and the length of label values
• Create alerts that fire when a scrape sample threshold is reached or when the target cannot be scraped

• Configuring the write relabel settings for remote write storage.
• Adding cluster ID labels to the metrics.
• Querying these labels to identify the source cluster or customer for a metric.

• API performance
• etcd
• Kubernetes compute resources
• Kubernetes network resources
• Prometheus
• USE method dashboards relating to cluster and node performance
• Node performance metrics

• Modify the settings for existing platform alerting rules by adjusting thresholds or by adding and modifying labels. For example, you can change the severity label for an alert from warning to critical to help you route and triage issues flagged by an alert.
• Define and add new custom alerting rules by constructing a query expression based on core platform metrics in the openshift-monitoring namespace.

Core platform alerting rule considerations

• New alerting rules must be based on the default OpenShift Container Platform monitoring metrics.
• You must create the AlertingRule and AlertRelabelConfig objects in the openshift-monitoring namespace.
• You can only add and modify alerting rules. You cannot create new recording rules or modify existing recording rules.
• If you modify existing platform alerting rules by using an AlertRelabelConfig object, your modifications are not reflected in the Prometheus alerts API. Therefore, any dropped alerts still appear in the OpenShift Container Platform web console even though they are no longer forwarded to Alertmanager. Additionally, any modifications to alerts, such as a changed severity label, do not appear in the web console.

• Minimize the number of new rules. Create only rules that are essential to your specific requirements. By minimizing the number of rules, you create a more manageable and focused alerting system in your monitoring environment.
• Focus on symptoms rather than causes. Create rules that notify users of symptoms instead of underlying causes. This approach ensures that users are promptly notified of a relevant symptom so that they can investigate the root cause after an alert has triggered. This tactic also significantly reduces the overall number of rules you need to create.
• Plan and assess your needs before implementing changes. First, decide what symptoms are important and what actions you want users to take if these symptoms occur. Then, assess existing rules and decide if you can modify any of them to meet your needs instead of creating entirely new rules for each symptom. By modifying existing rules and creating new ones judiciously, you help to streamline your alerting system.
• Provide clear alert messaging. When you create alert messages, describe the symptom, possible causes, and recommended actions. Include unambiguous, concise explanations along with troubleshooting steps or links to more information. Doing so helps users quickly assess the situation and respond appropriately.
• Include severity levels. Assign severity levels to your rules to indicate how a user needs to react when a symptom occurs and triggers an alert. For example, classifying an alert as Critical signals that an individual or a critical response team needs to respond immediately. By defining severity levels, you help users know how to respond to an alert and help ensure that the most urgent issues receive prompt attention.

• A user-defined alerting rule can include metrics exposed by its own project in addition to the default metrics from core platform monitoring. You cannot include metrics from another user-defined project.

  For example, an alerting rule for the ns1 user-defined project can use metrics exposed by the ns1 project in addition to core platform metrics, such as CPU and memory metrics. However, the rule cannot include metrics from a different ns2 user-defined project.

• To reduce latency and to minimize the load on core platform monitoring components, you can add the openshift.io/prometheus-rule-evaluation-scope: leaf-prometheus label to a rule. This label forces only the Prometheus instance deployed in the openshift-user-workload-monitoring project to evaluate the alerting rule and prevents the Thanos Ruler instance from doing so.

  Important

  If an alerting rule has this label, your alerting rule can use only those metrics exposed by your user-defined project. Alerting rules you create based on default platform metrics might not trigger alerts.

Alerting rule considerations

• The default alerting rules are used specifically for the OpenShift Container Platform cluster.
• Some alerting rules intentionally have identical names. They send alerts about the same event with different thresholds, different severity, or both.
• Inhibition rules prevent notifications for lower severity alerts that are firing when a higher severity alert is also firing.

• Minimize the number of alerting rules that you create for your project. Create alerting rules that notify you of conditions that impact you. It is more difficult to notice relevant alerts if you generate many alerts for conditions that do not impact you.
• Create alerting rules for symptoms instead of causes. Create alerting rules that notify you of conditions regardless of the underlying cause. The cause can then be investigated. You will need many more alerting rules if each relates only to a specific cause. Some causes are then likely to be missed.
• Plan before you write your alerting rules. Determine what symptoms are important to you and what actions you want to take if they occur. Then build an alerting rule for each symptom.
• Provide clear alert messaging. State the symptom and recommended actions in the alert message.
• Include severity levels in your alerting rules. The severity of an alert depends on how you need to react if the reported symptom occurs. For example, a critical alert should be triggered if a symptom requires immediate attention by an individual or a critical response team.