Chapter 11. Viewing cluster dashboards
The Logging/Elasticsearch Nodes and Openshift Logging dashboards in the OpenShift Container Platform web console show in-depth details about your Elasticsearch instance and the individual Elasticsearch nodes that you can use to prevent and diagnose problems.
The OpenShift Logging dashboard contains charts that show details about your Elasticsearch instance at a cluster level, including cluster resources, garbage collection, shards in the cluster, and Fluentd statistics.
The Logging/Elasticsearch Nodes dashboard contains charts that show details about your Elasticsearch instance, many at node level, including details on indexing, shards, resources, and so forth.
For more detailed data, click the Grafana UI link in a dashboard to launch the Grafana dashboard. Grafana is shipped with OpenShift cluster monitoring.
11.1. Accessing the Elastisearch and OpenShift Logging dashboards
You can view the Logging/Elasticsearch Nodes and OpenShift Logging dashboards in the OpenShift Container Platform web console.
Procedure
To launch the dashboards:
-
In the OpenShift Container Platform web console, click Monitoring
Dashboards. On the Dashboards page, select Logging/Elasticsearch Nodes or OpenShift Logging from the Dashboard menu.
For the Logging/Elasticsearch Nodes dashboard, you can select the Elasticsearch node you want to view and set the data resolution.
The appropriate dashboard is displayed, showing multiple charts of data.
- Optional: Select a different time range to display or refresh rate for the data from the Time Range and Refresh Interval menus.
For more detailed data, click the Grafana UI link to launch the Grafana dashboard.
For information on the dashboard charts, see About the OpenShift Logging dashboard and About the Logging/Elastisearch Nodes dashboard.
11.2. About the OpenShift Logging dashboard
The OpenShift Logging dashboard contains charts that show details about your Elasticsearch instance at a cluster-level that you can use to diagnose and anticipate problems.
| Metric | Description |
|---|---|
| Elastic Cluster Status | The current Elasticsearch status:
|
| Elastic Nodes | The total number of Elasticsearch nodes in the Elasticsearch instance. |
| Elastic Shards | The total number of Elasticsearch shards in the Elasticsearch instance. |
| Elastic Documents | The total number of Elasticsearch documents in the Elasticsearch instance. |
| Total Index Size on Disk | The total disk space that is being used for the Elasticsearch indices. |
| Elastic Pending Tasks | The total number of Elasticsearch changes that have not been completed, such as index creation, index mapping, shard allocation, or shard failure. |
| Elastic JVM GC time | The amount of time that the JVM spent executing Elasticsearch garbage collection operations in the cluster. |
| Elastic JVM GC Rate | The total number of times that JVM executed garbage activities per second. |
| Elastic Query/Fetch Latency Sum |
Fetch latency typically takes less time than query latency. If fetch latency is consistently increasing, it might indicate slow disks, data enrichment, or large requests with too many results. |
| Elastic Query Rate | The total queries executed against the Elasticsearch instance per second for each Elasticsearch node. |
| CPU | The amount of CPU used by Elasticsearch, Fluentd, and Kibana, shown for each component. |
| Elastic JVM Heap Used | The amount of JVM memory used. In a healthy cluster, the graph shows regular drops as memory is freed by JVM garbage collection. |
| Elasticsearch Disk Usage | The total disk space used by the Elasticsearch instance for each Elasticsearch node. |
| File Descriptors In Use | The total number of file descriptors used by Elasticsearch, Fluentd, and Kibana. |
| FluentD emit count | The total number of Fluentd messages per second for the Fluentd default output, and the retry count for the default output. |
| FluentD Buffer Availability | The percent of the Fluentd buffer that is available for chunks. A full buffer might indicate that Fluentd is not able to process the number of logs received. |
| Elastic rx bytes | The total number of bytes that Elasticsearch has received from FluentD, the Elasticsearch nodes, and other sources. |
| Elastic Index Failure Rate | The total number of times per second that an Elasticsearch index fails. A high rate might indicate an issue with indexing. |
| FluentD Output Error Rate | The total number of times per second that FluentD is not able to output logs. |
11.3. Charts on the Logging/Elasticsearch nodes dashboard
The Logging/Elasticsearch Nodes dashboard contains charts that show details about your Elasticsearch instance, many at node-level, for further diagnostics.
- Elasticsearch status
- The Logging/Elasticsearch Nodes dashboard contains the following charts about the status of your Elasticsearch instance.
| Metric | Description |
|---|---|
| Cluster status | The cluster health status during the selected time period, using the Elasticsearch green, yellow, and red statuses:
|
| Cluster nodes | The total number of Elasticsearch nodes in the cluster. |
| Cluster data nodes | The number of Elasticsearch data nodes in the cluster. |
| Cluster pending tasks | The number of cluster state changes that are not finished and are waiting in a cluster queue, for example, index creation, index deletion, or shard allocation. A growing trend indicates that the cluster is not able to keep up with changes. |
- Elasticsearch cluster index shard status
- Each Elasticsearch index is a logical group of one or more shards, which are basic units of persisted data. There are two types of index shards: primary shards, and replica shards. When a document is indexed into an index, it is stored in one of its primary shards and copied into every replica of that shard. The number of primary shards is specified when the index is created, and the number cannot change during index lifetime. You can change the number of replica shards at any time.
The index shard can be in several states depending on its lifecycle phase or events occurring in the cluster. When the shard is able to perform search and indexing requests, the shard is active. If the shard cannot perform these requests, the shard is non–active. A shard might be non-active if the shard is initializing, reallocating, unassigned, and so forth.
Index shards consist of a number of smaller internal blocks, called index segments, which are physical representations of the data. An index segment is a relatively small, immutable Lucene index that is created when Lucene commits newly-indexed data. Lucene, a search library used by Elasticsearch, merges index segments into larger segments in the background to keep the total number of segments low. If the process of merging segments is slower than the speed at which new segments are created, it could indicate a problem.
When Lucene performs data operations, such as a search operation, Lucene performs the operation against the index segments in the relevant index. For that purpose, each segment contains specific data structures that are loaded in the memory and mapped. Index mapping can have a significant impact on the memory used by segment data structures.
The Logging/Elasticsearch Nodes dashboard contains the following charts about the Elasticsearch index shards.
| Metric | Description |
|---|---|
| Cluster active shards | The number of active primary shards and the total number of shards, including replicas, in the cluster. If the number of shards grows higher, the cluster performance can start degrading. |
| Cluster initializing shards | The number of non-active shards in the cluster. A non-active shard is one that is initializing, being reallocated to a different node, or is unassigned. A cluster typically has non–active shards for short periods. A growing number of non–active shards over longer periods could indicate a problem. |
| Cluster relocating shards | The number of shards that Elasticsearch is relocating to a new node. Elasticsearch relocates nodes for multiple reasons, such as high memory use on a node or after a new node is added to the cluster. |
| Cluster unassigned shards | The number of unassigned shards. Elasticsearch shards might be unassigned for reasons such as a new index being added or the failure of a node. |
- Elasticsearch node metrics
- Each Elasticsearch node has a finite amount of resources that can be used to process tasks. When all the resources are being used and Elasticsearch attempts to perform a new task, Elasticsearch put the tasks into a queue until some resources become available.
The Logging/Elasticsearch Nodes dashboard contains the following charts about resource usage for a selected node and the number of tasks waiting in the Elasticsearch queue.
| Metric | Description |
|---|---|
| ThreadPool tasks | The number of waiting tasks in individual queues, shown by task type. A long–term accumulation of tasks in any queue could indicate node resource shortages or some other problem. |
| CPU usage | The amount of CPU being used by the selected Elasticsearch node as a percentage of the total CPU allocated to the host container. |
| Memory usage | The amount of memory being used by the selected Elasticsearch node. |
| Disk usage | The total disk space being used for index data and metadata on the selected Elasticsearch node. |
| Documents indexing rate | The rate that documents are indexed on the selected Elasticsearch node. |
| Indexing latency | The time taken to index the documents on the selected Elasticsearch node. Indexing latency can be affected by many factors, such as JVM Heap memory and overall load. A growing latency indicates a resource capacity shortage in the instance. |
| Search rate | The number of search requests run on the selected Elasticsearch node. |
| Search latency | The time taken to complete search requests on the selected Elasticsearch node. Search latency can be affected by many factors. A growing latency indicates a resource capacity shortage in the instance. |
| Documents count (with replicas) | The number of Elasticsearch documents stored on the selected Elasticsearch node, including documents stored in both the primary shards and replica shards that are allocated on the node. |
| Documents deleting rate | The number of Elasticsearch documents being deleted from any of the index shards that are allocated to the selected Elasticsearch node. |
| Documents merging rate | The number of Elasticsearch documents being merged in any of index shards that are allocated to the selected Elasticsearch node. |
- Elasticsearch node fielddata
- Fielddata is an Elasticsearch data structure that holds lists of terms in an index and is kept in the JVM Heap. Because fielddata building is an expensive operation, Elasticsearch caches the fielddata structures. Elasticsearch can evict a fielddata cache when the underlying index segment is deleted or merged, or if there is not enough JVM HEAP memory for all the fielddata caches.
The Logging/Elasticsearch Nodes dashboard contains the following charts about Elasticsearch fielddata.
| Metric | Description |
|---|---|
| Fielddata memory size | The amount of JVM Heap used for the fielddata cache on the selected Elasticsearch node. |
| Fielddata evictions | The number of fielddata structures that were deleted from the selected Elasticsearch node. |
- Elasticsearch node query cache
- If the data stored in the index does not change, search query results are cached in a node-level query cache for reuse by Elasticsearch.
The Logging/Elasticsearch Nodes dashboard contains the following charts about the Elasticsearch node query cache.
| Metric | Description |
|---|---|
| Query cache size | The total amount of memory used for the query cache for all the shards allocated to the selected Elasticsearch node. |
| Query cache evictions | The number of query cache evictions on the selected Elasticsearch node. |
| Query cache hits | The number of query cache hits on the selected Elasticsearch node. |
| Query cache misses | The number of query cache misses on the selected Elasticsearch node. |
- Elasticsearch index throttling
- When indexing documents, Elasticsearch stores the documents in index segments, which are physical representations of the data. At the same time, Elasticsearch periodically merges smaller segments into a larger segment as a way to optimize resource use. If the indexing is faster then the ability to merge segments, the merge process does not complete quickly enough, which can lead to issues with searches and performance. To prevent this situation, Elasticsearch throttles indexing, typically by reducing the number of threads allocated to indexing down to a single thread.
The Logging/Elasticsearch Nodes dashboard contains the following charts about Elasticsearch index throttling.
| Metric | Description |
|---|---|
| Indexing throttling | The amount of time that Elasticsearch has been throttling the indexing operations on the selected Elasticsearch node. |
| Merging throttling | The amount of time that Elasticsearch has been throttling the segment merge operations on the selected Elasticsearch node. |
- Node JVM Heap statistics
- The Logging/Elasticsearch Nodes dashboard contains the following charts about JVM Heap operations.
| Metric | Description |
|---|---|
| Heap used | The amount of the total allocated JVM Heap space that is used on the selected Elasticsearch node. |
| GC count | The number of garbage collection operations that have been run on the selected Elasticsearch node, by old and young garbage collection. |
| GC time | The amount of time that the JVM spent running garbage collection operations on the selected Elasticsearch node, by old and young garbage collection. |
