Chapter 12. Troubleshooting Logging

Procedure

1. Change to the openshift-logging project.

   $ oc project openshift-logging

   Copy to Clipboard Toggle word wrap

2. To view the OpenShift Logging status:

   1. Get the OpenShift Logging status:

      $ oc get clusterlogging instance -o yaml

      Copy to Clipboard Toggle word wrap

      Example output

      apiVersion: logging.openshift.io/v1
      kind: ClusterLogging
      
      ....
      
      status:  

      1

      
        collection:
          logs:
            fluentdStatus:
              daemonSet: fluentd  

      2

      
              nodes:
                fluentd-2rhqp: ip-10-0-169-13.ec2.internal
                fluentd-6fgjh: ip-10-0-165-244.ec2.internal
                fluentd-6l2ff: ip-10-0-128-218.ec2.internal
                fluentd-54nx5: ip-10-0-139-30.ec2.internal
                fluentd-flpnn: ip-10-0-147-228.ec2.internal
                fluentd-n2frh: ip-10-0-157-45.ec2.internal
              pods:
                failed: []
                notReady: []
                ready:
                - fluentd-2rhqp
                - fluentd-54nx5
                - fluentd-6fgjh
                - fluentd-6l2ff
                - fluentd-flpnn
                - fluentd-n2frh
        logstore: 

      3

      
          elasticsearchStatus:
          - ShardAllocationEnabled:  all
            cluster:
              activePrimaryShards:    5
              activeShards:           5
              initializingShards:     0
              numDataNodes:           1
              numNodes:               1
              pendingTasks:           0
              relocatingShards:       0
              status:                 green
              unassignedShards:       0
            clusterName:             elasticsearch
            nodeConditions:
              elasticsearch-cdm-mkkdys93-1:
            nodeCount:  1
            pods:
              client:
                failed:
                notReady:
                ready:
                - elasticsearch-cdm-mkkdys93-1-7f7c6-mjm7c
              data:
                failed:
                notReady:
                ready:
                - elasticsearch-cdm-mkkdys93-1-7f7c6-mjm7c
              master:
                failed:
                notReady:
                ready:
                - elasticsearch-cdm-mkkdys93-1-7f7c6-mjm7c
      visualization:  

      4

      
          kibanaStatus:
          - deployment: kibana
            pods:
              failed: []
              notReady: []
              ready:
              - kibana-7fb4fd4cc9-f2nls
            replicaSets:
            - kibana-7fb4fd4cc9
            replicas: 1

      Copy to Clipboard Toggle word wrap

      1

      In the output, the cluster status fields appear in the status stanza.

      2

      Information on the Fluentd pods.

      3

      Information on the Elasticsearch pods, including Elasticsearch cluster health, green, yellow, or red.

      4

      Information on the Kibana pods.

Procedure

1. Change to the openshift-logging project.

   $ oc project openshift-logging

   Copy to Clipboard Toggle word wrap

2. View the status of the OpenShift Logging environment:

   $ oc describe deployment cluster-logging-operator

   Copy to Clipboard Toggle word wrap

   Example output

   Name:                   cluster-logging-operator
   
   ....
   
   Conditions:
     Type           Status  Reason
     ----           ------  ------
     Available      True    MinimumReplicasAvailable
     Progressing    True    NewReplicaSetAvailable
   
   ....
   
   Events:
     Type    Reason             Age   From                   Message
     ----    ------             ----  ----                   -------
     Normal  ScalingReplicaSet  62m   deployment-controller  Scaled up replica set cluster-logging-operator-574b8987df to 1----

   Copy to Clipboard Toggle word wrap

3. View the status of the OpenShift Logging replica set:

   1. Get the name of a replica set:

      Example output

      $ oc get replicaset

      Copy to Clipboard Toggle word wrap

      Example output

      NAME                                      DESIRED   CURRENT   READY   AGE
      cluster-logging-operator-574b8987df       1         1         1       159m
      elasticsearch-cdm-uhr537yu-1-6869694fb    1         1         1       157m
      elasticsearch-cdm-uhr537yu-2-857b6d676f   1         1         1       156m
      elasticsearch-cdm-uhr537yu-3-5b6fdd8cfd   1         1         1       155m
      kibana-5bd5544f87                         1         1         1       157m

      Copy to Clipboard Toggle word wrap

   2. Get the status of the replica set:

      $ oc describe replicaset cluster-logging-operator-574b8987df

      Copy to Clipboard Toggle word wrap

      Example output

      Name:           cluster-logging-operator-574b8987df
      
      ....
      
      Replicas:       1 current / 1 desired
      Pods Status:    1 Running / 0 Waiting / 0 Succeeded / 0 Failed
      
      ....
      
      Events:
        Type    Reason            Age   From                   Message
        ----    ------            ----  ----                   -------
        Normal  SuccessfulCreate  66m   replicaset-controller  Created pod: cluster-logging-operator-574b8987df-qjhqv----

      Copy to Clipboard Toggle word wrap

Procedure

1. Change to the openshift-logging project.

   $ oc project openshift-logging

   Copy to Clipboard Toggle word wrap

2. To view the status:

   1. Get the name of the log store instance:

      $ oc get Elasticsearch

      Copy to Clipboard Toggle word wrap

      Example output

      NAME            AGE
      elasticsearch   5h9m

      Copy to Clipboard Toggle word wrap

   2. Get the log store status:

      $ oc get Elasticsearch <Elasticsearch-instance> -o yaml

      Copy to Clipboard Toggle word wrap

      For example:

      $ oc get Elasticsearch elasticsearch -n openshift-logging -o yaml

      Copy to Clipboard Toggle word wrap

      The output includes information similar to the following:

      Example output

      status: 

      1

      
        cluster: 

      2

      
          activePrimaryShards: 30
          activeShards: 60
          initializingShards: 0
          numDataNodes: 3
          numNodes: 3
          pendingTasks: 0
          relocatingShards: 0
          status: green
          unassignedShards: 0
        clusterHealth: ""
        conditions: [] 

      3

      
        nodes: 

      4

      
        - deploymentName: elasticsearch-cdm-zjf34ved-1
          upgradeStatus: {}
        - deploymentName: elasticsearch-cdm-zjf34ved-2
          upgradeStatus: {}
        - deploymentName: elasticsearch-cdm-zjf34ved-3
          upgradeStatus: {}
        pods: 

      5

      
          client:
            failed: []
            notReady: []
            ready:
            - elasticsearch-cdm-zjf34ved-1-6d7fbf844f-sn422
            - elasticsearch-cdm-zjf34ved-2-dfbd988bc-qkzjz
            - elasticsearch-cdm-zjf34ved-3-c8f566f7c-t7zkt
          data:
            failed: []
            notReady: []
            ready:
            - elasticsearch-cdm-zjf34ved-1-6d7fbf844f-sn422
            - elasticsearch-cdm-zjf34ved-2-dfbd988bc-qkzjz
            - elasticsearch-cdm-zjf34ved-3-c8f566f7c-t7zkt
          master:
            failed: []
            notReady: []
            ready:
            - elasticsearch-cdm-zjf34ved-1-6d7fbf844f-sn422
            - elasticsearch-cdm-zjf34ved-2-dfbd988bc-qkzjz
            - elasticsearch-cdm-zjf34ved-3-c8f566f7c-t7zkt
        shardAllocationEnabled: all

      Copy to Clipboard Toggle word wrap

      1

      In the output, the cluster status fields appear in the status stanza.

      2

      The status of the log store:

      • The number of active primary shards.
      • The number of active shards.
      • The number of shards that are initializing.
      • The number of log store data nodes.
      • The total number of log store nodes.
      • The number of pending tasks.
      • The log store status: green, red, yellow.
      • The number of unassigned shards.

      3

      Any status conditions, if present. The log store status indicates the reasons from the scheduler if a pod could not be placed. Any events related to the following conditions are shown:

      • Container Waiting for both the log store and proxy containers.
      • Container Terminated for both the log store and proxy containers.
      • Pod unschedulable. Also, a condition is shown for a number of issues; see Example condition messages.

      4

      The log store nodes in the cluster, with upgradeStatus.

      5

      The log store client, data, and master pods in the cluster, listed under 'failed`, notReady, or ready state.

1. In the OpenShift Container Platform console, click Monitoring Alerting.
2. Click the Alerts tab. The alerts are listed, based on the filters selected.

1. Navigate to the directory where you want to store the must-gather information.

2. Run the oc adm must-gather command against the OpenShift Logging image:

   $ oc adm must-gather --image=$(oc -n openshift-logging get deployment.apps/cluster-logging-operator -o jsonpath='{.spec.template.spec.containers[?(@.name == "cluster-logging-operator")].image}')

   Copy to Clipboard Toggle word wrap

   The must-gather tool creates a new directory that starts with must-gather.local within the current directory. For example: must-gather.local.4157245944708210408.

3. Create a compressed file from the must-gather directory that was just created. For example, on a computer that uses a Linux operating system, run the following command:

   $ tar -cvaf must-gather.tar.gz must-gather.local.4157245944708210408

   Copy to Clipboard Toggle word wrap
4. Attach the compressed file to your support case on the Red Hat Customer Portal.

Troubleshooting

1. Check the Elasticsearch cluster health and verify that the cluster status is red.

   oc exec -n openshift-logging -c elasticsearch <elasticsearch_pod_name> -- health

   Copy to Clipboard Toggle word wrap

2. List the nodes that have joined the cluster.

   oc exec -n openshift-logging -c elasticsearch <elasticsearch_pod_name> -- es_util --query=_cat/nodes?v

   Copy to Clipboard Toggle word wrap

3. List the Elasticsearch pods and compare them with the nodes in the command output from the previous step.

   oc -n openshift-logging get pods -l component=elasticsearch

   Copy to Clipboard Toggle word wrap

4. If some of the Elasticsearch nodes have not joined the cluster, perform the following steps.

   1. Confirm that Elasticsearch has an elected control plane node.

      oc exec -n openshift-logging -c elasticsearch <elasticsearch_pod_name> -- es_util --query=_cat/master?v

      Copy to Clipboard Toggle word wrap

   2. Review the pod logs of the elected control plane node for issues.

      oc logs <elasticsearch_master_pod_name> -c elasticsearch -n openshift-logging

      Copy to Clipboard Toggle word wrap

   3. Review the logs of nodes that have not joined the cluster for issues.

      oc logs <elasticsearch_node_name> -c elasticsearch -n openshift-logging

      Copy to Clipboard Toggle word wrap

5. If all the nodes have joined the cluster, perform the following steps, check if the cluster is in the process of recovering.

   oc exec -n openshift-logging -c elasticsearch <elasticsearch_pod_name> -- es_util --query=_cat/recovery?active_only=true

   Copy to Clipboard Toggle word wrap

   If there is no command output, the recovery process might be delayed or stalled by pending tasks.

6. Check if there are pending tasks.

   oc exec -n openshift-logging -c elasticsearch <elasticsearch_pod_name> -- health |grep  number_of_pending_tasks

   Copy to Clipboard Toggle word wrap

7. If there are pending tasks, monitor their status.

   If their status changes and indicates that the cluster is recovering, continue waiting. The recovery time varies according to the size of the cluster and other factors.

   Otherwise, if the status of the pending tasks does not change, this indicates that the recovery has stalled.

8. If it seems like the recovery has stalled, check if cluster.routing.allocation.enable is set to none.

   oc exec -n openshift-logging -c elasticsearch <elasticsearch_pod_name> -- es_util --query=_cluster/settings?pretty

   Copy to Clipboard Toggle word wrap

9. If cluster.routing.allocation.enable is set to none, set it to all.

   oc exec -n openshift-logging -c elasticsearch <elasticsearch_pod_name> -- es_util --query=_cluster/settings?pretty -X PUT -d '{"persistent": {"cluster.routing.allocation.enable":"all"}}'

   Copy to Clipboard Toggle word wrap

10. Check which indices are still red.

    oc exec -n openshift-logging -c elasticsearch <elasticsearch_pod_name> -- es_util --query=_cat/indices?v

    Copy to Clipboard Toggle word wrap

11. If any indices are still red, try to clear them by performing the following steps.

    1. Clear the cache.

       oc exec -n openshift-logging -c elasticsearch <elasticsearch_pod_name> -- es_util --query=<elasticsearch_index_name>/_cache/clear?pretty

       Copy to Clipboard Toggle word wrap

    2. Increase the max allocation retries.

       oc exec -n openshift-logging -c elasticsearch <elasticsearch_pod_name> -- es_util --query=<elasticsearch_index_name>/_settings?pretty -X PUT -d '{"index.allocation.max_retries":10}'

       Copy to Clipboard Toggle word wrap

    3. Delete all the scroll items.

       oc exec -n openshift-logging -c elasticsearch <elasticsearch_pod_name> -- es_util --query=_search/scroll/_all -X DELETE

       Copy to Clipboard Toggle word wrap

    4. Increase the timeout.

       oc exec -n openshift-logging -c elasticsearch <elasticsearch_pod_name> -- es_util --query=<elasticsearch_index_name>/_settings?pretty -X PUT -d '{"index.unassigned.node_left.delayed_timeout":"10m"}'

       Copy to Clipboard Toggle word wrap

12. If the preceding steps do not clear the red indices, delete the indices individually.

    1. Identify the red index name.

       oc exec -n openshift-logging -c elasticsearch <elasticsearch_pod_name> -- es_util --query=_cat/indices?v

       Copy to Clipboard Toggle word wrap

    2. Delete the red index.

       oc exec -n openshift-logging -c elasticsearch <elasticsearch_pod_name> -- es_util --query=<elasticsearch_red_index_name> -X DELETE

       Copy to Clipboard Toggle word wrap

13. If there are no red indices and the cluster status is red, check for a continuous heavy processing load on a data node.

    1. Check if the Elasticsearch JVM Heap usage is high.

       oc exec -n openshift-logging -c elasticsearch <elasticsearch_pod_name> -- es_util --query=_nodes/stats?pretty

       Copy to Clipboard Toggle word wrap

       In the command output, review the node_name.jvm.mem.heap_used_percent field to determine the JVM Heap usage.

    2. Check for high CPU utilization.

Troubleshooting

1. Increase the node count by adjusting nodeCount in the ClusterLogging CR.

Troubleshooting

1. Identify the node on which Elasticsearch is deployed.

   oc -n openshift-logging get po -o wide

   Copy to Clipboard Toggle word wrap

2. Check if there are unassigned shards.

   oc exec -n openshift-logging -c elasticsearch <elasticsearch_pod_name> -- es_util --query=_cluster/health?pretty | grep unassigned_shards

   Copy to Clipboard Toggle word wrap

3. If there are unassigned shards, check the disk space on each node.

   for pod in `oc -n openshift-logging get po -l component=elasticsearch -o jsonpath='{.items[*].metadata.name}'`; do echo $pod; oc -n openshift-logging exec -c elasticsearch $pod -- df -h /elasticsearch/persistent; done

   Copy to Clipboard Toggle word wrap

4. Check the nodes.node_name.fs field to determine the free disk space on that node.

   If the used disk percentage is above 85%, the node has exceeded the low watermark, and shards can no longer be allocated to this node.

5. Try to increase the disk space on all nodes.
6. If increasing the disk space is not possible, try adding a new data node to the cluster.

7. If adding a new data node is problematic, decrease the total cluster redundancy policy.

   1. Check the current redundancyPolicy.

      oc -n openshift-logging get es elasticsearch -o jsonpath='{.spec.redundancyPolicy}'

      Copy to Clipboard Toggle word wrap
      Note

      If you are using a ClusterLogging CR, enter:

      oc -n openshift-logging get cl -o jsonpath='{.items[*].spec.logStore.elasticsearch.redundancyPolicy}'

      Copy to Clipboard Toggle word wrap
   2. If the cluster redundancyPolicy is higher than SingleRedundancy, set it to SingleRedundancy and save this change.

8. If the preceding steps do not fix the issue, delete the old indices.

   1. Check the status of all indices on Elasticsearch.

      oc exec -n openshift-logging -c elasticsearch <elasticsearch_pod_name> -- indices

      Copy to Clipboard Toggle word wrap
   2. Identify an old index that can be deleted.

   3. Delete the index.

      oc exec -n openshift-logging -c elasticsearch <elasticsearch_pod_name> -- es_util --query=<elasticsearch_index_name> -X DELETE

      Copy to Clipboard Toggle word wrap

Troubleshooting

1. Identify the node on which Elasticsearch is deployed.

   oc -n openshift-logging get po -o wide

   Copy to Clipboard Toggle word wrap

2. Check the disk space on each node.

   for pod in `oc -n openshift-logging get po -l component=elasticsearch -o jsonpath='{.items[*].metadata.name}'`; do echo $pod; oc -n openshift-logging exec -c elasticsearch $pod -- df -h /elasticsearch/persistent; done

   Copy to Clipboard Toggle word wrap

3. Check if the cluster is rebalancing.

   oc exec -n openshift-logging -c elasticsearch <elasticsearch_pod_name> -- es_util --query=_cluster/health?pretty | grep relocating_shards

   Copy to Clipboard Toggle word wrap

   If the command output shows relocating shards, the High Watermark has been exceeded. The default value of the High Watermark is 90%.

   The shards relocate to a node with low disk usage that has not crossed any watermark threshold limits.

4. To allocate shards to a particular node, free up some space.
5. Try to increase the disk space on all nodes.
6. If increasing the disk space is not possible, try adding a new data node to the cluster.

7. If adding a new data node is problematic, decrease the total cluster redundancy policy.

   1. Check the current redundancyPolicy.

      oc -n openshift-logging get es elasticsearch -o jsonpath='{.spec.redundancyPolicy}'

      Copy to Clipboard Toggle word wrap
      Note

      If you are using a ClusterLogging CR, enter:

      oc -n openshift-logging get cl -o jsonpath='{.items[*].spec.logStore.elasticsearch.redundancyPolicy}'

      Copy to Clipboard Toggle word wrap
   2. If the cluster redundancyPolicy is higher than SingleRedundancy, set it to SingleRedundancy and save this change.

8. If the preceding steps do not fix the issue, delete the old indices.

   1. Check the status of all indices on Elasticsearch.

      oc exec -n openshift-logging -c elasticsearch <elasticsearch_pod_name> -- indices

      Copy to Clipboard Toggle word wrap
   2. Identify an old index that can be deleted.

   3. Delete the index.

      oc exec -n openshift-logging -c elasticsearch <elasticsearch_pod_name> -- es_util --query=<elasticsearch_index_name> -X DELETE

      Copy to Clipboard Toggle word wrap

Troubleshooting

1. Check the disk space of the Elasticsearch node.

   for pod in `oc -n openshift-logging get po -l component=elasticsearch -o jsonpath='{.items[*].metadata.name}'`; do echo $pod; oc -n openshift-logging exec -c elasticsearch $pod -- df -h /elasticsearch/persistent; done

   Copy to Clipboard Toggle word wrap

   Check the nodes.node_name.fs field to determine the free disk space on that node.

2. If the used disk percentage is above 95%, it signifies that the node has crossed the flood watermark. Writing is blocked for shards allocated on this particular node.
3. Try to increase the disk space on all nodes.
4. If increasing the disk space is not possible, try adding a new data node to the cluster.

5. If adding a new data node is problematic, decrease the total cluster redundancy policy.

   1. Check the current redundancyPolicy.

      oc -n openshift-logging get es elasticsearch -o jsonpath='{.spec.redundancyPolicy}'

      Copy to Clipboard Toggle word wrap
      Note

      If you are using a ClusterLogging CR, enter:

      oc -n openshift-logging get cl -o jsonpath='{.items[*].spec.logStore.elasticsearch.redundancyPolicy}'

      Copy to Clipboard Toggle word wrap
   2. If the cluster redundancyPolicy is higher than SingleRedundancy, set it to SingleRedundancy and save this change.

6. If the preceding steps do not fix the issue, delete the old indices.

   1. Check the status of all indices on Elasticsearch.

      oc exec -n openshift-logging -c elasticsearch <elasticsearch_pod_name> -- indices

      Copy to Clipboard Toggle word wrap
   2. Identify an old index that can be deleted.

   3. Delete the index.

      oc exec -n openshift-logging -c elasticsearch <elasticsearch_pod_name> -- es_util --query=<elasticsearch_index_name> -X DELETE

      Copy to Clipboard Toggle word wrap

7. Continue freeing up and monitoring the disk space until the used disk space drops below 90%. Then, unblock write to this particular node.

   oc exec -n openshift-logging -c elasticsearch <elasticsearch_pod_name> -- es_util --query=_all/_settings?pretty -X PUT -d '{"index.blocks.read_only_allow_delete": null}'

   Copy to Clipboard Toggle word wrap

Troubleshooting

1. Get the disk space of the Elasticsearch node.

   for pod in `oc -n openshift-logging get po -l component=elasticsearch -o jsonpath='{.items[*].metadata.name}'`; do echo $pod; oc -n openshift-logging exec -c elasticsearch $pod -- df -h /elasticsearch/persistent; done

   Copy to Clipboard Toggle word wrap
2. In the command output, check the nodes.node_name.fs field to determine the free disk space on that node.
3. Try to increase the disk space on all nodes.
4. If increasing the disk space is not possible, try adding a new data node to the cluster.

5. If adding a new data node is problematic, decrease the total cluster redundancy policy.

   1. Check the current redundancyPolicy.

      oc -n openshift-logging get es elasticsearch -o jsonpath='{.spec.redundancyPolicy}'

      Copy to Clipboard Toggle word wrap
      Note

      If you are using a ClusterLogging CR, enter:

      oc -n openshift-logging get cl -o jsonpath='{.items[*].spec.logStore.elasticsearch.redundancyPolicy}'

      Copy to Clipboard Toggle word wrap
   2. If the cluster redundancyPolicy is higher than SingleRedundancy, set it to SingleRedundancy and save this change.

6. If the preceding steps do not fix the issue, delete the old indices.

   1. Check the status of all indices on Elasticsearch.

      oc exec -n openshift-logging -c elasticsearch <elasticsearch_pod_name> -- indices

      Copy to Clipboard Toggle word wrap
   2. Identify an old index that can be deleted.

   3. Delete the index.

      oc exec -n openshift-logging -c elasticsearch <elasticsearch_pod_name> -- es_util --query=<elasticsearch_index_name> -X DELETE

      Copy to Clipboard Toggle word wrap