Chapter 4. Troubleshooting

To extract Elasticsearch logs

1. Log in to the Metrics Store virtual machine as root, and run the following command. Where openshift-logging is the namespace for the Elasticsearch pod:

   for espod in $(oc -n openshift-logging get pods -l component=es -o jsonpath='{.items[*].metadata.name}') ; do
       oc -n openshift-logging exec -c elasticsearch $espod -- logs > $espod.log 2>&1
   done

   Copy to Clipboard Toggle word wrap
2. Optionally, you can use the use the OpenShift logging-dump tool located at logging dump tool script

1. To search for all RHV/oVirt logs, ordered by timestamp from newest to oldest.Log in to the Metrics Store virtual machine as root, and run the following command, where openshift-logging is the namespace for the Elasticsearch pod:

   oc -n opesnshift-logging exec -c elasticsearch $(oc -n openshift-logging get pods -l component=es -o jsonpath='{.items[0].metadata.name}') -- es_util --query=project.ovirt-logs*/_search?sort=@timestamp:desc​\&pretty | more

   Copy to Clipboard Toggle word wrap
2. The output is presented in the human readable JSON format.

1. Log in to the Metrics Store virtual machine as root, and clone the script repository:

   # git clone https://github.com/jcantrill/cluster-logging-tools -b release-3.11

   Copy to Clipboard Toggle word wrap

2. Run the script:

   # cd cluster-logging-tools/scripts
   [OLDEST=3h] ./get-last-rec-from-host

   Copy to Clipboard Toggle word wrap

3. To change the interval, change the OLDEST value to a longer or shorter interval in hours.

   For example, to go back 1 day (24 hours), use:

   OLDEST=24h ./get-last-rec-from-host

   Copy to Clipboard Toggle word wrap