Chapter 4. Troubleshooting

To extract Elasticsearch logs

1. Log in to the Metrics Store virtual machine as root, and run the following command. Where openshift-logging is the namespace for the Elasticsearch pod:

   for espod in $(oc -n openshift-logging get pods -l component=es -o jsonpath='{.items[*].metadata.name}') ; do
       oc -n openshift-logging exec -c elasticsearch $espod -- logs > $espod.log 2>&1
   done

2. Optionally, you can use the use the OpenShift logging-dump tool located at logging dump tool script

1. To search for all RHV/oVirt logs, ordered by timestamp from newest to oldest.Log in to the Metrics Store virtual machine as root, and run the following command, where openshift-logging is the namespace for the Elasticsearch pod:

   oc -n opesnshift-logging exec -c elasticsearch $(oc -n openshift-logging get pods -l component=es -o jsonpath='{.items[0].metadata.name}') -- es_util --query=project.ovirt-logs*/_search?sort=@timestamp:desc​\&pretty | more

2. The output is presented in the human readable JSON format.

1. Log in to the Metrics Store virtual machine as root, and clone the script repository:

   # git clone https://github.com/jcantrill/cluster-logging-tools -b release-3.11

2. Run the script:

   # cd cluster-logging-tools/scripts
   [OLDEST=3h] ./get-last-rec-from-host

3. To change the interval, change the OLDEST value to a longer or shorter interval in hours.

   For example, to go back 1 day (24 hours), use:

   OLDEST=24h ./get-last-rec-from-host