이 콘텐츠는 선택한 언어로 제공되지 않습니다.
Chapter 4. Troubleshooting
The following sections explain how to resolve issues that may occur in Metrics Store.
4.1. Information Is Missing from Kibana
If Kibana is not displaying metric or log information as expected, use journalctl to investigate the collectd and rsyslog log files as follows:
- If only metrics information is missing, check the collectd log files.
- If only log information is missing, check the rsyslog log files.
If both metrics and logs information are missing, check both log files.
To investigate collectd log files, log in as
rootand run the following command:# journalctl -u collectd
To investigate rsyslog log files, run the following command:
# journalctl -u rsyslog
To learn about additional journalctl options, see journalctl in Linux man pages .
4.2. Extracting Elasticsearch logs
To extract Elasticsearch logs
Log in to the Metrics Store virtual machine as
root, and run the following command. Whereopenshift-loggingis the namespace for the Elasticsearch pod:for espod in $(oc -n openshift-logging get pods -l component=es -o jsonpath='{.items[*].metadata.name}') ; do oc -n openshift-logging exec -c elasticsearch $espod -- logs > $espod.log 2>&1 done- Optionally, you can use the use the OpenShift logging-dump tool located at logging dump tool script
4.3. Searching Elasticsearch Logs
To search for all RHV/oVirt logs, ordered by timestamp from newest to oldest.Log in to the Metrics Store virtual machine as
root, and run the following command, whereopenshift-loggingis the namespace for the Elasticsearch pod:oc -n opesnshift-logging exec -c elasticsearch $(oc -n openshift-logging get pods -l component=es -o jsonpath='{.items[0].metadata.name}') -- es_util --query=project.ovirt-logs*/_search?sort=@timestamp:desc\&pretty | more- The output is presented in the human readable JSON format.
4.4. Searching log record results
To search the log report results, you can use the get last rec from host search tool.
The search tool shows how long it has been since the Metrics Store virtual machine received a record from each host that it knows about during a given time interval (last 3 hours by default).
For each host that the Metrics Store virtual machine receives logs from over the defined duration (default 3h) duration, it prints out "green", "yellow", or "red", depending on whether the Metrics Store virtual machine received a record from that host recently or not.
This is followed by the number of seconds it has been since the last record was received from that host, and the number of records received.
Log in to the Metrics Store virtual machine as
root, and clone the script repository:# git clone https://github.com/jcantrill/cluster-logging-tools -b release-3.11
Run the script:
# cd cluster-logging-tools/scripts [OLDEST=3h] ./get-last-rec-from-host
To change the interval, change the
OLDESTvalue to a longer or shorter interval in hours.For example, to go back 1 day (24 hours), use:
OLDEST=24h ./get-last-rec-from-host
Some hosts may not be listed, if no records were received from that host during the given time interval.
