8.15. Grafana 问题故障排除

流程

• 通过执行以下命令，验证 pmlogger 服务是否已启动并正在运行：

  $ systemctl status pmlogger

• 运行以下命令，验证是否在磁盘中创建或修改了文件：

  $ ls /var/log/pcp/pmlogger/$(hostname)/ -rlt
  total 4024
  -rw-r--r--. 1 pcp pcp   45996 Oct 13  2019 20191013.20.07.meta.xz
  -rw-r--r--. 1 pcp pcp     412 Oct 13  2019 20191013.20.07.index
  -rw-r--r--. 1 pcp pcp   32188 Oct 13  2019 20191013.20.07.0.xz
  -rw-r--r--. 1 pcp pcp   44756 Oct 13  2019 20191013.20.30-00.meta.xz
  [..]

• 运行以下命令验证 pmproxy 服务是否正在运行：

  $ systemctl status pmproxy

• 通过查看 /var/log/pcp/ pmproxy /pmproxy.log 文件，验证 pmproxy 是否正在运行、时间序列支持以及到 valkey 的连接是否已建立：

  pmproxy(1716) Info: valkey slots, command keys, schema version setup
  Here, 1716 is the PID of pmproxy, which will be different for every invocation of pmproxy.
  Verify if the valkey database contains any keys by executing the following command:
  $ valkey-cli dbsize
  (integer) 34837

• 运行以下命令，验证 valkey 数据库和 pmproxy 中的任何 PCP 指标是否能够访问它们：

  $ pmseries disk.dev.read
  2eb3e58d8f1e231361fb15cf1aa26fe534b4d9df
  $ pmseries "disk.dev.read[count:10]"
  2eb3e58d8f1e231361fb15cf1aa26fe534b4d9df
      [Mon Jul 26 12:21:10.085468000 2021] 117971 70e83e88d4e1857a3a31605c6d1333755f2dd17c
      [Mon Jul 26 12:21:00.087401000 2021] 117758 70e83e88d4e1857a3a31605c6d1333755f2dd17c
      [Mon Jul 26 12:20:50.085738000 2021] 116688 70e83e88d4e1857a3a31605c6d1333755f2dd17c
  [...]
  $ valkey-cli --scan --pattern "*$(pmseries 'disk.dev.read')"
  pcp:metric.name:series:2eb3e58d8f1e231361fb15cf1aa26fe534b4d9df
  pcp:values:series:2eb3e58d8f1e231361fb15cf1aa26fe534b4d9df
  pcp:desc:series:2eb3e58d8f1e231361fb15cf1aa26fe534b4d9df
  pcp:labelvalue:series:2eb3e58d8f1e231361fb15cf1aa26fe534b4d9df
  pcp:instances:series:2eb3e58d8f1e231361fb15cf1aa26fe534b4d9df
  pcp:labelflags:series:2eb3e58d8f1e231361fb15cf1aa26fe534b4d9df

• 运行以下命令，验证 Grafana 日志中是否有错误：

  $ journalctl -e -u grafana-server
  -- Logs begin at Mon 2021-07-26 11:55:10 IST, end at Mon 2021-07-26 12:30:15 IST. --
  Jul 26 11:55:17 localhost.localdomain systemd[1]: Starting Grafana instance...
  Jul 26 11:55:17 localhost.localdomain grafana-server[1171]: t=2021-07-26T11:55:17+0530 lvl=info msg="Starting Grafana" logger=server version=7.3.6 c>
  Jul 26 11:55:17 localhost.localdomain grafana-server[1171]: t=2021-07-26T11:55:17+0530 lvl=info msg="Config loaded from" logger=settings file=/usr/s>
  Jul 26 11:55:17 localhost.localdomain grafana-server[1171]: t=2021-07-26T11:55:17+0530 lvl=info msg="Config loaded from" logger=settings file=/etc/g>
  [...]