3.5. Monitoring consensus latency for etcd

Procedure

1. Enter the following command:

   # oc get pods -n openshift-etcd -l app=etcd

   Example output

   NAME      READY   STATUS    RESTARTS   AGE
   etcd-m0   4/4     Running   4          8h
   etcd-m1   4/4     Running   4          8h
   etcd-m2   4/4     Running   4          8h

2. Enter the following command. To better understand the etcd latency for consensus, you can run this command on a precise watch cycle for a few minutes to observe that the numbers remain below the ~66 ms threshold. The closer the consensus time is to 100 ms, the more likely the cluster will experience service-affecting events and instability.

   # oc exec -ti etcd-m0 -- etcdctl endpoint health -w table

   Example output

   +----------------------------+--------+-------------+-------+
   |          ENDPOINT          | HEALTH |    TOOK     | ERROR |
   +----------------------------+--------+-------------+-------+
   | https://198.18.111.12:2379 |   true |  3.798349ms |       |
   | https://198.18.111.14:2379 |   true |  7.389608ms |       |
   | https://198.18.111.13:2379 |   true |  6.263117ms |       |
   +----------------------------+--------+-------------+-------+

3. Enter the following command:

   # oc exec -ti etcd-m0 -- watch -dp -c etcdctl endpoint health -w table

   Example output

   +----------------------------+--------+-------------+-------+
   |          ENDPOINT          | HEALTH |    TOOK     | ERROR |
   +----------------------------+--------+-------------+-------+
   | https://198.18.111.12:2379 |   true |  9.533405ms |       |
   | https://198.18.111.13:2379 |   true |  4.628054ms |       |
   | https://198.18.111.14:2379 |   true |  5.803378ms |       |
   +----------------------------+--------+-------------+-------+