Chapter 2. Recommended etcd practices

Because etcd writes data to disk and persists proposals on disk, its performance depends on disk performance.

Although etcd is not particularly I/O intensive, it requires a low latency block device for optimal performance and stability. Because the consensus protocol for etcd depends on persistently storing metadata to a log (WAL), etcd is sensitive to disk-write latency. Slow disks and disk activity from other processes can cause long fsync latencies.

Those latencies can cause etcd to miss heartbeats, not commit new proposals to the disk on time, and ultimately experience request timeouts and temporary leader loss. High write latencies also lead to an OpenShift API slowness, which affects cluster performance. Because of these reasons, avoid colocating other workloads on the control-plane nodes that are I/O sensitive or intensive and share the same underlying I/O infrastructure.

Run etcd on a block device that can write at least 50 IOPS of 8KB sequentially, including fdatasync, in under 10ms. For heavy loaded clusters, sequential 500 IOPS of 8000 bytes (2 ms) are recommended. To measure those numbers, you can use a benchmarking tool, such as the fio command.

To achieve such performance, run etcd on machines that are backed by SSD or NVMe disks with low latency and high throughput. Consider single-level cell (SLC) solid-state drives (SSDs), which provide 1 bit per memory cell, are durable and reliable, and are ideal for write-intensive workloads.

The following hard drive practices provide optimal etcd performance:

Some key metrics to monitor on a deployed OpenShift Container Platform cluster are p99 of etcd disk write ahead log duration and the number of etcd leader changes. Use Prometheus to track these metrics.

etcd requires low network and disk I/O latency so that Raft consensus can commit changes quickly and recover from leader failures without destabilizing the cluster.

Address two constraints to provide a low-latency, high-availability network for etcd:

etcd uses the Raft consensus algorithm, and every change must replicate to a majority of cluster members before it commits. This process is highly sensitive to network and disk performance. The minimum time for an etcd request is the round-trip time (RTT) between members, plus the time required to write data to permanent storage.

To achieve high availability, etcd must detect and recover from a leader failure quickly. This depends on two key tuning parameters:

In a healthy cluster, the round-trip time between members should be less than 50 ms to eensure stability and avoid frequent leader elections. For this reason, etcd clusters are often deployed within a single data center or availability zone to minimize physical distance and network latency.

To support a low-latency, high-availability network, especially during the leader election process, an arbiter site should be located where it provides an RTT latency of less than 10 ms. The arbiter component of a network maintains consistency and availability in a distributed system.

Validate control plane disk performance with fio before or after you create the OpenShift Container Platform cluster so that you can confirm that storage meets etcd latency requirements.

The output of the following procedure reports whether the disk is fast enough to host etcd by comparing the 99th percentile of the fsync metric captured from the run to see if it is less than 10 ms.

A few of the most important etcd metrics that might affected by I/O performance are as follows:

Because etcd replicates the requests among all the members, its performance strongly depends on network input/output (I/O) latency. High network latencies result in etcd heartbeats taking longer than the election timeout, which results in leader elections that are disruptive to the cluster. A key metric to monitor on a deployed OpenShift Container Platform cluster is the 99th percentile of etcd network peer latency on each etcd cluster member. Use Prometheus to track the metric.

The histogram_quantile(0.99, rate(etcd_network_peer_round_trip_time_seconds_bucket[2m])) metric reports the round trip time for etcd to finish replicating the client requests between the members. Ensure that it is less than 50 ms.