One control-plane node, [NODE_B], kept flapping. It would go Ready → NotReady → Ready multiple times a day. Each flap triggered a cascade of DaemonSet alert noise: Traefik, zfs-localpv, Velero, and Promtail all reported numberMisscheduled warnings. The node's local kube-apiserver had restarted 273 times over its lifetime, compared to just 13 restarts on the cluster's etcd leader node.
During each flap, the kubelet logs showed etcdserver: request timed out. The apiserver liveness probe returned HTTP 500, the container was killed (exit 137), and for 1–2 minutes the node's port 6443 returned connection refused.
We looked at the usual suspects: OOM kills? None. Network latency between control-plane nodes? Sub-millisecond. Hardware errors in dmesg? None. So why was this node so much worse than its peers?
We ran iostat on the node. The system disk (dm-0) was at 96% utilisation with ~38% iowait. On the stable leader node, disk utilisation was around 7%. We then checked etcd's internal metrics: on the flapping node, the lifetime counter for Write-Ahead Log (WAL) fsyncs faster than 2ms was ~91. On the leader, it was ~83,000. That is three orders of magnitude difference.
What was saturating the disk? The control-plane nodes had no NoSchedule taint. They were running general workloads, including two Rook-Ceph OSDs, Velero node-agents, and CI runners. The OSDs were pounding the same RAID/HDD stack that etcd used for its WAL.
Severe disk I/O contention on the system disk. etcd requires low-latency, durable writes for every Raft log entry. When the disk is saturated by unrelated workloads, fsync latency spikes from microseconds to hundreds of milliseconds. The local apiserver, which reads from etcd for every request, starts timing out. Kubelet cannot renew its lease. The node is marked NotReady. When the apiserver eventually restarts, the node recovers, only to repeat the cycle when disk contention returns.
etcd uses the Raft consensus algorithm. Every write must be durably persisted to the Write-Ahead Log on all members before it is committed. The leader sends entries to followers, and followers acknowledge after fsyncing the WAL to disk. If a follower's disk is slow, it delays the entire quorum, because the leader waits for a majority of acknowledgements.
fsync is a system call that forces data from the OS buffer cache to the physical storage device. On a heavily utilised HDD RAID with competing I/O (e.g., Ceph OSD writes), fsync calls can take 100–500ms or more. etcd has a hard-coded warning threshold of 100ms; beyond that, it logs apply request took too long. When fsync latency consistently exceeds the apiserver's timeouts, the apiserver starts failing probes.
We migrated the Rook-Ceph OSDs off all control-plane nodes and applied a NoSchedule taint to prevent future workloads from landing there. We also fixed the provider's failover controller (which was itself crashing) to ensure the floating IP could move if a node went down.
etcd_disk_wal_fsync_duration_seconds) proactively. If you see a node with an order-of-magnitude worse fsync latency, you have a ticking bomb.At Obmondo, we built KubeAid to provide a production-ready Kubernetes observability foundation. As part of KubeAid, we enforce taints on all control-plane nodes and include Grafana dashboards dedicated to monitoring etcd disk latency per member. If the gap between nodes exceeds 10x, we investigate before it flaps.