Kubernetes assumes an infinite datacenter

What breaks when the control plane runs on hardware you already own.

Versions: Cluster API v1.11+ (v1beta2) with CAPH on Hetzner bare metal. v1beta1 field paths are given inline where they differ. v1.12 changed the ending, which we get to at the end.

The rollout had been running for six minutes and nothing had happened.

Three control plane nodes, all on Hetzner bare metal, all needing a config change. kubectl get machines showed a fourth Machine that had appeared out of nowhere, sitting in Provisioning, and a host association condition underneath it that said, in substance:

no available host (all hosts are in use - found 3 hosts)

The rollout was not slow. It was never going to start. Every HetznerBareMetalHost we owned was already bound to a Machine, and Cluster API was politely waiting for a fourth server that does not exist and cannot be conjured.

The assumption nobody writes down

Deployments roll by surging: create the new pod, wait for it to be healthy, then destroy the old one. KubeadmControlPlane does the same, with a default maxSurge of 1.

That default is correct on a cloud. A fourth control plane node on HCloud costs a few cents for the twenty minutes it exists, and you get a rollout with zero reduction in etcd fault tolerance. It is the right trade, and it is so obviously the right trade that it became a default, and defaults become assumptions, and assumptions stop being visible.

On bare metal the assumption is just wrong. You have three servers because you bought three servers. There is no fourth one to surge into, and no API call that will make one appear in the next twenty minutes.

The interesting thing is that Kubernetes already has a controller that does not make this assumption. StatefulSets never surge. A StatefulSet rolling update deletes and recreates each pod in place, one at a time, highest ordinal first, waiting for each to be Running and Ready before touching its predecessor. That is exactly the shape you need on finite hardware, and it is the shape the rest of this post is about getting out of KCP.

First guess: the provisioner is broken

We spent a while looking in the wrong place. The surge Machine was stuck, so the natural read is that provisioning failed: bad image, bad rescue boot, credentials expired, CAPH bug. We went through the CAPH controller logs looking for the failure.

There was no failure. That was the confusing part. Nothing had errored, nothing had retried, nothing had timed out. The controller had correctly determined there was no host available and was correctly waiting for one, forever, in a cluster where "forever" is the accurate estimate.

The failure mode is not an error. It is an absence of one.

Why does a Machine sit in Provisioning with nothing in the logs?

Then it got worse: the rollout that eats itself

Once you do free up a host, a second cloud assumption fires.

A bare metal provision is not a boot. It is rescue boot, then install-image, then reboot, then cloud-init, then kubeadm join. Measured on our hosts, that chain takes 20 to 40 minutes.

Upstream MachineHealthCheck defaults nodeStartupTimeout to 10 minutes. Our own HCloud template shipped 900 seconds, fifteen minutes, which is generous for a VM and nowhere near enough for a physical server. Either way the outcome is the same: somewhere around the fifteen minute mark, while the replacement node is still installing packages, the health check declares it failed to start. KCP remediates by deleting it and provisioning another one, which also takes 20 to 40 minutes, and is also reaped.

Reinstall, reaped, retry. The same physical host, re-imaged repeatedly, by a health check that is faster than the thing it is checking.

Worth being precise about the blast radius, because this is where people over-read the danger. KCP remediation has real guardrails: it will not remediate if doing so would break etcd quorum, it requires at least two control plane machines, it remediates strictly one machine at a time, and MachineHealthCheck short-circuits entirely once too many machines are unhealthy. What you get instead is a remediation loop: a node reprovisioned, reaped, and reprovisioned again, forever, because remediation cannot fix a template that is wrong. The cluster survives. The rollout never finishes and the host never stabilises.

Why does the replacement node never finish joining?

The fix is one field, and it inverts the algorithm

# KubeadmControlPlane, Cluster API v1beta2 (v1.11+)
rollout:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 0

On v1beta1 the same setting lives at spec.rolloutStrategy.rollingUpdate.maxSurge. The spec.rollout block is new in v1beta2, which folded the old rolloutBefore, rolloutAfter and rolloutStrategy fields together. maxSurge accepts only 0 or 1, nothing in between.

maxSurge: 0 switches KCP from a surge rollout to a scale-in rollout. It deletes one outdated control plane Machine first, deprovisioning frees that host, and then it provisions the replacement onto the host it just freed. Destroy, then create. The exact inverse of what every rolling update tutorial tells you to do, and the only order that terminates on finite hardware.

This is not a trick. It is in the original KubeadmControlPlane proposal, which says maxSurge: 0 "could be used in resource constrained environment like bare-metal, OpenStack or vSphere resource pools, etc when there is no capacity to Scale up the control plane." It is the documented answer. It is just not the default, and nothing tells you to go looking for it.

One precondition the docs are easy to skim past: maxSurge: 0 requires at least 3 control plane replicas. KCP verifies that before it will scale down first. A two node control plane cannot use this, which is fine, because a two node control plane cannot survive the swap anyway.

What changes when you flip one field from 1 to 0? The left column keeps full redundancy and needs a spare server. On bare metal you do not have one, which makes the right column the only one that finishes.

And the timeout, which needs to describe your hardware rather than someone's VM fleet:

# MachineHealthCheck, v1beta2. On v1beta1: spec.nodeStartupTimeout: "60m0s"
checks:
  nodeStartupTimeoutSeconds: 3600

One hour covers the slowest install we have observed, with headroom, while still catching a server that is genuinely dead. We kept 900s on the HCloud path, because there a fifteen minute boot really is a broken machine. Note the v1beta2 rename: this was spec.nodeStartupTimeout as a duration string, and it is now an integer of seconds under spec.checks.

The trade-off, stated plainly

maxSurge: 0 costs you something real, and anyone who tells you otherwise has not run it.

During each swap, a three node control plane runs two members. etcd still has quorum, so the cluster is up and writable. It has zero fault tolerance while that swap is in flight. Lose another node in that window, for any reason, and you have lost quorum on a cluster that is mid-upgrade.

When exactly is the cluster running without fault tolerance?

With a three node control plane and 20 to 40 minutes per node, you are accepting roughly two hours of degraded fault tolerance per rollout, in three separate windows. That is the price of being able to roll at all. On HCloud we do not pay it, because there the fourth server is cheap and we keep the surge default.

So the honest version of the rule is not "use maxSurge 0 on bare metal". It is: on fixed hardware, highly available and upgradeable pull against each other, and you have to decide which one you give up for the next half hour. If that window is unacceptable to you, the answer is not a config flag, it is buying a fourth server.

Two other things follow from the same constraint, and both bit us before we understood the pattern.

Node names are not cosmetic. Kubernetes node names are immutable, so in a CAPI cluster renaming a node means destroying and reprovisioning it. If your bare metal hosts come up with random hashes in their hostnames instead of stable ones, CAPI eventually treats them as ephemeral and recycles them, and every node name change breaks the PVC attachments bound to the old name. We ended up alerting on the hostname pattern itself, before it can cause damage:

absent(kube_node_labels)
  or
(kube_node_labels{label_beta_kubernetes_io_instance_type!~"^(cax|cx|cpx|ccx).*"}
  unless
 kube_node_labels{label_kubernetes_io_hostname=~"^bm-.*-[0-9]{7}$"})

The absent() on the front is deliberate: the alert also fires when its own dependency is missing, because kube-state-metrics only exposes those labels if you pass --metric-labels-allowlist=nodes=[beta.kubernetes.io/instance-type,kubernetes.io/hostname]. An alert that silently stops evaluating is worse than no alert.

One rot risk, since we are handing you a query to copy: beta.kubernetes.io/instance-type is deprecated in favour of node.kubernetes.io/instance-type. Both are still populated today. When the beta label finally goes, the left side of that unless stops matching anything and the alert quietly stops firing, which is the exact failure the absent() guard is there to catch. Match on both labels if you want it to outlive the deprecation.

If you run Ceph, a node rename is a two-system operation. The OSDs keep pointing at the old node name and stay down until you rename the CRUSH bucket to match:

ceph osd crush rename-bucket <old-node-name> <new-node-name>

The part where automation makes it worse

The reason we care so much about the health check timeout is an incident on 26 April.

A Hetzner Floating IP was pointed at a control plane node whose IP was not in the apiserver certificate SANs. Workers went NotReady immediately:

tls: failed to verify certificate: x509: certificate is valid for
10.96.0.1, 203.0.113.11, 203.0.113.12, not 203.0.113.99

(Addresses replaced with documentation ranges. The last one is the floating IP.)

We regenerated the certificate on the node holding the IP at 10:17 and considered it handled. It was not. Workers were still failing, because a second control plane node was still serving the old cert, which we did not find until 10:35. Eighteen minutes lost on a fix that was correct and incomplete, which is the worst kind.

A detail worth stealing from that hour: kubeadm certs renew apiserver does not pick up new SANs. Renewal rebuilds the cert from the attributes of the existing one, so adding a SAN needs a regeneration, not a renewal. The declarative fix is to put the floating IP in KubeadmControlPlane.spec.kubeadmConfigSpec.clusterConfiguration.apiServer.certSANs and let the resulting rollout bake it into new machines.

Meanwhile machinehealthcheck-controller had been watching. At 11:05 it decided one of the control plane nodes was unhealthy and deleted it. The replacement was provisioned from the same template, with the same missing certSANs, and failed the same way.

The lesson we wrote into the RCA at the time:

CAPI is a "double-edged sword" during configuration errors; it will delete nodes that it perceives as broken, which can turn a configuration error into a quorum loss.

We would soften that sentence now. As covered above, KCP remediation will not knowingly break etcd quorum, so "turns a config error into quorum loss" overstates it. What it reliably does is turn a config error into a replacement loop on hardware that takes 40 minutes per attempt, during an incident, while you are trying to think. The cluster stays up. Your ability to reason about it does not, because the inventory keeps moving underneath you.

That is the real cost, and it is bad enough without exaggerating it. Remediation is not wrong. It is calibrated for a world where machines are disposable and replacement is cheap, and yours are not.

This is all in the open, and free

Everything above lives in KubeAid, our Kubernetes platform, which is the same code we run for the clusters we manage. The bare metal control plane chart is argocd-helm-charts/capi-cluster/charts/hetzner, AGPL, comments and scars included. We open source it because the alternative is asking you to trust that we handled this, and you should not have to.

If you want the working version of this rather than the explanation, KubeAid CLI drives cluster lifecycle, including the rollout behaviour in this post:

curl -fsSL https://raw.githubusercontent.com/Obmondo/kubeaid-cli/main/scripts/install.sh | sh
kubeaid-cli config generate hetzner

That generates a Hetzner cluster config with maxSurge: 0, the one hour startup timeout, and the rest of what this post cost us to learn, already set. cluster bootstrap, cluster upgrade and cluster recover do what they sound like. It is AGPL, it costs nothing, and it works whether or not you ever talk to us. There is also a devenv command that spins up a local K3D cluster if you want to try the whole thing without buying a server.

One thing to run

If you have a bare metal control plane under Cluster API, run this:

kubectl get machines -o wide

Any Machine sitting in Provisioning with no error is the deadlock, not a slow boot. Then check the two fields that decide whether your next upgrade completes:

kubectl get kcp <cluster>-control-plane -o jsonpath='{.spec.rollout.strategy.rollingUpdate.maxSurge}{"\n"}'
kubectl get mhc -o jsonpath='{.items[*].spec.checks.nodeStartupTimeoutSeconds}{"\n"}'

On v1beta1, those paths are {.spec.rolloutStrategy.rollingUpdate.maxSurge} and {.items[*].spec.nodeStartupTimeout}.

If the first is 1 and every host you own is already bound, your control plane cannot roll. If the second is under about half an hour and your servers take that long to install, your rollout will loop before it finishes.

The postscript: upstream fixed this properly

While writing this up we checked whether anyone had solved surge rollouts on fixed hardware. Someone had. Upstream.

Cluster API v1.12 shipped a delete-first rollout strategy, which the release notes describe as making it "easier to do immutable rollouts on bare metal / environments with constrained resources". That is this post's workaround, promoted to a supported strategy. It also shipped in-place updates behind the InPlaceUpdates feature gate, which let KubeadmControlPlane and MachineDeployments change existing machines without replacing them at all.

Read that second one again if you run bare metal. This whole post argues that swapping a physical host is expensive, so swap them in the least-bad order. In-place updates say: for a large class of changes, do not swap them.

The reasoning above still transfers, and most people are not on v1.12 yet. But if you are starting fresh on bare metal today, evaluate in-place updates before you spend a week tuning maxSurge and timeouts like we did.

The question we still have: how much of a real bare metal upgrade can in-place updates actually cover before you fall back to replacing the host anyway?

We are Obmondo. We run Linux and Kubernetes for other people, and we open source the entire toolchain we use to do it: KubeAid, KubeAid CLI, LinuxAid. All AGPL, all free, no demo required.

Ready to take your IT operations to the next level? Book a meeting or get in touch.