kubernetesbare-metalhigh-availabilityhetznercontrol-planegitopsopensource
When the Floating IP Stays Stuck, A Bare-Metal HA Control-Plane Failure
The Symptom
It started with worker nodes going NotReady one by one. The cluster runs on Hetzner bare-metal with three control-plane nodes. The Container Network Interface (CNI) agents on the workers lost connectivity to the API server and began crash-looping. Soon, the cluster's storage layer (Rook-Ceph) reported that OSDs could not be scheduled, and stateful workloads like Redis master were stuck in Pending. The platform was effectively down.
What made this particularly puzzling was that the API server was supposed to be highly available. A Hetzner Failover IP acts as the single stable endpoint for all worker kubelets and external clients. In an HA setup, if the master node holding that IP fails, a controller should automatically detect it and move the IP to a healthy node. That did not happen. The IP remained attached to a node that had entered a continuous hardware reset loop.
The Failover IP never moved. Workers kept dialling it, and it kept pointing at a dead node, while two healthy masters sat unused.
The Chase
We checked the obvious suspects first. Was the network interface down? No, the healthy master nodes had working network stacks. Was the API server certificate valid? Yes, for the node's own IPs. But workers were trying to reach the Failover IP, and that IP was routing traffic to the dead node.
We audited Helm releases in the cluster. That's when the gap appeared: the hetzner-robot Helm chart, the component responsible for detecting node failures and calling the Hetzner API to migrate the Failover IP, was not deployed. The cluster had never had it. Without it, there was no automation to swing the IP.
The Root Cause
A missing infrastructure dependency. The cluster provisioning checklist did not include the failover controller. When the first master node hard-crashed, the IP stayed pinned to it, creating a routing black hole. Worker kubelets could not reach the API server, CNI could not initialise, and the taint node.cilium.io/agent-not-ready:NoSchedule spread across the cluster, blocking all storage pods.
Underlying Technology: Floating IPs and Controller Automation
In bare-metal Kubernetes, a floating IP (or virtual IP) provides a stable endpoint that is not tied to a single machine. The IP is assigned to a network interface on one node at a time. To make this work in an HA cluster, you need a failover controller that:
- Continuously monitors the health of each control-plane node (e.g., via API server health endpoints or node
Readystatus). - When a node stops responding, the controller calls the cloud provider's API to reassign the IP to another node.
Hetzner provides the Robot API for this purpose. The hetzner-robot Helm chart wraps that API into a Kubernetes controller. It watches node conditions and, upon detecting a failure, executes an HTTP request to Hetzner to migrate the IP. The controller also handles re-registration of the IP if the new node reboots. Without this component, you have a single point of failure, the IP itself becomes a static anchor to whatever node first acquired it.
The Fix and Lessons
We deployed the hetzner-robot chart immediately. We also manually reassigned the IP via the Hetzner console to a healthy master to restore service. Beyond that, we audited all other bare-metal clusters to ensure the chart was present, and updated our provisioning automation (Terraform + ArgoCD) to make it a mandatory deployment step.
- Lesson 1: In an HA cluster, the failover mechanism is not optional, it is part of the control plane. Treat it as a Tier-0 dependency.
- Lesson 2: Provisioning checklists must include all controllers that interact with external infrastructure. A missing Helm chart can cause a full cluster outage.
- Lesson 3: Test failover explicitly during cluster setup (and recovery drills) by taking a master node down to verify the floating IP moves automatically (remembering to reactivate it afterward).
At Obmondo, we now treat failover automation as a day-0 requirement for every bare-metal cluster we deploy. If your control plane depends on a floating IP, make sure you have a controller that can move it, before the failure happens.
Don't Let a Checklist Be Your Failover Plan
The controller was missing because a human had to remember it. That's the part worth fixing.
KubeAid is our open-source Kubernetes platform, built so components like hetzner-robot are declared in Git and reconciled by ArgoCD rather than ticked off a list during setup. It won't stop you leaving a chart out. It makes the omission show up in a diff, instead of the first time a master node stops answering.
If Lesson 3 is the one you keep postponing, Obmondo runs bare-metal and cloud Kubernetes as a managed service, failover drills included. Same open-source stack, no lock-in.