Helm chart defaults come from someone else's environment

Notes from maintaining 124 of them.

Install the Loki Helm chart with its defaults and you get a StatefulSet with no pods.

Not crashing pods. No pods. Zero of them, because the chart ships singleBinary.replicas: 0. A StatefulSet with zero replicas is a valid object in a healthy state, so helm install exits zero and ArgoCD reports Synced and Healthy. Nothing is logging. Nothing you own is going to mention it.

Why it ships that way isn't obvious. Probably so the chart can be a dependency without deploying anything, which is fair enough, and it does mean a new user's first release looks perfect and does nothing.

And that's one chart. Nobody runs just one. Every upstream release of every chart you run is another chance for a default to mean something new, and none of it fails loudly. It looks fine right up until somebody goes looking for logs that were never shipped.

The expensive defaults are the ones that work

A default that breaks the install is helpful. You hit it in the first minute, you fix it, you forget it ever happened.

The expensive ones do exactly what they say. They pass CI and helm lint, they go green, and they stay green for months, because a health check only ever answers one question: is this object in the state it was asked to be in? It is. Nobody asked whether that state was any good.

Chart maintenance gets treated as a version problem, and the version half is genuinely easy. A bot opens the bump PRs, you read the diff, you merge. But almost nothing that actually reaches production is a version. It's a number, or a boolean, that made sense to the person packaging the chart and doesn't make sense on your cluster.

That's not a criticism of maintainers. They're guessing at your CNI, the plugin that wires up pod networking, your storage class, how many tenants you have. You'd guess "single tenant, cloud, one demo workload" too, because it's the guess that works for the most people.

It installed fine and it does nothing

Loki is the blunt case, and the fix is one line:

  singleBinary:
    # Chart default is 0, which deploys a StatefulSet with no pods.
    replicas: 1

Tetragon is the same trick with higher stakes. Install it and it runs, healthily, exporting process, network and file events as JSON. It also ships no TracingPolicy at all, so what you have is a security tool that detects nothing:

# Tetragon ships no TracingPolicy of its own, so without these it is a firehose
# of process/network/file events with no curated detections.

Green dashboard, no detections. You find out the day something happens and nothing fires.

The blast radius is not the chart

Most bad defaults can only hurt their own release. Admission webhooks are the exception.

A webhook that ships failurePolicy: Fail is making the defensible choice: if I can't run, don't admit pods I haven't checked. Now install that operator before it's configured, or catch it in the gap before cert-manager has issued its serving certificate. The webhook isn't ready, so every pod creation in the cluster fails. Storage provisioning, the CSI drivers that attach volumes, anything a control plane roll needs to bring back. An optional feature nobody finished setting up now holds a veto over unrelated workloads.

For the netbird-operator we set failurePolicy: Ignore, and added a namespace selector so un-labelled namespaces never reach the webhook at all. That's weaker. It's the right weaker: an optional mesh injection should fail to "no injection", not to "no pods". If injection is load bearing for you, keep Fail and fix your install ordering instead.

Cilium has a quieter version of the same problem. With kube-proxy replacement, socket-level load balancing rewrites a Service's virtual IP to a real pod address at connect() time. That covers almost everything, and it misses one case completely:

  # Packets a pod FORWARDS never touch a socket, leave the pod still
  # addressed to the VIP, and blackhole on the node ...
  socketLB:
    hostNamespaceOnly: true

The comment ends with the symptom, which is the bit to remember: SYNs leaving eth0, the opening packet of each connection going out, and no reply ever. Anything that forwards traffic rather than originating it, a VPN gateway or a mesh router, hits this, and it looks like a CNI bug for as long as you let it.

Some values depend on your kernel version

Here the assumption isn't about your cluster at all. It's about the kernel on every node in it.

Tetragon's version of this is the easy one, and it's written down plainly. Every node needs eBPF, the in-kernel mechanism it hooks into, plus the BTF type data that goes with it, which means kernel 5.4 or newer built with CONFIG_DEBUG_INFO_BTF=y. Rook's is easier to miss and a great deal more expensive to get wrong.

Rook lets you set cephx keyType to aes256k, replacing a cipher that Ceph 20.2.4 now flags as insecure. Fair enough, until you reach the CSI keys, which the kernel itself consumes:

# csi.keyType is the one with a node requirement: those keys are consumed by
# the kernel clients, so aes256k needs kernel 7.0+, i.e. Ubuntu 26.04 or newer,
# on every node. Override it to aes on clusters with older nodes.

The CSI keys are the only ones that can strand a mount, so an older node is exactly where this bites. The fix has a trap of its own. Override the CSI key type to aes and you also have to widen the allowed-cipher list to match, or the cluster locks out its own CSI keys. The comment beside that list is blunter about how far a wrong list goes:

# ... an excluded cipher locks out mon. and client.admin with no way back.

That's a Ceph cluster nobody can administer any more, from one list in a values file.

That same Rook block documents a Helm limitation that is easy to go years without meeting:

# Helm cannot null a subchart default from a values override, so do not add
# a daemon.keyType default here; clusters that need one set it themselves.

You can change a subchart's default. You can't un-set it. Rook passes --mon-auth-emergency-allowed-ciphers to every mon the moment daemon.keyType has any value at all, whatever your allowed-cipher list says, and Ceph answers with AUTH_EMERGENCY_CIPHERS_SET, a HEALTH_WARN that surfaces as Degraded in ArgoCD. The only way to have no value is to never put one there.

CRDs that arrive after the things that need them

If a chart ships custom resources, the CustomResourceDefinitions that teach Kubernetes those kinds have to ship with it. Tetragon's upstream default doesn't: crds.installMethod is "operator", so the operator creates them once it's running. An operator that creates CRDs after it starts is always too late for resources applied in the same pass. The TracingPolicy resources go in, nothing recognises the kind, the sync fails. One line fixes it, crds.installMethod: helm, and the CRDs arrive with the chart instead.

Worth checking on any chart that ships its own custom resources, because the ordering is not something the values file will tell you.

We ship defaults too, and this is the one we get asked about most: KubeAid leaves ArgoCD auto-sync off. Changes land in Git and wait for you. You can turn it on per application, and plenty of people do, but the default assumes you'd rather decide when a cluster changes than find out afterwards.

Your alert rules describe a different cluster

Alert rules are worse than values, because a rule that cries wolf teaches people to ignore the channel it fires into.

Take KubeHpaMaxedOut, which ships upstream and fires when a HorizontalPodAutoscaler sits at its maximum replicas. Sensible, until you meet an HPA with min == max. Plenty of people configure those deliberately to pin a workload at a fixed size. It's at maximum permanently. It cannot scale. The alert is true forever and means nothing. One clause sorts it out:

kube_horizontalpodautoscaler_spec_min_replicas{job="kube-state-metrics"}
  !=
kube_horizontalpodautoscaler_spec_max_replicas{job="kube-state-metrics"}

And then there's everything upstream has no reason to ship, because it assumes your disks belong to somebody else. Several of the thirteen mixins in KubeAid only exist because there's real hardware underneath: smartmon, mdraid, zfs, node-count, constant-node-name. Degrade a disk in a RAID array on a physical node and stock kube-prometheus has nothing at all to say about it.

Steal orphan-pvc first. It fires once per cluster when a PersistentVolumeClaim is Bound but no running pod mounts it, and the annotation runs a second query at notification time to name every offender. One alert rather than forty, and it tells you which volumes you're paying for and nobody is using.

Every override is a merge you will pay for

None of this is free.

Every upstream release becomes a merge. An override you can't explain is worse than no override, because in six months nobody remembers whether it was a workaround or somebody's preference. The rule worth stealing: a changed default needs a comment saying what breaks without it. If you can't write that sentence, leave the value alone.

There's a reason to pay that merge cost beyond tidiness. Vendoring means nothing is pulled from upstream at deploy time, so what lands in your cluster is what somebody actually read. Referencing a chart by version is far less work and leaves you with a supply chain you don't control.

Write the exit condition next to the override, or the override never leaves. Ours on ceph-csi says to drop it once upstream ships the fixed version by default. Upstream has. The line is still there.

Some of these will be wrong for you, too. failurePolicy: Ignore is a security decision, not a bug fix. Read the reasoning, then decide. Don't copy the value.

All of that, already decided

A release with no pods. A security tool that detects nothing. A webhook that can veto every pod you have. A cipher list that can lock you out of your own Ceph cluster for good. CRDs that arrive after the resources that need them. An alert that fires forever and means nothing. Every one of those is a decision somebody makes once, then keeps making every time upstream ships.

That's the actual work, and it's the same work for everyone running these charts. So we do it in public, on a schedule rather than whenever somebody remembers. A job opens the upstream changes as pull requests every week and releases are cut from what gets merged, so what you review is a version diff instead of an archaeology session in a values file.

The charts are in argocd-helm-charts/, the alert rules in build/kube-prometheus/mixins/. Every quote in this post is a line in one of those files, so check me rather than believe me.

You don't have to adopt anything to use them. If you run Loki, or Cilium, or Rook, open that chart's values.yaml, read the comments, and take the two lines that apply to you.

The rest of KubeAid applies the same idea further out: Cluster API templates for AWS, Azure and Hetzner, KubeOne for SSH-only bare metal, plus Kyverno policies, sealed secrets, and monitoring and compliance defaults that ship in the install rather than separately.

curl -fsSL https://raw.githubusercontent.com/Obmondo/kubeaid-cli/main/scripts/install.sh | sh
kubeaid-cli config generate

That prompts for the provider (AWS, Azure, Hetzner, bare metal, or a local K3D cluster) and writes your config. kubeaid-cli devenv create gives you the local one if you want to poke at it first. No signup, no demo call, no license key.

We are Obmondo. We run Linux and Kubernetes for other people, and we open source the entire toolchain we use to do it: KubeAid, KubeAid CLI, LinuxAid. All AGPL, all free, no demo required.

Ready to take your IT operations to the next level? Book a meeting or get in touch.