argocdgitopshelmkuberneteskyvernoci-cdopensource
ArgoCD Deadlock — How an "example.yaml" Broke GitOps!
The Symptom
The ArgoCD application for Kyverno had been unrenderable for a long time. Every sync attempt failed with a fatal JSON unmarshaling error. Engineers could not update the Kyverno policies, and the application was stuck in ComparisonError state. Worse, because the application was broken, any manual live changes made to the cluster were not being reconciled by GitOps.
The error was:
failed to unmarshal manifest: error unmarshaling JSON: Object 'Kind' is missing
The Chase
We looked at the Helm chart source for Kyverno in our KubeAid repository. In the /templates/ directory, there was a file named example.yaml. It contained raw Helm values, dictionary keys and values, but no apiVersion, kind, or metadata. Helm treats every file in /templates/ as a resource to render and send to Kubernetes. ArgoCD tried to decode this raw YAML as a Kubernetes object, found no kind, and fatally crashed.
How did we get here? An engineer had placed example.yaml as a reference file while testing. The PR was merged into master, and the feature branch was deleted. ArgoCD was still pointing to the deleted branch. When we tried to switch ArgoCD back to master in the UI, ArgoCD first attempted to render the current branch state (the deleted branch) to validate the change, which failed because the branch no longer existed. We were deadlocked: we could not sync to a working branch because the broken branch prevented ArgoCD from even processing the UI request.
The Root Cause
A misplaced reference file (example.yaml) in the Helm templates directory. Helm includes everything in /templates/ without discrimination. Raw values files do not belong there; they belong in /examples/ or /tests/. Additionally, the feature branch was deleted before ArgoCD was updated to point away from it, creating a GitOps deadlock.
Underlying Technology: Helm Template Rendering
Helm processes every file in the templates/ directory (unless excluded by .helmignore) as a Go template. The rendered output is expected to be a valid Kubernetes YAML manifest (or multiple manifests separated by ---). If the template produces a YAML that is not a valid Kubernetes resource, e.g., missing kind, tools like ArgoCD or helm template will fail to unmarshal it.
ArgoCD's branch management works as follows: when you change the target branch in the UI, ArgoCD fetches the new branch's manifest and compares it with the live cluster state. However, it must first stop the current sync operation. If the current branch is deleted or unrenderable, ArgoCD cannot perform the necessary operations to change the target, it is stuck.
The Fix and Lessons
We restored the deleted branch as a dummy placeholder so ArgoCD could perform the lookup. We then changed the target branch to master (which had already been fixed by removing example.yaml). Once ArgoCD was pointing to a valid branch, we synced the application. We also added a CI check that runs helm template on every PR to ensure the charts are renderable.
- Lesson 1: Never put raw values files in Helm
/templates/, use/examples/. - Lesson 2: When using ArgoCD, update the target branch to
masterbefore deleting the feature branch. Or use a PR-based sync workflow that does not leave ArgoCD pointing at ephemeral branches. - Lesson 3: CI validation of Helm renderability is non-negotiable, it costs seconds but saves hours of recovery.
At Obmondo, we now have a mandatory helm template step in our CI pipeline for every Helm chart change. We also require that engineers point ArgoCD back to master after testing, or we use the --sync-option to avoid leaving dead branches behind.
This One Was Our Own Chart
The example.yaml that deadlocked ArgoCD was sitting in our repository. We merged it.
KubeAid is our open-source Kubernetes platform: Kyverno, ArgoCD and the rest of the stack, declared in Git and reconciled by ArgoCD. It won't stop a stray file landing in templates/. It does mean the chart that breaks you is one you can open, read and fix yourself, rather than one you file a ticket about.
Or Don't Spend a Morning Breaking the Deadlock Yourself
Restoring a deleted branch purely so ArgoCD will let you point away from it is not a skill worth acquiring. Obmondo runs Kubernetes as a managed service with 24/7 SRE cover, chart changes reviewed before they reach your cluster.
Same open-source stack. No lock-in.