postgresqlcloudnativepgbackups3kubernetesmonitoringstorageopensource
CloudNativePG WAL Archives — When an S3 Bucket Kills Your Backups
The Symptom
Three weeks after setting up a CloudNativePG (CNPG) PostgreSQL cluster for a customer, we noticed the Persistent Volume for the primary pod was filling up. WAL (Write-Ahead Log) files were accumulating on the local disk instead of being archived to S3. The backup strategy was broken, and we did not know it.
The CNPG logs showed:
ERROR: WAL archive check failed for server relate-pgsql: Expected empty archive
The barman-cloud-check-wal-archive command was failing with exit code 1. The S3 bucket s3://[BUCKET_NAME] existed, credentials were correct, and we could push files manually. So why was barman rejecting it?
The Chase
We inspected the bucket. It was not empty. It contained stale data from a previous, unrelated test. Barman has a strict requirement: the target bucket must be empty when you first configure archiving for a new server. This is a safety mechanism to prevent mixing backups from different clusters.
We had set up the backup configuration three weeks prior but never verified that the first WAL archive actually succeeded. There was no alert, no post-setup validation, and no dashboard showing the last successful archive. The failure was silent, except for the PV filling up.
The Root Cause
The S3 bucket was non-empty at the time of configuration. Barman rejected it with Expected empty archive, and CNPG did not surface that error prominently. The WAL archiving job failed silently, and WAL files piled up on the PV for three weeks, eventually causing disk pressure alerts.
Underlying Technology: Barman and WAL Archiving
PostgreSQL writes WAL segments (16MB each) to the pg_wal directory. For point-in-time recovery (PITR), you need to archive these segments to durable, long-term storage. CloudNativePG uses Barman Cloud, a tool that ships WALs to object storage: Amazon S3, Azure Blob Storage (ABS), Google Cloud Storage (GCS).
Barman requires a fresh, empty bucket on initialisation for a given server ID. This ensures that there is no leftover data from a previous cluster with the same name, which could cause version mismatches or recovery conflicts. The check barman-cloud-check-wal-archive performs this validation. If it fails, the archiving sidecar in the CNPG pod will not start pushing WALs.
CNPG provides status fields in the cluster custom resource:
status:
lastSuccessfulBackup: ...
backupStatus: ...
But these fields only update if archiving succeeds at least once. If it never succeeds, they remain empty. There is no obvious WALArchivingFailed condition in the status by default.
The Fix and Lessons
We created a new dedicated bucket, ensured it was empty, updated the CNPG cluster spec to point to it, and verified archiving started immediately. We also added a Prometheus alert using CNPG metrics (e.g., cnpg_wal_archive_failed_total) and updated our cluster setup runbook to include a verification step: wait for the first WAL archive to succeed before declaring the cluster ready.
- Lesson 1: Always verify backups after configuring them. A backup configuration that has never succeeded is not a backup, it is a false sense of security.
- Lesson 2: Understand your tools' preconditions. Barman's empty-bucket requirement is well-documented; we missed it.
- Lesson 3: Use monitoring to alert on WAL archive failures, not just on PV usage. By the time the PV fills up, you are already in danger.
At Obmondo, we now have a standard post-setup validation checklist for every CNPG cluster. We also automatically create a fresh, timestamped bucket per cluster to guarantee it is empty.
A Backup You Never Verified Isn't a Backup
Three weeks of "we have backups" was three weeks of WAL files filling a local disk.
KubeAid is our open-source Kubernetes platform, and CloudNativePG is one of the charts in it, deployed from Git and reconciled by ArgoCD. It won't tell you the bucket is non-empty. That check is still yours to run on day one, before you call the cluster ready.
Or Let Someone Else Watch the Archive
If nobody on your team owns the question "did the first WAL actually land," Obmondo runs Kubernetes and PostgreSQL as a managed service with 24/7 SRE cover. The Lesson 1 checklist is already written.
Same open-source stack. No lock-in.