Don't Use "increase()" on Gauges — A False Alert Story

The Symptom

Critical ContainerDown alerts started firing for containers that were clearly running. mailcow-nginx-mailcow-1 and obmondo_monitoring_cadvisor both triggered pages, yet docker ps showed both containers Up continuously. The alerts were false, but they woke people up at night.

A screen full of ContainerDown, and a terminal in the corner quietly reporting Up. Both were telling the truth about what they measured.

The Chase

The alert rule used this PromQL expression:

increase(container_last_seen[30m]) < 1200

container_last_seen is a Unix timestamp (a gauge) that cAdvisor exports. The rule expected the value to increase by at least 1200 seconds (20 minutes) over a 30-minute window. If the increase fell below that, it fired.

We checked the Prometheus logs. The local agent was dropping samples with out-of-order sample errors, and remote writes to the central Prometheus were returning HTTP 400 (duplicate timestamp). The cAdvisor version in use was v0.39.0, released in 2021. Newer versions fixed a timestamp-consistency bug that caused exactly these issues.

Because samples were being dropped, the increase() computation over the 30-minute window saw gaps. Fewer data points meant the "increase" appeared smaller than 1200 seconds, triggering the alert, even though the containers were healthy.

The Root Cause

Two compounding issues:

1. Outdated cAdvisor emitting inconsistent timestamps, causing sample drops both locally and during remote write.
2. Misuse of increase() on a gauge. increase() and rate() are designed for counters, monotonically increasing metrics. Applying them to a gauge (which can go up and down) is semantically wrong. A timestamp gauge resets on every scrape; it does not represent a cumulative value. The alert became hypersensitive to any scrape gap.

Underlying Technology: Prometheus Metric Types and increase()

Prometheus distinguishes between four core metric types:

• Counter: Monotonically increasing (e.g., request count). increase(counter[1h]) gives the total number of events in the last hour.
• Gauge: Can go up or down (e.g., temperature, memory usage, or a timestamp).
• Histogram and Summary for distributions.

increase() works by subtracting the first sample in the range from the last sample, adjusting for counter resets. It assumes the metric is monotonically increasing. When you apply it to a gauge, the result is meaningless, it effectively measures the difference between two arbitrary timestamps, which can be negative or artificially low if samples are missing.

The correct way to check for staleness (container not reporting) is to measure how long ago the container was last seen:

time() - max by (certname, name) (container_last_seen) > 1200

This expression is self-healing: a single successful scrape resets the value. It does not "remember" gaps across a 30-minute window.

The Fix and Lessons

We upgraded cAdvisor to v0.56.2 (latest) to fix the timestamp corruption. We rewrote the alert to use the staleness check instead of increase(). We also added an alert on remote-write failure rates so upstream ingestion problems surface before they cause false pages.

• Lesson 1: Know your metric types. If the metric is a gauge, do not use increase() or rate().
• Lesson 2: For heartbeat-style alerts, use time() - last_seen. It is simpler, more robust, and self-healing.

At Obmondo, we now audit all alert rules for metric-type correctness as part of our PR review process. We also maintain a version tracking system for exporters to prevent outdated software from running in production.

The Exporter Was Three Years Old

Nobody decided to run cAdvisor v0.39.0 in production. It just never came up.

LinuxAid is our open-source Linux fleet manager: exporters and packages converge from Git every 30 minutes, so versions don't drift quietly. It wouldn't have caught the increase() bug, bad PromQL still needs a human reading a diff, but the stale-exporter half of this outage stops happening.

• See how it works →
• Star it on GitHub →

Or Stop Being Paged for Nothing

A false page at 3 AM costs the same as a real one. If you'd rather someone else own alert quality, Obmondo runs Linux and Kubernetes infrastructure as a managed service, alert-rule review and exporter currency included.

Same open-source stack. No lock-in. If you leave, you keep it.

• Book a free consultation →