ServiceMonitor "namespaceSelector.any: true" — How a Permissive Setting Broke Prometheus

The Symptom

Prometheus pods in all of a customer's clusters started logging a persistent Role-Based Access Control (RBAC) error:

failed to list *v1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User "system:serviceaccount:monitoring:prometheus-k8s" cannot list resource "endpointslices" at the cluster scope

Scraping targets from ServiceMonitor and PodMonitor resources stopped working. Metrics for several applications (including RabbitMQ) went missing. The error was consistent across four clusters.

The Chase

We first checked the Prometheus ServiceAccount permissions. It had RoleBindings to several namespaces, but no ClusterRole for endpointslices. This is intentional, we use roleSpecificNamespaces to enforce least privilege. Prometheus should only list resources in the namespaces it is explicitly allowed to scrape.

So why was it making a cluster-scoped list request? The error message clearly said at the cluster scope. We looked at the generated Prometheus configuration. For the RabbitMQ scrape jobs, the kubernetes_sd_configs section had no namespaces restriction at all. When no namespaces are listed, the Kubernetes SD client defaults to a cluster-wide informer.

We traced this back to the ServiceMonitor resource deployed by the RabbitMQ Helm chart. It had:

namespaceSelector:
  any: true

The Prometheus Operator translated any: true into an empty namespace list in the generated Prometheus config. This forced Prometheus to use the cluster-scoped informer, which requires cluster-level RBAC permissions.

The Root Cause

A configuration clash. The RabbitMQ ServiceMonitor (managed by the customer in their kubeaid-config repository) used namespaceSelector.any: true, a setting intended for cross-namespace monitoring. But our Prometheus deployment used roleSpecificNamespaces, which grants only namespace-scoped roles. The combination resulted in a forbidden cluster-scoped API call. Prometheus could not discover any endpoints, and all ServiceMonitor-based scraping broke.

Underlying Technology: Prometheus Operator and Kubernetes SD

The Prometheus Operator translates ServiceMonitor and PodMonitor resources into Prometheus scrape_config entries. A key part of that translation is the kubernetes_sd_config for endpoints. The Operator looks at the namespaceSelector field:

• If matchNames is specified, it passes that list to Prometheus. Prometheus then creates a namespaced informer for each namespace.
• If any: true is specified (or the namespaceSelector is omitted), the Operator passes no namespace list. Prometheus then uses a single cluster-scoped informer for that resource type.

A cluster-scoped informer performs LIST requests against the cluster-wide API endpoint (/api/v1/endpointslices). This requires a ClusterRole with permissions for that resource. When the Prometheus ServiceAccount only has namespace-scoped Roles, the API server rejects the request with forbidden at the cluster scope.

The roleSpecificNamespaces feature in kube-prometheus creates separate RoleBindings for each namespace in the prometheus_scrape_namespaces list. It explicitly avoids granting a ClusterRole to minimise blast radius in multi-tenant clusters.

The Fix and Lessons

We updated the RabbitMQ Helm chart to replace any: true with an explicit matchNames list pointing to its own release namespace. We also added any missing namespaces to the prometheus_scrape_namespaces list in the cluster's JSONNET configuration.

• Lesson 1: namespaceSelector.any: true is a foot-gun in environments with strict RBAC. While granting Prometheus a ClusterRole is standard for cluster-wide monitoring, it is bad practice in multi-tenant environments where strict least privilege is required. When Prometheus relies on namespace-scoped Roles, always specify matchNames.
• Lesson 2: Understand the Operator's translation behaviour, an empty namespace list is not the same as a list with one namespace. The former triggers cluster-scoped informers.
• Lesson 3: When deploying third-party Helm charts, always review their ServiceMonitor configurations for overly permissive selectors.

At Obmondo, we now have a policy: all ServiceMonitors must explicitly list the namespaces they scrape. We also added a CI check to detect any: true in our Helm charts and reject it unless explicitly overridden.

Least Privilege Is Why This Broke

A ClusterRole would have made the error disappear in seconds. That is exactly why these clusters don't have one.

KubeAid is our open-source Kubernetes platform: kube-prometheus configured per cluster in Jsonnet, where the namespaces Prometheus may scrape are an explicit list (prometheus_scrape_namespaces) rather than a blanket cluster-wide grant. It won't catch an any: true buried in somebody else's chart. It does mean the blast radius is a list you can read.

• Deploy the stack →
• Star it on GitHub →

Or Let Someone Else Read the Third-Party Charts

Lesson 3 is not a task, it is a standing obligation: every chart, every ServiceMonitor, every upgrade. Obmondo runs Kubernetes as a managed service with 24/7 SRE cover, monitoring stack and its RBAC included.

Same open-source stack. No lock-in.

• Book a free consultation →