October 2026: what changed upstream, and what we did about it

Hi from team Obmondo,

This is the October edition of our monthly newsletter, where we catch up on what changed in the open source projects your platform runs on, and what we did about it. Every story has two halves: the news upstream, and the work in KubeAid or LinuxAid that followed from it.

Kyverno retires its old policy format, and every KubeAid policy was rewritten before the deadline

News: Kyverno 1.19 retires the old policy format

Kyverno is an open source policy engine for Kubernetes. It runs inside the cluster and checks every resource that is created or changed against a set of policies. A policy can block a change that breaks a rule, adjust a resource on its way in, or create other resources alongside it, so the cluster enforces its own rules instead of relying on people to remember them.

KubeAid ships Kyverno policies for the patterns most clusters need: sending image pulls through your registry cache, copying registry pull secrets into every namespace, and giving each namespace default resource quotas and limits. On top of that, we write custom policies for customers where their setup calls for it.

Kyverno 1.19 was released on 20 August 2026, and KubeAid has shipped it since 24 August. It deprecates the legacy policy types; their removal is planned for 1.20, estimated November 2026.

Read the Kyverno 1.19 announcement →

Maintenance: Every policy rewritten before the old format disappears

Every policy KubeAid shipped, and every policy customers ran on top, was the legacy type. All of them would have stopped working at the next Kyverno upgrade. The new format splits the one old ClusterPolicy into five focused types, each named after what it does: ValidatingPolicy checks a resource and can reject it, MutatingPolicy changes a resource on its way into the cluster, GeneratingPolicy creates other resources from it, ImageValidatingPolicy verifies container images, and DeletingPolicy cleans up.

We rewrote every KubeAid policy as one of these. Each policy was tested in an environment replicating real production clusters, including the switch from the old policy to the new one as it will happen on your cluster. The changes will be rolled out in the service windows coming ahead.

Loki's old Helm chart is retired, central logging is rebuilt, and an OpenTelemetry pipeline into OpenObserve

News: Loki's old Helm chart is retired

Grafana Loki is an open source log store for Kubernetes. Unlike a search engine, it does not index the text of every log line; it indexes a small set of labels, such as namespace and container, and stores the lines compressed next to them. That keeps it cheap to run, and it is queried from the same Grafana that already shows your Prometheus metrics, so logs and metrics sit side by side in one place.

Two things changed upstream. The loki-stack Helm chart, the all-in-one chart most installations started from, is marked deprecated by Grafana and still ships Loki 2.9, while Loki itself is at 3.7. And since 16 March 2026 the Loki chart is maintained in the grafana-community repository by the community, with the copy in Grafana's own repository kept in maintenance-only mode.

Read the migration notice from the Loki maintainers →

News: OpenObserve reaches 1.0

OpenObserve is an open source observability platform that keeps logs, metrics and traces in one place. It stores everything as columnar files on object storage instead of in a search index, which is what keeps it cheap to run at volume, and it speaks OpenTelemetry natively, so anything that emits OTLP can send to it without an agent of its own. Version 1.0.0, the first release the project calls generally available, came out on 11 September 2026, adding service level objectives with burn-rate alerts, composite alerts and database monitoring.

Read the OpenObserve 1.0.0 release notes →

Maintenance: Central logging in KubeAid rebuilt on the community chart

KubeAid shipped Loki through loki-stack. We replaced it with the community-maintained chart, on Loki 3.7, and rewrote the defaults around it: one Loki instance per cluster with 30 days of retention, enforced by the compactor; Fluent Bit as the log shipper, chosen over the alternatives for its low memory use on every node.

When one node is no longer enough, the setup grows into Loki's distributed mode on S3-compatible object storage; we skip the intermediate "simple scalable" mode, which the chart marks as deprecated and due for removal in Loki 4. The new chart is in KubeAid since release 32.7.1 (2 September 2026).

Feature development: Building an observability pipeline with OpenTelemetry, Hubble and OpenObserve

This is how a complete pipeline into OpenObserve fits together on KubeAid. We set it up recently for one customer, and every piece of it is in KubeAid for anyone to use. The OpenTelemetry Operator manages the collectors as Kubernetes resources. An agent on every node forwards container logs, the node's systemd journal and Kubernetes events. Hubble, the network observability layer of Cilium, adds a record of every connection between pods, allowed or denied, so a network policy question is answered with a search instead of a packet capture. Sign-in goes through Keycloak SSO.

Metrics stay with the kube-prometheus stack. The first design let the collector gather them too, but every alert rule and dashboard in KubeAid is built on kube-prometheus label conventions, so kube-prometheus keeps metrics and alerting exactly as before and remote-writes a copy to OpenObserve. The result is one interface where a log line, a Kubernetes event and the metric that spiked at the same minute sit next to each other, with OpenObserve's ready-made dashboards for kube-prometheus metrics included.

We hope this is useful as a pattern. You can set it up yourself from your cluster's KubeAid configuration, or tell us and we will set it up with you.

The backup exporter now covers MariaDB and alerts with less noise

Tooling: The backup exporter: did the backup actually land in the bucket?

Every backup tool reports on its own runs. Velero says the volume snapshot succeeded, the Postgres operator says the dump finished, and a dashboard shows green. None of them answer the question that matters when you need to restore: is there a recent, complete copy sitting in object storage right now? A bucket policy can expire files early, credentials can rotate, a schedule can be paused and forgotten, and the tool keeps reporting on the last run it knows about.

The backup exporter is a service we wrote for exactly that gap. It runs on your cluster next to the KubeAid agent, reads the backup buckets directly, and dates what is really there, for Velero volume backups, Postgres logical and WAL backups, MongoDB and MariaDB dumps, and the Sealed Secrets master key. From that it exports Prometheus metrics, and the alert rules shipped with it fire when a backup is missing or older than the recovery point objective (RPO) agreed for your cluster. It is live on our customers' clusters today, and the alerts it raises are the ones our on-call team acts on.

Maintenance: What changed in the exporter this month

The exporter went through three releases in September, with the matching alert rules in KubeAid:

• MariaDB joins the list. Dumps made by the MariaDB operator are now detected and dated, with their own missing and exceeded-RPO alerts.
• A check right after start. The exporter checks the buckets as soon as it starts instead of waiting for its first scheduled run, and a long check no longer overlaps with the next one.
• One alert per namespace. Missing and exceeded-RPO alerts are grouped by namespace and list every affected volume.
• No alerts for what nobody asked to back up. Volumes excluded by label or policy, volumes still waiting for their first run, and volumes deleted from the cluster stay silent.

Running container images are scanned for known vulnerabilities, with the findings in the Obmondo dashboard

Tooling: Scanning the images that are running, not the ones that were pushed

A container image is built once and runs for months, and the vulnerabilities published against the packages inside it keep arriving after the build. A registry scan at push time catches what was known that day; it says nothing about an image that has been running since spring, or about an image pulled from a public registry that never passed through your registry at all.

Trivy is an open source scanner, it reads an image layer by layer, lists its operating system packages and application dependencies, and matches them against the distribution and upstream security advisories. Trivy Operator runs it inside the cluster: it watches every Deployment, StatefulSet, DaemonSet, Job and CronJob, scans each image they run, writes a VulnerabilityReport per workload, and rescans as new advisories are published, so a finding appears for an image that has not changed in months.

Read the Trivy Operator documentation →

Feature development: Every finding in the Obmondo dashboard, with its upgrade path

KubeAid now ships Trivy Operator with a shared in-cluster scanner, set to report only vulnerabilities of Critical or High severity for which a fixed version exists. A new security exporter in the KubeAid agent collects the reports from the cluster, resolves each finding to its CVE score, severity, affected package and version, whether a fixed version exists and the link to the advisory, and forwards it to Obmondo. The exporter only reads.

In the Obmondo dashboard you see which running containers carry known vulnerabilities, ranked by severity, and for each fixable one the version that closes it. An alert is raised when a running image carries a Critical or High severity CVE and an upgrade that closes it is available, so the upgrade can be booked into a service window. The findings are snapshotted against that booking, and the dashboard shows which CVEs the service window closed.

ExternalDNS 0.22 reads a new annotation, and a Service that still carries the old one can lose its DNS record

News: ExternalDNS 0.22 changes the annotation it reads

ExternalDNS is the component that keeps DNS in step with the cluster. When a Service or Ingress carries a hostname annotation, it creates the record at the DNS provider, and when the Service goes away, it removes the record again. KubeAid ships it as a managed chart, so it is installed and updated like every other application on the cluster.

Release 0.22.0, published on 20 August 2026, changed the annotation it looks for. The prefix used since the project began, external-dns.alpha.kubernetes.io/, is replaced by external-dns.kubernetes.io/, and the old prefix is no longer read unless ExternalDNS is explicitly told to keep it. A Service that still carries the old annotation is no longer seen, and with the sync policy in place that is worse than being ignored: the controller treats the live record as orphaned and deletes it at the provider, taking the endpoint offline. The same release makes the sync policy a required setting: upsert-only is no longer the default. The Helm chart followed on 11 September as 1.22.0, and KubeAid 32.9.0 picked that up in September.

Read the ExternalDNS 0.22.0 release notes →

Maintenance: Every cluster moved to the new annotation without losing a record

• Audit first: Every Service and Ingress on every managed cluster was checked for the old prefix. Database endpoints, gateways and application hostnames still carried it.
• Dry run on staging: The new release went to staging clusters with --dry-run=true, so the controller logged what it would create, change or delete without touching the provider. That log was read for deletions before anything else.
• Both prefixes during the move: ExternalDNS was configured to keep reading the old prefix next to the new one, so a manifest could be updated in Git at its own pace while the records stayed in place.
• Manifests first, controller second: The annotations moved to external-dns.kubernetes.io/hostname in version control, and the controller upgrade ran in the regular service windows.

KubeOne 1.14, a round of kubeaid-cli fixes, and the KubeAid agent on arm64 with live update progress

News: KubeOne 1.14 moves the supported range to Kubernetes 1.34 to 1.36

KubeOne is the tool from Kubermatic that installs and upgrades kubeadm-based Kubernetes on servers you already have. KubeAid uses it for generic bare metal and plain virtual machines, the hosts with no cloud API behind them; clusters on AWS, Azure and Hetzner are built with Cluster API instead. Version 1.14.0 came out on 27 July 2026 and supports Kubernetes 1.34, 1.35 and 1.36, with 1.33 dropped. The tooling for migrating off in-tree cloud providers is gone in this release, so a cluster still on one has to finish that move on 1.13 first. Patch releases followed through August, up to 1.14.3 on 24 August.

Read the KubeOne 1.14.0 release notes →

News: The KubeAid agent: the link between your cluster and Obmondo

Every KubeAid cluster that Obmondo looks after runs a small agent. It authenticates to the Obmondo API with a client certificate and reports what the platform needs to know: how many nodes the cluster has, which Kubernetes version it runs, and, through the exporters described earlier in this edition, the state of its backups and the vulnerabilities in its running images. It also does the work during an agreed service window: it prepares the KubeAid update as a change to your cluster's configuration repository, waits for the merge, and runs the Argo CD syncs application by application. The agent holds the Obmondo credential and nothing else; the exporters that need cluster-wide read access run as separate workloads with their own permissions.

See the kubeaid-agent chart in KubeAid →

Maintenance: kubeaid-cli: what changed around cluster setup and upgrade

kubeaid-cli is the KubeAid command line tool that bootstraps a cluster and upgrades it. For a bare metal cluster it renders the KubeOne configuration from your cluster's configuration repository, runs KubeOne, and handles what comes before and after: the Argo CD bootstrap, the drain of each node, the check that every node came back on the target version. Over the last two months it picked up a set of improvements, most of them found while upgrading real clusters:

• Kubernetes 1.35 and 1.36 upgrades. KubeOne 1.14.3 and Cilium 1.20.2 are bundled; Cilium 1.19 is only tested up to Kubernetes 1.34, so the two go together.
• One home per cluster. Configuration, kubeconfig and logs for a cluster now live in one directory of their own. Two clusters bootstrapped from the same working directory no longer overwrite each other's kubeconfig, and a kubeconfig never lands inside a git checkout any more.
• No silent reuse. Starting a fresh configuration next to an existing one now asks before reusing it, instead of handing back the old cluster's name as the default.
• Upgrades on virtual machines. The check for cgroup v2 that runs before an upgrade now recognises KVM and QEMU guests, where it used to stop the upgrade.
• Two alerts that never cleared. The scheduler and controller manager on KubeOne clusters now expose their metrics so Prometheus can reach them, which ends the two "instance unreachable" alerts that fired permanently.

Every upgrade we run for a customer is rehearsed first on a virtual machine that replicates the cluster, with the same kubeaid-cli release that will be used in the service window. That rehearsal is where most of the fixes above came from.

See the kubeaid-cli releases →

Feature development: KubeAid Agent runs on arm64, and shows the update as it happens

Two improvements landed in September, both in release 1.1.3 of 28 September 2026.

Images for amd64 and arm64. Until September the agent image was built for amd64 only. On a cluster that mixes architectures, such as one with Ampere-based arm64 servers from Hetzner next to amd64 nodes, the pod could be scheduled onto an arm64 node and crash at start, and with it the node count reporting stopped and an alert was raised. Every release is now published for both architectures under the same tag, so the scheduler can place the agent on any node. Releases also carry major and major.minor tags, so a cluster can follow a line of releases without pinning each patch.

Live progress in the Obmondo portal. During an update window the agent now reports the state of the update to Obmondo as it goes: the plan, each application as it is syncing, synced or failed, and whether the update is waiting for the merge. The Obmondo portal shows this live, so you can follow a service window while it runs instead of waiting for the report at the end or reading Argo CD yourself. The reporting is asynchronous, so a slow connection to the API never delays a sync.

The NVIDIA GPU Operator is now a KubeAid chart

Tooling: What a GPU node needs before it can run a model

Running AI models on your own cluster takes more than a node with a GPU in it. The kernel driver, the container toolkit, the device plugin that tells Kubernetes how many GPUs a node has, and the exporter that publishes GPU metrics all have to be installed, at matching versions, on every GPU node, and upgraded together. The NVIDIA GPU Operator does that as a Kubernetes operator: it finds the nodes that carry a GPU, installs the components on them as containers, and keeps them in step. Release 26.7.0 came out on 21 August 2026.

See the GPU Operator release →

Feature development: The GPU Operator is now a KubeAid chart

Until September the GPU Operator was an upstream chart that KubeAid did not manage. KubeAid 32.9.1 adds it as a KubeAid chart, so it is installed and upgraded like every other application on the cluster.

MinIO is archived and RustFS 1.0 takes its place as backup storage, with an off-site copy

News: MinIO's community edition is archived, and RustFS 1.0 is out

MinIO is an open source, S3-compatible object store, and for years it was the popular choice for running one on your own servers. In December 2025 MinIO put its open source repository into maintenance mode, with development continuing only in its commercial product, and on 25 April 2026 the repository was archived.

RustFS is an S3-compatible object store written in Rust under the Apache 2.0 licence, built as a replacement for MinIO: erasure coding, versioning, object lock, lifecycle rules and bucket replication. Version 1.0.0, the first release the project calls production-ready, came out on 16 September 2026.

Read the RustFS 1.0.0 release notes →

Feature development: RustFS as backup storage, installed by LinuxAid, with an off-site copy

A backup has to live outside the cluster it protects. After MinIO's retirement the backups first moved to Zenko CloudServer, which runs on a plain server but not on Kubernetes, and we wanted one store that runs in both places. We also looked at Garage, another S3-compatible store, but it is early in its life and its permissions are too coarse for what a backup needs. RustFS won on two counts: credentials with limited access are easier to set up, and the configuration is simpler.

We wrote a Puppet module, published as puppet-rustfs, so LinuxAid installs and manages RustFS on a Linux server: the service runs as a container, its data sits on a dedicated filesystem, and the S3 endpoint is exposed through HAProxy with a TLS certificate obtained automatically. On the Kubernetes side, KubeAid's RustFS chart moved to 1.0.0 the day after the release, and the MinIO charts were removed from KubeAid in September.

The second half is the off-site copy. Bucket replication in RustFS synchronises every backup to a second RustFS at another location, also provisioned through LinuxAid. Access follows least privilege: a backup job gets credentials that can only add objects, and with versioning and object lock switched on, nothing a backup job can do overwrites or deletes an earlier backup. A compromised cluster cannot take its own history with it.

HAProxy 3.2 LTS is the front door on Linux hosts, so no application port faces the internet on its own

News: HAProxy 3.2 is a long-term support release, maintained until 2030

HAProxy is the open source load balancer and reverse proxy that sits in front of services on a Linux host. It terminates TLS, routes requests by hostname and passes them to the application behind it, so the application itself never listens on the internet.

Version 3.2 is a long-term support release, maintained by upstream until 2030. It balances work across thread groups so that many-core machines spend less time waiting on locks, extends the certificate store introduced in 3.0, and adds a first built-in ACME client so the proxy can obtain and renew its own certificates. It also tightens the handling of malformed protocol traffic.

The reason this matters on a Linux host is the services that tend to end up there: an S3-compatible object store listening on its own port, a remote administration panel on another. Each one exposed directly is a separate thing to encrypt, protect and rate-limit. Behind one LTS reverse proxy they share a single front door.

Read the HAProxy 3.2 LTS announcement →

Maintenance: One entry point per host, certificates watched, backend ports closed

LinuxAid's edge routing on Linux hosts was rebuilt around HAProxy 3.2:

• Tracks the LTS stream: HAProxy is installed from the upstream 3.2 repositories, so the host receives the security and stability updates of that branch and stays on an even-numbered LTS release.
• One profile for any backend: A standard profile puts a backend, the RustFS S3 endpoint or the RustDesk web interface for instance, behind a hostname on port 443 with TLS terminated by HAProxy.
• Certificates watched, not scripted: Renewal moved from external scripts to HAProxy's certificate store. A certificate inspector reports days to expiry to Prometheus and drops the thresholds of domains that no longer exist.
• Uptime checks scoped per host: Blackbox exporter checks are tied to the host that serves the domain, so two hosts no longer probe each other's domains and the uptime graph in Grafana is accurate.
• Firewall as policy: Firewall rules are declared in the LinuxAid host policy. Backend ports are closed to the internet and reachable only through HAProxy.

The result is an HTTPS front end for storage and administration tools that needs no manual renewal, with monitoring that stays quiet unless a certificate or a domain is really in trouble.

• Thank you for reading. We hope you found something of value in this edition. If anything here raised a question, write to us at info@obmondo.com and we will answer. See you next month.