K8s now routes LLM calls like a dispatcher

Datadog shipped an extension for Kubernetes that acts as an LLM routing layer. You point it at your cluster, feed it a list of models — open-source, hosted, whatever — and it decides which one gets the call.

It tracks latency, cost, and error rates per model. If a model starts lagging or burning through tokens, the router can switch traffic to the next one on the fly. No app changes needed.

Think of it like a cousin running a side hustle. You tell the cousin what you need, they check who's got the fastest response right now, and they hand it off. That's the router.

The extension plugs into existing K8s deployments. You don't rewrite your app to talk to it. You just tell the cluster: "route these LLM calls through me."

Datadog's pitch is cost control. If you're running multiple models — one for fast answers, one for accuracy, one for cheap volume — this lets you shift traffic based on real-time performance instead of guessing.

It's not a new model. It's a dispatcher sitting in front of the models you already pay for.

Why this matters for us: Most of us are already juggling too many tools and too many subscriptions. A routing layer that cuts costs without making us rewrite our apps? That's the kind of infrastructure that lets small teams compete without burning cash on unused model capacity.