Goodput beats throughput when your LLM is actually fast — not just busy

Goodput is the throughput of useful tokens — the ones that end up in the final answer. Throughput counts everything, including wasted tokens the model generated and then threw away because the prompt was too long or the answer was already complete. When the model is churning through tokens but most of them are noise, the throughput number looks great and the service still feels slow.

The post sketches the math: with a 100k context window, a single long prompt can consume 90% of the GPU just to pass data around, while the actual inference uses a fraction of that. That's the kind of waste that shows up as high utilization and low real speed. The fix is usually prompt compression, KV-cache reuse, or chunking — whichever cuts the fat before it hits the GPU.

This is one of those things that sounds obvious once you see it, but nobody writes about it until their latency spikes at 3 AM. It's the difference between a service that looks healthy in Grafana and one that actually answers fast.

Why this matters for us: when we're paying for GPU time or running our own boxes, goodput is what we're actually buying — and the cheap models that look great on paper can be the most expensive once you count the wasted tokens.