How an AI token actually travels through a model — and why variance matters

Instacart's engineering team published a long-form deep dive on variance reduction, and it's one of those pieces that earns its length. The idea is straightforward: when you sample a model, you get a distribution of outputs, not a single answer. The variance — how much those answers spread — is what makes or breaks reliability. Below the randomization grain, the post walks through the actual mechanics: temperature, top-p, logit bias, entropy — how each knob shifts the distribution and what it costs in compute.

The piece lands at 11,000 words because it doesn't skip the hard parts. It covers the math behind why variance reduction works (re-ranking, self-consistency, and the newer techniques like P-bias) and the engineering tradeoffs — latency, cost, correctness. The author, who writes for TLDR Data, frames it as a practical guide for anyone building with models, not a theoretical tour.

What makes this worth reading is that it connects the theory to the code. You can see how variance reduction changes the actual outputs of your model, and why the right technique depends on your use case — a chatbot needs different variance control than a code generator.

Why this matters for us: the models we use are getting better, but they're still noisy — variance reduction is how we get them to stop guessing and start delivering.