Speculative decoding: the cousin who guesses the answer

Large language models are slow because they generate one word at a time, waiting for the full computation before they can write the next token. It's like a chef who only chops one onion, waits for it to finish, then chops the next. You could hire a sous chef who guesses what's next. Speculative decoding does exactly that. A tiny, fast model — often called the "draft" model — predicts the next few tokens ahead of time. The big model then checks the draft in one pass. If the draft is right, the big model accepts the whole block. If it's wrong, the big model stops at the first mistake and corrects it. You get the speed of the small model with the quality of the big one. The big model still does the heavy thinking, but it does it in batches instead of one at a time. Think of it like a family dinner. The auntie on the phone guesses what the whole table wants before the main course is even ready. She might be wrong about the dessert, but she's right about the main dish. The kitchen doesn't throw away her guess; it just trims the part that's off. That trimming is the rejection step. The draft model is cheap because it's small. The big model is expensive because it's big. You pay the big price only for the parts you actually need. The result is fewer tokens generated per second, lower latency, and the same output quality. No one notices the difference in the final answer. They just notice it's faster. This is why your phone can run a decent model offline now. It's also why companies are racing to make draft models smaller and smarter. The race isn't about the big model anymore. It's about who can guess better, faster, and cheaper.