What are tokens? The AI unit of measurement explained

Tokens are how AI models count and process language. When you type a question, the AI doesn't read it word by word — it chops it into chunks called tokens.

A token is roughly four characters. One token is usually about three-quarters of a word. So the sentence "Tokens are how AI models count language" breaks into about eleven tokens, not eleven words.

Think of tokens like the little plastic chips you buy at the bodega for the arcade machine. Each chip is a unit. You hand the clerk a handful of chips. The register tallies them up. The machine doesn't care that some chips came from your aunt and some from your uncle — it just counts.

Why tokens matter: the more tokens in your prompt, the more the AI charges you. You can't just keep feeding it paragraphs forever. Eventually the register rings and you stop.

That's also why AI sometimes feels like it's hallucinating. When the model runs low on tokens, it starts guessing at the end. Like a cook who's run out of ingredients and just throws in whatever's in the fridge.

If you're building apps or using AI tools, tokens are the new currency. They're how your usage gets measured, how your costs stack up, and how much of the model's brain gets used up.

Ask your dev cousin: does your tool count tokens by word or by character? The answer changes your bill.