Hamel's eval-smell: how to tell when your AI agent is lying to you

Hamel Hashmi put up eval-smell, a quick diagnostic tool for AI agents. You run it against your evaluation set and it flags the tells — when the agent is overconfident, when it's hedging, when it's hallucinating with a straight face.

The problem it's solving is the one everyone in the trenches knows: your agent looks fine in demos and then breaks the second you put it in front of real data. The evaluation numbers lie. Hamel's tool makes the lie visible.

The writeup is lean — no hand-wringing, no 20-slide deck. Just the problem, the fix, and how to use it. Worth bookmarking if you ship agents for a living.

Why this matters for us: when we build tools for Brown working families — la migra app, the IEP assistant, the screener — we need evals that actually catch mistakes, not ones that smooth them over.