Formal verification for AI is finally getting real

Georg Wiese is writing about formal verification — the old math proof approach to software — and why it's suddenly worth looking at again for AI systems. The idea is simple: instead of hoping a model does the right thing, you prove it does. And with open weights and open models, you can actually do this at scale.

The piece walks through how this differs from testing. Testing says the model worked on the examples you gave it. Formal verification says it can't work incorrectly in ways you've specified. For a chip routing model that's already beating traditional tools, that's the difference between a research toy and something you'd actually put in production.

The real question is whether this scales to the models we'll be running in 2026. Georg's arguing yes, and the evidence is accumulating — not just in chip design, but in finance, in the tools that already use Coq and Lean to prove their code is right. AI systems are getting complex enough that testing alone isn't cutting it. We need proofs.

Why this matters for us: the tools we rely on — routing, trading, the apps our abuelos use to check their pension — are now written by models. If they break, they break in ways we can't easily see. Formal verification is how we make sure they don't.