Model inversion: stealing secrets from your AI model

Researchers can pull private data out of a trained model by probing its outputs — not by hacking a server, but by feeding it inputs and reading the probabilities it spits back. The attack works on classification models, on image classifiers, and even on large language models. You can recover training examples, sometimes whole sentences, sometimes faces. The paper calls it model inversion, and the attacks are getting more practical.

The reason this isn't just a paper exercise is that the models people are running are the same ones in production. If your model was trained on user data — customer reviews, medical notes, photos — an attacker with API access can start asking it questions and watch the confidence scores shift. That's enough to reconstruct parts of the training set. OpenAI, Google, and Anthropic all ship models that expose logits or token probabilities; the attack needs only a few hundred queries.

The fix is not to stop training models. The fix is to limit what the model reveals. That means dropping confidence scores, clamping logits, adding calibrated noise, and treating model outputs as sensitive data the same way you treat a database. If you're shipping an API, you're shipping a side channel — and the side channel is leaking.

Why this matters for us: The apps and bots a lot of our community already uses run on these models, and if the model was trained on people's data, the data might be recoverable through the very buttons they tap every day.