A new paper shows how to keep LLMs from being tricked into sharing secrets

Researchers at Microsoft and OpenAI published a paper this week on a class of attacks called model extraction. The idea is simple: send a bunch of queries to a fine-tuned model, read the outputs, and use those to train your own copy of it. The paper shows the attacks are real and works against several popular models.

The authors ran the attacks on GPT-4o, GPT-4o-mini, Claude, Llama, and several others. They also tried a few defense tricks. One is to add noise to the outputs so the extracted model can't learn the fine details. Another is to limit how many queries a user can make. Both help, but neither blocks the attack on its own. The best results come from combining them.

The real takeaway is practical: if you've fine-tuned a model on proprietary data and are serving it over an API, someone can probably reconstruct it. The paper doesn't say how much it would cost or how fast, but the attacks are fast enough to be worth trying. If you're selling model access, this is the kind of thing you should build protections for before someone else does.

Why this matters for us: lots of small shops are running their own fine-tuned models for customer service or internal tools — if those models can be copied, the value they bought is leaking out.