Let me click — the quiet problem with AI screenshots

Ilya Birman wrote a short note about how AI tools are starting to work off screenshots instead of plain text. The idea is simple: the model sees the desktop, reads what's on it, and does the work. No copy-paste. No API glue. You point, click, done.

It sounds convenient until you realize what it actually is. The model is looking at pixels and pretending to read the screen. That's a workaround, not a feature — and workarounds break when the layout changes, the font shifts, or the window moves by a few pixels. The model doesn't actually understand what it's looking at; it guesses. Every time.

The real problem is that this becomes the default interface for a lot of work. When you start routing tasks through screenshots, you lose the clean contract between tool and text. Everything gets fuzzy at the edges. And the more you lean on it, the harder it is to tell when the model is actually doing something useful versus just looking confident.

There's a better way: let the model read the DOM, parse the JSON, hit the API. It's more work upfront, but it works every time. Screenshot-based interaction is a bandaid. Bandaids wear out.

Why this matters for us: the same thing is happening with AI tools in the shop and the restaurant — models trying to read receipts, invoices, and work orders off photos. It looks convenient until it breaks on a Tuesday morning and nobody knows why. Build the clean path first.