Hex is benchmarking AI agents that write SQL

Hex has released a benchmark called DataBench for evaluating agentic analytics — AI agents that look at a dataset, figure out what you're asking, and write the SQL to answer it. The benchmark tests how well these agents handle real queries on real tables, not toy examples.

The idea is to measure whether an agent can actually do the work instead of just sounding confident. It covers the kind of queries people actually write: joins across tables, aggregations, time windows. The ones that fail usually fail because the agent picked the wrong table, missed a column, or wrote something that looks right but returns the wrong numbers.

This matters because teams are starting to let AI agents touch production data. If a system can't reliably write SQL on a real schema, it's not ready to sit in front of anything people use every day.

Why this matters for us: When we start letting AI agents query our databases, we need to know they won't quietly return the wrong numbers — especially when the wrong numbers get sent to a boss or a client.