Hudi indexes on data that never stops changing

Hudi is a table format for Apache Iceberg. It sits on top of your data lake and lets you write records without rewriting the whole file. The trick is it keeps an index—like a table of contents—that gets updated as records arrive.

Most table formats are built for batch work: you dump files, then read them. Hudi is built for the opposite. Data keeps flowing in—clicks, sensor readings, transactions—and the index keeps up. Without it, every query has to scan the whole dataset just to find the rows that changed.

The moving-target problem is real. When a table is being written to, a query might see stale rows or miss new ones. Hudi's index makes writes atomic at the row level. You get consistency without stopping the pipeline.

The TLDR post walks through how the index stays current as rows are inserted, updated, and deleted. It's not a new idea—Lucene has done this for decades—but it's the right architecture for streaming workloads where the data never sits still.

If you're running a data warehouse on object storage and writing more than you read, this is the kind of thing that saves you hours of refactoring.

Why this matters for us: the hustlers, the side-business owners, the abuelas sending remittances—all of them are data now, and formats like Hudi let that data be updated in real time without breaking the whole system.