Why Databricks Built Delta Lake When Parquet Was Already Good Enough

A nightly job is halfway through writing a batch when the cluster dies. Nobody runs a rollback, because there's nothing to roll back to. The next morning a dashboard is quietly serving half-written garbage, and no one can tell the good files from the wreckage.

That's not a bug in Parquet. Every one of those files is valid, beautifully compressed, doing its job perfectly. The problem lives one level up, in the one thing a folder of Parquet files has never had and a real table cannot live without.

In this episode:
- Why a folder of Parquet files was never actually a table, and what a crashed…