Skip to content
Global sound
The Databricks Data Engineer

The Databricks Data Engineer

Databricks broadcast joins: when the memo beats the meeting (and when it kills a task)

· 11 min

Show notes

Two engineers on the same team join the same big orders table to the same small lookup table. Same cluster, same data, one line of code different. Sarah's finishes in the time it takes to get a coffee. Mike's dies, and the error isn't about the data at all. A task ran out of memory building a hash map.

Broadcast joins get passed around as a tip instead of a model: small table equals fast, flip the switch when a join runs long. But the mechanism in almost every write-up is years out of date, and the size Spark checks is not the size that has to fit in memory.

In this episode:
- How to…

Website: podcasters.spotify.com

Live, last hour: 13,775 songs on 7,116 stations in 143 countries

Discover the world through radio

More ways to discover (9)

Heard something good? Save it, or open it straight away in Spotify.

★ Artist of the week The fastest riser on the radio this week
Your song here? From € 9 per week →
94.7 The WaveContinue listening