Μετάβαση στο περιεχόμενο
Global sound
The Databricks Data Engineer

The Databricks Data Engineer

Data skew is a checkout line - how to unjam Databricks jobs without bigger clusters

· 12 λεπ.

Σημειώσεις επεισοδίου

Your nightly Databricks job ran long, so you open the Spark UI. Two hundred tasks. One hundred ninety-nine finished in about a minute. One is still running, and it's been running for forty. You double the cluster and rerun it. Same forty minutes. Nearly twice the bill.

The cluster was never the problem. The rule that keeps your joins and group-bys correct is the same rule that lets one task carry the whole job, and no amount of extra workers moves that work off it. Sometimes the stuck task turns out to be worse than slow.

In this episode:
- Why more workers and more shuffle partitions both…

Ιστότοπος: podcasters.spotify.com

Ζωντανά, τελευταία ώρα: 9.594 τραγούδια σε 5.173 σταθμούς σε 126 χώρες

Ανακαλύψτε τον κόσμο μέσα από το ραδιόφωνο

Περισσότεροι τρόποι ανακάλυψης (9)

Ακούσατε κάτι ωραίο; Αποθηκεύστε το ή ανοίξτε το απευθείας στο Spotify.

★ Artist of the week The fastest riser on the radio this week
Your song here? From € 9 per week →
SCCN Smooth FMΣυνέχεια ακρόασης