The Databricks Data Engineer
by Jakub Lasak
Latest episode
Medallion architecture vs. reality - when your Databricks bronze…
Medallion architecture vs. reality - when your Databricks bronze…
About this podcast
Short episodes on the Databricks skills that get data engineers hired and promoted. One theme per episode, built for a commute: how senior engineers make the calls, what interviews actually test, and the career moves that change your level. By Jakub, a Databricks Data Engineer (ex-Uber).
Episodes
The newest 20 of 33 episodes
- Medallion architecture vs. reality - when your Databricks bronze, silver and gold are just folders · 12 minThree schemas. Bronze, silver, gold. The diagram got nods in the architecture review. Then two dashboards land in the same leadership meeting with two different active customer…
- How the always-sure Databricks engineer caps their own promotion (and never sees it) · 11 minMark's manager asks the room one question: if we move the biggest fact table onto Liquid Clustering, will it fix the slow dashboards? Mark answers first: yes, problem solved…
- Two Databricks jobs, one Delta table, same moment: 3 endings (and the 1 that needs a human) · 12 minTwo in the morning. A nightly merge is folding yesterday's orders into a Delta table. A backfill someone kicked off before logging off is correcting March. Neither job knows the…
- Unity Catalog Can't Manufacture Trust: Your Most-Read Databricks Table Might Be the Least Trusted · 12 minThe rollout is done. Every production table migrated to Unity Catalog, every one owned by a team group, and the revenue table carries a certified check mark. The platform lead…
- Your Databricks director: what they fear, why they fund the duller proposal, what gets yours through · 12 minTwo proposals sit on a head of data's desk, both about the nightly pipeline that feeds the finance dashboard. One swaps the full reload for incremental loads with Auto Loader: it…
- Data skew is a checkout line - how to unjam Databricks jobs without bigger clusters · 12 minYour nightly Databricks job ran long, so you open the Spark UI. Two hundred tasks. One hundred ninety-nine finished in about a minute. One is still running, and it's been running…
- Can You Reach Senior Databricks Engineer Without Ever Leading an Architecture Review? · 11 minSam is four years into a Databricks role, up for senior this cycle. Merged pull requests on the ingestion layer. The nightly job he rescued at three in the morning when the…
- Why Databricks serverless exists, and which of your jobs should stay on classic clusters · 12 minOpen the workspace. Start a cluster. Get coffee. Come back, and sometimes it's ready. If you've been doing this a few years you can feel that wait in your shoulders, and it's the…
- How Databricks engineers lose the final round (and the 4 places it happens) · 11 minYou cleared the recruiter screen. You cleared the Databricks technical round, and cleared it well. Then the final round, then the email that says strong candidate, tough…
- The real reason your Databricks week never got simpler - it isn't the lakehouse · 12 minYour org finished the migration. Two systems became one. The nightly copy nobody wanted to own is gone, and so is the quarterly meeting where finance and the data science team…
- The Databricks incident where the most senior engineer never touched the keyboard · 12 minThursday, just after two in the afternoon. The first message doesn't come from an alert, it comes from a person in revenue operations: the biggest region jumped overnight and…
- 9 Databricks decisions that reveal your real seniority (and one that only looks senior) · 12 minThree in the morning. The nightly job that loads the orders table died halfway. Half the tasks are green, and the rerun button is right there. What you do in the next ten seconds…
- The Databricks job market in 2026: there are two of them, and only one answers · 11 minOne engineer tells you hiring is back and recruiters are warm again. The next one is sixty applications deep and hearing nothing. Same month, same platform, sometimes the same…
- Databricks broadcast joins: when the memo beats the meeting (and when it kills a task) · 11 minTwo engineers on the same team join the same big orders table to the same small lookup table. Same cluster, same data, one line of code different. Sarah's finishes in the time it…
- Build the platform or use the platform: the two Databricks data engineer tracks nobody names · 12 minTwo Databricks engineers sit at adjacent desks. Same title, same pay, same words in the ladder document. Watch a year go by, and they are not doing the same job. Both get rated…
- Why Unity Catalog exists: the Databricks governance chaos it was built to end · 11 minYou inherited Unity Catalog already switched on. You learned the catalogs, the schemas, the grant statements, and never once saw the problem all of it was built to solve. Then…
- How to get credit for Databricks platform work without becoming the on-call martyr · 11 minYour pipeline hasn't failed in months, and nobody noticed. Then it breaks at 2am, you fix it in twenty minutes, and Slack fills with fire emojis and thank-yous. You're the…
- Why Databricks Built Delta Lake When Parquet Was Already Good Enough · 10 minA nightly job is halfway through writing a batch when the cluster dies. Nobody runs a rollback, because there's nothing to roll back to. The next morning a dashboard is quietly…
- The Databricks pipeline that ran green for a year and was wrong the whole time · 8 minA revenue-attribution pipeline runs overnight, bronze to silver to gold, and lands a clean set of numbers the whole company uses to decide where the money goes. Every morning…
- How Photon actually makes your Databricks queries faster (and when it silently doesn't) · 11 minTwo engineers run the same SQL on the same Delta table. Same data, same cluster size, copy-pasted code. Alex goes to make a coffee and comes back to a query still running. Sam's…
