The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
by Astronomer
Orchestrating Qualcomm’s Semiconductor Chip Design At Scale
About this podcast
Welcome to The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI— the podcast where we keep you up to date with insights and ideas propelling the Airflow community forward.
Join us each week, as we explore the current state, future and potential of Airflow with leading thinkers in the community, and discover how best to leverage this workflow management system to meet the ever-evolving needs of data engineering and AI ecosystems.
Podcast Webpage: https://www.astronomer.io/podcast/
Episodes
The newest 20 of 123 episodes
- Orchestrating Qualcomm’s Semiconductor Chip Design At Scale · 29 minSemiconductor chip design is not where most people expect to find Apache Airflow, but Qualcomm's CPU team runs millions of pre-silicon simulations and post-silicon validation…
- Orchestrating cloud data migrations at HCL Tech · 20 minLarge-scale cloud migrations can result in a lot to manage when it comes to orhcestration: multi-cloud targets, on-prem extraction, dynamic workloads, and strict operational…
- Building self-service Dags with GitHub Actions at Lyft · 30 minAt Lyft, a small data engineering team supports dozens of scientists and analysts who tune driver pay configurations on demand, all under SOX controls. In this episode, Ken…
- Orchestrating cost-efficient data pipelines at Insignis · 31 minRunning Airflow with a one-person data engineering team means every decision about cost, tooling, and best practices has to earn its keep. In this episode, Kenten sits down with…
- Scaling to 1,000+ Airflow Dags at Descartes / Idelic · 23 min · Season 1 Episode 98Scaling Airflow to over 1,000 active DAGs on a team of 3-4 engineers takes structure. In this episode, Kenten Danas talks with Matt Stavinga, Data Engineer at Idelic (now part of…
- Orchestrating local LLMs for sensitive data with Airflow · 27 min · Season 1 Episode 97Running LLMs over sensitive data means external APIs are off the table, but open-weight models still need reproducibility, validation, and a paper trail before downstream systems…
- Improving security in Airflow 3 · 22 min · Season 1 Episode 96Security is one of the least visible parts of Airflow for most users, but it shapes what features teams can turn on in production. In this episode, Kenten Danas talks with Anish…
- Orchestrating data across 30 companies at itti · 20 min · Season 1 Episode 95Orchestrating data across more than 30 companies means most Airflow users on the platform aren't data engineers. In this episode, Kenten Danas talks with Lucas Trubiano, Data…
- Building self-healing Airflow pipelines at ATC Drivetrain · 27 min · Season 1 Episode 94Manufacturing data pipelines can't afford silent failures. When quality decisions and shop-floor visibility depend on Airflow, a broken DAG at midnight can cost real money. In…
- Orchestrating strictly sequential ETL pipelines at Synechron · 24 min · Season 1 Episode 93Strict sequential execution across DAGs sounds simple until you have scheduled pipelines and event-driven pipelines writing to the same MongoDB collections. Ivana Isailovic…
- Managing financial datasets with Airflow at Wise · 29 min · Season 1 Episode 92Financial data pipelines have to be right the first time. On this episode, Kenten sits down with [Antonello Benedetto](linkedin.com/in/anbento4), Staff Data Engineer at Wise, to…
- Managing travel platform data with Airflow at Headout · 29 min · Season 1 Episode 91Running a travel platform means dealing with fast-moving inventory, real-time fraud detection, and heavy performance marketing attribution, all while keeping data trustworthy for…
- How Airflow orchestration decisions impact Spark performance · 19 min · Season 1 Episode 90Airflow and Spark are one of the most common combinations in modern data platforms, but the orchestration decisions made on the Airflow side often determine whether Spark jobs…
- Using Airflow for diverse client projects at Accion Labs · 28 min · Season 1 Episode 89When a real-time loan eligibility scoring pipeline is built on cron jobs, midnight pages are inevitable. In this episode, [Chandan…
- Consolidating legacy job schedulers with Airflow at Trading Technologies · 31 min · Season 1 Episode 88Migrating two decades of legacy job orchestration is the kind of project most teams quietly avoid. In this episode, Marc Lamberti is joined by [Sanket…
- Orchestrating AI video intelligence and evaluation pipelines at Firework · 23 min · Season 1 Episode 87How do you orchestrate AI workflows when LLM outputs are non-deterministic and evaluation costs can quietly exceed compute costs? Shawn Feng, Head of Data at…
- Orchestrating Retail Data Pipelines at Saks Global · 25 min · Season 1 Episode 86Saks Global runs one of the largest retail data operations in the US, with around 8 million SKUs flowing across point-of-sale, e-commerce, catalog, and fraud detection systems…
- What's New in Apache Airflow® 3.3 · 27 min · Season 1 Episode 85Airflow 3.3 is here, with a set of features to help with the messy realities of production pipelines: persisting state across retries, reacting intelligently to different failure…
- Running Airflow 3 in a regulated environment at OTPP · 19 min · Season 1 Episode 84Running Apache Airflow at a major pension fund means balancing strict compliance requirements with the need to move fast on new capabilities. On this episode, Kowsy Narayan…
- Managing a Customer Analytics Platform with Airflow at Skimlinks · 23 min · Season 1 Episode 83Skimlinks runs a reporting platform that serves around 2,000 weekly publisher users, and the data infrastructure behind it runs on Airflow. In this episode, Julian Larralde…







