Skip to content
Global sound
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

Orchestrating local LLMs for sensitive data with Airflow

· 27 min · Season 1 Episode 97

Show notes

Running LLMs over sensitive data means external APIs are off the table, but open-weight models still need reproducibility, validation, and a paper trail before downstream systems can depend on their output. In this episode, Kenten is joined by Chhayank Jain, Senior ML Engineer at Visa, to talk through how Airflow orchestrates local LLM pipelines end to end: nightly inference over millions of records, silent failure handling with Pydantic and distribution checks, and using a separate DAG to ma...

Live, last hour: 11,099 songs on 5,399 stations in 145 countries

Discover the world through radio

More ways to discover (9)

Heard something good? Save it, or open it straight away in Spotify.

Indie88Continue listening