The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
Orchestrating local LLMs for sensitive data with Airflow
· 27 min · Season 1 Episode 97
Show notes
Running LLMs over sensitive data means external APIs are off the table, but open-weight models still need reproducibility, validation, and a paper trail before downstream systems can depend on their output. In this episode, Kenten is joined by Chhayank Jain, Senior ML Engineer at Visa, to talk through how Airflow orchestrates local LLM pipelines end to end: nightly inference over millions of records, silent failure handling with Pydantic and distribution checks, and using a separate DAG to ma...