The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
Orchestrating AI video intelligence and evaluation pipelines at Firework
· 23 Min. · Staffel 1 Folge 87
Shownotes
How do you orchestrate AI workflows when LLM outputs are non-deterministic and evaluation costs can quietly exceed compute costs? Shawn Feng, Head of Data at [Firework](firework.com), joins the show to walk through how his team uses Airflow to coordinate ingestion, embeddings, and evaluation pipelines for AI features in a video commerce platform. The conversation covers the technical stack, the challenges of validating LLM output, cost guardrails, and what Shawn wants to see next from the Air...
