The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
Orchestrating strictly sequential ETL pipelines at Synechron
· 24 dk · Sezon 1 Bölüm 93
Bölüm notları
Strict sequential execution across DAGs sounds simple until you have scheduled pipelines and event-driven pipelines writing to the same MongoDB collections. Ivana Isailovic, Senior Big Data Engineer at Synechron, joins Marc Lamberti to walk through the three-layer DAG architecture her team built to solve exactly that, plus how they generate 200-task DAGs and how they rebuilt subdag-style group retries in Airflow 3. Key Takeaways: (00:00) Introduction. (02:16) The stack: Snowflake source, Mo...
