Data Engineering Podcast
by Tobias Macey
Latest episode
Entity Resolution at Scale: Graphs, LLMs, and Real-World Data…
Entity Resolution at Scale: Graphs, LLMs, and Real-World Data…
About this podcast
This show goes behind the scenes for the tools, techniques, and difficulties associated with the discipline of data engineering. Databases, workflows, automation, and data manipulation are just some of the topics that you will find here.
Episodes
The newest 50 of 521 episodes
- Entity Resolution at Scale: Graphs, LLMs, and Real-World Data Messiness · 59 min · Episode 520Summary In this episode, I spoke with Steven Renwick, CEO and co-founder of Tilores, about the realities of entity-oriented data processing and the practical role of LLMs in…
- How Amazon Uses Telemetry and AI to Automate Physical Logistics · 50 min · Episode 519Summary In this episode, I spoke with Keerti Kulkarni about the data and AI systems behind physical logistics automation at Amazon. She explained how her team supports dock-door…
- Why Metadata Still Matters: From Data Catalogs to AI-Ready Semantic Hubs · 1 h 24 min · Episode 518Summary In this episode, I spoke with Christian Bremeau and Simon Dynin about their decades-long work in metadata management and how that experience shaped the evolution of Meta…
- Reducing Data Debt with Agile Ledger Architecture · 50 min · Episode 517Summary In this episode Christopher Doidge talks about his Agile Ledger Architecture (ALA) approach to data warehousing and how it aims to reduce data debt while shortening the…
- What Context Really Means in Data Engineering and AI · 53 min · Episode 516Summary In this episode Soham Mazumdar, co-founder and CEO of Wisdom.ai, talks about what “context” really means in data engineering and AI systems. He explores why context has…
- Specialized AI for Data Engineers: Inside Astronomer’s Otto · 46 min · Episode 515Summary In this episode Yetunde Dada discusses Otto, Astronomer’s AI agent for Airflow, and the broader challenge of making agentic tooling actually useful for data engineers…
- Why Multi-Agent Systems Need Shared State, Graph Semantics, and Governance · 1 h 2 min · Episode 514Summary In this episode Ragnor Comerford talks about OmniGraph, a lakehouse-native graph storage layer designed around the needs of agentic systems. He explores how graphs are…
- Building the Context Flywheel for AI Data Agents · 1 h 0 min · Episode 513Summary In this episode Prukalpa Sankar, co-founder of Atlan, talks about what it takes to build a “context flywheel” for AI agents in data-intensive organizations. She explained…
- Holding Kafka Right: Product-Friendly Streaming with TypeStream · 50 min · Episode 512Summary In this episode Jevin Maltais talks about the practical realities of building reliable, product-focused streaming systems with Kafka. Jevin shares lessons from roles at…
- Text to Data Products: Kaarvi’s End-to-End AI for Ingestion, Quality, and Dashboards · 53 min · Episode 511Summary In this episode Shravan Gunda, founder and CEO of Kaarvi AI, talks about building an AI-native, agent-driven data platform designed to eliminate the janitorial work that…
- Scaling Graph Analytics Without ETL: Inside PuppyGraph’s Architecture · 54 min · Episode 510Summary In this episode Weimo Liu, co‑founder of PuppyGraph, talks about the engineering behind their “zero-copy” graph querying engine for lakehouse and database sources. He…
- Maximizing GPU Utilization: Heterogeneous Pipelines with Ray and Kubernetes · 59 min · Episode 509Summary In this episode Robert Nishihara, co-founder of Anyscale and co-creator of Ray, talks about maximizing hardware utilization for AI and data-intensive workloads. He…
- The AI-First Data Engineer: 10–50x Productivity and What Changes Next · 59 min · Episode 508Summary In this episode, I sit down with Gleb Mezhanskiy, CEO and co-founder of Datafold, to explore how agentic AI is reshaping data engineering. We unpack the leap from…
- Treat Metering Like Finance: Building Data Platforms for Consumption Economics · 50 min · Episode 507Summary In this episode Himant Goyal, Senior Product Manager at Salesforce, talks about how data platform investments enable reliable, accurate metering for consumption-based…
- Beyond the PDF: Rowan Cockett on Reproducible, Composable Science · 43 min · Episode 506Summary In this episode Rowan Cockett, co-founder and CEO of CurveNote and co-founder of the Continuous Science Foundation, talks about building data systems that make scientific…
- Beyond Prompts: Practical Paths to Self‑Improving AI · 1 h 1 min · Episode 505Summary In this episode Raj Shukla, CTO of SymphonyAI, explores what it really takes to build self‑improving AI systems that work in production. Raj unpacks how agentic systems…
- Orion at Gravity: Trustworthy AI Analysts for the Enterprise · 1 h 5 min · Episode 504Summary In this episode of the Data Engineering Podcast, Lucas Thelosen and Drew Gilson, co-founders of Gravity, discuss their vision for agentic analytics in the enterprise…
- From Models to Momentum: Uniting Architects and Engineers with ER/Studio · 45 min · Episode 503Summary In this episode of the Data Engineering Podcast, Jamie Knowles (Product Director) and Ryan Hirsch (Product Marketing Manager) discuss the importance of enterprise data…
- From Data Models to Mind Models: Designing AI Memory at Scale · 58 min · Episode 502Summary In this episode of the Data Engineering Podcast, Vasilije "Vas" Markovich, founder of Cognee, discusses building agentic memory, a crucial aspect of artificial…
- Prompt Management, Tracing, and Evals: The New Table Stakes for GenAI Ops · 51 min · Episode 501Summary In this episode of the Data Engineering Podcast, Aman Agarwal, creator of OpenLit, discusses the operational groundwork required to run LLM-powered applications reliably…
- From Legacy to AI-Ready: How MongoDB AMP Accelerates Modernization · 47 min · Episode 500Summary In this episode, Shilpa Kolhar, SVP of Product and Engineering at MongoDB, discusses using MongoDB as a unified foundation for AI-driven and agentic applications. She…
- Branches, Diffs, and SQL: How Dolt Powers Agentic Workflows · 57 min · Episode 499Summary In this episode Tim Sehn, founder and CEO of DoltHub, talks about Dolt - the world’s first version‑controlled SQL database - and why Git‑style semantics belong at the…
- Logical First, Physical Second: A Pragmatic Path to Trusted Data · 41 min · Episode 498Summary In this episode of the Data Engineering Podcast Jamie Knowles, Product Director for ER/Studio, talks about data architecture and its importance in driving business…
- Your Data, Your Lake: How Observe Uses Iceberg and Streaming ETL for Observability · 1 h 12 min · Episode 497Summary In this episode Jacob Leverich, cofounder and CTO of Observe, talks about applying lakehouse architectures to observability workloads. Jacob discusses Observe’s decision…
- Semantic Operators Meet Dataframes: Building Context for Agents with FENIC · 57 min · Episode 496Summary In this episode Kostas Pardalis talks about Fenic - an open-source, PySpark-inspired dataframe engine designed to bring LLM-powered semantics into reliable data…
- Beyond Dashboards: How Data Teams Earn a Seat at the Table · 49 min · Episode 495Summary In this episode Goutham Budati about his Data–Perspective–Action framework and how it empowers data teams to become true business partners. Gautham traces his path from…
- Unfreezing The Data Lake: The Future-Proof File Format · 59 min · Episode 494Summary In this episode PhD researcher Xinyu Zeng talks about F3, the “future-proof file format” designed to address today’s hardware realities and evolving workloads. He digs…
- From Context to Semantics: How Metadata Powers Agentic AI · 1 h 6 min · Episode 493Summary In this episode Suresh Srinivas and Sriharsha Chintalapani explore how metadata platforms are evolving from human-centric catalogs into the foundational context layer for…
- From Data Engineering to AI Engineering: Where the Lines Blur · 27 min · Episode 492Summary In this solo episode of the Data Engineering Podcast, host Tobias Macey reflects on how AI has transformed the practice and pace of data engineering over time. Starting…
- Malloy: Hierarchical Data, Semantic Models, and the Future of Analytics · 59 min · Episode 491Summary In this episode Michael Toy, co-creator of Malloy, talks about rethinking how we work with data beyond SQL. Michael shares the origins of Malloy from his and Lloyd Tabb’s…
- Blurring Lines: Data, AI, and the New Playbook for Team Velocity · 1 h 0 min · Episode 490Summary In this crossover episode, Max Beauchemin explores how multiplayer, multi‑agent engineering is transforming the way individuals and teams build data and AI systems. He…
- State, Scale, and Signals: Rethinking Orchestration with Durable Execution · 52 min · Episode 489Summary In this episode Preeti Somal, EVP of Engineering at Temporal, talks about the durable execution model and how it reshapes the way teams build reliable, stateful systems…
- The AI Data Paradox: High Trust in Models, Low Trust in Data · 52 min · Episode 488Summary In this episode of the Data Engineering Podcast Ariel Pohoryles, head of product marketing for Boomi's data management offerings, talks about a recent survey of 300 data…
- Bridging the AI–Data Gap: Collect, Curate, Serve · 51 min · Episode 487Summary In this episode of the Data Engineering Podcast Omri Lifshitz (CTO) and Ido Bronstein (CEO) of Upriver talk about the growing gap between AI's demand for high-quality…
- Beyond the Perimeter: Practical Patterns for Fine‑Grained Data Access · 1 h 5 min · Episode 486Summary In this episode of the Data Engineering Podcast Matt Topper, president of UberEther, talks about the complex challenge of identity, credentials, and access control in…
- The True Costs of Legacy Systems: Technical Debt, Risk, and Exit Strategies · 1 h 4 min · Episode 485Summary In this episode Kate Shaw, Senior Product Manager for Data and SLIM at SnapLogic, talks about the hidden and compounding costs of maintaining legacy systems—and practical…
- Context Engineering as a Discipline: Building Governed AI Analytics · 52 min · Episode 484Summary In this episode of the Data Engineering Podcast, host Tobias Macey welcomes back Nick Schrock, CTO and founder of Dagster Labs, to discuss Compass - a Slack-native…
- The Data Model That Captures Your Business: Metric Trees Explained · 1 h 1 min · Episode 483Summary In this episode of the Data Engineering Podcast Vijay Subramanian, founder and CEO of Trace, talks about metric trees - a new approach to data modeling that directly…
- From GPUs-as-a-Service to Workloads-as-a-Service: Flex AI’s Path to High-Utilization AI Infra · 57 min · Episode 482Summary In this crossover episode of the AI Engineering Podcast, host Tobias Macey interviews Brijesh Tripathi, CEO of Flex AI, about revolutionizing AI engineering by removing…
- From RAG to Relational: How Agentic Patterns Are Reshaping Data Architecture · 53 min · Episode 481Summary In this episode of the AI Engineering Podcast Mark Brooker, VP and Distinguished Engineer at AWS, talks about how agentic workflows are transforming database usage and…
- Duck Lake: Simplifying the Lakehouse Ecosystem · 1 h 10 min · Episode 480Summary In this episode of the Data Engineering Podcast Hannes Mühleisen and Mark Raasveldt, the creators of DuckDB, share their work on Duck Lake, a new entrant in the open…
- Aligning Business and Data: The Essential Role of Data Modeling · 1 h 6 min · Episode 479Summary In this episode of the Data Engineering Podcast Serge Gershkovich, head of product at SQL DBM, talks about the socio-technical aspects of data modeling. Serge shares his…
- From Academia to Industry: Bridging Data Engineering Challenges · 51 min · Episode 478Summary In this episode of the Data Engineering Podcast Professor Paul Groth, from the University of Amsterdam, talks about his research on knowledge graphs and data engineering…
- High Performance And Low Overhead Graphs With KuzuDB · 1 h 1 min · Episode 477Summary In this episode of the Data Engineering Podcast Prashanth Rao, an AI engineer at KuzuDB, talks about their embeddable graph database. Prashanth explains how KuzuDB…
- Bridging Data and Decision-Making: AI's Role in Modern Analytics · 1 h 10 min · Episode 476Summary In this episode of the Data Engineering Podcast Lucas Thelosen and Drew Gilson from Gravity talk about their development of Orion, an autonomous data analyst that bridges…
- From Bits to Tables: The Evolution of S3 Storage · 50 min · Episode 475Summary In this episode of the Data Engineering Podcast Andy Warfield talks about the innovative functionalities of S3 Tables and Vectors and their integration into modern data…
- Revolutionizing Python Notebooks with Marimo · 52 min · Episode 474Summary In this episode of the Data Engineering Podcast Akshay Agrawal from Marimo discusses the innovative new Python notebook environment, which offers a reactive execution…
- Warehouse Native Incremental Data Processing With Dynamic Tables And Delayed View Semantics · 55 min · Episode 473Summary In this episode of the Data Engineering Podcast Dan Sotolongo from Snowflake talks about the complexities of incremental data processing in warehouse environments. Dan…
- Streamlining Data Pipelines with MCP Servers and Vector Engines · 52 min · Episode 472Summary In this episode of the Data Engineering Podcast Kacper Łukawski from Qdrant about integrating MCP servers with vector databases to process unstructured data. Kacper…
- Foundational Data Engineering At Two Sigma · 55 min · Episode 471Summary In this episode of the Data Engineering Podcast Effie Baram, a leader in foundational data engineering at Two Sigma, talks about the complexities and innovations in data…
