Catatan episode
Everyone says they do evals. Almost nobody does. LangChain actually put a number on it: around 89% of teams have observability set up, and only about a third ever do anything with what it collects.
So I got five people who do this for a living in a room, from LangChain, Mercor, Galileo, Langfuse, and CoreWeave. Side note, three of those companies got acquired in the past year. Galileo to Cisco, Langfuse to ClickHouse, Weights & Biases to CoreWeave. The layer is getting bought up faster than most teams can figure out how to use it.
A few things I didn't expect going in. Grading an agent while…

