Skip to content
Global sound
DataTalks.Club

DataTalks.Club

Building AI Evaluations You Can Actually Trust - Tejas Kumar

· 1 h 6 min · Season 25 Episode 9

Show notes

Tejas Kumar joins Alexey Grigorev to discuss building evaluation datasets, calibrating LLM judges against human preferences, and testing agents beyond their final answers.

CHAPTERS
00:00 Podcast intro
00:08 Tejas's path to AI engineering
12:24 Why evals go beyond a manual check
17:37 Sourcing evaluation cases and calibrating a judge
28:33 Refreshing datasets and pinning severe cases
33:10 Judge agreement versus agent performance
50:01 Retraining judges and keeping data fresh
55:59 Human preferences, sample sizes and audience questions
59:46 Evaluating final answers versus the agent's trajectory
1:02:46 Faster judging and further evaluation resources

Resources
AI evals: https://tej.as/blog/ai-evals-llm-judge
Code, demos and slides: https://github.com/TejasQ/basically-agent-evals
Tejas: https://tej.as

Watch the original recording: https://www.youtube.com/watch?v=oWVmQDqZ7tw
Join DataTalks.Club: https://datatalks.club/slack.html

Website: podcasters.spotify.com

Live, last hour: 6,785 songs on 2,950 stations in 127 countries

Discover the world through radio

More ways to discover (7)

Heard something good? Save it, or open it straight away in Spotify.

★ Artist of the week The fastest riser on the radio this week
Your song here? From € 9 per week →
Costa Hit RadioContinue listening