Can data agents do real analysis?

Can an AI agent do the work of a junior data analyst? Shreya Shankar, an incoming database professor at Carnegie Mellon University built a benchmark to find out. Her benchmark uses messy conditions: multiple databases, tribal knowledge, and unstructured text. In her tests, the best agents solved about 61% of queries on the first try.
Tristan and Shreya get into why agents tend to pull everything into pandas instead of pushing computation down to the source, and why a good semantic layer makes such a big difference. They also cover why plugging agents into raw data goes badly while gold-layer…