Siirry sisältöön
Global sound
AI Post Transformers

AI Post Transformers

Investor's Last Exam: Benchmarking AI Equity Research Without Future Leakage

Jakson kuvaus

This episode examines a paper that builds a benchmark for testing whether AI agents can do equity investment research without seeing the future. It centers on a Tesla task whose search used a "before March 24, 2026" date filter. The filter still returned a July page reporting realized Q2 deliveries of 480,126, because pages can keep an old publication date after their content is updated. The benchmark's fix is version-level availability: every document revision is stored separately and is eligible only if it predates the decision time. The database holds about 3.4 million versions, 1.52 TB, and every tool call is checked against it. The discussion explains why finance lacks a pass/fail oracle like SWE-bench. It describes the expert rubrics and dependency graphs that grade the chain from evidence to assumptions to judgment. It also covers the three yardsticks of answer quality, forecast outcome and simulated return, which the paper says rank models differently. The hosts separate retrieval leakage from "parametric leakage," meaning outcomes already in a model's pretraining weights, and tie that to prior work on lookahead bias in finance. The episode closes by introducing the 86 buy-side-style tasks, the 11 agent systems and the shared six-tool setup.

Sources:
1. Investor's Last Exam: Can AI Agents Do Equity Investment Without Seeing the Future? — Huangfei Jiang, Anning Hu, Kai Sheng, Yuchen Cai, Rui Ge, Xiangyu Kong, Jianghao Lin, Shuzi Chen, Jiyi Weng, Junchi Yan, 2026
http://arxiv.org/abs/2610.38342
2. FinanceBench: A New Benchmark for Financial Question Answering — Pranab Islam, Anand Kannappan, Douwe Kiela, Rebecca Qian, Nino Scherrer, Bertie Vidgen, 2023
https://scholar.google.com/scholar?q=FinanceBench%3A+A+New+Benchmark+for+Financial+Question+Answering
3. BloombergGPT: A Large Language Model for Finance — Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, Gideon Mann, 2023
https://scholar.google.com/scholar?q=BloombergGPT%3A+A+Large+Language+Model+for+Finance
4. TradingAgents: Multi-Agents LLM Financial Trading Framework — Yijia Xiao, Edward Sun, Di Luo, Wei Wang, 2024
https://scholar.google.com/scholar?q=TradingAgents%3A+Multi-Agents+LLM+Financial+Trading+Framework
5. Financial Statement Analysis with Large Language Models — Alex G. Kim, Maximilian Muhn, Valeri V. Nikolaev, 2024
https://scholar.google.com/scholar?q=Financial+Statement+Analysis+with+Large+Language+Models
6. Assessing Look-Ahead Bias in Stock Return Predictions Generated By GPT Sentiment Analysis — Paul Glasserman, Caden Lin, 2023
https://scholar.google.com/scholar?q=Assessing+Look-Ahead+Bias+in+Stock+Return+Predictions+Generated+By+GPT+Sentiment+Analysis
7. Lookahead Bias in Pretrained Language Models — Suproteem Sarkar, Keyon Vafa, 2024
https://scholar.google.com/scholar?q=Lookahead+Bias+in+Pretrained+Language+Models
8. Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest…

Livenä, viimeisin tunti: 5 621 kappaletta 2 184 asemalla 118 maassa

Löydä maailma radion kautta

Lisää tapoja löytää (7)

Kuulitko jotain hyvää? Tallenna se tai avaa se suoraan Spotifyssa.

★ Artist of the week The fastest riser on the radio this week
Your song here? From € 9 per week →
A Fine Jazz Gumbo RadioJatka kuuntelua