Bayesian LLM Agents — Reasoning Does the Math, Code Handles the Rest

From Prompts to Posteriors (Filbry, Melo & Gal, Oxford): LLMs break Bayes' rule without reasoning, but reasoning models compute simple posteriors almost exactly, and with code even intractable ones.
Key takeaways:

• Three tests. Dutch-book coherence (Gemini-2.5-Pro, Gemma-4-31B-it, Qwen-2.5-32B-Instruct), then posterior fidelity (KL to a Bayesian oracle) and experimental design (posterior-mass gap) on 20 Questions and the new ProteinWorld, with five open models.
• Coherence depends on the chat. In separate chats Gemini gave 50% to a Democrat winning in 2028 and 55% to a non-Democrat…