Brown University — Is Difference-in-Differences Actually Broken?

Is difference-in-differences broken? Brown economist Peter Hull used AI agents to replicate 1,346 staggered-adoption results from 108 papers: TWFE and modern DiD estimates correlate about 0.9, and significant sign flips are rare.
Key takeaways:

• The worry. In staggered rollouts, two-way fixed effects (TWFE) can put negative weights on some treated observations, so a policy that helps everyone could in theory look harmful.
• The test. AI agents (OpenAI Codex) screened 16,094 records from ten economics journals, with the author overseeing. TWFE was compared with DiD-averaging (Callaway and…