AI Analysts — Keeping Type I Error Under Control as They Evolve

Lifetime-valid decisions by continually evolving AI analysts (Sun, Zeng & Wang, Purdue): every prompt edit or model swap gives an AI analyst another chance at a false positive. Continual validation tests each update on simulated null data before it ships.
Key takeaways:

• The risk accumulates. Over 20 runs of 10 updates proposed by Claude Sonnet 5.5, the worst deployed version without validation reached a Type I error of 0.098 (bioequivalence) and 0.183 (A/B tests), against a 0.05 target.
• Continual validation. Each frozen update is tested on fresh data simulated under the null before…