Notas del episodio
This research explores whether transformers, a type of neural network architecture, can learn to reason implicitly over knowledge. The authors find that transformers can learn to reason implicitly, but only through a phenomenon called grokking, where training extends far beyond overfitting. The study investigates two reasoning types: composition and comparison. They find that while the transformers generalize well on in-distribution examples for both types, they struggle with out-of-distribution generalization for composition but succeed for comparison. Through mechanistic analysis of the…
Web: podcasters.spotify.com

