Skip to content
Global sound
Rooted Layers

Rooted Layers

The Transformer Attractor

· 25 min

Show notes

In 2023, Mamba promised to replace attention with elegant state-space math that scaled linearly with context. By 2024, the authors had rewritten the core algorithm to use matrix multiplications instead of scans. Their paper explains why:
“We restrict the SSM structure to allow efficient computation via matrix multiplications on modern hardware accelerators.”
The architecture changed to fit the hardware. The hardware did not budge.
This is not a story about hardware determinism. It is a story about convergent evolution under economic pressure. Over the past decade, Transformers and GPU…

Website: lambpetros.substack.com

Live, last hour: 5,240 songs on 2,088 stations in 106 countries

Discover the world through radio

More ways to discover (9)

Heard something good? Save it, or open it straight away in Spotify.

Radio 538 luisteren - Live!Popular right now