Shownotes
One attention head can only average its past, so it smooths out a spring that needs a minus sign. Through a single sensor it can still rebuild a hidden state from history, if it has room inside to unfold it and is told which regime it is in. And predicting the next step well isn't the same as having an internal state you can read the physics from.

