Exact Learning Dynamics in Deep Linear Neural Networks

This episode explores the Saxe, McClelland, and Ganguli paper on exact learning dynamics in deep linear neural networks. It explains why these networks are worth studying even though they collapse to a single linear map: their gradient-descent dynamics are still coupled and non-convex, and they show the long plateaus followed by sudden error drops seen in real deep nets. The discussion covers the Baldi and Hornik result that every critical point is a saddle except the best low-rank solution. It then covers the "decoupled invariant manifold," where each input-output mode evolves independently. The mode-strength equation, whose growth rate is proportional to its own value, explains why small initial weights produce long plateaus. The episode argues that depth need not slow learning if the initial conditions are chosen well, and ties this to unsupervised pretraining and to random initialization. Listeners get a clear, step-by-step derivation of why training time depends on the starting point.

Sources:
1. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks — Andrew M. Saxe, James L. McClelland, Surya Ganguli, 2013
http://arxiv.org/abs/1312.6120
2. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks — Andrew M. Saxe, James L. McClelland, Surya Ganguli, 2013 (arXiv:1312.6120; ICLR 2014)
https://scholar.google.com/scholar?q=Exact+solutions+to+the+nonlinear+dynamics+of+learning+in+deep+linear+neural+networks
3. Deep Learning without Poor Local Minima — Kenji Kawaguchi, 2016
https://scholar.google.com/scholar?q=Deep+Learning+without+Poor+Local+Minima
4. On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization — Sanjeev Arora, Nadav Cohen, Elad Hazan, 2018
https://scholar.google.com/scholar?q=On+the+Optimization+of+Deep+Networks%3A+Implicit+Acceleration+by+Overparameterization
5. Implicit Regularization in Deep Matrix Factorization — Sanjeev Arora, Nadav Cohen, Wei Hu, Yuping Luo, 2019
https://scholar.google.com/scholar?q=Implicit+Regularization+in+Deep+Matrix+Factorization
6. A Fast Learning Algorithm for Deep Belief Nets — Geoffrey E. Hinton, Simon Osindero, Yee-Whye Teh, 2006
https://scholar.google.com/scholar?q=A+Fast+Learning+Algorithm+for+Deep+Belief+Nets
7. Greedy Layer-Wise Training of Deep Networks — Yoshua Bengio, Pascal Lamblin, Dan Popovici, Hugo Larochelle, 2006 (NIPS 2007)
https://scholar.google.com/scholar?q=Greedy+Layer-Wise+Training+of+Deep+Networks
8. Why Does Unsupervised Pre-training Help Deep Learning? — Dumitru Erhan, Yoshua Bengio, Aaron Courville, Pierre-Antoine Manzagol, Pascal Vincent, Samy Bengio, 2010 (JMLR)
https://scholar.google.com/scholar?q=Why+Does+Unsupervised+Pre-training+Help+Deep+Learning%3F
9. Understanding the difficulty of training deep feedforward neural networks — Xavier Glorot, Yoshua Bengio, 2010
https://scholar.google.com/scholar?q=Understanding+the+difficulty+of+training+deep+feedforward+neural+networks
10. Delving…