Show notes
This episode examines "Backprop as Functor," which argues that gradient descent on a whole network equals the composition of the gradient-descent learners of its parts. It defines a learner as four pieces: a parameter space, an implementation, an update, and a "request" function. The request tells an upstream module what input it should have received, and this supplies the missing intermediate training target. The discussion explains the categorical machinery in plain terms (categories, functors, monoidal structure, and the Para and Learn categories). It presents the main theorem: a faithful, strong symmetric monoidal functor sends parametrised functions to gradient-descent learners, with the request given by backprop. The episode compares this to difference target propagation, which only approximates backprop, and to lenses and open games, where the request plays the role of the backward "put". It notes that the paper has no experiments and asks whether the formalism does more than restate the fact that the chain rule is functorial. It then starts a worked quadratic-error example showing why the composite update matches plain gradient descent. Listeners get a clear view of what a compositional treatment of backprop does and does not buy.
Sources:
1. Backprop as Functor: A compositional perspective on supervised learning — Brendan Fong, David I. Spivak, Rémy Tuyéras, 2017
http://arxiv.org/abs/1711.10455
2. Backprop as Functor: A compositional perspective on supervised learning — Brendan Fong, David Spivak, Rémy Tuyéras, 2017 (arXiv); LICS 2019
https://scholar.google.com/scholar?q=Backprop+as+Functor%3A+A+compositional+perspective+on+supervised+learning
3. Categorical Foundations of Gradient-Based Learning — G.S.H. Cruttwell, Bruno Gavranović, Neil Ghani, Paul Wilson, Fabio Zanasi, 2021 (arXiv); ESOP 2022
https://scholar.google.com/scholar?q=Categorical+Foundations+of+Gradient-Based+Learning
4. The Simple Essence of Automatic Differentiation — Conal Elliott, 2018
https://scholar.google.com/scholar?q=The+Simple+Essence+of+Automatic+Differentiation
5. Position: Categorical Deep Learning is an Algebraic Theory of All Architectures — Bruno Gavranović, Paul Lessard, Andrew Dudzik, Tamara von Glehn, João G. M. Araújo, Petar Veličković, 2024
https://scholar.google.com/scholar?q=Position%3A+Categorical+Deep+Learning+is+an+Algebraic+Theory+of+All+Architectures
6. Difference Target Propagation — Dong-Hyun Lee, Saizheng Zhang, Asja Fischer, Yoshua Bengio, 2015
https://scholar.google.com/scholar?q=Difference+Target+Propagation
7. Compositional Game Theory — Neil Ghani, Jules Hedges, Viktor Winschel, Philipp Zahn, 2016 (arXiv); LICS 2018
https://scholar.google.com/scholar?q=Compositional+Game+Theory
8. Combinators for Bidirectional Tree Transformations: A Linguistic Approach to the View-Update Problem — J. Nathan Foster, Michael B. Greenwald, Jonathan T. Moore, Benjamin C. Pierce, Alan Schmitt…
