HyperTuning: Adapting Language Models Without Back-propagation Using Hypernetworks

This episode examines "HyperTuning," an ICML 2023 paper that uses a hypernetwork (HyperT5) to adapt a frozen language model to a new task in a single forward pass, with no back-propagation through the downstream model. It explains how the hypermodel reads up to sixteen labeled examples and emits prefix or LoRA parameters that the downstream model reuses for every query without seeing the examples again. The discussion covers why per-task adaptation is costly, how parameter-efficient methods still require a full backward pass, and how the work relates to earlier hypernetwork, meta-learning, and instruction-to-parameter approaches. It also sets up a skeptical reading of the paper's three claims: that HyperT5 roughly matches full fine-tuning on held-out tasks, that it beats shared-parameter PEFT baselines, and that its output works as a PEFT initialization. The skeptical questions are where the training cost actually moves to, and whether the headline margins hold beyond single-run noise.

Sources:
1. HyperTuning: Adapting Language Models Without Back-propagation Using Hypernetworks
https://proceedings.mlr.press/v202/phang23a/phang23a.pdf
2. Multitask Prompted Training Enables Zero-Shot Task Generalization (T0) — Victor Sanh, Albert Webson, Colin Raffel, et al., 2022 (ICLR; arXiv 2021)
https://scholar.google.com/scholar?q=Multitask+Prompted+Training+Enables+Zero-Shot+Task+Generalization+%28T0%29
3. Finetuned Language Models Are Zero-Shot Learners (FLAN) — Jason Wei, Maarten Bosma, Vincent Zhao, et al., 2022 (ICLR; arXiv 2021)
https://scholar.google.com/scholar?q=Finetuned+Language+Models+Are+Zero-Shot+Learners+%28FLAN%29
4. Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks — Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, et al., 2022
https://scholar.google.com/scholar?q=Super-NaturalInstructions%3A+Generalization+via+Declarative+Instructions+on+1600%2B+NLP+Tasks
5. MetaICL: Learning to Learn In Context — Sewon Min, Mike Lewis, Luke Zettlemoyer, Hannaneh Hajishirzi, 2022
https://scholar.google.com/scholar?q=MetaICL%3A+Learning+to+Learn+In+Context
6. HyperNetworks — David Ha, Andrew Dai, Quoc V. Le, 2016
https://scholar.google.com/scholar?q=HyperNetworks
7. HINT: Hypernetwork Instruction Tuning for Efficient Zero-Shot Generalisation — Hamish Ivison, Akshita Bhagia, Yizhong Wang, Hannaneh Hajishirzi, Matthew Peters, 2022
https://scholar.google.com/scholar?q=HINT%3A+Hypernetwork+Instruction+Tuning+for+Efficient+Zero-Shot+Generalisation
8. Boosting Natural Language Generation from Instructions with Meta-Learning — Budhaditya Deb, Guoqing Zheng, Ahmed Hassan Awadallah, 2022
https://scholar.google.com/scholar?q=Boosting+Natural+Language+Generation+from+Instructions+with+Meta-Learning
9. Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks (HyperFormer) — Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, James Henderson…