Catatan episode
Here's a basic argument that describes "misalignment traps" lying in wait as a mundane threat that will not be straightforward to fix. The argument is as follows:
• Models are very sample efficient, and larger models tend to be more so. They generalise broadly from a few bits.
• Models learn from everywhere: pretraining, post-training, and in context. We may also figure out online learning eventually.
• Narrow data can cause a model to generalise broadly in unexpected ways. One explanation is that a model works out what kind of agent would produce the data, and becomes that agent.
• Such…
Situs: lesswrong.com

