Skip to content
Global sound
OAPN

OAPN

OAPN #5: Distributed Training 101

· 20 min

Show notes

In OAPN #5, we introduce the basics of distributed language model training.
Using educational material from Nathan Lambert’s RLHF Course and Stanford’s CS336, we walk through the foundations of conventional model training and explain the core concepts in a way that connects directly to what OpenAgents is building.
We then introduce DiLoCo, or Distributed Low-Communication Training, and discuss how projects like Bittensor, Templar, and Prime Intellect have approached distributed training, along with how the Psionic and Pylon approach differs.
This episode is meant to help people understand…

Website: openagents.substack.com

Live, last hour: 13,653 songs on 7,361 stations in 144 countries

Discover the world through radio

More ways to discover (9)

Heard something good? Save it, or open it straight away in Spotify.

★ Artist of the week The fastest riser on the radio this week
Your song here? From € 9 per week →
Radio 538 luisteren - Live!Popular right now