Naar de inhoud
Global sound
Mechanical Dreams

Mechanical Dreams

Pretraining Mixture of Experts for Emergent Modularity

· 22 min

Beschrijving

In this episode:
• Introduction to Monolithic MoEs: Linda introduces EMO and Professor Norris discusses the memory bottlenecks of standard monolithic Large Language Models.
• The Document-Level Routing Constraint: Linda explains EMOs core mechanism of restricting tokens in a document to a shared pool of experts.
• Overcoming Training Instability: The hosts discuss the conflict with local load balancing and how global load balancing and dynamic pool sizing solved it.
• Semantic Specialization and Extreme Pruning: Linda reveals the performance numbers, showing how EMO can drop over 85 percent…

Website: mechanicaldreams.mechanicaldirk.com

Live, afgelopen uur: 10.774 nummers op 4.267 zenders in 129 landen

Ontdek de wereld via de radio

Meer manieren om te ontdekken (9)

Iets moois gehoord? Bewaar het, of open het direct in Spotify.

★ Artiest van de week De snelste stijger op de radio deze week
Jouw nummer hier? Vanaf € 9 per week →
Onda Beat Web RadioVerder luisteren