Beschrijving
In this episode:
• Introduction to Monolithic MoEs: Linda introduces EMO and Professor Norris discusses the memory bottlenecks of standard monolithic Large Language Models.
• The Document-Level Routing Constraint: Linda explains EMOs core mechanism of restricting tokens in a document to a shared pool of experts.
• Overcoming Training Instability: The hosts discuss the conflict with local load balancing and how global load balancing and dynamic pool sizing solved it.
• Semantic Specialization and Extreme Pruning: Linda reveals the performance numbers, showing how EMO can drop over 85 percent…
Website: mechanicaldreams.mechanicaldirk.com

