Notes d’épisode
A counterintuitive experimental result has been reported in the technology of distillation, which makes large language models smaller.
In knowledge distillation, normally the large teacher model passes the entire probability distribution of the next token to the student model. The design involves calculating probabilities across the entire vocabulary and using that as a clue for learning. What this research showed was that this entirety might have been significantly excessive.
In experiments conducted on Qwen3, a prominent model from China, using just 0.01% of the vocabulary with the…
Site web: tech.airadio.news

