Episodebeskrivelse
One AI request may soon begin on one computer and finish on another. WHAT?!
AMD and Cerebras are separating the two phases of LLM inference: Helios processes prompts and long context, while Cerebras generates tokens. They claim up to 5x more tokens per second per watt, although the figure is based on internal modeling.
Meanwhile, OpenAI reportedly cut inference costs for one segment of ChatGPT by more than half through an undisclosed optimization. The inference race is shifting from installing more chips to extracting more useful work from them.
Attention Span explains how the race to…
Nettsted: inference.mave.digital

