Conversation about Quantization(Also about hosting DeepSeekv4:Flash)

READ THE FULL EPISODE PAGE
https://devmesh.tech/podcast/how-a-284-billion-parameter-model-fits-on-one-machine
A 284-billion-parameter AI model should not fit on one local machine.
DeepSeek V4 Flash does.
In Episode 20 of System Prompt, Peter and Val explore how quantization, mixed precision, importance-aware compression, and speculative decoding make it possible to run a massive mixture-of-experts model on hardware such as a single DGX Spark.
Peter breaks down how Antirez compressed DeepSeek V4 Flash to about 81 GB while preserving enough reasoning, coding, and tool-use ability to remain…