Notatki do odcinka
Episode 443: The Rise of Compact, Specialized AI
🔗 Read the full guide here: https://smartkeys.org/small-language-models/
In this episode of the SmartKeys podcast, we challenge the dominant narrative of the artificial intelligence boom: the assumption that bigger is always better. While frontier large language models (LLMs) like ChatGPT, Claude, and Gemini demonstrate staggering general intelligence, deploying a massive, data-center-backed model simply to route support tickets, extract invoice fields, or summarize internal meetings is the corporate equivalent of hiring a brilliant polymath with multiple PhDs just to sort office mail. It creates massive computational overhead, inflates recurring inference costs, and introduces significant data privacy risks.
In this episode, you will learn:
• The Problem with Universal Frontier Models: Why sending routine, repetitive corporate workflows to cloud-hosted frontier LLMs burns unnecessary compute budget and introduces data residency risks.
• Defining Small Language Models (SLMs): How models under 10 billion parameters compare to frontier giants across parameters, memory footprints, and compute requirements.
• Architectural Designs Explained:
• Encoder-only (e.g., DistilBERT): Reading entire inputs simultaneously for rapid classification and sentiment tagging.
• Decoder-only (e.g., Gemma, Phi, Qwen, Llama): Predicting next tokens sequentially for generative chat and drafting.
• Encoder-decoder: Combining ingestion and output generation for structured summarization and translation.
• Model Compression Mechanics:
• Knowledge Distillation: Training a compact "student" model to replicate a giant "teacher" model's outputs without inheriting its parameter bloat.
• Pruning: Removing inactive neural connections and retraining remaining pathways to preserve accuracy.
• Quantization: Compressing 32-bit floating-point weights down to 8-bit or 4-bit integers (e.g., Llama 3.2 shrinking by 56% while running 2–4x faster).
• Low-Rank Factorization: Decomposing large weight matrices to slash computation cycles.
• Real-World Extraction Benchmarks: Why GLiNER2 (205M parameters) matched GPT-4o on CrossNER F1 extraction accuracy (0.590 vs. 0.599) while running in 163 ms on a standard CPU compared to 425 ms.
• The 4 Strategic Business Advantages:
• Inference Cost Slashes: Dropping cost per query to fractions of a cent.
• Ultra-Low Latency: Eliminating network lag for real-time customer support chat and voice assistants.
• Zero Cloud Leakage: Running completely on-premise or on-device to comply with GDPR and sovereign data residency.
• Energy Efficiency: Slashing electricity draw and carbon footprint compared to hyperscale data centers.
• The 80/20 Agentic Routing Strategy: Routing 80% of routine categorization and extraction to local SLMs, escalating only the remaining 20% of open-ended creative reasoning to larger models.
• RAG vs. LoRA Fine-Tuning: When to use Retrieval-Augmented Generation for…
Strona: podcasters.spotify.com

