Global Medical Device Podcast powered by Greenlight Guru
#464: Why LLM Unpredictability is a Liability in MedTech
· 53 min · Epizoda 464
Poznámky k epizodě
Artificial intelligence has officially entered the mainstream cultural zeitgeist, creating a wave of excitement—and a fair share of fatigue—across the medical device industry. In this episode, host Etienne Nichols sits down with Tyler Harmon, biomedical engineer and CEO of Iaso Automated Medical Systems, to cut through the marketing buzzwords. Together, they explore the technical realities behind the technology stack, shifting the conversation away from generic AI toward specific, actionable engineering frameworks.
The discussion highlights a critical distinction between traditional machine learning models and consumer-oriented Large Language Models (LLMs). Harmon explains that while technologies like convolutional neural networks (CNNs) have successfully processed medical imaging for years, modern LLMs introduce an intentional element of randomness to mimic human conversation. This lack of predictability presents unique challenges for medical device developers who operate in a deterministic, safety-critical environment where reproducibility is paramount.
Looking toward practical deployment, the episode addresses how companies can responsibly govern these tools both within their software architectures and their internal Quality Management Systems (QMS). From classifying external AI models as Software of Unknown Provenance (SOUP) under IEC 62304 to leveraging machine learning for early detection of Acute Respiratory Distress Syndrome (ARDS) in the ICU, this conversation serves as an essential guide for innovators looking to build the next generation of safe, compliant, and effective medical technologies.
Key Timestamps
• 00:05 – Introduction to the dual nature of AI in MedTech: embedded clinical algorithms versus internal process optimization.
• 02:14 – Demystifying the math: Breaking down artificial intelligence into linear and non-linear algorithmic transformations.
• 04:30 – The Turing Test, Markov chains, and why consumer LLMs are mathematically designed to be unpredictable.
• 07:15 – Real-world success stories: How convolutional neural networks (CNNs) revolutionized emergency stroke triage.
• 09:42 – Inside Iaso Automated Medical Systems: Using non-LLM machine learning to identify Acute Respiratory Distress Syndrome (ARDS) in critical care.
• 12:10 – AI Governance in the QMS: Designing specialized Standard Operating Procedures (SOPs) and Machine Learning Management Systems (AIMS).
• 15:35 – Evaluating recent FDA 510(k) clearances for LLM-adjacent technologies and managing third-party stacks as SOUP.
Quotes
"If we as innovators can't explain things to a more general audience, we generally don't understand them ourselves. And if you can't do that, it's probably not the best idea to be implementing it into your products." - Tyler Harmon
"I am probably going to be the biggest advocate you'll ever talk to about 'doctors need enablement, not replacement.' We need to give them the tools, the force multipliers to tackle the challenges they're…
Web: podcast.greenlight.guru
