Skip to content
Global sound
Snacks Weekly on Data Science

Snacks Weekly on Data Science

Compressing LLM Agent context [Shopify]

· 9 min · Episode 163

Show notes

In this episode, we discuss how Shopify tackled a recurring cost in its AI agent by compressing detailed system instructions into a smaller learned representation. The approach preserved prediction quality in the team’s evaluations while reducing latency and GPU requirements. As businesses expand their use of LLMs, the cost of processing tokens at scale becomes an increasingly important operational consideration. Approaches like this show how improving serving efficiency can help businesses scale their AI capabilities while keeping costs manageable.
For more details, you can refer to their…

Website: podcasters.spotify.com

Live, last hour: 11,670 songs on 5,185 stations in 132 countries

Discover the world through radio

More ways to discover (9)

Heard something good? Save it, or open it straight away in Spotify.

Radio 538 luisteren - Live!Popular right now