Skip to content
Global sound
Search Off the Record

Search Off the Record

Google crawlers behind the scenes

· 25 min

Show notes

Developers often talk about Googlebot as if it were a single program you could just run as "googlebot.exe", but that is not how Google's crawling actually works. In this episode of Search Off the Record, Martin and Gary from the Search Relations team unpack how Google's crawling infrastructure is really built and operated.
They cover why "Googlebot" is a misnomer and how it relates to a central crawling software-as-a-service used by many Google products , how crawl behavior is controlled centrally to avoid overwhelming sites (throttling, handling 503s, and "don't break the internet"…

Website: search-off-the-record.libsyn.com

Live, last hour: 6,892 songs on 2,849 stations in 119 countries

Discover the world through radio

More ways to discover (9)

Heard something good? Save it, or open it straight away in Spotify.

★ Artist of the week The fastest riser on the radio this week
Your song here? From € 9 per week →
Radio KJCKContinue listening