Ir para o conteúdo
Global sound
System Design on the Road

System Design on the Road

S2 E15 — Design a Web Crawler: URL Scheduling, Deduplication, and Politeness

· 33 min · Temporada 2 Episódio 15

Notas do episódio

A crawler has ten million queued URLs, but many point to one small website. Daniel and Maya design a frontier whose scheduling rules limit pressure on remote origins instead of letting queue size drive request volume.

Discovery, deduplication, fetches, parsing, and publication form a complete path. Shared politeness, infinite calendars, changing pages, redirects, and hostile content test its limits. Recovery must preserve both accepted results and the permissions and timing rules governing the next fetch.

Chapters
00:00:00 Welcome and a crowded URL frontier
00:02:56 Crawl scope and URL…

Site: podcasters.spotify.com

Ao vivo, última hora: 8.255 músicas em 4.335 rádios de 126 países

Descubra o mundo pelo rádio

Mais formas de descobrir (7)

Ouviu algo legal? Salve ou abra direto no Spotify.

★ Artista da semana A maior subida na rádio esta semana
A tua música aqui? Desde € 9 por semana →
Radio 538 luisteren - Live!Popular agora