Because most of the good sites won't let you crawl them any more, unless you're ... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		Animats on Nov 13, 2024 \| parent \| context \| favorite \| on: Two upstart search engines are teaming up to take ... Because most of the good sites won't let you crawl them any more, unless you're Google.

immibis on Nov 13, 2024 [–]

Crawl them anyway. Adversarial interoperability is the name of the game.

Consider applying for YC's Summer 2026 batch! Applications are open till May 4
Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact