Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Pardon me for asking, but how did you crawl Google for a whole weekend. From what I know, Google blocks you if you request too many search queries in a short period of time. Did you use proxies?


Maybe now, I don't remember I had do anything sneaky back then (8-10 years ago)


In 2004 I attempted to do some automatic crawling of Google for my masters thesis and was astonished to get an unfriendly server response saying it was disallowed and "don't even bother asking for an exception for research, it won't be granted."

So at least 11 years ago it was blocked.

(I didn't know about spoofing a user agent back then, so it might not have been as easy as that to get around it.)




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: