I don't understand how these pages could have been crawled - could someone enlig... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

cleverjake on Nov 2, 2012 | parent | context | favorite | on: More than 1MM Facebook accounts exposed

I don't understand how these pages could have been crawled - could someone enlighten?

nico-roddz on Nov 2, 2012 [–]

It's seems that Facebook uses robots.txt to block this pages

https://www.facebook.com/robots.txt

But, depending of the amount of inbound links, Google will index the urls anyway.

It's a common issue.

cleverjake on Nov 2, 2012 | [–]

Google ignores robots.txt if the number of inbound links is > N?

Also - any speculation as to how so many sites were lining to peoples login pages?

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact