User-agent: \* Disallow: / I don't remember the robots.txt rules for sure, but d... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		Tichy on Dec 16, 2010 \| parent \| context \| favorite \| on: Yahoo is Shutting Down Del.icio.us User-agent: * Disallow: / I don't remember the robots.txt rules for sure, but doesn't that mean they don't allow crawlers at all?

ivank on Dec 16, 2010 | [–]

That's the rule for crawlers that aren't Slurp, Googlebot, Teoma, or msnbot.

Tichy on Dec 16, 2010 | | [–]

I noticed the extra rules, but I am neither Slurp, Googlebot, Teoma nor msnbot :-(

mfukar on Dec 17, 2010 | [–]

robots.txt is merely a suggestion.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact