Googlebot won't follow the link because it's listed in robots.txt. | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		Ded7xSEoPKYNsDd on May 18, 2015 \| parent \| context \| favorite \| on: What one may find in robots.txt Googlebot won't follow the link because it's listed in robots.txt.

ceequof on May 18, 2015 | [–]

Googlebot caches robots.txt for a very, very long time. If you disallow a directory it may take months for the entire googlebot fleet to start ignoring it. Google's official stance is that you should manage disallow directives through webmaster tools.

ryanlol on May 18, 2015 | [–]

Yes, but it will index it.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact