Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

If you look at http://wap.bing.com/robots.txt, the URLs that Google is returning are actually all set to 'allow', not disallow.

It also looks like m.bing.com/robots.txt blocks /search while their actual URLs are /Search - I guess Googlebot treats robots.txt as case-sensitive.



Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: