Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Not having robots.txt just results in Google defaulting to their reasonable crawl settings. For most sites, a hastily configured robots.txt usually results in problems rather than SEO help.

The indexed pages are also missing most of the 'important' landing pages and other big pages, so most likely some sort of automated spam detection was triggered.

My ISP seems to time out on the domain, sshing and curling from various servers returns the page.



Just saying, the absence of `robots.txt` can be sign of not really caring about SEO.




Consider applying for YC's Summer 2026 batch! Applications are open till May 4

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: