Hacker News
new
|
past
|
comments
|
ask
|
show
|
jobs
|
submit
login
sjwright
on Aug 24, 2016
|
parent
|
context
|
favorite
| on:
Web Scraping in 2016
And if you have some way to identify yourself as a potential competitor to google and not some jackass trying to scrape email addresses or spam comments forms, I'm all ears.
wumpus
on Aug 24, 2016
[–]
A majority of the websites that blekko, a google competitor, contacted to ask for robots.txt access ignored us.
sjwright
on Aug 24, 2016
|
parent
[–]
I agree, it's a difficult conundrum. It sucks.
wumpus
on Aug 24, 2016
|
root
|
parent
[–]
There are worse barriers to entry for a search engine! DMCA take-downs... Right to be forgotten... click history...
Guidelines
|
FAQ
|
Lists
|
API
|
Security
|
Legal
|
Apply to YC
|
Contact
Search:
