> Switching costs for a search engine are pretty damn low.
Are they, though? I seem to recall conversations here where some larger sites were only allowing googlebot to crawl their site in robots.txt rules. Their thinking was that the vast majority of traffic is coming from google, and it wasn't worth it to have multiple also-ran search engines crawling their site frequently. There's a case to be made that search engines are actually a natural monopoly.
I blocked a number of non Google search bots on one of my smaller sites the other day, increasing the number of bots I have blocked from about 20 to 30. That reduced the load on my database server by over 80%. It's a small site that only generates about $5/day from AdSense, so I need to keep the expenses down on it, and I don't want all of the revenue paying for bots that don't bring in any significant traffic.
If I didn't block any search bots, I bet the bots would make up at least 95% of the traffic.
On my larger sites I don't mind serving a million page views a day to random search bots, but on that smaller one it doesn't make sense.
I meant end-user switching costs, but you raise good points.
Barrier to entry and startup costs to a competitor have never been lower. I believe recent court wins make “unauthorized” scraping more openly available, and anti-trust regulations should look at allowing competitors to ignore a Google-only robots.txt as long as reasonable throttling is enforced.
> Their thinking was that the vast majority of traffic is coming from google, and it wasn't worth it to have multiple also-ran search engines crawling their site frequently. There's a case to be made that search engines are actually a natural monopoly.
The other case to be made that people doing this sort of decisions should loose their jobs.
Are they, though? I seem to recall conversations here where some larger sites were only allowing googlebot to crawl their site in robots.txt rules. Their thinking was that the vast majority of traffic is coming from google, and it wasn't worth it to have multiple also-ran search engines crawling their site frequently. There's a case to be made that search engines are actually a natural monopoly.