I don't understand why a decentralized, open, free index is a bad thing. It's not the same as it is today and there's no reason one person would have to carry all the load.
Is anyone here capable of having a conversation instead of jabs, quips or snark? Or at least a good reason why a distributed or copied, shared index is a bad thing for anyone?
Sometimes there's a point to the snark, which is missed by the downvoters.
There's more to an index than "hey, let's share an index". That's WHY we have multiple search engines: there will be differing views of what should be in that shared index, and how it should be constructed to facilitate varying needs. There are multiple search engines because each one plays off "hey, our index is better than theirs".
My snark makes the point: why not just have a single distributed/copied/shared library system, and do away with pricy competition? Answer: because that single system does not provide everything everyone wants, and people are willing to pay for a different selection & service.
The search engine and index are tightly coupled. They're huge, they're complex, they're expensive - and people think they can make a good buck by somehow doing it different. Create a "universal index", and someone will realize they can make money by making their own, leading right back to what we have today.
The TFA's key issue isn't really that multiple indexes are crawling his website, it's that they're gathering his data thru the most inefficient means possible - polling every page as often as practical. You don't want universal index, because short of banning competing indexes there will always be competing indexes. You want an agreed-on search interface: a means to serve the indexes what they want at a cost lower than what it costs them to poll-mine your site.
Yeah, a universal index is a bad idea. It ignores the fact (you'd think visitors to ycombinator of all sites would get this) that there is money in competition, ergo there will never be a single index. That, and it still doesn't solve the problem: it may reduce the number of crawlers sucking your bandwidth, but it's still polling every page as fast and often as it can.
I'm not sure we're talking about the same thing. From what I understand "index" is just a copy of the database of sites. There is little to no value in simply having that index. The index is near useless.
Yeah, the more I read your post, the more I don't think you understand the point of the idea. The idea would be that many people could contribute to indexing the web, faster, and sharing that in a decentralized fashion. With open access to that index, anyone is able to innovate on that data and build their own search engine.
Having fewer crawlers would be a good thing and would indeed decrease the number of hits to your website. More importantly, the work of those crawlers could be distributed and thus page updates could happen even faster.
Even if you're right, that some how the indices themselves can be tuned and can be search-engine specific, who cares? That doesn't preclude or prevent an open index from existing.
BTW, I didn't really downvote anyone, nor did I have a problem with your comment. I just didn't understand it, and it was hot on the tails of another indecipherable comment. I think I understand your point now, but I feel we're either talking about two different things or thinking about them in vastly different manners.
