Here's an honest question: Why should a search engine cost so much?
Look at Wikipedia - it's a search engine and content repository for all of the world's knowledge, and its operating expenses are around $100M / year.
The Internet Archive has an annual budget of $10M / year.
Google on the other hand, has annual revenues of $181B / year, or about 2000 times Wikipedia's spending.
Is the crawling part hard? The folks at Common Crawl (https://commoncrawl.org/) crawl the entire web every month and release it free of charge.
What about the search algorithms? Well, you can do a bunch with TFIDF and PageRank, for which patient protection has expired.
For the sake of argument, let's assume that the cost of hosting a ubiquitous search engine would be double Wikipedia's annual costs. Still about a thousand times less expensive than Google.
Ads don't pay for a search engine. They pay for vast amounts of excess.
What keeps such a solution from us? Here are a few of many reasons:
* Google pays billions of dollars a year to make it the default option on browsers.
* Exclusive deals with portals such as LinkedIn which exclude open engines such as Archive.org from their content.
* Limits on peer-to-peer connections perpetuated by IPv4 which prevent us from hosting a distributed engine ourselves.
Posing the problem as one of "there's no other way to pay for this expensive machine" is pretty lame isn't it?
"Businesses should keep adding engineers to work on optimization until the cost of adding an engineer equals the revenue gain plus the cost savings at the margin. This is often many more engineers than people realize." https://danluu.com/sounds-easy/
Yeah, I think the amount of Google employees is completely rational if you consider it for what it is.
The key takeaway from that article:
>Businesses that actually care about turning a profit will spend a lot of time (hence, a lot of engineers) working on optimizing systems, even if an MVP for the system could have been built in a weekend.
Google's goal is to make as much profit as possible. Wikipedia's isn't. If Google had stayed a non-profit search engine the whole time (as proposed in the original paper for Google), then its costs would probably be not too different from Wikipedia's.
Look at Wikipedia - it's a search engine and content repository for all of the world's knowledge, and its operating expenses are around $100M / year.
The Internet Archive has an annual budget of $10M / year.
Google on the other hand, has annual revenues of $181B / year, or about 2000 times Wikipedia's spending.
Is the crawling part hard? The folks at Common Crawl (https://commoncrawl.org/) crawl the entire web every month and release it free of charge.
What about the search algorithms? Well, you can do a bunch with TFIDF and PageRank, for which patient protection has expired.
For the sake of argument, let's assume that the cost of hosting a ubiquitous search engine would be double Wikipedia's annual costs. Still about a thousand times less expensive than Google.
Ads don't pay for a search engine. They pay for vast amounts of excess.
What keeps such a solution from us? Here are a few of many reasons:
* Google pays billions of dollars a year to make it the default option on browsers.
* Exclusive deals with portals such as LinkedIn which exclude open engines such as Archive.org from their content.
* Limits on peer-to-peer connections perpetuated by IPv4 which prevent us from hosting a distributed engine ourselves.
Posing the problem as one of "there's no other way to pay for this expensive machine" is pretty lame isn't it?