With respect to the author, the conclusion here is very flawed.
If you search for Bing in Google, you get Bing all over page 1. If you search for Google in Bing, you get Google all over page 1. That's not the result of Google capturing click stream data from Google Chrome and copying Bing's results, nor is it the result of Microsoft capturing click stream data from IE8 and copying Google's results. That's just the nature of indexing.
As for robots.txt disallowing those URLs, there is no standard for robots.txt behavior. I have observed some user agents treat it as case insensitive, and others treat it as case sensitive.
Honestly, this isn't even in the same ballpark as the Google accusations made earlier this week, and it smacks of just looking for things to accuse Google of in response to the "Binggate" (ugh, I typed it) drama. Can't we go back to more productive things?
Except for the schema and host parts (which are not part of robots.txt anyway) URLs are case sensitive (ref: RFC 3986, sections 6.2.2.2 and 6.2.3).
The problem here is that Microsoft servers respond to /search, /Search and /SeaRCh without distinction. They are all distinct URLs. If it was the intended behavior (stupid, but understandable, coming from Microsoft), then robots.txt should contain all variants in capitalization for each path. A better solution would be to force a 301 redirect to a canonical path, and have this path in robots.txt. Google would work as expected.
The original article is totally bogus. I can't imagine how it has over 90 votes.
I don't understand why everyone is using the term "copying the results". I think what Bing did was very smart, they incorporated user clickstream data. One could accuse this method of walking a thin line morally, but I suspect that Google's accusation wouldn't have stood any water as a lawsuit.
Because by incorporating clickstream data from Google, they're effectively copying Google search results. Bing should blacklist Google from its clickstream data.
Let's say tomorrow DDG is the search engine with the largest market share. Then Bing would be getting all the clickstream data from DDG. I hope you do realize that this "algorithm" is not Google specific. Its just a novel ranking technique that incorporates a human user feedback loop and is a pretty well known technique in the information retrieval field.
Highly ironic, though, as DDG uses Yahoo as a backend, which uses Bing, which uses Google, which would use...DDG? I think there's a cycle in that list somewhere...
Not necessarily. It is possible that Google incorporates clickstream data too.
The problem that Google's little experiment highlighted was: given the utter lack of any other signal, Bing uses the fact that that URL was ranked #1 by Google's search engine and clicked on by a user.
Having said that: had I been doing this experiment at Google, I would have also added the following variations:
- for some search terms, rank the honeypot URL #1 but don't click on it
- for some search terms, rank the honeypot URL #1 on some _other_ search engine's list and click on it. How can they do that? There are search engines out there which use Google in the backend.
Experiment #1 would have shown more blatant copying. Experiment #2 would have shown whether it's just Google, or any other search engine.
Google has said they do not use clickstream data for ranking from Google toolbar.
They did have variations of their tests. Cutts mentioned this during the bigthink panel. Sometimes they went to Bing first or not, sometimes they clicked on the links and sometimes not, and various other things.
Google has said they do not use clickstream data for ranking from Google toolbar.
Please provide a reference. I've been looking for this statement and haven't found it. When Googlers are directly asked, they pointedly don't answer or say they don't know.
Amit Singhal's statement was carefully worded to be ambiguous on this matter, and Google has apparently confirmed that page-load-time data (at the very least) from the Toolbar does affect rankings.
Such use by search is definitely allowed by Google's written privacy policy. The confirm dialog a user passes when installing the Toolbar refers to that privacy policy.
It'd be very easy for an official Google spokesperson to say clearly that Toolbar data doesn't drive search rankings, if that were true. That they haven't strongly suggests it is used.
Search expert Danny Sullivan made the same observation in his 'Bing: Why Google’s Wrong In Its Accusations' article:
As For The Google Toolbar
Meanwhile, I’m on my third day of waiting to hear back from Google about just what exactly it does with its own toolbar. Now that the company has fired off accusations against Bing about data collection, Google loses the right to stay as tight-lipped as it has been in the past about how the toolbar may be used in search results.
Google’s initial denial that it has never used toolbar data “to put any results on Google’s results pages” immediately took a blow given that site speed measurements done by the toolbar DO play a role in this. So what else might the toolbar do?
You are being disingenuous. That you cannot find them explicitly denying something does not therefore make it true nor provide any evidence that it is true.
“Absolutely not. The PageRank feature sends back URLs, but we’ve never used those URLs or data to put any results on Google’s results page. We do not do that, and we will not do that,” said Singhal.
"Put any results" is vague, perhaps intentionally finessed, language – as I (and the Sullivan quote) already highlighted in the grandparent comment. It could mean, especially in the context of the Bing allegations, that Google Toolbar data never adds a new URL to the index or a result-set, but is still used for relative ranking of already-known URLs.
In the link you provide, Matt Cutts says: "I’m not going to say definitively that Google doesn’t/won’t use toolbar data (or other signals) in ranking." And: "I’m not going to say whether Google uses a particular signal in our ranking." Cutts simply says Toolbar data could be problematic because it could be gamed. Well, links can too – didn't stop Google from building its empire on that impure signal. This seems to me more of the same finesse that creates the impression of a denial without a denial.
Further, it's clear from previous Google statements that page-load-timing from the Toolbar is used to affect rankings. That alone invalidates the 'strong' interpretation of Amit Singhal's statement. So Singhal means something other than "Toolbar data never effects search rankings". What does he mean? Just requoting that vague statement doesn't clear anything up.
And since everything Google does with this data is a closely-guarded secret, how can we be sure of anything, short of awaiting (and then trusting) definitive Google statements? And I can't yet find any clear statement about Toolbar data usage – even though lots of people seem to think they have seen them. (I think general warm feelings towards Google are creating this mistaken impression.)
I don't expect a clear statement; I believe Toolbar clicktrails are a big part of Google' secret sauce. But it means Google could be using equivalent techniques to Bing's, and simply have better filters against the blatant dominance of a single website or only 20 clickers on any result sets.
I do not think it would matter if Sergey and Page personally delivered a stone slab engraved with a statement that Google does not use their toolbar's clicks. As you say, you would have to rely on trusting their statements. You seem to require extraordinary evidence that they do not and have no evidence to back up that they actually do use it, and yet cling to belief that they do.
Matt spent 5 paragraphs on the subject, and yet you have laser focus on one statement that is not even contradicted by other evidence and try to derive whatever you want from it.
Here are some other comments from the panel at bigthink, but I'm sure you will find enough wiggle in them to assert that Google faked the moon landing.
Matt Cutts: "I'm not sure that users realize...when they search on Google and click... those results appear--those clicks appear to be encrypted and sent to Microsoft which then appear to be used in Google's [sic] rankings?"
[stuff about EULAs]
Harry Shum: "Everyone does this Matt you know.."
Matt: "Google... I want to categorically deny that Google does this."
Harry: "That Google does what?"
Matt: "We don't use clicks on Bing's users in Google's rankings."
It's a subtle point, but the whole point of this issue is that Bing actually does copy more than just "clickstream data". Think about what "clickstream" data is - user clicks a link, the link and the text of the link get sent to Microsoft. But in Google's text the search term was not in the link text, nor was it in the target page the user went to. So then how does the search term end up in Bing's index? They have to get it from somewhere - where? The logical conclusion is they get it from the URL of the page the click is performed on, or from the referrer header when the link is clicked (essentially these are the same thing). But how do you do that generically? The search URL contains the query terms in a format that is unique to Google. The only way Bing can be seeing it is if they deliberately are parsing out the Google search terms by specifically targeting how Google encodes them.
So rather than just generically "incorporating clickstream data" this is actually using a special procedure to extract search terms from clicks that are determined to be google searches and then putting those search terms into the bing index.
What Bing did is very smart! It actually was, but not crediting Google there just makes it look like a cheap shot.
You won't quote someone without citing their name now would you?
Sadly you are right, they won't be able to push a lawsuit. Not enough grounds for it, however Bing should acknowledge what they did and are doing crediting Google.
If Google hadn't caught them, we would all be thinking Bing did it on its own, which really isn't the case.
Taking liberty to concoct a scenario. If Walmart asked shoppers to take a photograph of the product layout on display at their favorite shop (which say happens to be Target because its the most popular in town) and used that to make small modifications to its own layout, would you say Walmart needs to credit Target, or that its copying Target? This is an arrangement between Walmart and its shoppers and there is nothing Target can do about it other than making a brouhaha. I don't see why things have to be different in the digital world. We all know how user interfaces historically have been blatantly ripped off.
Actually, Bing should credit the users providing all that clickstream data and all the websites they are using. It just happens that Google is one of many... what's your point?
It's possible the author may have intended that:
.1 since search results are only accessed through a search action, and not links in the wild
.2 since even if links in the wild are followed the google bot should have respected the disallow /search rule (which may have no standard, but google usually respect the format used in that robots.txt, and results are in both cases as of this writing)
3. therefore, they got to know such pages by using clickstream data
Hmm, I'm not sure you RTFA. He didn't search for the term "bing", he searched for the term "site:.bing.com/search". So it IS kinda a big deal that they're disrespecting the robots.txt and listing those pages anyway.
I have also seen Google listing one of my domains for which I specifically disallowed all spiders (Bing doesn't show those domains FYI). My feeble attempt at separating my personal and professional personas was defeated by frigging Google's blatant disregard for Internet etiquette.
Since I'm trying to be anon here, I won't be able to list the search term. Sorry.
There's a bit of a weird myth with robots.txt and the idea that it prevents pages from showing up in search engine indexes. Robots.txt means that the search engine cannot crawl the page - it can still include it in the search results if it sees enough sites linking to it. It can take a guess at what the page title can be, but there's usually no descriptive snippet because it's unable to see what's on the page.
If you don't want the page to be included in the index at all, you can use the meta noindex tag, by putting this in the head:
<meta name="robots" content="noindex" />
Pro tip: you need to also unblock that page from robots.txt - if Google isn't allowed to crawl the page, it can't see the meta noindex tag, which means it would stay indexed.
So, I have to let Google see any page which I want to prevent it from showing it to others. If I want Google to show NONE of my pages to others, I should show them ALL my pages. Conveniently, there is no wildcarded noindex, is there? Nonsensical, but since Google has more power here I'll probably have to bend to their whim.
> I have also seen Google listing one of my domains for which I specifically disallowed all spiders [...] Since I'm trying to be anon here, I won't be able to list the search term. Sorry.
Unsubstantiated accusations. If you truly think Google is disregarding robots.txt but you don't want to divulge the original site, you should set up an experiment as Google has. Otherwise we have no way of determining if your accusations are true, and therefore they should not be taken seriously.
Fine. I reason I stated my point was so that others could chime in if they've had a similar experience.
Let me ask you and others this: Is the following robots.txt supposed to exclude all pages from my domain from showing up in Google results? Am I missing something? According to http://www.robotstxt.org/robotstxt.html I think I'm doing the right thing. Same file is returned for www.<domain>.com/robots.txt and <domain>.com/robots.txt. Google lists <domain>.com/<subdir> in results. I don't think it should be.
> Let me ask you and others this: Is the following robots.txt supposed to exclude all pages from my domain from showing up in Google results?
I believe that robots.txt is a way to prevent your site from being crawled by a robot, but it is not a blacklist against your site appearing in Google search results if it finds a link to your page on a site that does allow robots.
Specifically check out the section "I want to completely remove a page from search results." It appears that if you use the "noindex" meta tag, you can prevent the site from showing up in search results even if other pages link to it. The noindex meta tag is documented here: http://www.google.com/support/webmasters/bin/answer.py?answe...
So it appears the only way to not appear is to let them crawl to see your NOINDEX tag.
And while it's clear NOINDEX prevents a page from appearing in results, it's not clear that it excludes the page contents from analysis by any of Google's algorithms, once collected. (Is it still used to train the spell-checker, for example?)
Great. This link helps. Google's site basically says that in addition to specifically excluding their crawler via robots.txt, I also have MANUALLY submit a request to them. As I said in a reply below, this is nonsensical and Bing is being more reasonable, but I'll grit my teeth and do it since Google has more power here.
It definitely seems like Google is exploiting a loophole in spirit of the definition of robots.txt. Robots.txt is an ancient standard, and I don't think it was anticipated at that time that search engines would gain enough confidence about pages' relevance to list them even if they had not indexed/crawled them.
As I recall, the spirit of robots.txt was not about appropriateness of search results so much as "this URL space can generate an unbounded graph, please don't DoS my server by trying to exhaustively traverse it."
> So it IS kinda a big deal that they're disrespecting the robots.txt and listing those pages anyway.
Did you RMFC? The robots.txt doesn't match what is indexed. As has been exhaustively pointed out in this thread by more than a few commenters, "Search" != "search". On that very search results page, we see #1 which is:
> OLAC search - Bing
OLAC is an unrelated site, and at one point it apparently linked to "www.bing.com/search" with the anchor text "OLAC search - Bing", which Google faithfully indexed but absolutely did not fetch, as you notice there is no description. What else would "OLAC search" mean? This gives us insight into how Google indexes and that they actually respect robots.txt case-sensitively.
The second URL, m.bing.com/Search/Results.aspx, was indexed and fetched because "Search" != "search".
This is highly ridiculous. HTTP URLs are not mandated to be case-sensitive (though it's recommended), and clearly lots of sites use them in a case-insensitive way. Robots should either consider robots.txt in a case-insensitive way (even if I'm conscious that lot of them, including major ones, currently don't do that, which is precisely what I consider to be a problem, and this is supported by what happened here -- where google is risking to appear as a fool). The following article has perfectly good arguments in favor of case-insensitivity or even more clever handling: http://www.slicksurface.com/blog/2007-04/be-careful-robotstx...
Also; failing to respect conditions of use of a service because an automatic process is not safe enough is not a completely exonerating excuse for the operator of such process...
An HTTP client must always assume URL paths and query strings are case-sensitive. You can't rely on the bing.com web servers always returning the same resource for http://www.bing.com/search?q=test and http://www.bing.com/Search?q=test just because the default filesystem for their OS is case-insensitive today. Actually it's unlikely the main entry point for their search engine is a file named "search" in some top-level directory. The only way for an author to tell you many URLs are equivalent is to use "301 Moved Permanently" responses to the canonical one from all the others, and Bing doesn't do that.
If you search for Bing in Google, you get Bing all over page 1. If you search for Google in Bing, you get Google all over page 1. That's not the result of Google capturing click stream data from Google Chrome and copying Bing's results, nor is it the result of Microsoft capturing click stream data from IE8 and copying Google's results. That's just the nature of indexing.
As for robots.txt disallowing those URLs, there is no standard for robots.txt behavior. I have observed some user agents treat it as case insensitive, and others treat it as case sensitive.
Honestly, this isn't even in the same ballpark as the Google accusations made earlier this week, and it smacks of just looking for things to accuse Google of in response to the "Binggate" (ugh, I typed it) drama. Can't we go back to more productive things?