This Compete study was interesting to me (found on Mashable). The blog posts that summarizes it asks:
An interesting data point got me thinking recently. According to Compete data there are roughly 7.5 billion search queries performed every month by the US Online Population. However there are only about 5 billion search referrals every month. This means that roughly 1/3 of all searches in some sense go unanswered. People search for something and then don’t click on a search result. So the obvious question is which search engine is doing the best job from a “search fulfillment” standpoint?
It then answers:
Yahoo! pretty much takes the cake on this one with about 75% of searches performed on Yahoo! in August resulted in a referral. By comparison, searches on Google result in a referral about 65% of the time and searches on MSN/Live result in a referral about 59% of the time. Lower search fulfillment numbers mean that on a percentage basis fewer search queries on that engine resulted in the searcher clicking on a result link. So from this perspective one might consider Yahoo! more effective at getting consumers the results they want.
I can imagine that these results have to do with any number of contextual issues – do people who use Yahoo search exclusively, for example, have habits that Google searchers do not? For example, I wonder if folks who do tons and tons of searches each day – the most high volume, hard core users of search – might be Google users, and also might use Google the way I do – very iteratively, in other words, I try a lot of query strings till I get the one I want, then I drill down.
Just a thought. When I see studies like this, I tend to not buy the obvious conclusion. That’s not to say it might not be true, but it strikes me more is most likely in play here than meets the eye.
Could it be that Google refers less because questions are often answered in Google itself?
I don’t know about the typical Google searcher, but when I’m looking to define a word, get a quick stock quote, figure out the spelling of someone’s name, or remember the date of a historical event, I don’t usually have to click a result to get an answer — it’s in a onebox or in the snippet.
Are Google’s lower referral numbers evidence of better relevance?
I doubt it, mb — especially for knowledgable users, there are plenty of more foculased, far more effective engines available. Google, Yahoo and MSN are more for the “one-size, fits-all” (novice) users.
For example, http://onelook.com is a quite effective meta-engine that searches across many “reference” properties such as dictionary.com and wikipedia.org (and many dozens of others, too). So if you want to find a good answer to your question rather than simply a “quick and dirty” answer, you might use such a site as that.
I also expect that the “conversion rate” at such sites is far higher. A smart advertiser should be interested in co-branding with such “preferred” properties — but then again there is the issue of whether these sites remain “independent” or not. Certainly, if they are taken over by a big media company, then their reliability would be far more questionable.
Thousands of such sites exist, but few rank well on some of the major search engines (perhaps due to competition — much like Google’s VP of Engineering said at last year’s “Press Day” that Google did not want to see http://creditcards.com ranking highly for a search on “credit cards”).
Most knowledgable users recognize that Google results are filled with spam. I hardly use “one-size, fits-all” search engines at all any more for precisely this reason. In this light, Jeff Weiner’s remarks during “Reinventing the Audience” session at the Conversational Marketing summit (see http://federatedmedia.net/events/index ) rang quite true: Yahoo has more or less been “all about community” all along — whether that was the portal audience looking for expert guidance in the early years, or whether it’s the newer, more “collaborative” communities like flickr.com and/or del.icio.us: it’s very much in the “social” space (versus Google, which used to be algorithmic [before the “miserable failure” censorship took over]).
The other “one-size, fits-all” engines? I really cannot say — they still seem to be “developing” quite rapidly.
What about focused engines such as http://hotels.com , http://pizza.de or http://cellreception.com ? Do you suppose the the “search fulfilltment” rate is higher here? I would think so — but that’s just a guess (and much like the above example [ http://onelook.com ], I would expect that here the “experienced” user will be able to separate the wheat from the chaffe [so I would expect that REPUTABLE advertisers would steer clear of spamming techniques and instead cough up and pay for co-branding instead]).
John, by your logic Live search is the best! Thanks for giving a logic to us to justify ourselves as the best:)
I agree with John – you cannot conclude anything just from looking at these numbers. In order to say anything, one would have to have access to some of the queries and see which ones resulted in referral, which ones were reformulated, which ones look like queries that may have been answered directly by the engine, say through the result snippets. Of course, as an academic at a University I realize that I will never get access to such data. mb’s guess is an interesting one. It might also be that google users first try simpler queries and then reformulate, while yahoo users immediately type in the longer query. Lot’s of possibilities there.
I strongly disagree with nmw, however. Many advanced searchers use standard engines such as google. I use it and am not exactly naive about search. Whether you use a general engine or some other tool is not so much a matter of novice or advanced, but of what you are looking for. For my most common needs (politics, computer science, general trivia, and many navigational searches) a general engine works just fine.
Yes, TS: I do agree that most “general” engines (as you call them) usually work fine for “navigational” searches (basically: looking up a name like “TS” in the WHITE PAGES). I would be very careful about looking up topics on either politics or computer science with such a “general” (“one-size, fits-all”) search engine. As I have observed above: CENSORSHIP HAPPENS. And when I want to learn something (e.g. about computers and/or computer science) I will usually go chat in one of the geeky communities I belong to and thereby recieve far more useful (or whatever you want to call it) results than a “general” SERP which “runneth over” with spam.
Don’t know much about trivia — but I do know what a wonderful web this could be….
;D nmw
nmw – I think there are two questions here: (1) do advanced users primarily use general engines or other tools, and in particular do enough of them use other tools for it to make a difference in those stats that were presented, and (2) whether they should (which doesn’t directly relate to those charts).
For (1) I suspect that the answer is negative, and that the users that do use other tools are a fairly small group, probably overrepresented amount searchblog readers. In particular, your first statement seemed to say that advanced users don’t use the basic engines anyway and thus should be discounted as an explanation for the differences. I doubt this, and think many advanced users use general engines. Thus, there may be significant differences between the general engines in terms of user sophistication – in particular the ways in which MSFT acquires traffic to their engine may significantly impact the types of queries they get.
For (2), I don’t argue against using other tools. But I think even for a topics such as politics that are subject to a lot of manipulation, you can get good results via general engines. It all depends on how you interact with the engine and its results, and what questions you ask, and on your personal habits and preferences which are often not rational. (Do you strongly prefer a single interface or are you willing to go to different places for different things? Do you prefer to type or to look over more results? etc.) I can also see that for certain occupations, say in the context of media or law, other tools may be really preferable, but for many other cases it is not so clear.
Personally, I prefer simply dumping a single word or two into google, hitting return, and then going from there. Key to this working for me are the sub-second response times (think of it as hitting return quickly before typing the second word, instead of waiting until the end of the query) and the spelling correction (I type queries fast). By the time other have reached their specialized engine, I have my first result page. And since I know a little about how engines work internally, I can often guess what term combinations might work. But a lot of this is just habits, and few people study their habits to discover possible efficiency gains, unless it is about something they do a lot. So maybe I would do better by changing my habits.
I should add that by computer science I meant mostly the academic side of the field, which is not really spammed that much. For general technology issues (product reviews, programming etc) things are different of course.
Well, as you said in your first response, our answers to this issue are probably at best “educated guesses”. I should add that I am quite amazed that you feel typing in 1 or 2 words into Google might get you a result that was worth moving your fingers for (unless the strings are something like “ebay” or “yahoo com” — BTW: I just searched for “delicious” and the top 3 results had nothing to do with “food that tastes good”, they were entirely about the site del.icio.us; this further substantiates the idea that Google is primarily useful for people who still have difficulties navigating to sites they wish to visit).
My work (in “information retrieval”) is very interdiscipliary in nature. I talk with professionals and academics from many fields/disciplines about their “search behavior” (and/or “information gathering”) quite often. Few mention Google. I would say that “colleagues” are a primary source, but also professional journals and/or conferences or even just *random people on the street*. Especially people who have had the experience of “Garbage In, Garbage Out” WRT trying to find reasonable answers to questions by using “dirty data” would (I guess) be prone to be skeptical about Google’s rather simplistic algorithms.
I mentioned above that “censorship happens”. Well, of course it does! Most people do not want to see a 9 year-old kid walking out of the children’s library with a copy of Hustler magazine. But beyond such obvious examples, it is also quite reasonable for people to apply one measuring stick in one instance (like mb’s example of “finding a definition”) and another measuring stick in another instance “finding a telephone number”) — especially if they are focusing on one discipline of knowledge and hence using that discipline’s jargon and not the “GENERAL” language of the wider community. So when a computer scientist refers to “memory” then that would most likely have nothing to do with what a psychologist might refer to using the same term. Therefore, traditional “finding aids” (in the professional and/or academic arenas the primary example of such tools are often referred to as “indexing services”) usually focus on particular topics of interest to the “information searcher” and quite often also specify an “authorized” indexing vocabulary (with short definitions or “scope notes”).
Not only does Google do nothing of the sort, but spammers actually pretend to portray apples as oranges (and vice versa). The chance of getting a relevant result for non-navigational searches are quite dim (I guess that answering a “trivia” question such as “what is Benjamin Franklin’s birthday” is to some degree also a navigational question, insofar as it is primarily a matter of navigating to the entry for “Benjamin Franklin” — but even here I suspect that historians and/or genealogists would tend to use more specialized engines).
In the early days Google was useful because the Internet was filled with academics (and the method of using citations actually has a very long tradition in academia — though here too “citation analysis” produces only “ballpark” figures [at best]). The Internet is no longer as “academic” as it was when Google started out — and that is perhaps one primary reason why it no longer works as well as it did back then. And perhaps the reason it has remained popular in the meantime is that the teenagers who grew up with it had virtually nothing to compare it with. Now that’s changing — and I would say that enabling each person to choose their own search and/or ranking algorithm is a good thing.
And I feel that is another point where we do agree after all: You asked “Do you strongly prefer a single interface or are you willing to go to different places for different things?” I think children’s library exist for one reason, pornoshops exist for other reasons, and research libraries exist for yet other reasons. And yes, I do feel that that is a good thing. That the video cited in http://www.telegraph.co.uk/news/main.jhtml?xml=/news/2007/09/28/wburma528.xml was yesterday the second-most viewed video on youtube.com is a little disconcerting. I find that the video shows EXTREMELY disturbing images — vile and repugnant images that I do not wish tender young children to be exposed to (at least not without *some* parental guidance).
I’m with mb. Rather than saying “Yahoo fulfills the most” I’d say “Google already answers the query in its snippet results”. I also use Google a lot for spelling checks, telephone number searches which with the right query 9/10 times show in the snippets right away. Even when I need a definition the snippets it picks from say Wikipedia are often enough to get the gist.
There’s also the site: link: and other commands in Google which normally don’t lead to a click through. Yahoo has Site Explorer for this but I doubt they took that subdomain into equation for this research.
Image search is another area I rarely click through and I asusme here Google Images is bigger than Yahoo’s.
Fascinating! I do, however, feel that there are people who actually type “delicious” into Google rather than into the location bar — simply because they do not want to type in the dots.
But apart from that, we have:
1. telephone numbers
2. spelling check, definitions (hmm — does mb mean the definitions provided by http://answers.com ?)
3. trivial pursuit
4. stock quotes
So including “navigational search” (i.e. the person prefers to type “ebay com” into Google rather than typing “ebay.com” into the location bar), that’s 5 ways people have suggested that Google’s results are useful to them. Are there any OTHER ways in which people have found Google to be useful?
Very interesting discussion. I have to admit that many of my queries fall into the above 5 categories. I never type into the address bar because it is a pain – positioning the cursor, doing the dots, the http:, etc. And many of my computer science queries are either navigational, or very specific (not “information retrieval” but with much more specialized terms and often names), and I find google to work well in that case. Often I just use google as an entry point into citeseer, where I browse for literature, but I rarely search within citeseer. And quite a lot of my (non-CS) questions are answered by the result page.
While these 5 categories may not seem very challenging for an IR system, I would suggest that these 5, and other types of queries between these and “traditional informational queries”, are the drivers in terms of what makes an engine successful and in terms of how search impacts technology at large. You can’t really separate these from “real search” — if you try then for most people there will be very little left. And the boundary between navigational and informational is not always that clear – so do not disrespect the humble navigational query 🙂
I have great respect for the navigational query — the question is: Does Google have a similar respect for it? AFAIK, Google employs a large army of lawyers in legal battles concerning how Google “sells” results for trademarked strings to the highest bidder (often to competitors of the trademark owners). And AFAIK the “snippets” also may someone else’s intellectual property (i.e. protected by copyright). So for most of the 5 examples described above, Google is actually in “deep doodoo”.
The chance of getting a relevant result for non-navigational searches are quite dim
I tend to agree with this statement. Google is being consciously optimized for navigational searches, at the expense of other types of searches, such as explorational, informational, transactional, etc.
I suppose that’s ok, on the surface of things. It certainly is the “lowest common denominator” approach to information retrieval, something that will satisfy the majority of the crowd, the majority of the time. So it is completely understandable why they optimize in this direction.
The danger, in my mind, is when we take this type of information retrieval and equate it with all of information retrieval. When we stop looking for something better, something more, that is not good. When we fall into the habit of using Google a certain way, because we “kinda know how search engines work” and know that we can get millisecond response times, we’ve basically fallen into a local minimum. If we never even try to get out of that minimum, it might as well be game over. If customers are not demanding more of the search engine, the search engine is never going to bother making any real improvements beyond constant spam monitoring. The search engine will forever remain in navigational mode.
Is that what we want?
A new search tool called PredictAd is offering a service that offers individual websites and blogs similar features to what the big guys like Google, Yahoo and MSN have added to their search engines (autocomplete, rss live search).
Check out this article on PredictAd to get an idea of what it does:
http://www.ringofblogs.com/2007/10/01/predictad-navigation-monetization-and-search/
In a previous job, my team performed a great deal of analysis into the quality of search engine results, as measured by user response to a results page. In one project, we took four leading search engines and transformed their UIs into wireframes (stripped out all the formatting and branding elements). Then we asked users to perform identical searches on each of the four engines and evaluate the quality of results. In this brand-neutral test, users consistently reported that one or more engines outperformed Google. Yet, when we revealed the brands behind each wireframe, existing Google users maintained their preference for Google. Never underestimate the power of a brand.
My team also looked at search referrals as a measure of user satisfaction and, as with the Compete study, found that users more frequently found a “clickable” result on engines other than Google.
The findings of these research studies held up for each of the three types of searches: navigational, informational, and commercial. And let’s not flatter ourselves by thinking that Google users are more sophisticated or advanced, and thus need only to read the text string in the search results to find out what they need to know. There’s no evidence for this.
Put simply, research evidence consistently suggests that the leading search engines, incuding Yahoo, Ask, and some metasearch engines, produce user satisfaction levels that match or exceed those of Google users.
“we can get millisecond response times”
Yes: *time is money* (why does it always seem like Benjamin Franklin was 10 times smarter than anybody else? 😉
Are Mozilla and IE all there is — I mean: is that all there is?!? (no there’s another one isn’t there? what was that one called again? — oh, never mind…)
Has anyone ever used ctrl-L ? OK, that’s one millisecond.
“[Google] will forever remain in navigational mode”
I’m not even sure that Google will forever remain a search engine (right now it seems to be more of an ad agency). What’s really surprising is that there is this MYTH that Google is a search engine (to be reckoned with 😉 — and people continue to believe it! Typing “ebay” into Google is actually a WASTE of time — so if “time is money”, then it’s *time to move on*….
Another amazing thing is that it seems like there is a yearning for some HERO and/or ROLE MODEL. If I go to http://www.christianity.com then I can EITHER search the site or the Bible (that’s kinda neat-o! 🙂 And if I go to the blog section of the site, I can do a keyword search (I tried “help”, then I tried “support”: first result was “how to handle a money crisis” — not so bad! 😉 I am not sure that all of my problems need to be solved from a particularly Christian.COM perspective — what other measuring sticks are there? (again: is that all there is?!?)
How about http://help.com? I can search for tags: first “money”, then the site offers further tags to hone in on — I click on “loans”… — this is also neat-o! ;D I type in “save money” (minus quotes) and then I see something new: I can limit my search to “posts” or “replies” or “both” (COOL!! 😉 — I can add user names, … — is this a search engine? Is this advanced? Is this explorational? informative? …?
When I type “help” into Google, help.com shows up as the first result. Is that good or bad? What is the most efficient / effective method an advanced searcher should use?
One way to solve this mystery might be to by using the process of elimination — which methods are NOT efficient / effective?
Here’s a start:
http://groups.google.com/group/blogger-help/search?q=find+blog
interested in navigational search?
click on find: http://find.net
;D nmw
Hello, my name is David Kim and I’m a Marketing and Sales Director for an Import/Wholesale company in Dallas. To complete my research I use google and yahoo everyday. Yes, I use both. First I would like to say that having a lot of information is good but I rather have few best informations to save me time and so many other factors. I always used yahoo until two months ago when I just kept on hearing about google this and google that. So I tried using Google and stuck with it for two months until I couldn’t find all the results I wanted. So I tried typing the same thing in yahoo.com and realized not only there is more accurate sources but gives me more subcategories of the topic I chose to research. However, google seems to have a bigger system but for some reason I get this strange feeling that google concentrates more on the ads that pop out on the right side then the results. I also feel more satisified getting results from yahoo either for business research, business listing, etc. You shouldn’t rely on the internet for million dollar business plans but I have used yahoo 80% of the time for business related things compared to google. I don’t understand what’s all this big fuss about google. I was just researching something on google and I couldn’t find it, I tried all possible ways mixing up the words. I tried the same thing on yahoo and got the answer I wanted in one try. I was so frustrated with google (months of piling up) I typed “yahoo is better then google” and landed here. Use yahoo my friends.
I have to strongly disagree with comments above that suggest that Google does not provide the information sought directly on the results page. I am a physician who uses Google every day to augment my medical knowledge. Google replaces all the cheatsheets, pocket pharmaceutical books and little formula compendiums that used to make our lab-coats ungodly heavy when we were residents in the pre-web days. I get what I am looking for in a second without going to a secondary referral page. Try it yourself. Search for something like “pediatric dose lasix”. It’s right there on the top line. … and no I don’t need to use Yahoo to get that kind of result.
John’s comment about focused search engines being more useful than Google is somewhat off the mark. I use a number of good medical search engines, but in the office, speed is paramount. Its much quicker to use Google as the home page, then include the name of the particular specialty search engine in the query. Google pulls the result out of the other search engine, and I have just waited for one page load instead of two. This method also shoves the “google-spam” results off the top of the page. If I have the need and the time to dig deeper, then I will go directly to a specialty search engine, but 90% of the time its not necessary.
I personally feel that Google delivers better results for any search query. However, it would be intersting to see how the scene changes when Microsoft takes over Yahoo.
The worst thing of all is that Google (and the other two are guilty also) has been tweaking results outside their advertising section to provide an unfair advantage to the Google services. Do a search for a stock quote or for a song or for a product and you will see what I mean… The Google results are at the top of the natural results area, an area that users expect to be “unbiased”!
So, based on what Google’s VP of Engineering said at 2006’s “Press Day”, Google is more than welcome to compete with websites like http://www.cardhub.com when someone searches for “credit cards” but the regulators need to step-in and not allow Google to be able to compete by biasing their natural results. Users expect objectivity in that area and consumers should demand it given that we will soon have 2 search engines controlled 90% of all searches!