No one study should be seen as definitive, but it’s interesting nevertheless to see this study from French Professor Jean Veronis and his students, which shows a tie between Google and Yahoo in terms of relevance. What’s even more interesting: the study found that *all* the engines failed, for the most part. From the post:
Google and Yahoo tied for first place, with a rating of 2.3, but the most striking result is undoubtedly the extremely low level of user satisfaction. None of the search engines even passed (2.5 out of 5) and some of the grades were extremely low…
This is not too surprising to me. When you watch users use the Web, you know how often they fail and how often they get search results that are not as helpful as they should be.
In fairness to the search engines, what *is* surprising is that they can find any kind of reasonable documents from the billions on the Web, given only 2-3 words of hints as to the user’s current problem.
But there is certainly much room for improvement.
On a related matter: users don’t necessarily click the first hit, though many do. We know from eyetracking that users often scan the first few hits before deciding where to click. Thus, a more realistic assessment of search quality is the maximum relevance of the best of the first N hits, given users’ estimates of information scent from scaning the summaries of these hits.
Sadly, collecting this data would require an even more elaborate study than the one conducted by Jean Véronis. But he could still re-analyze his data under the (unrealistic) assumption that the proximal cues are sufficiently good indicators of distal relevancy that users would always click the best link. In other words, analyze MAX(relevance) rather than MEAN(relevance).
Jakob> This is an excellent suggestion. I just recomputed the results along the lines you suggest, i.e. for each result page, select the maximum score, and then take the mean for a given search engine. The results greatly improve, and Yahoo takes the lead:
Yahoo 4.5
Google 4.3
MSN 4.2
Exalead 4.1
Voila 3.6
Dir 3.4
Here is the percentage of pages that contain at least one result with the max possible grade (5 on the 0-5 scale):
Yahoo 67%
Google 61%
MSN 54%
Exalead 51%
Voila 34%
Dir 34%
Still room for improvement.
Thanks for the suggestion!
Jean:
We have never head about Exalead which is at position 4. Very interesting study 😉
I’ve done additionnal computations following Jakob’s suggestions. Results and charts here:
http://aixtal.blogspot.com/2006/03/search-yahoo-google.html
Thanks!
The recalculated figures appear to align with the often-quoted market research findings that ca. 80% of users report they’re “satisfied” with what they receive as search results. Of course, some of those same investigations discovered that 40% of users couldn’t tell the ads from the search results, too. 🙂
I believe the lack of satisfaction stemies from over-diversification without integration over services. My roommates don’t know about Google Scholar, Google Book Search, or the myriad of other Google services. I show them how to elicit certain information that they didn’t know they could even search for and they are mystified.
The tools are still too complex for the average joe.
Wow, this is great: I suggest a new analysis before going to bed, and when I wake up the next morning, it’s already done. Time zones working in our favor 🙂 Thank you Jean. Very interesting new numbers.
I think the fact that the study excludes commercial results inhibits it’s usefulness. Depending on who you believe commercial searches constitute 35-60% of total searches and very often the sponsored links are the most relevent, so I think the study would be more useful if those were included. In addition, I think both Yahoo and Google would argue that their user interfaces allow searchers to find what they are looking for quicker, which is the ultimate goal.
will we be looking back in 10 years and laughing at how bad the search engines were? (like we already do now)
I am proud of the frenchy, this results interested me a lot when I stumbled accross them a few weeks ago.
The point of commercial results is interesting and question by the Yahoo change on its websearch since you can tell what’s the commercial part you’d like in your results. A cursor that makes you give on key data : Commercial / non commercial ! It’s a key for search especially for soft when they’re free version outhere !
PS : exalead is a french search engine that works well / has a nice filetype search options and give results that other folks don’t come with.
Sorry for any english mistakes … i’m french too.
Tied… to me google and yahoo are loosing… spending our time with a trashbox on the search results.
Tied… to me google and yahoo are loosing… spending our time with a trashbox on the search results.