Tim posts on fellow editor Sarah Milstein’s observations. Sarah wrote (and updates) Google: The Missing Manual. From her thoughts:
As the Web gets bigger, search results contain more irrelevant stuff. In many cases, it’s getting harder to find what you want. Appreciably harder.
What do you all think? Yes or no?
and
Other search companies are doing some cool stuff with their main results. For many searches, Microsoft Live presents a super-useful list of related searches; also, somebody hit them with the clean-interface stick. Ask.com does a nice job with simple natural-language questions. Clusty has been offering very handy clustered search results since at least 2004. Daylife (still in alpha) does what is essentially clustering with a thoughtful interface.
My thoughts: It’s about a year away, but in one year, there will be precious little difference between results on Google, MSFT, and Yahoo. But will it matter? Yes or no…
and
I wonder when/if search is going to be real-time (i.e. live Web) rather than index-based. And I wonder if the main barrier to it now is hardware or software (to the degree you can separate them). At Web2, I met a woman from Intel R&D who’s working on a continuous refresh data system that would allow real-time searching but for which you need multi-core processors that aren’t yet ready for primetime. Still, an interesting glimpse of the possible future.
This is exciting, but what I want is even more is a web time axis….
and
During the two-year period from one edition of Google TMM to the next, Google began adding features and services at such an increasingly quick clip that while the second edition was at the printer, we missed out on eight new tools. That was in a 10-day period. (Some of those new tools are search-related; many are not.)
Tell me about it. I missed about 100 from the hardback to the paperback, and now I’m hopelessly behind. No, wait…that’s why we have Searchblog….
One reason that it may be harder to get relevant results is the reactive strategies search engines MUST use to filter out spam in their SERPs – thus increasing the liklihood of False positives, exemplified by the Google’s famous ’40 – 40 – 20′ policy.
The idea of assigning a high value to Trusted Sites that link to other sites can only go so far – and of course is bias towards the large, well connected sites – which are more likely to get noticed.
There is now a Zero-Tolerance trend by GYM to cleanse their Databases – but a previous approach – where there WAS minor spam – but more OVERALL relevant SERPs, appeared to produce OVERALL more relevant SERPs.
Well the Blog search engines by Google and Technorati are very close to live search. They are good to find the latest posts in blogs but they depend on a mechanism of pinging and need user co-operation too. Another challenge is the spam – search engines will have to have stricter algorithms to filter out spam websites.
getting easier and easier all the time. not only are search capabilities getting better and better, but more and more of the information i want is coming online – from wikipedia to universities putting their courses online, to mom and pop shops setting up online, to google enabling small players/orgs/groups to get online very easily in a big way, to amazon doing more and better stuff with their book-related previewing, to the improving quality of many blogs, to the variety of political points of view i’m able to access, to the amount and quality of video i’m able to access, etc.
if anything, i might only be frustrated these days by not getting exactly what i want, down to the most minute detail. two years ago i’d have been happy to have ‘X’ number of semi-relevant results, whereas now i’m not completely happy if the very first search result isn’t the best one that i could possibly find, and if it doesn’t contain my answer in the first 3 words of text i scan.
riya just added image search. google added blog search. more and more government documents are online now than ever before – helped by google’s gubment search.
can’t find what you want? i’d suggest a ‘google for dummies‘.
“In many cases, it’s getting harder to find what you want.”
Found it! http://what-you-want.com/
😀 nmw
My thoughts: It’s about a year away, but in one year, there will be precious little difference between results on Google, MSFT, and Yahoo. But will it matter? Yes or no…
A year? I just read a peer-reviewed study last week in which users were show results for their exact same query, side by side, from both Google and Yahoo. Identifying information (branding, page stylistics, etc) were stripped away, so that only the links and the summaries were shown. The study was double blind, and randomized (sometimes Yahoo would be in the left column and Google on the right, and sometimes vice versa.) Then users were asked which ranked list they prefered.
The conclusion? There was no statistically significant difference between user preferences of G and Y. Once you stripped the branding away, users did not clearly see one engine as “more relevant” than the other.
It appears we might be approaching singularity already.
So if the ranked lists are no better or worse, what distinguishes one search engine from another? Well, it would have to be the branding, and the “query assist” wrappers, as you say: “Other search companies are doing some cool stuff with their main results. [i.e.: clustering, related search suggestion, query narrowing, query broadening, etc.]”
But therein lies the problem, a supreme paradox. I think these other search engines can offer all these cool tools, because their corporate revenue stream isn’t solely reliant on “AdWord”-like advertising. They can afford to sacrifice advertising page real estate, because they have other sources of income. This is a problem for Google. In order to offer some of these cool clustering/etc. tools, it has to either sacrifice real estate, sacrifice the number of ads it shows, or else clutter up the page. And we all know how sacrosant clutterlessness is to Google. But if Google sacrifices ad space for helpful tools, that will lead to a (significant?) drop in revenue.
So my point here is that, ultimately, I have to disagree that in one year there will be precious little difference between MSN, Google, Yahoo, etc. And the reason for this has to do with El Goog’s reliance on advertising. As the other search engines continue to innovate with result clustering and other forms of result modification, Google is locked in, economically, to their current approach. Unless Google can change this, in a year I think you will see even more differences between those other search engines who have been able to innovate, and Google, which will still be stuck.
Yes or no?
JG: In order to offer some of these cool clustering/etc. tools, it has to either sacrifice real estate, sacrifice the number of ads it shows, or else clutter up the page.
I doubt these are the reasons that you don’t see, e.g., clustering on Google. Rather, I’d suspect they’ve user-tested clustering (as we did at Excite in 1997) and found that most folks don’t find clustering useful. Melissa Mayer’s Web 2.0 presentation is relevant. She noted that, e.g., folks might think they’d like more results per page, but, if you watch them, they actually find what they want more often when fewer results are displayed per page. So folks might say they like clustering, related search suggestions, etc., but in fact these features might actually hinder their ability to find what they’re after.
Doug – are you and users and Marissa suggesting that “useful” = “used”? People use simple approaches even while they’d be happier if they learned and used advanced queries (which they won’t do unless navigation can be simplified). It’s one of many human, not technological, challenges and IMHO our human inadequacies will define search for many years.
I personally find “related searches” feature very useful. Often I know how to modify my query but having a readymade link and clicking it make my search-session super-fast.
At some point in past, Google was my default search engine. I do not think I can go back to Google until this feature is implemented there. For now, Live search would remain my default, even if I leave my super-cool Microsoft job.
Doug: I need to echo Joe Hunkins’ comments about “useful” != “used”. And I will go back to Marissa Mayer to do so. A while ago, she was making the rounds, talking about various things Google had learned over time. One of those things was spelling correction. Google long had spelling correct, but users were not using it. Google was puzzled, because they found that a large number of users did misspell their queries (“Britnay”, anyone?) But they never clicked the “Did you mean?” spelling correction link at the top of the page.
Then Google had that “ah hah” moment, when they realized users would unconsciously skip the spelling suggestion and just look down to the SERPS. After scrolling through all 10 SERPS, they realized their query had failed, and would type it again. Once Google understood this, they simply put another, duplicate, link at the bottom of the page, with the “did you mean” spelling correction. Once they did that, usage of the tool jumped by a few orders of magnitude.
So Google learned from this that they had to do two things: (1) Dedicate more screen real estate to the tool, and (2) Dedicate the correct screen real estate to the tool. They couldn’t just put it anywhere.
And what I learn from this is that this spelling correction query modification tools was ALWAYS useful. It just wasn’t until Google actually made it usable that users actually used it.
So why should query expansion, query narrowing, query clustering, etc. be any different? We know from the academic literature that these things are useful, just like spelling correction is useful. But to make them actually used, Google has to dedicate the right screen real estate for it. But that means showing fewer ads!
JG: We know from the academic literature that these things are useful.
Really? Can you provide citations? Where average users more frequently find what they seek before giving up?
Something has to be usable and used to be useful. Those are prerequisites. To use your example, moving spelling correction to the bottom of the page made it useful. Marissa was observing that faster result pages are more useful, since folks can more quickly iterate, repairing their query. The top few hits give folks a pretty good indication of whether they’re on target, and I assume she meant that seeing those faster (i.e., trying more queries before giving up) is more useful than scanning longer lists of hits.
I’m not an unqualified Google fan. But I do think that they’re trying hard to create more useful search interfaces. And I suspect that if they found that, e.g., displaying clusters made results more useful at the expense of some revenue, that they’d probably go with the more useful results. They’re after long-term success and believe that happier users are a better route to that than short-term revenue gains. That’s what they say over and over, and I’ve not seen many actions demonstrating the contrary.
Doug: I just tried listing a few citations. Unfortunately, Battelle’s blog chokes when there are multiple hyperlinks.. it thinks the comment is spam. So I am removing the hyperlinks from my post. You’ll have to do a search and find these papers yourself; apologies. FWIW, they are all available on Citeseer.
The first paper is “Improving Interactive Retrieval by Combining Ranked Lists and Clustering” by Leuski and Allan. See section 3.4. Note that for very short (2-3 word, typical of web search) queries, the authors observed a > 30% increase in precision using clustering methods.
Next is “Web Search Intent Induction via Search Results Partitioning” by Daume and Brill. These authors perform a very interesting experiment in which users are show the top 5 ranked documents, as normal, but then a set of clustered results, after that. This is neat, because if the user’s query succeeds, there is no need to use the clusters. But if the user’s query fails, the clusters are there, ready to be used. And they find that the clusters actually are used. That is a perfect example of putting clustering in the right place using the right amount of real estate. Evaluating their work using real web data and real user usage, the authors find that the total cost of the clustering system usage is much less than of the ranked list alone. In other words, clustering it both used and useful; it helps the users find what they need, quicker.
Finally, the last paper is “Improving Automatic Query Expansion” by Mitra, Singhal, and Buckley. The focus of this study is coming up with automated query expansion techniques that are robust to the query drift problem. So while the authors do not focus on manual methods, I find it very interesting that they use as their comparison/baseline a manual (user feedback-driven) method. Their claim is that their automated technique is good, because it yields a 6 to 13% improvement, which is almost as good as the manual method (7 to 22% improvement). While automation is good, that also says to me that manual mechanisms definitely have their place.
So maybe you are correct, when you say that you “do think that [Google is] trying hard to create more useful search interfaces.” Perhaps I (as usual) am stating my case a bit too harshly. I really have next to zero insider knowledge of what Google is really doing.
But I look at the insane amount of products they are releasing all the time, all the chat and spreadsheet and widgets and rss readers, etc. And I think to myself: hmm.. all these products are coming out of 20% of their efforts (i.e. the 70-20-10 rule). And supposedly there is that 70% dedicated to search. That means we should be seeing 3.5 times the innovation on the search end, as we are seeing on the chat & spreadsheet end, right? So why are there not even optional clustering or query expansion tools that I can turn on inside of Google search? In other words, don’t release them to the mass market just yet, but let me at least have a personalized version of Google, where I myself can play with some of these tools, and determine if I like them or not. So I am just astounded by the fact that, with 3.5 times the amount of effort going in to search, there isn’t an option like this, at all. For those of us who want it. For the long tail of users who aren’t satisfied with fully automated methods.
Do you know what I mean?
By the way, a couple of other minor points: In regards to the Marissa notion about speed trumping number of results, there was a problem with that experiment. Users didn’t actually say “we want more results, in more time”. They said “we want more results, in the same amount of time”. Since Google didn’t actually give them more results in the same amount of time, you can’t really conclude anything about the number of results that users actually find “useful”.
But how can you give the users more results in the same amount of time? By loading docs 1-10 in the normal quick time, and then using Ajax to dynamically append results 11-30 onto the original list. By the time the user has scanned the first few items in the 1-10 list, results 11-30 will have appeared. It will seamlessly appear to the user as if all 30 results had appeared at once.
Google needs to rerun this experiment, and actually test what the users asked for, instead of throwing up their hands and saying “sorry, it takes longer — therefore you must really not want more results”.
And a final minor point: I have to disagree that “something has to be usable and used to be useful. Those are prerequisites.” Eating right and exercising regularly are very useful activities for prolonging one’s life. But most (American?) users do not find these activities very useable, nor used. Most of us are too lazy. But does that make exercise and a good diet any less useful? No.
I do not mean to start a flame war; it’s hard to tell from text, but the tone of my reactions here are actually quite neutral. These are just questions that I have, that I can’t seem to get answers from anywhere else. So it’s great when a discussion arises on-list.
And by the way, thanks for Lucene! 🙂
I doubt these are the reasons that you don’t see, e.g., clustering on Google. Rather, I’d suspect they’ve user-tested clustering (as we did at Excite in 1997) and found that most folks don’t find clustering useful.
Well, saying that we tested clustering in 1997 or that other folks did more recently and users didn’t find it useful, is a bit like saying we tried a GUI and users didn’t like it so GUIs aren’t worthwhile.
Quality matters. Usefulness depends on it.