This is a fascinating post from Google on how we track data on search results.
I think we as an industry (social media, new forms of digital media) need to do a better job of proving the engagement we all know exists online. It’s presumed in television (and there’s tons of research to back it up). We need to do more, and a better job, of showing the impressive engagement of online.
The darker the pattern, the more time they spent looking at that part of the page. This pattern suggests that the order in which Google returned the results was successful; most users found what they were looking for among the first two results and they never needed to go further down the page.
Hmm. Let’s see here…
They give us an example of what a successful query looks like on an eye-track heatmap.
They give us no examples of what an unsuccessful query looks like on an eye-track heat map.
Furthermore, they offer no explanations or examples about how to distinguish between an unsuccessful query, in which the users have looked further down in the list because they didn’t find their answer in the top 2 results, and a successful query, but one in which the user needed to find 15 links to satisfy his or her information need rather than just 1 link. (For example, when the user is looking for product reviews. In that case, completeness — as many reviews as possible — is more valuable than just one top review.)
Seems to me that the heatmap would be fairly evenly distributed for both a failed query and successful, but “completeness-oriented”, query.
How does Google tell the difference? As usual, this blogpost obfuscates more than it enlightens. I give them a B+ for trying, but, sometimes I wonder..
Just a quick note… You’re probably already on to it but here’s a link to Google’s brand new Social Web Blog!
http://googlesocialweb.blogspot.com/
Without audio, the usability of video is greatly reduced.
;D nmw
John, could you delete my multiple posting instances, and leave just one? There was something funky happening with your server over the past 1/2 day.
Cheers.
JG – I deleted your duplicate comments. Searchblog was moved to a new hosting environment, which introduced some commenting issues.
@JG
An unsuccessful SERP (according to useit.com, who has done far more than this ridiculous video) is when there’s alot of clicks (and eye movement) to the next page (and apparently that happens a lot).
But none of this is actually meaningful in any way — the only point of the post seems to have been yet another PR stunt to remind you of the Google brand name (in case you were beginning to forget about it).
What would be better would be to do “exit interviews” (along the lines of: “are you satisfied?” — and if people are happy, maybe it wouldn’t even matter whether the answer was wrong or right! LOL 😉
So it’s considered unsuccessful when there are a lot of clicks, and viewing of multiple pages? Really? Unfrickinbelieveable.
It’s as bad as I thought. This is the metric that Google is using?
That means they are punishing folks like me, when I have more complex information needs.. because when they observe my behavior they consider it a “failed” search, and then try to make an adjustment to their algorithm so as not to give me those same results again. When in fact, it could very well have been the case that I found exactly what I was looking for.. all 15 links!
How brain dense is that?
And I know I’m not the only one. I was reading Robert Scoble recently, and noticed that he was having a lot of the same troubles with Google that I’ve been talking about for years on this blog. Let me quote Robert. First he writes:
http://scobleizer.com/2009/02/09/is-the-real-time-web-a-threat-to-google-search/
if I need to know who the best retailer is to buy, say, a Canon 5D Mark II, is it better to ask the people I know, like I did here on friendfeed, or go to Google and deal with the SEOs? Try doing that search over on Google. I did. Do you find a single retailer? I didn’t.
Basically, Scoble’s information need involves finding not just one retailer.. but finding multiple retailers, so that he can evaluate all of them, and pick the best one.
Google, by optimizing their engine to push Scoble toward the 1 or 2 top links and not give him a way of sorting through the rest of the links if those are bad, fails by design to meet Robert’s information need.
Next, Scoble writes:
How about we search for all Tweets that talk about the Australian Fires?
Scoble is not trying to navigate to a home page, he is trying to find as much information, from as many different sources as possible. Again, Google does not support this sort of information seeking behavior at all; moreover, as nmw points out, their eye-tracking studies show that they consider it a failure when someone engages in this type of search behavior.
Scoble continues: Can Google search show you all the Upcoming.org events that mention SXSW? No, but friendfeed search can.
Just like Scoble’s previous information need, the emphasis here is on the word “all”. Google does not optimize their engine in that direction. It’s very frustrating.
Two more comments from Scoble:
Can you easily see all the YouTube videos that have the word Grammy in them? Probably over on YouTube you could do that. But can you now constrain the videos to the ones that have gotten some comments? With friendfeed you can.
and
But try doing THIS with Google: try finding everytime Dave Winer has commented on an item about netbooks. On friendfeed that’s easy. On Google? They don’t have the metadata to study.
I would argue with Scoble that it’s not just about Google having the metadata to study or not. Instead, the emphasis here is on Scoble’s need to find “everytime” that Dave Winer has commented on a netbook item, and “all” grammy videos with comments.
It is very interesting to me that every single one of these information needs that Scoble mentions really isn’t about the “real time web”, so much as it is about a type of information need in which Scoble needs a set of information to fill his request, rather than the single top ranked link.
If Google only measures success in terms of top ranked links, and considers searches in which I’ve clicked on 15+ results to be failures, then its not just a matter of having a better algorithm or better metadata. The engine itself is defective by design.
Sometimes I really wonder if Google is listening or not. I hope they are, because I want them to be better; I want to be able to use them for my queries again.
But it’s been ten years now, and they still don’t support my information seeking behaviors. I see no evidence of any innovation in this direction, either. Furthermore, it appears, from Robert Scoble, that there is a growing need and awareness for more of these types of searches. I hope Google can do something better, soon.
Anyone else notice that they were no ads on either of the SERPs? I think that was not a coincidence.
John, could you delete my multiple posting instances, and leave just one? There was something funky happening with your server over the past 1/2 day.
Eye tracking is fascinating stuff. The case studies at http://thinkeyetracking.com/?page_id=72 are also worth reading.