Jeremy posts a few insights and such on Yahoo’s forthcoming new search…
Dave Sifry, who is also at Demo, has posted a Demo conference conversation page on Technorati…looks like there’s some healthy competition going between Feedster and Technorati! This page is a combination of keyword search (which Technorati recently added), and link cosmos around Demo-related URLs…
(thanks for the pointer Dave)
This Times piece explores the Yellow Pages market of small businesses (plumbers, etc) who don’t have websites but increasingly feel the pressure to have an online advertising presence. Why? Because the customers are increasingly using search instead of the Yellow Pages (the Times based that conclusion on research from the Kelsey Group and Bizrate.) The article provides an overview of the current state of Overture and Google’s local search solutions.
Feedster is demo’ing at the Demo conference today, and Scott sent along news of their latest feature: Feedpapers. They’ve got proof of concept here: Political Feedpapers for each candidate. In essence a Feedpaper is a search of the blogophere for posts on a particular topic, which are then organized and presented for you. It’s well done.
Feedster has a page with more on Feedpapers:
When you think about what Feedster really does, it lets you listen to the web. A Feedpaper is a sophisticated daily newspaper, a “Feedpaper”, automatically constructed from what people are saying about a subject. We offer a handful of general interest Feedpapers and you or your organization can sign up for a personal Feedpaper (not yet available; coming soon).
I’m looking forward to creating Feedpapers on topics like search, TiVo, etc.
Another example of how Feedster feeds the political sphere is on this Kerry site, which uses Feedster to search the blogosphere for Kerry news.
UPDATE: I should have noted the public “builder” they’ve got as well, it’s a tool for rolling your own feed or Feedpaper…
The Washington Post prints something of a Sunday rumination on how search engines might evolve, a rather flippant piece of magazine writing that reads like a poorly edited Wired rant from the early days (I should know). Overall the piece bothers me – it takes search seriously in word, but the tone finds a way to be dismissive at the same time, and only gives a cursory answer to the question it sets up (what might search look like in the future). The set up illustrates what I mean:
Only now in the bright light of the Google Era do we see how dim and gloomy was our pregooglian world. In the distant future, historians will have a common term for the period prior to the appearance of Google: the Dark Ages.
Well, in fact, I’ll warrant that when historians look back at this era, they’ll disagree. But enough about that. The piece does provide an interesting signpost of popular culture: our most respected institutions of journalism are trying to make sense of this phenomenon as more than just a business story. Thank God.
Well I was not expecting this one: I head into a conversation with the CEO of TiVo, fully expecting him to look at me cross-eyed when I suggest that TiVo might be understood to be a search-driven application. Instead, he wholeheartedly agrees. In fact, Ramsay was adamant about the role search plays, and how much innovation can come from understanding video through the lens of the internet.
As usual I must save stuff for my book (and column), but here’s one of the coolest things he mentioned: the idea of folks building video content websites that TiVo could search and download – using exactly the same search interface TiVo already has. Ramsay pointed out that with television search, you often don’t know what you want till you see it (“I feel like a foreign film tonight – hey…there’s an old Seinfeld episode on!”), but on the net, you often you know what you want, but not where it is (“I’m looking for a 1965 Ford Mustang in perfect condition”). What would happen if the two merged?
Ramsay gave an example of a typical TiVo wishlist (an ongoing set of instructions you give your TiVo) that includes, say, anything from Martin Scorcese. As it stands today, TiVo offers up all the films he’s directed which happen to be showing in the next few days, and possibly documentaries featuring the director. But that’s it. What if there were a great Scorcese interview available on a fan website, and that website was built to work with a hypothetical TiVo search API? TiVo would present that piece of content as an option as well, and you could simply “record” it, just like you do any other show. TiVo would then send a request over the net to the site, and download the content for viewing (TiVo might employ “drizzle” like technology to download the massive files overnight). Imagine the possibilities, both for non-commercial content as well as for advertising. It makes my head spin. I am sure the cable companies (and networks) simply loathe the idea – it takes distribution completely out of their hands. One can imagine any number of scenarios: a “hit” show that lives solely on a website, for one. Advertising-only websites providing paid search-like content for which TiVo becomes a market maker. Crippled Comcast PVRs which refuse to cross-pollinate with the net (I am sure Mike would love this, as it would provide a major differentiator for TiVo, were they to implement this kind of feature).
What this heralds is the long-promised melding of broadband and broadcast. I asked Ramsay about TiVo’s recent acquisition of Strangeberry, and he was not very forthcoming, but he did say “they are doing some cool stuff in this space.”
Our conversation ranged across many subjects, from business models to collaborative filtering to his relationship with Hollywood and the IP cabal (“improving”). I’ll be writing up the whole conversation for my column in Business 2.0, so stay tuned for that.
A good overview of various approaches to search, with sections on Mooter, Dipsie, and Microsoft, as well as an interview with Google employee #1, Craig Silverstein. A minor but interesting piece of infoporn: Craig tells the reporter that Google engineers spend 10 percent of their time on outside projects. That’s down from 20 percent. Or maybe it was a mistake. Thanks to Gary for the pointer….
The last entry on Yahoo’s new search got me thinking about search results, and in particular Google’s, which nearly everyone imitates in one form or another. We all know about the endless list of results, 10 to a page, stretching past what Tim Bray calls “the Google event horizon.” I used to think that horizon was 100 or so entries – no one will ever look further than that. But the truth is, it’s usually one page of listings, if not less.
I’ve gotten to thinking – what’s the use of having all those results? I mean, really, from a user interface point of view, the only information we gain from “Results 1 – 10 of about 3,950,000″ is the rather attenuated sense that the search engine is, in fact, pretty darn thorough. That used to be a big deal, back when engines were really crappy. But these days we expect engines to be thorough. What’s the point of giving me a list of more than 3 million results when I am never, ever, ever going to go through them?
Seems to me it’s time to change the interface. Clearly many others have thought about this, from Grokker to Mooter to Vivisimo and beyond. But it’s the big guys, Google and Yahoo, that make the standards, and I think we’re getting close to the point where a new user interface paradigm is needed for search. Danny talks about invisible tabs, and that’s a good idea. But I’m not talking about intuiting what the user wants – that’s the hard stuff, and I know there are plenty of PhDs working on that. I’m talking about something much less difficult – changing the way results we get are presented.
Here’s what I’d like to see, as a small step in a new direction: A button that I can hit when the results come up which reshuffles the search in an intelligent way. In a fit of originality, I’ll call it the “reshuffle” button. Show me the first ten pages, and only those first ten. Just as I do now, I’ll scan them. If there’s nothing there, I’ll hit “Reshuffle”, and the engine shows me another 10 results, only this time, it eliminates pages that are similar to the ones it showed me before. This way, you can quickly and intuitively sift through all those results, grokking and pruning your search as you go. This is not some massively new visual approach, it’s just a quick hack that allows me to drill down. It’s this kind of stuff, I think, the simple stuff, which ends up being the most elegant and useful. I know there’s much work to be done, there are plenty of NP-hard problems to solve in search (I know because I’m trying to grok them and write them up in plain english for my book). But solving those problems will take years.
In my discussions with folks at Google and elsewhere, I often hear a resistance to changing search approaches due to technical reasons – clustering, for example, is not used at Google because the results are not considered relevant enough. But what about the user interface for results? The most frustrating thing in the world is seeing “Results 1 – 10 of about 3,950,000″ and knowing that somewhere in that haystack is your needle. But why sift blindly through the event horizon? Maybe some UI innovation on top of the current results can help.
(While I’m ranting, I’d like the engine to suggest better query terms for me. It can’t be that hard to store user queries and cluster those which have similar constructions, query words, or results/paths taken. I’d like to hit a button that says “show me similar searches.” I think this exists somewhere, but I can’t remember where (yeah, I know about Direct Hit, that’s not what I mean exactly). It’s not quite collaborative filtering, but it points that way.)
Any readers out there know about tools or research that does some of what I’m on about, or a response as to why UI innovation is a bad/too difficult idea?
Met with Jeff yesterday, and we didn’t have nearly enough time, so we’re going to meet again next week. But the time we did get was quite interesting. This against the backdrop, of course, of Yahoo’s stated intent to shift from Google results to its own native technology. Jeff was coy on when the switch would occur, but extremely enthusiastic about the end result (Yahoo CEO Semel has announced the switch will occur by the end of the first quarter, and that’s not too far away).
I think Yahoo search will be new from the ground up. It’s not just Inktomi in place of Google, it will be an entirely new product. Jeff wouldn’t give me details on what to expect, but he is a man clearly sitting on his hands – he’s proud of the work his team has done. “People don’t realize how scarce search engineering talent is,” he told me. “And we’ve got critical mass.”
On more general topics, we had a robust discussion around the issue of paid inclusion. This issue is almost always painted in black and white – Paid Inclusion Bad, “Pure” Search Good. But Weiner defended the practice against the metric of user value – when sites pay to insure their content is indexed, they also insure it will be available as potentially relevant results to the user. If Yahoo fails to give the user relevant, quality results, and instead spams the user with commercial fare, Yahoo will lose that user. In other words, it’s not in Yahoo’s best interest to value the advertiser over the user’s needs. In fact, it’s in the advertiser’s interest for Yahoo to value the user over the advertiser. This, of course, is Publishing 101.
What Weiner seemed to be saying was: don’t judge us by the past practices of Inktomi, and certainly not by how MSN implements Inktomi. We’re going to do it in a way that delivers value to the searcher.
I get the sense (and it’s just that, a guess, as I have no direct facts yet on what the next rev of Yahoo Search will look like) that in many cases, Yahoo search won’t initially show a list of algorithmic results to the user, but will instead show what the site *thinks* the user is trying to get at, and go from there. This raises a larger question about basic approaches to search results. Yahoo seems far more comfortable making explicit editorial decisions in its approach – intuiting the intent behind a particular search, and delivering results from any number of sources – its commercial deals, its directory, its algorithmic web results, etc. Google, on the other hand, continues, for the most part, to maintain a “purist” approach to search, claiming that its secret algorithmic sauce will deliver the most relevant results regardless of editorial judgment.
I asked Weiner about this idea of “purity” – a concept which is a clear differentiator for Google. Staying on message, Weiner said that the issue is not “purity,” but rather “value.” As in, which site will give the user the most value for the search query. He clearly believes Yahoo can play on that metric. By the end of March or even sooner, we’ll all have a chance to find out.