Rich Skrenta, he of Topix and Web OS posting fame, is turning feed searches over in his mind. This is a neat conversation, referencing our pals Jeremy, Rafer, and others. In essence, why have all the majors punted on the presentweb – the fresh stuff that’s being discussed *right now* via blogs, feeds, and pings?
He sets up the question: But still it seems odd, that with the biggies supposedly in cutthroat competition for search, that they’ve left the field for Feedster as the best resource for this class of searches. Why?
Then asks a bigger one: Feeds & Blogs: Fad or something big?
He points to usenet as the precursor to the current blogosphere, and notes that many are skeptical of the current “wild eyed enthusiasm” w/r/t same. (He also pokes two well known bloggers in the eye with the links he chooses, but I’ll let him deal with that fight!) He wonders if Google is not just waiting and seeing – will this thing just be another search tab, like usenet became?
His conclusion: nope. Content is king, search is the king’s phonebook (or index…). I certainly agree, but the two are entirely intertwingled. Google et al do well when content does well, otherwise, what is there to index? The big question, however, is whether Google wants to start playing in the content aggregation space – the new model of “search, find, subscribe” that Jeremy discusses. Yahoo has already decided it does (and the buzz is that more is coming…). So has MSN and AOL. But at the end of the day, the tail is far too big on this beast, and no one place can own the conversation.
Search engines can’t construct compelling results pages for ‘fresh stuff’ because without some form of ‘popularity’ yardstick to guide them they go blind. At present this is the main reason they can’t build a relevancy index good enough to deliver an intuitively compelling result.
The only way you can deal with this problem currently is to ‘go vertical’: construct a framework which reflects the structure of the data in a particular field, scan a lot of current content and slot particular fragments into appropriate parts of the framework.
This is in fact what bloggers currently do manually.
More automated approaches are possible, and will be increasingly needed as rapidly changing content becomes more widespread on the web, but are still I think more likely to cover vertical subject areas than address the ‘presentweb’ as a whole.
If you are looking for the ‘fresh stuff’ *right now*, then the only engine that is really built for that purpose is the matching engine of PubSub.com. It is a “matching engine” not a “search engine” precisely because the problem of capturing the “fresh stuff” is distinctly different from the problem of retrieving data from a search engine’s archives. This is the difference between retrospective and prospective search.
We take “pings” from hundreds of thousands of sites, immediately scan the pinging sites, and publish results to our users with a delay that is often less than a minute. To see our new “PubSub Sidebar” that builds a listener directly into your IE browser, go to http://pubsub.com/sidebar. You can also see our support for the “PresentWeb” built into the Gush news aggregator. (http://2entwine.com/features/pubsub.html).
Because “relevance” in the “PresentWeb” is a different concept than relevance in the historical web, we’ve created new, time-sensitive measures of relevance that can be used to filter messages… See: http://www.pubsub.com/linkranks-about.php
The search engine folk seem to think that all problems can be solved by their search engines. “If you have a hammer, every problem looks like a nail.” We need both search engines and matching engines. Ideally, the two would be well integrated…
Bob Wyman
I probably think that part of the answer lies in the analogy that Rich provided regarding his previous reliance on Usenet –
Personally, I find Feedbot a much more compelling solution than Feedster. And the way it is implemented just screams, “Assimilate me, Google!” 🙂
http://www.feedbot.org
From my blog:
Battelle (and his commentors) discuss how “…all the majors punted on the presentweb – the fresh stuff that’s being discussed *right now* via blogs, feeds, and pings…”
Let’s be honest here… if you were the EVP of Search at AOL, responsible for a LOB that generated north of $300MM of revenue FY2003, and had finite resources to pursue a seemingly unlimited number of resources, where would feed search sit on your list of priorities. The reality is, probably pretty low at present.
Why? Because, while you do want to innovate (and differentiate your offering), the reality is that feed search is firmly in the “entreprenuerial” stage of its market lifecycle. By definition, competition will increase once the concept is proven and the value apparent, which happens in the growth and maturity stages. Now, of course, different technologies, products, solutions (what have you) progress through the market lifecycle at different paces; it’s entirely possible that the lifecycle of feed search will develop substantially more quickly due to the state of web search, local search et al.
But for now, it is rightfully the domain of the entreprenuer, where it will remain until investment from the “majors” is warranted.