Search engines can't construct compelling results pages for 'fresh stuff' because without some form of 'popularity' yardstick to guide them they go blind. At present this is the main reason they can't build a relevancy index good enough to deliver an intuitively compelling result.

The only way you can deal with this problem currently is to 'go vertical': construct a framework which reflects the structure of the data in a particular field, scan a lot of current content and slot particular fragments into appropriate parts of the framework.

This is in fact what bloggers currently do manually.

More automated approaches are possible, and will be increasingly needed as rapidly changing content becomes more widespread on the web, but are still I think more likely to cover vertical subject areas than address the 'presentweb' as a whole.

• Posted by: KJ
• August 27, 2004 2:33 PM

If you are looking for the 'fresh stuff' *right now*, then the only engine that is really built for that purpose is the matching engine of PubSub.com. It is a "matching engine" not a "search engine" precisely because the problem of capturing the "fresh stuff" is distinctly different from the problem of retrieving data from a search engine's archives. This is the difference between retrospective and prospective search.

We take "pings" from hundreds of thousands of sites, immediately scan the pinging sites, and publish results to our users with a delay that is often less than a minute. To see our new "PubSub Sidebar" that builds a listener directly into your IE browser, go to http://pubsub.com/sidebar. You can also see our support for the "PresentWeb" built into the Gush news aggregator. (http://2entwine.com/features/pubsub.html).

Because "relevance" in the "PresentWeb" is a different concept than relevance in the historical web, we've created new, time-sensitive measures of relevance that can be used to filter messages... See: http://www.pubsub.com/linkranks-about.php

The search engine folk seem to think that all problems can be solved by their search engines. "If you have a hammer, every problem looks like a nail." We need both search engines and matching engines. Ideally, the two would be well integrated...

Bob Wyman

• Posted by: Bob Wyman
• August 27, 2004 5:17 PM

I probably think that part of the answer lies in the analogy that Rich provided regarding his previous reliance on Usenet - “But I was part of a small technical minority."

The major search engines are mass market products and if you look at the Zeitgeist, Buzz Index, you will see that what they are actually doing is accommodating Pareto’s 80:20 rules. Will feed engines actually make any real substantial differences for queries such as “Madonna” – they have got most recent news / pr events covered by the newsbot’s.

I do not believe that this is a technical problem, the R&D labs at the moment can extract structured data out of unstructured – extracting knowledge from structured is as task that’s been accomplished already. The R&D guys at Yahoo published a paper called “The Ying to the Semantics Web’s Yang,” which basically gives you the companies approach to data structures etc moving forward. Google are even performing real time searches from streaming media (e.g. TV, Voice.)

Sorry kind of went off track slightly, but I think you can see where I’m coming from. If you compare these engines to plane's, Concorde was your feed engine, but was only for the elite – and sadly died (not that feeds will), whilst “jo public” we more than happier with a big Boeing buss – which was mass market. Eventually though we’ll hit that point where supersonic travel is the norm for everyone….but I’m sure that it will happen a lot quicker.

• Posted by: ID:entity
• August 28, 2004 7:50 AM

Personally, I find Feedbot a much more compelling solution than Feedster. And the way it is implemented just screams, "Assimilate me, Google!" :)

http://www.feedbot.org

• Posted by: mneptok
• August 28, 2004 2:03 PM

From my blog:

Battelle (and his commentors) discuss how "...all the majors punted on the presentweb - the fresh stuff that's being discussed *right now* via blogs, feeds, and pings..."

Let's be honest here... if you were the EVP of Search at AOL, responsible for a LOB that generated north of $300MM of revenue FY2003, and had finite resources to pursue a seemingly unlimited number of resources, where would feed search sit on your list of priorities. The reality is, probably pretty low at present.

Why? Because, while you do want to innovate (and differentiate your offering), the reality is that feed search is firmly in the "entreprenuerial" stage of its market lifecycle. By definition, competition will increase once the concept is proven and the value apparent, which happens in the growth and maturity stages. Now, of course, different technologies, products, solutions (what have you) progress through the market lifecycle at different paces; it's entirely possible that the lifecycle of feed search will develop substantially more quickly due to the state of web search, local search et al.

But for now, it is rightfully the domain of the entreprenuer, where it will remain until investment from the "majors" is warranted.

• Posted by: Tony
• August 31, 2004 5:08 PM