In practice, if you search for “REM” and find 43,000,000 results, it’s an indication that maybe you should refine your search to “REM music”. On the other hand, if you search for “Giraffe osteoporosis” and get 34 results, you have a good idea that if the first 10 don’t suit you, refinement probably won’t help very much.

Think of adding a second search term as a user-directed reshuffling.

I think you should look, and use for a while, http://www.Kartoo.com (one of the first engine to make clustering) and is new Kapitalyser who keep a history of your searches and learn about it. They should re-released a plain Html version soon.

(reposted here with his permission)

ohn, a couple of comments about your post. I hope they are useful to you. If
you think it’s worthy, feel free to repost on your site.

cheers,
gary

—-
—

1) For most web search engine total number of hits are junk and mean very
little.First, they are far from accurate. Second, they also include duplicate
and near duplicates. Third, they also mean nothing because most search engines
will only show you about 1,000 results (if you would even look that far down
the list). In other words, they are really just a promotion tool to make the
use think their search is “seeing it all.”

2) Don’t forget that you’re only seeing what the web crawler can find. I don’t
need to tell you that a great deal of material is not on the open web. It might
be in a fee-based system or reside on the invisible or opaque webs. How much
time are people wasting searching the open web for something that doesn’t exist
or isn’t current?

In many cases, the searcher can save time and aggravation by using a specialized
database that is focused on a specific topic. Instead of searching the massive
open web (feel free to take a guess at its size) but instead direct the user to
a speciality database (smaller in size) and then run the search on it. This is
another approach to increase the precision of your search and lower the recall.
Said another way, a bigger database doesn’t always mean better results.

In the library world these days a great deal of interest is in what was once
called “federated search” but has been renamed meta search. This is not the
same way the web world thinks of metasearch. These products will allow an
organization to build categories of search tools, for example “Business” or
“International Relations” or “Biomechanical Engineering.” Then, the search
technology will parse the query for each underlying database, run the search,
remove duplicates, and merge the results onto a results page that has also been
designed by the local organization. In other words, you can present the results
in a way that will best serve the user group.

This paper (which uses the another term for the technology “common user
interface”) reviews a great deal of the technology:
http://www.natlib.govt.nz/files/CUI_Report_Final.pdf

This technology will work with open web resources (Google), specialized and
Invisible Web databases (ResearchIndex), and proprietary info systems (what a
library or company might pay for (LexisNexis). Because some of these products
are not dependent on any standard, advanced search features can be easily
mapped to the search interface.

3) A new user interface from Yahoo (http://search.yahoo.com) allows the user to
pick which tabs/databases will be visible on their Yahoo search page. Of
course, many people don’t use the tabs.

4) I think something that is very interesting/exciting is Ask.Com’s Smart Answer
technology. Where instead of just getting links it takes a stab at an answer
and places it at the top of the results page. This is not using the Jeeves old
idea of pre-supposing question/answer sets but rather by using NLP.

For example:
What is the capital of Spain.
http://web.ask.com/web?q=what+is+the+capital+of+spain&o=0&qsrc=0

Who won the academy award for best actor in 1972.
http://tinyurl.com/2x5en

5) Suggesting better query terms? AltaVista has had Prisma for a couple of
years.
http://www.altavista.com/web/results?q=battelle+media&kgs=0&kls=1&avkw=aapt

Teoma does a good job by using the names of “user communities” to help narrow
and focus.

Gigablast just launched Giga Bits.
http://www.gigablast.com/search?q=john+battelle+journalist

Also, Yahoo’s SmartSort is very useful and works well at getting the user to
what he or she is looking for.
http://shopping.yahoo.com/smartsort

Reading:
This article by Silverstein and Henzinger of Google and a Stanford associate
discusses several problems that engines face getting the material into the
database in the first place.
“Challenges in Web Search Engines”
http://www.acm.org/sigs/sigir/forum/F2002/henzinger.pdf

This White Paper from Vivisimo is also interesing reading.
“Needed: A More Selective Ignorance”
http://vivisimo.com/docs/overlook.pdf

cheers,
gary

“You have chosen ‘fiscal responsibility.’ More Americans have chosen ‘war on terrorism.’ Use that to reform your query? (Y/N/Escape to Canada)”