Our dilemma is introducing “relevant” search within business application – which are driven by more than just associated quality links, but also ratings, meta-data and resume match to a job amongst others. The current world of resume search largely depends on “sql” query – I think the future will be a combination of many more data elements like feedback, relevancy, background and many more meta data for which algorithms would need to be developed to matched appropriately. This problem can easily be extended to other business place applications like vendor management, marketplace ratings and recommendations etc. We want to go beyond traditional queries to intelligent relevancy

One of the story in your book resonated close to home. We have another product DBSync for Salesforce.com and QuickBooks which is one of highly recommended and downloaded as compared to our competition. We had listed it on Salesforce.com Marketplace – AppExchange and it used to rank quite high on “QuickBooks” keyword as its initial “relevance” algorithm seemed to handle other meta data. Then suddenly the ranking fell on its “Relevance” page, and you had now a product listed on top which had no review or rating and limited functionality – all other “relevant” applications fell way below. When I checked, they said they had switched to “Google Search”.

The above begs the discussion that how misunderstood search is and how un-planned organizations are while pursuing their search strategy and implementations. Salesforce.com AppExchange would have been better off following the Amazon model of recommendation and search – which I feel is more relevant in such situations. Perhaps at some point you might want to write another book for Amazon Search…

While I will continue my research on implementing search within our application, which will not likely be Google, I will certainly look forward to your blog for ideas and suggestions.

Rajeev Gupta
CTO, Avankia

More in this post:
http://anand.typepad.com/datawocky/2008/05/why-yahoo-glue.html

http://searchdns.netcraft.com/?restriction=site+contains&host=tennis

Now if I could search the content within these domains for “racquet”, then I think Google would start to look very old school.

I made http://browse.name in a couple hours like 3 or 4 years ago (well, that’s what it looks like, too — I guess back then most of the time was just to upload the framesets It’s pretty crappy and quite outdated, but probably still 10x better than hoping to find information using Google.COM ;P

However: If Google worked more *like* browse name it might function at least a *little* better (there was someone who posted here on battellemedia.com about half a year ago who is apparently developing such an engine, BTW).

The basic idea is that hotels.com is actually not merely a “navigational” result (when searching for “hotels”) — hotels.com actually delivers information about hotels (and in particular: commercial hotels)). But I don’t need google to know that. Just like Spock on Star Trek, I can simply find “logical” answers: hotels.de is about hotels in Germany, hotels.be is about hotels in Belgium, hotels.no is about hotels in Norway, etc.

So the point is that Google is simply helping out the novices who haven’t figured out how to use their browser to find information — at least in theory. In practice I think you are right: Google is not interested in producing any useful results whatsoever, since that would negatively affect the immediate revenue streams.

The main issue is, however, that Google pretends to have a “secret formula” that magically returns hotels.com when using google.com (and that returns hotels.de when using google.de) to search for “hotels” — and that’s why they need to have 15,000 Ph.D.s working on it. So far, a lot of people actually still believe this! :O

;D nmw

Topic is Findability, and it’s focused more on inside the Enterprise than outside, but still, might be of interest to you and your crowd here.

Closing the survey down next week, but can always use more data. >400 responses right now, but would love to raise that significantly to get an even bigger picture.

Project is described more at:
http://www.biztechtalk.com/2008/04/findability-sur.html

And the survey itself is live at:
http://aiimMarketIntelligence.questionpro.com

Average time to complete the survey is between 20-25 minutes at this point.

Participants have a chance to win 1 of 25 gift certificates at Amazon.com, as well as early notification of the free finished report, and companion free public webinar discussing the results, both available the last week of June.

I’ll second the ongoing discussion here – while frequently people use search to find “known information” there are plenty of times when people need to discover new information, and be more adequately lead in the direction of useful “leads” on such information.

That’s why our survey is on Findability rather than Search as the umbrella term. What about taxonomies? ontologies? relevancy ranking algorithms? dynamic clustering? impact of social interaction with information back on relevancy, discover, etc.?

Text retrieval (search) is over 30 years old, and yet we’re still barely taking advantage of what could be done – not just from a “wow, that’s a great demo” but from a true, “made a difference to my business” aspect.

I realize the enterprise angle is not your primary focus here, but certainly relates. How do consumer, web-focused search solutions address the differing needs of the enterprise? Does one corrupt the other? What do both sides have to learn from each other?

Interesting times!

Cheers,
Dan
http://www.twitter.com/dankeldsen

I think Google suffers from a second problem in this space, too: The innovator’s dilemma.

Because Google so heavily aligned itself with doing navigation-only queries, to the complete detriment of the 1/3 to 1/2 of the informational queries passing through its search box, it has succeeded in turning itself into an AdWords money-printing machine.

Think about it: Google is trying to condition people to think of it only as a navigation/transaction engine, only, rather than a true “search” engine. And the reason it does that is there is a higher chance of people clicking on ads, if the intent behind the user of Google navigation/transaction engine is to find a place to buy something.

A user arriving at a search engine with an informational query is not going to spend much, if any, time clicking on ads. Because what ads have to offer are not the intent or goal of an informational search!

Let me say that again, because it is one of the biggest conceits that we are collectively operating under in this industry: If someone has an informational query, an ad is (almost) never going to satisfy their information need. Ads presuppose a navigational or transactional query type!

So the innovators dilemma for Google is that if they actually altered their search engine to be able to handle informational queries, two things would happen: (1) It would require more effort from Google, both in terms of algorithm designers and programmers, as well as CPU usage, memory, disk space, bandwidth, etc. Informational queries are more computationally expensive! So costs for Google go up.

At the same time, (2) ad clicks would go down, because advertisements tend not to be relevant to informational queries, as mentioned above.

So in order to innovate (provide good algorithms and interfaces for informational queries), Google would need to destroy the very business that got them where they are today (navigational/transactional queries, laced with ads).

That’s another reason why I cringe every time Google says its ads are independent of its search results. They’re not. Google consciously avoids developing certain type of search results and search result interfaces, because that would interfere with ad click throughs.

Oh, I notice that. Believe me, I’ve noticed that for years. And ranted about that for years. I’ve asked Google, publicly in forums like this, privately to their engineers, semi-publicly at conferences, to get over it already, and get on with the business of organizing the world’s information by doing more than just known item search.

They’ve refused to budge.

That’s why I find it laughable when Marissa Mayer gets up there and talks about how they’re improving their search engine all the time. About how there were like 450 changes (improvements) to the search engine last year alone. So what? 450 changes to an engine that does known item search is.. still an engine that does known item search. And nothing else. Nothing has changed. Google is giving me no new insights, no new ways to help organize the world’s information. 10 years into it, Google’s still only 5% of the way there, as Teddie above mentions.

You continue: In other words, people do not generally use Google to find something they don’t already know about.

Do they not use Google because they do not want to use Google? Or do they not use Google because Google is not providing a search engine that meets their needs?

I would argue that it’s only 50% of the former reason, and simultaneously 100% of the latter.

What I mean is this: Go back and read Andrei Broder’s 2002 paper, “A Taxonomy of Web Search”:

http://www.sigir.org/forum/F2002/broder.pdf

In it, he makes the case that Web search is not like traditional IR.. that in addition to “informational” queries, web engines need to be able to answer navigational (known item) and transactional queries.

Fine. That’s all good. Kudos to the big web search engine companies for finally bringing us good navigational searches.

But what this paper also says is that between 39% (from a user study) and 48% (from log analysis) of web search engine queries are informational, not known item.

This means than more than a third, if not closer to a half, of every single query running through Google’s engine is a query that Google has specifically designed its search engine to not be able to answer!

That’s completely ridiculous.

(Conversely, it says that between 52% and 61% of all queries are NOT informational, which is why I do concede that not every person, all the time, is doing an informational query. But it’s also not true that no one ever does informational queries! 1 out of every 3, if not 1 out of every 2 queries is informational!)

So I’ll say again: Search engines need to evolve again, and get beyond doing navigation-only searches. Look at what John Battelle wrote about it 5 years ago:

http://battellemedia.com/archives/test/000100.php

John says that search engines are moving from navigational to transactional. But that was 5 years ago, and they appear to have stopped moving. Almost 1 out of every 2 queries, informational queries, passing through Google are not only not being answered, they are not even attempting to be answered.

if you review http://battellemedia.com/archives/003991.php , you may notice that many (indeed, perhaps even most) people use Google for “white pages” lookup.

In other words, people do not generally use Google to find something they don’t already know about. “New information” (actually somewhat of a redundant tautology) is found elsewhere — and Google is primarily used for “known item searches”).

I’ll never forget the way some librarians in the old days of paper card catalogs proudly smiled that “their” catalog could be used to find reliable information such as the birth year and/or year of death of an author (and therefore no additional resources were needed).

Similarly, Google can be used to “find” the url of a domain name (in other words: you can use the space bar instead of actually using the “dot” key, and then the top result will normally have the name with the dot) — I agree: not a great feat, but as long as browsers require people to use the “dot” (“period”) rather than the space bar, it may still be a millisecond or two faster than typing in the correct URL directly. In some cases, when brand names are well secured / protected, the TLD extension can also be left off — and then the time savings may be even greater (perhaps half a second or more!)

This is the primary reason Google is useful: It has little to do with discovering anything new, but rather it’s mostly about saving the effort of having to type out brand names.

Search in the sense of “discovery” is usually done elsewhere (if it’s done at all) — e.g. “vertical search engines”, such as baby.com (and/or similar “targeted” keyword domain names). The main reason why Google is hesitant to divulge the stats of the amount of money it earns from such domains (parking pages) is that they are afraid that people will realize that Google’s search engine is so totally ineffective that it’s laughable that anyone would waste their time and/or money by giving it to Google.

Sooner or later, people will catch on, and then Google’s share price will really tank big time….