(image from Scientfic American – thanks ID:entity)
I am writing the final chapter of my book (no, not the last…just the last one, I’m writing them out of order, don’t ask….)
In any case, I got the utterly lazyweb idea of asking all the folks I’ve interviewed, in particular the professional thinkers and Big Idea folks, the relatively simple question of: What might the world look like if we had perfect search?
Now, in the process of putting the book together, I’ve been mining my blog quite a lot, and I’ve noticed that the comments section is always better than my posts. As Dan says, our readers always know more.
So I thought I’d ask you guys to indulge me once again. Here’s the email I sent out:
——————
Battelle here, contacting you one last time about search (well, perhaps not the last, but at least I’m close.) The last chapter of my book is entitled “Perfect Search”, in it I run through the many developments in search which might lead us to the Holy Grail – a perfect (or at least the best possible) answer to every question.
From the early draft, I write:
Imagine the ability to ask any question and get not just an accurate answer, but your perfect answer – an answer that suits the context and intent of your question, an answer that is informed by who you are and why you might be asking. The engine providing this answer is capable of incorporating all the world’s knowledge to the task at hand – be it captured in text, video, or audio. It’s capable of discerning between straightforward requests – who was the third president of the United States? – and more nuanced ones – under what circumstances did the third president of the United States foreswear his views on slavery?
This perfect search also has perfect recall – it knows what you’ve seen, and can discern between a journey of discovery – where you want to find something new – and recovery – where you want to find something you’ve seen before.
That’s a long way from the typical search engine of today, but imagining such a service no longer falls in the realm of science fiction. It’s the stated goal of nearly major player in IT today – be it IBM, Microsoft, Google, Yahoo, Amazon, and scores of others.
I then go on (and on…) describing various interesting forays into creating more perfect search – domain specific, federated, semantic, personal, local, etc. etc. etc.
But the real payoff is toward the end. This is where I want to stretch out and imagine a world where perfect search exists, and conjure up the implications of such a place. What opportunities arise when knowledge can be so easily gathered? What threats? How might this change our social structures, our politics, our economy?
I am sending this note to a special set of thinkers and visionaries with whom I have conversed in the course of writing this book, and beyond. Because you have suffered me to date, I ask you to suffer me once more, so that I might gather your insight, and those of your peers, into a special section of the book.
Please accept my thanks in advance for asking this of you, any response, no matter how trivial or considered, will be most appreciated and recognized.
My best,
John
——————
So, what do you think?
I look forward to the day. I would love nothing less than to walk in the room, tell (speak) my computer a phrase, for example “ham and pineapple pizza” and it returns me all my local pizza shops that sell “ham & pineapple pizza”, their timings and whether they deliver. All this without even going to the site as yet. I know what I want, I know when I want it, now I just want to know who’s got it. So, my perfect idea is a search engine with voice recognition, whether directly or combines through desktop search integrated within the PC combined with accurate, descriptive local search. What bliss that would be. All I’d need then is for the search engine and computer to talk directly to the shop and order the damn thing for me! O’well.
John,
I wrote something on my SolveDating website. It relates to finding people but it can be extrapolated to everything else that needs to be searched on the Internet.
http://www.solvedating.com/soulmatemoutcome.html
Looking forward to your book,
Chau Vuong
http://www.socialgrid.com
I think, your ponder has a lot of permutations… perfect search in a world of imperfect justice would have different implications, than perfect search in a world of perfect justice. Perfect search really might not be much more than current search without lies, deception and easily prevented chaos. The problems in search is not that search engines don’t know who I am “personally”, the problem is they don’t know –and don’t want to know– who is providing the content they “suggest”. Content providers are obligated to guarantee nothing to be awarded the #1 ranking as an answer to a question. Search engines impose no accountability beyond a minor responsibility not to use aggressive, gratuitously deceptive “optimizing” techniques.
Back to the subject: “Perfect Search” would obviously greatly enhance human productivity. Efficient knowledge distribution is certainly a good thing…. the problem is the-powers-that-be, the privileged “owners” of the economy, and most earthly assets, will likely selfishly consume all the benefits of productivity gains and leave the masses smarter, but no better off (economically) except for maybe the gains reaped from accelerated technical progress.
Perhaps most people would just search for
“people I could sue for money and win” 🙂
Your search – ” people I could sue for money and win” – did not match any documents.
till now?
John, perfect search is a great goal to aspire to, but I think an equally important goal is real-time translation as that is necessary to expand the value of perfect search to more people who use the Internet in non-English-dominant nations.
I think it’s a fundamental mistake to believe that Perfect Search is even possible. It seems to me to assume the existence of a single, authoratitive, objective standpoint from which any question can be answered. Take a question like “Why did the third president of the United States foreswear his views on slavery?”. Even simple ‘what happened’ questions just aren’t that simple – see Rashomon.
In response to Gen Kai, I’d like to say that the web (and search tools) need to become much more English-dominated as English is currently the best candidate for a worldwide lingua franca. If you want to be a pilot you need to speak English and frankly the same should be true if you want to use the internet. Investing time and money in translation services is a waste and a distraction from the real holy grail of natural language understanding, the ultimate AI problem. (Funnily enough, translation will become much easier once NLU has been perfected. If that ever happens).
Are there any techies out there who have thought about how an intelligent search engine would “understand” the web pages it spiders? My idea is to use “clause mapping” to map a myriad of different sentences that essentially mean the same thing to one simple, basic sentence. For example, suppose a user typed in “Who killed Abraham Lincoln?”, the search engine would have to be pretty lucky to have a page in its database that contained the text “John Wilkes Booth killed Abraham Lincoln”. It’s far more likely to have stuff like “Abraham Lincoln’s life was ended in 1878 by an assassin called John Wilkes booth” or “Abraham Lincoln was murdered in a theatre. The killer’s name was John Wilkes Booth” etc etc. The real problem is being able to think of all the different ways of saying the same thing and getting a computer to create the mappings to the same basic sentence.
And finally, what if there’s a web page that says “Abraham Lincoln was killed by Britney Spears”? It makes sense gramatically and semantically, it just happens to be factually inaccurate. An intelligent search engine therefore should report this as a valid answer. The real purpose of search is to help a human brain reach a valid conclusion. The better the search, the more relevant the results obtained.
Google uses the Page Rank voting method to assess relevance, but analyzing meaning should, in theory, be a better way to judge relevance. Any comments?
The world of the Perfect Search would not be interesting. Asimov was great in exploring in his “I Robot” short stories the shortcomings and unintended consequences of his own laws. I can much more easily imagine someone building a vision of the future where the Perfect Search has been promised but underdelivers, then the utopistic version.
Mathematicians are well equipped in exploring the theoretical limits of their models, and Godel did so very well with his Theorem: we now know thanks to him that scientific exploration is an endless adventure. The memetic field of intensely interacting intentional agents (pardon my cacophony) must be similarly unbounded.
I comment regularly on this blog, but I don’t feel qualified to answer these questions.
Basically the one person who has provided to date the most though provoking answer is Sir Tim Berners Lee. Socio/economic/political thoughts were covered off best in his book “Weaving the World Wide Web.”
In terms of opportunity this interview with Scientific America, is the best I’ve come across to date:-
http://www.sciam.com/article.cfm?articleID=00048144-10D2-1C70-84A9809EC588EF21
Here is an extract on the vision bit…(so apologies for the long post, but your more likely to read it this way.)
————————–
The entertainment system was belting out the Beatles’ “We Can Work It Out” when the phone rang. When Pete answered, his phone turned the sound down by sending a message to all the other local devices that had a volume control. His sister, Lucy, was on the line from the doctor’s office: “Mom needs to see a specialist and then has to have a series of physical therapy sessions. Biweekly or something. I’m going to have my agent set up the appointments.” Pete immediately agreed to share the chauffeuring.
At the doctor’s office, Lucy instructed her Semantic Web agent through her handheld Web browser. The agent promptly retrieved information about Mom’s prescribed treatment from the doctor’s agent, looked up several lists of providers, and checked for the ones in-plan for Mom’s insurance within a 20-mile radius of her home and with a rating of excellent or very good on trusted rating services. It then began trying to find a match between available appointment times (supplied by the agents of individual providers through their Web sites) and Pete’s and Lucy’s busy schedules. (The emphasized keywords indicate terms whose semantics, or meaning, were defined for the agent through the Semantic Web.)
In a few minutes the agent presented them with a plan. Pete didn’t like it
Would businesses be able to tap into perfect search? Could they search for the perfect customer? Could Ford do a perfect search for the consumer who’s in the market for an SUV, is predisposed to American cars, and doesn’t tend to haggle much over the sticker price?
Could politicians likewise conduct the perfect search? Could Kerry and Bush perform a perfect search for the Ohio undecided voters with whom the candidates’ messages would resonate? What happens when the pro-choice, anti-g*y-marriage moderate comes up on both lists? Or would that person be excluded from all lists since they are too complex to be perfect? [Note that in trying to spell out the “g” word here, the post was censored, and the three-letter “s” word in the middle of “homos*xual” fared no better.]
Can perfect search only apply to things so simple that they’re perfect? It may be rather easy for a young lover to do a perfect search for the best music to get it goin’ on, so to speak (and the perfect answer already exists to that – it’s Marvin G*ye – so it seems there are some universal truths in the world). When that same person conducts a search for a form of media that will bring him enlightenment, nirvana, and/or total consciousness, things might get a little murkier. Perfect search, if perfect, would lead to universal happiness for all who seek it. The fountain of youth, the Garden of Eden – all become accessible. [And I hate to offer another distraction here, but I couldn’t even spell out Marvin’s last name without the Content Submission Error telling me, “Your comment could not be submitted due to questionable content: G*y”.]
Personally, I wouldn’t want perfect search to exist. If it did, I would hope and pray that I would never be found by another’s search, as I would never aspire to be the round peg fitting so smoothly in another’s predetermined round hole.
Perfect Search gives us Comprehensive Information, unfortunately it won’t necessarily give us practical answers or even solutions, e.g. the answers we get may include contradiction.
However, what I find very exiting, perfect search provides a good starting point for Perfect Decision Making, a whole new industry. Among many other dimensions, this quest will introduce the need for opinions and quantity of opinions.
An example: What company will be most successful in the software business?
The Perfect Search will give a Comprehensive Answer but it will include contradictions. The next step will be to find out what parts of the Comprehensive Answer are supported by whom and why. Andsoforth.
Perfect Search is the beginning of decision making, not the end.
Philosophical, ethical and other similar issues aside, I would love some technology that will do more work for me. That means:
– a small input device that can record my ‘search ideas’ whenever I think of them (in the subway, at the beach, at work, after seeing a movie…).
– this device would allow audio input
– this device can be hooked to a proper computer (or to an omni-present wi-fi connection) and send out all my queries to appropriate destinations (various search engines and specialized databases it known of and chooses based on my queries)
– I’d like to be able to hear the search results, or rather the vital information scrubbed from found sources (web pages, database records, my old email, my bookmarks…), in addition to getting a report delivered somewhere for easy access (e.g. my PIM’s dashboard). It would also be good if the device had a screen, so I can also read the locally cached results and browse through them (think commute home after work).
In summary, I need something that is _asynchronous_ (I don’t want to have to sit and wait for results and then analyze 10-20 of them. We try to parallelize with browser tabs, but that quickly becomes a pain, too).
I need something that requires the _least__time__and__effort_ from me to give it my search idea, my query. Hence, _voice__recognition_.
I do not care if my search takes 5 minutes or even _6__hours_ sometime, as long as I don’t have to spend time looking. I’d rather spend my time, half a day later, listening to high quality findings.
Now I have to go work. I need a robot to do my job, too, so I can just learn interesting things all day, using this perfect search tool. For instance, I’ve been wanting to learn about the anatomy of octopuses, but have not had the chance to do proper research yet.
In my opinion, the perfect “search” answer isn’t a list of ten blue links, it is (more or less) a wikipedia entry.
The response would be appear to have been written by close, well-informed set of friends to answer your non-trivial question (hmm). Being well-engineered, the algorithm would provide multiple balanced perspectives in some detail, but it would know that you have only a certain amount of time, so it would put a concise summary at the beginning of the article and an insightful comment at the end. It would include references when appropriate, media when illustrative, and separate fact from falsehood from opinion. It would not be (overly) biased by the distribution of knowledge that’s easily available on the web and it would be able to provide references to offline material.
In fact, if you squint when you look at it, your blog entry IS a query, constructed for that perfect search engine before it exists. These comments, your other private conversations, and your subsequent interpretation and editorialization of them in your book will also be a response to a query.
Unfortunately, we’re still doing the equivalent of the manual concordances of yesteryear, 3×5 cards and all (see page 1 of Managing Gigabytes to understand the reference to 3×5 cards).
Instant search returns on medical questions, ie. drug interactions, prognosis speculations and rare disease sypmtomology would be of great value to practicing physicians and their patients.
Emergency personnel responding to hazzardous material spills could query instantly the effects of environmental elements such as wind, and accurately project danger zones. Same for wildland firefighters.
Police confronted with a suspect would ask for data in the field that would assist him in making a good arrest since under you scenario all is known and documented about him/her.
These examples are real world uses of the perfect search.
If I need to find an open face berry pie that uses sugar substitute and no msg. made by a woman of hispanic decent between the ages of 65 and 70 while visiting Tokyo I guess it would be useful.
I can’t say what ‘perfect search’ will be delivering 10 years from now, but I can say something about how it will be doing it. The standard towards which all web services have to converge is I believe Google’s response time.
Response time for me is made up of four components:
(a) Time to access the computer
(b) Time to specify the query
(c) Time to load the results page
(d) Time to sort out the results I want from everything else on the page
Google sets the standard on all four:
(a) All it needs is a browser – I can use whatever computer is at hand
(b) For me 99% of the time it needs only two or three words typed in and a click on a button
(c) It’s a clean, uncluttered, text-based interface which loads fast
(d) I can choose instantly to focus on the content or the ads, because I know where they are and there is no distracting ‘eye candy’
All this stuff which requires users to download an application (Picasa), sit and wait for two minutes to get a result (Farechase), won’t work with most browsers (both), is crammed with visually confusing content, mixes neutral content with paid content etc., is going to be swept away (most of it is, anyway).
The key reason why this is inevitable is that the number of things the average person wants to achieve in a day which are mediated via the web is going up all the time.
Already literally every time I want to walk out the door, no matter what the purpose, leisure or business, local or not, trivial or challenging, I realize I can save myself time/make my trip more effective by punching up my browser and either researching or using a web resource.
Yet the number of minutes a day I want to spend on the web is limited. And the novelty factor associated with some cool new site/capability is frankly wearing off – just as the ‘jet set’ appeal of air travel did decades ago, so that we now think of a plane as little more exciting than a ‘bus in the sky’. The ever-expanding range of available web-enabled services will and in my opinion already is driving us relentlessly towards simplified, fast, tightly focused services and interfaces.
The interesting part is that while in some areas this will mean primarily tossing out a lot of junk content (cough… MSN search… cough), in plenty of other areas serious engineering work is needed before we can get there.
I’m with Peter Ferne. Perfect Search is science fiction. It requires Artificial Intelligence, which is not possible. Intelligence is part-and-parcel of humanity.
We’ll continue to see incremental improvements in search, much as we did in cars through the last century. Think power windows, not teleportation.
Having done quite a bit of construction, I think I’m more inlined to want to be able to personally maneurve the tools to seek my own results.
Now software as a tool often does some convennient stuff on its own…nice to have the algorithims create a foot noe at the click of a button.
But “Search” and the use of “artificial intelligence” seems to knock heads a bit. A “more perfect” search for me, wouldn’t have the search engine anticpating my needs but would allow me to in an equalizer type fashion turn up and down Parsing toools.
Basiscally, a custom parse…sort first by chamber of commerce directory, then by work content, then by picture titles. Maybe on my dual 20 inch screens quickly (I trade stocks in real time some months show me the top ten in 4 categories simalaneuously, then let me merge collumn two and three with 1- 4 used as 1/4 influence…and perhaps let me custom modify individual collums before I merge…necessitating “exact phrases” in some categories (like don’t you hate it when you are trying to find out about someone and say you get a marathon list with Tom Jones, and 500 entries down Jim Norian? but still as Tom Norian might be who you want to know more about you don’t want to exclude marathon searches entirely.
If you are looking into me for my skills as a soccer coach, the marathon information might be more important than if you were doing due dilegence on me as a real estate broker. I can’t see AI entirely getting that any more than if you bump into someone at a tailgate party and ask a general question about a mutual aquaintance. Being askes “what do you want to know” is sort of irrelevant…its general impressions or perhaps some sort of information you might use in a bon jovial way like “hey, mack…heard they called you “opie” at davis”
Of course if the fellow responded after a sip of beer, “I think of Mack now and then and you know, he’s one of the most brilliant guys I’ve ever met”. Now at that party, you might know you are asking a guy that on the faculty of McGeorge, and so to who your question was addressed was far from random.
Or think about questions ones asks your wife, and maybe you get a “why do you want to know” or “what do you care about that for” and maybe you don’t really know exactly what you are thinking about, certainly not able to put it into words. I really don’t think that search AI is going to get too much better than your wife in knowing you and in a similar way if the person anwering spends to much time trying to ponder your question you’re going to get different results in answeres.
Really, to get maximum information you’d like to keep peppering with questions, reading quickly, refining questions…
Why? because anctipating what you are looking for will change the results given, and you’d sort of want to double check to make sure that your assistant was giving you what you wanted sometimes without disclosing entirely your intentions.
The search engine that immediately presents me with the perfect answer in response to the most succinct of my questions is the search engine that knows more about my personal interests than I would feel comfortable with.
For me, the perfect engine would converse with me as an anonymous individual, asking highly pertinent questions where necessary to help discover my intent.
Asking questions back is similar to various toggles. Still I think that as we all think differently the AI is sometimes something we’d like to be able to avoid.
Tivo try’s pretty well and gets to know things I’d like…so I won’t through out the concept…but its times when you want to drive it yourself in a timely manner. (i.e I like documentries but don’t like dramas on the same subject…evnetually it might firuge it out but I want to wear the pants and know what qualitative decisions they are making on my behalf).
Human nature will bias algorithms the way the writers have incentives to do…being able to overide incentives is quite important.
But I do think a question back would help. Last night, my wife was watching Nip and Tuck on the Tivo and wanting to unwind, I figured…I haven’t played shogi for while. I didn’t have a program so I searched shogi, was a bit overwhelmed and then thought “oh yeah, freeware” and I got an adequate program in the first or second google screen downloaded and was playing within a minute.
Now, next time I type “shogi” I probably won’t want freeware, and I could just see some A.I figuring thats what I wanted last time and understimating stragie sites…so yeah a question back might help.
Soometimes too you’re using a tool for diagnosis. I want to use the hammer to detect hollow spots in a wall, if I tell it to tap I don’t want it to tell me there is no nail to tap always.
There are all sorts of reasons why you might want to use search as more of a study tool, to explore social correlations etc….degrees of seperation and overlap…you know like figuing the overlap of people frequently mentioned in relation to swimming pools an people mentioned in relation to breast cancer.
In my spare time, I sort of like to explore correlations.
In deciding to trust the character of managment of oa public company, especially illiquid ones, I’ll dig into the social and charitable activities of the owners and directors. Certainly you’d want to know about allegations of substance abuse and/or try to gage to what extent they’re shake on it sorts of guys or gals or folks who’ve made big problems with their condo association they belong to which you can often read publically.
I think I’d confuse the heck out of any A.I but given control of parsing I can often get closer to what I’m hoping for.
Fantastic discussion. I have to agree with folks like Peter Ferne, Doug Cutting, and Mark Harwood in that Perfect Search is not possible, and if it was it would make me uncomfortable.
But to extend the question a bit (too far, perhaps?), why do we have any belief or expectation that this is possible? To my knowledge, despite the amazing advances we’ve made via industry and technology over the past couple hundred years, nothing perfect has yet been created. Has it?
Further, who’s to decide when Perfect Search has been achieved? What earth-bound authority is in position to declare that Perfect Search exists Everywhere for Everyone?
As far as I am concerned, the danger is not what would happen if such perfect search existed – the danger is that
If perfect search existed, you’d have a solution to Frege’s Puzzle and therefore understand the relation between symbols and the real world, and presumably also therefore understand how much we can know about the real world itself. The world would probably be a less interesting place — although, on the other hand, I’m not sure that we’d recognise perfect search even if we did create it
In a world with perfect search, society would self-categorize into the “indexed” and the “non indexed.” People, groups, corporations, and other entities would make their decision to be indexed (or not) based on the need (or lack thereof) for privacy and/or transparency. The indexed would have little visibility into the non-indexed. As a result, the indexed may believe that the non-indexed do not exist, as search is presumably “perfect” and therefore knows all.
Both groups may also place too much faith in the Information Brokering God (a.k.a. the Search Engine).
Hmmm. Methinks this is already happening.
In a world with perfect search, an individual’s capacity for wisdom (from understanding, then knowledge, then information, then data) would be limited. Perfect search gives the perfect answer. However, wisdom may be a collection of imperfect answers that, when considered together, inform the desired eureka. In a world with perfect search, serendipity is limited, reductionist answers are more likely than holistic understanding, regurgitation of an old answer is more likely than the synthesis of something new, and no one would look beyond the first half page of results.
Hmmm. Methinks this is already happening.
And let’s consider two extreme cases:
1. Assume humans and technology merge, a la Singularity. In this case, I’d like my very own Miriam the Singularity Librarian to consult (as humans [even posthumans] will very likely always do it better).
2. Assume society enters a non-indexed Dark Age (the inaccessible data on that floppy of yours from high school on a much grander scale). In this case, who will be our Information Brokers? Who will still have the skills to process, analyze, and synthesize the remaining information (assuming it exists)? Who will still be able to think, as opposed to merely construct the perfect query? In this case, I’d like my very own Miriam the Luddite Librarian to consult.
And that’s my set of random pennies. And, no, I’m not a librarian (although I may be the only woman who has commented on this post? hard to tell from some of the initial-or-gender-neutral names…).
As per previous posters (Peter Ferne, Doug Cutting, Mark Harwood, Matt McGee) have argued, Perfect Search is not likely to exist anytime soon.
You’re probably familiar with the work of the TREC information retrieval evaluation series of workshops held over the last decade plus(http://trec.nist.gov), but if not, it’s worth repeating one of the most fundamental results that these people discovered. (Apologies – can’t remember the exact person/group and reference for this result off the top of my head. It’s likely to be Donna Harman or Ellen Vorhees however.)
Basically, in human judgements of information retrieval tasks, on average (measured over a number of queries – say 50 or more) two human judges agree on whether a document is relevant to a particular search query only 80% of the time!
In other words, for the exact same query, two people will disagree 2 out of 10 times whether or not a given result is in fact relevant. Without personalisation, it’s therefore nigh on impossible for a search engine to ever do better.
In John’s world of super-personalised search, the retrieval engine may be able to tailor its results to better meet the individual biases of the searcher, and bring them results which the searcher considers relevant, even if they would be irrelevant to another person.
But that said, in most TREC experiments, best performing search engines are mostly achieving an average precision (number of relevant results returned at a particular retrieval level) of only 40%.
Of course, there is a category of information retrieval queries for which one relevant result is sufficient (e.g. home page of John Battelle’s search blog), but there are a large number where this is not so.
All up then, there is a long way to go. But that’s what keeps it fun right?!
I have been interested in the concept you call “Perfect Search” for many years now, and have been looking at the application of pragmatic AI solutions to the issue. Clearly, the challenge lies in the (effectively) infinite search space that the Internet represents to today’s computers coupled with the fact that search engines only see the surface-web. Our solution, called Memogo, executes a, well, near-perfect search on a much smaller search space, i.e. material that the user has gathered as well as information that her buddies have thought she’d enjoy sharing.
We’re still in the early beta phase, but we’d be interested in hearing feedback from sensible, tolerant and informed beta users 😉
I don’t think you can define ‘perfect search’ as simply that which returns the right answer. There are too many other factors to consider.
In my own attempts to define what would make a perfect public or ‘open’ index (http://www.openindex.org/anidealindex.htm), I made up a list of features which I thought were desirable in such an index – what makes a ‘perfect’ index besides providing the right answer? With some modification, they include:
Complete – It would index all resources that are available to the public over the Internet.
Fast – It would be responsive and give good results quickly.
Timely – Up-to-date. It would be able to quickly discover and index resources and revisit them at appropriate times to record changes.
Relevant – Results from searches would be useful.
Accurate – The results would be correct and authoritative (I know: that’s hard!)
Honest and Fair – It would present search results fairly and without bias.
Private – Users of the index would be protected from surveillance and identification. It couldn
So – the perfect search is (a) technologically unlikely for a fair while yet; (b) potentially very dangerous if it gets into the wrong hands, and (c) philosophically unlikely as it suggests the existence of a perfect answer (for every question). But, if it did exist it would certainly be a multiple stage process (What are the 10 best blogsites? As adjudged by who? etc.). It would also lead, I think, to a change in emphasis – the skills of interpreting and extrapolating would be valued far more highly than those of researching and exploring – and, consequently, mindset.
In a postmodern world, perfect search equates to a flourishing economy for search in which a range of tools and options are available. This openness not only allows technical innovation to thrive, but gives people market freedom to not only use interesting research tools, but to start companies and put forward arguments that have the potential to make the whole business “better.”
This is to some extent realized in today’s environment, but insofar as market forces and lack of imagination have led to the undoing of a lot of promising entities called “search engine companies,” we are unfortunately some ways away from even my very forgiving definition of “perfect search.” Danny Sullivan’s keynote in August seemed to be hopeful that we would be heading towards a “four-crawler” race (with healthy ongoing research at and consumer usage of Google, Teoma, Yahoo, and MSN) where things looked pretty bleak and one-sided for awhile there. But a four-crawler world is still behind where we were a few years ago, with at least five or six active web indexing companies alive at any given time. And the state of comprehensive human-edited directories is currently terrible. It, too, was better a few years ago.
There are a number of forces which could prevent us evolving into a relatively vibrant search economy (I say *could*):
– Monopoly — too few major search tech companies, too few web indexes, and no one seeming to care
– Privatism — good info if you pay a huge fee, bad info if you don’t, without a middle-ground low-fee option for the average person
– Standards (too few participants in some hypothetical future metadata scheme)
– Repression (governments)
– Commercialization via portalization — similar to monopoly; portals enforce a creeping, poorly-disclosed regime with too much paid inclusion and too many sponsored links
– Spam — same as above, but by cowboys
– Lack of imagination — if consumers and experts put the same old template on the idea of “what search is,” they may miss out on the idea that “a big list of keyword-search results, ranked based on a particular algorithm” is only one way of looking for stuff
– Illiteracy (too many people spend too much of their disposable income on 68″ TV’s, watch crap, and become stupid, thus no demand for “search”)
– Faddism in capital markets or gov’t policy — insufficient investment in search tools and basic research because something else becomes ‘hot’
In conclusion, I don’t really much believe in asking frivolous questions about perfect search. When Googlers are asked about it, don’t they just give the stock answer about the search engine reading your mind, and then quickly move on?
Is perfect search connected with access to information? It might be interesting to take stock of how far we’ve come in the past 100 or 500 years; to ask whether there are today any particularly premodern practices cropping up that fly in the face of trends towards fuller disclosure; and to consider how far the information revolution is likely to go. And probably it’s also worth considering that even in a society where there is a veritable smorgasbord of news and facts and information available, there are widely divergent ways of thinking based on different selection or different interpretations of facts.
The notion of “perfect search” is like the notion of “fixed ontology.” The way the world is seen depends on the way you categorize it. Those who advocate fixed categorizations of any kind (like arttworks, who admirably understands that global corporations are very powerful but cannot get past his interest in creating a fixed democratic alternative that might resemble a Stalinist regime) are suspect.
In a world where much information is privately held, one definition of perfect search would be to allow us to find out everything, to look under the veils of corporate and governmental power. If we live in a democratic society the trend must be in that direction, for sure. But these things unfold in shades of grey. I think China is a repressive society and would like to see some companies and some government agencies in the West disclose much more about their operations. But only a wide-eyed revolutionary would expect or wish every bit of information to be disclosed.
In light of that, “public” search engines perform a relatively limited service. Insofar as semi-private and personal information might become more available with the advent of “peer-to-peer” (etc.) search technologies (“hey, go ahead and search my desktop for the good of science or for the improvement of my own search experience”), privacy issues arise.
“Perfect search” works both ways. When the Big Spotlight is turned on you as a private citizen, private company, employee, or celebrity, it’s no longer so much fun.
More germane than perfecting search, whatever that could possibly mean, is finding an acceptable balance. Companies like Google are at a crossroads whereby they could make more money or create delightful new tools and features, just as long as they get more people to agree with creeping behavioral targeting and a slight loss of privacy. (GMail was one example of this, and there was a significant outcry. I thought the backlash was overdone, but the fears people have are not completely out of line.) Clearly some in the advertising industry (Claria) want to proceed full-out with behavioral targeting, spying, whatever you call it. Companies like Google have been much more cautious because they presumably believe that people like to be left alone and will punish the company that fails to respect that.
So on the privacy issue, there are already a number of “takes” on that, all of them active in the marketplace. Claria and Google alike have rights to offer tools and information in vastly different ways, instead of mostly frivolous lawsuits against them. That’s the postmodern let-multiple-flowers-bloom thing I’m talking about.
Future developments in search will grapple with complex socioeconomic equations, not just algorithms. We don’t build the perfect bridge, because bridges cause money and the perfect bridge would bankrupt us. We don’t run the perfect school, because perfect obedience is against children’s natures. But we do want better bridges for the money, and on average, we want better education for children (such as not teaching them things that are completely wrong or outdated, or making them inhale asbestos). How to get there becomes a complex policy debate.
In a perfect world, search would be moving in the right direction, but there would be no “ultimate” goal. Consumers and experts would evaluate and use a variety of information-gathering tools in such a way as to enhance their productivity and enrich their lives. Absolutes are not only dangerous, they’re silly.
Like you, I look forward to the day that the perfect search arrives. There is a complete absence of context using current methods and as a result, the “infoglut” problem is only going to get worse. Simple keyword matching just won’t cut it The reason is that any solution that does not understand the language and vocabulary of the user cannot ultimately succed in meeting the needs of the user.
To this end, we are trying to bring this time forward by developing our own solutions. From our background in encyclopedia and linguistic developments, we have created “sense engine” technology. Textonomy enables the search to be placed in context and filters the results to meet the search profile. This is enabled through the combination of linguistics based technology and comprehensive super index of concepts and lexemes.
So far, we have applied this to search and to enabling truly “contextual advertising” . Sorry if this sounds like an infomercial but this is an area in which we feel very passionate.
Perfect Search:
Aren’t we wrong in making the assumption that we can
1) Collect information faster than its growth rate
2) That we can actually predict how and in what form more information will get added.
The perfect search will be a combination of known selected information AND some pattern of collating constantly evolving additional information.
And it will point out not the perfect answer but a few sets of data which will comprise the 3 or 4 most likely answers.
Fragmentation:
Also as we evolve more and more with person or audience specific search we will have more and more fragmentation in vocabularies, thought processes, and ways of living.
This actually might be scary enough to bring us back to a more ‘utilitarian’ search paradigm rather than individual specific search.
Already it’s difficult to get people from a different line of work to understand/comprehend you well.
This post made pleasant reading within the book, was nice that architecture of participation became a reality. The blog/book I feel encapusulated journalistically what was lacking at the time. Now comes that hard part, write the sequal ~ look to Ricky Gervais for insparation.
The definition of a perfect search is in the user itself. Search is a science that continues to evolve. It is definitly in the same category that has been pleaging the medical field for a lifetime. What makes each of us unique, once we understand that, then the “Perfect Search” can be established. We all define and see information differently, thats why we will continue to venture outward to find the perfect search..
I’d look to the poets for answers on the perfect search. Tarkovsky in his film “Nostalghia” leads us into the room, the room where our questions, our deepest darkest wishes, will be answered, and we have to assume these will be perfect answers to perfect searches, and what happens? No one can bring themselves to go into the room. Why not? We do not want the perfect answer. We do not have the perfect question. We are far far from formulating the perfect question and from embarking on the perfect search because we are so far removed from our place, society, desires, needs, connections… The perfect search must be the search we undertake, physically and mentally, alone. It seems it is as individual as we are. How one art deviates into another. How inventions are stumbled upon. How collaborations form. The one perfect search, question, answer, uniform and rigid, might end all variation, flatten our art. Ask James Lee Byars (no longer living, but available through his artwork and writings). He was devoted to the search for the perfect. I believe that all perfects are the same. There is one truth. If you are looking for the perfect search, you are looking for the perfect life, the perfect dinner, the perfect relationship, the perfect night.. Ask the artists. What about the perfect do you desire, does one desire? What about imperfection requires annihilation?