free html hit counter The Database of Intentions - John Battelle's Search Blog

The Database of Intentions

By - November 13, 2003

So nothing really new in the news today, I wanted to take a graf or two and explain what I mean by The Database of Intentions, referred to in this post. That way I can use it again and again and just link the phrase to this post. Hey, we love the web, Ted Nelson lives….

The Database of Intentions is an idea central to the book I’ve been working on for the past year or so, which is tentatively titled “The Search: Business and Culture in the Age of Google” (Penguin/Putnam/Portfolio 2004). As with many in this industry, it all started with the Macintosh. Back in the mid 80s I was an undergraduate in Cultural Antropology, and I had a class – taught by the late Jim Deetz,which focused on the idea of material culture – basically, interpreting the artifacts of everyday life. It took the tools of archaeology – usually taught only in the context of civilizations long dead – and merged them with the tools of Cultural Anthropology, which interpreted living cultures. He encouraged us to see all things modified by man as expressions of culture, and therefore as keys to understanding culture itself. I began to see language, writing, and most everyday things in a new light – as reflecting the culture which created them, and fraught with all kinds of intent, contreversies, politics, relationships. It was a way to pick up current culture and hold it in your hand, make sense of it, read it.
(more via link below)

]]>< ![CDATA[

At the same time I was making extra money beta testing some software on a brand spanking new Mac, vintage 1984. Anthropology and technology merged, and I became convinced that the Mac represented mankind’s most sophisticated and important artifact ever – a representation of the plastic mind made visible. (Yeah, college – exhaaaaale – wasn’t it great!).

Anyway, the idea that a graphical user interface and, later, a network connecting many GUIs, could provide a medium between many minds drove much of my fascination with reporting on technology, from MacWeek to Wired to The Standard to now. The “Mac as the greatest artifact” meme became one of my standard riffs, from discussions with potential writers at Wired, to discussions with partners at The Standard. The idea that we could better understand ourselves by looking at how we employ technology was and remains the driving force of my work as a journalist.

This is all a long-winded way of saying, I’ve now come to the conclusion that humankind has created a far more fascinating and important artifact, one that surpasses the Macintosh (and its badly drawn descendant, Windows). And before you roll your eyes and say “Oh God, not the Internet…”, no, it’s not the Internet. It’s something that is a product of the Internet, what I call the Database of Intentions.

The Database of Intentions is simply this: The aggregate results of every search ever entered, every result list ever tendered, and every path taken as a result. It lives in many places, but three or four places in particular hold a massive amount of this data (ie MSN, Google, and Yahoo). This information represents, in aggregate form, a place holder for the intentions of humankind – a massive database of desires, needs, wants, and likes that can be discovered, supoenaed, archived, tracked, and exploited to all sorts of ends. Such a beast has never before existed in the history of culture, but is almost guaranteed to grow exponentially from this day forward. This artifact can tell us extraordinary things about who we are and what we want as a culture. And it has the potential to be abused in equally extraordinary fashion.

Once I grokked this idea (late 2001/early 2002), my head began to hurt. This was A Big Idea, one that certainly was not new (I edited this piece in 1995), but my comprehension of it was new; and it explained the recent surge in paid listings as a successful advertising vehicle – the first truly robust commercial exploitation of the Database of Intentions.

So I decided to focus on this idea in book form. I started looking around for folks who understood these ideas better than I did, and I found many – an entire industry of people devoted to search , and a subset of academics, writers, entreprenuers and visionaries engaged in the exploration of this idea and its implications. The goal of my book, then, is to tell the story of this idea, and how it drove the rise of computing, the internet (and the bubble), and where it might be going. Clearly Google will play an important role, but I’m not out to write yet another boring and opportunistic book about today’s hot company. I’m just starting the writing, and reporting continues apace. And if you’ve managed to read all the way down to this point, I hope you will join the conversation.


Related Posts Plugin for WordPress, Blogger...

87 thoughts on “The Database of Intentions

  1. Jackie Cohen says:

    Hi there.
    I came across your blog by clicking through a series of links, starting with Matt’s newsletter. What is going to improve search in a dramatic way is something that sounds mundane at first: the storage market is moving toward interoperability, after eons of pundits bitching about the lack thereof. (One of my recurring freelance gigs that isn’t on newsstands is with an outfit called The Storage Reporter, so I’m right in the trenches on this topic.) Search has suffered from the technical difficulties stemming from the fact that most companies’ data now doubles in volume ANNUALLY (three years max, however). The data doesn’t go away, it just piles up in disparate siloes; CRM and search vendors have been touting products that allegedly pull data from multiple sources, but such software is stymied by the limitations of the hardware architectures underlying them. That’s why the interoperability efforts of the Storage Networking Association, and the new SMI-S specs are such a big deal. Feel free to drop me a line if you want to learn more — but I have a feeling you might be on this already.
    Cheers,
    Jackie Cohen :)

  2. Steve Krause says:

    John,

    For the sake of defintion, let’s say that search engines index digital representations of “things”: a blog entry, an image, a CD review, and so on. In this context, the Database of Intentions (as I understand your explanation) comprises an unprecedented view of how people relate to things, as well as how things relate to each other. Maybe it also includes derivatives like how people relate to other people due to their common interest in certain things.

    If I’ve got your definition right, why aren’t the players with the highest potential leverage not the big search engines but rather the big ISPs? Set aside who has so far played the game better–Google clearly wins that–and consider what happens if a big ISP gets serious about building a Database of Intentions. We’ll take AOL as an example.

    AOL can track every clickstream by each of its millions of users, wherever they go on the Web. That includes whatever they do on search engines: the words they search for, the results they get back. Thus, AOL has exactly the same search-behavior information as the search engines themselves, except AOL has that information across *all* search engines. On top of that, AOL sees what a user does after he/she clicks to a result, which has massive implications for the fidelity of the database.

    So if the point is to generate information that drives better knowledge about “who we are” and–of particular commercial interest–“what we want,” then ISPs have proximity to far better raw materials. If the technical and privacy issues of mining those raw materials can be managed, duplicating any other bits of the search engines’ infrastructure (like PageRank-style link analysis) should be possible.

    Of course, the question of “big ISP” versus “big search engine” is messy because companies like AOL, Microsoft, and Yahoo all have anti-Google internal search efforts, and those companies also own or have close partnerships with ISP units. But that just says the ingredients are already at hand; it’s a question of who puts them together first.

    –Steve

  3. sounds a bit like a godzilla-scale version of something out of Microsoft Research’s Social Computing Group (http://www.research.microsoft.com/scg/)

  4. Euphrosyne says:

    As a quick comment to Steve, your comments appear to rely on the idea of a single user per ISP account for its power. What about corporate ISP accounts that account for a huge portion of data? My office has a thousand different people using search engines on the same line, and IP addresses vary daily, so aggregation can’t really be done at that level.

    And just an FYI to John: people use search engines in a lot of different ways that could easily confound your goals (without ever giving evidence of the inaccuracy of your results). As an example: I rely on search engines to provide me with certain links, rather than bookmarking them. One reason is that I use many different computers and have no single bookmark file, but the reason is unimportant. My point is that the links I truly use/need/like don’t go through search engines because I have them bookmarked and/or know them by heart. On the other hand, for a site I reference once every other week, I may use Google to look up every time, because I know how and where it appears on the results list. Thus, the information I use *less* often is actually much more highly represented in search engines than the pages I use all the time…

    I guess in general I am wary of notions that aspire to “explain it all” and your choice of wording (Database of Intentions) sets off that alarm in my head. Examine your premises very, very carefully.

  5. Nick Douglas says:

    Doesn’t building a Database of Intentions violate user rights? Marketers would jump on this like vultures on carrion, and that would encourage the database managers to clamp down on the information and auction it off. This widens the information (and thus power) gap between marketers and users.

  6. My thought is that the DBoI is already “built”- it’s how it gets used that’s the interesting question. It’s an astounding artifact, from an academic standpoint, and also quite a haystack for the spooks of the world. I agree, Euphrosyne, that one idea can’t “explain it all.” Bias exists, results can be misinterpreted. But the fact is, this thing exists. And that alone means it has power, and is worth examining.

  7. Slav Kandyba says:

    John,
    I came across this web log following your link from an e-mail. I’m a tech biz reporter at the San Fernando Valley Business Journal, who thought he would venture into news and investigative stuff (still will probably), but some days I reminisce over my younger years … reading Jules Verne like there’s no tomorrow. I read Michael Crichton all the time too. To summarize (here’s my lede) I simply am a science fiction addict-cum-journalist, although I really want to just be a journalistic jack-of-all-trades.

    Anyway, just thought I’d drop a note. The Google book idea is brilliant, and I hope it materializes.

  8. The DBofI is a good idea. But it relies primarily on the fact that people’s online footprints/mindprints are more easily traceable than their offline counterparts, and the offline ones are more telling.

  9. Jay says:

    Perhaps the Zeitgeist info (popular searches) on Google tells us nothing more than just that – Pop Culture trends. It does not really reveal the common man’s thought. Let me build an analog for a search here. If I was curious about Einstein, and I was in a local library, I can borrow just a few books, not all, as I have limited time. The felllow next to me would probably not be there for the same thing, but I can bet that we are probably both aware of who was running for president, and if there was a Time magazine nearby we both may sneak a peek into it, but our real search intent and purpose is each separate and unique (not about the election), of which will not get a tally or popular score in some rating. That intent gets lost when the metric is through counts, which is really on a superficial level.

  10. Jay says:

    Another question is, who drives what. If I see that Britney Spears is on top of the list, it gets me curious too, so it’s the chicken before the egg question. What people get curiouis about is certainly driven by outside (Marketing) influences, and reveals more about the marketing engine rather than the intrinsic individual. It’s just an electronic tally of what’s popular, plain and simple. Far from intention, which is self-driven.

  11. Hi John
    Great to see you on 60 min. last night speak on the Corporate culture at Google- really enjoyed your persepectives on the future of search -keep up the great work.

  12. hangton says:

    I guess in general I am wary of notions that aspire to “explain it all” and your choice of wording (Database of Intentions) sets off that alarm in my head. Examine your premises very, very carefully

  13. Lissa Klein says:

    Corporate Information per Jackie Cohen ” most companies’ data now doubles in volume ANNUALLY”. Some of it really great data of potential long-term. Legislation like Sarbanes-Oxley (SOX) was lead to deliberate obfuscation. Removal of search functionality, new layers of password access. In my experience the information piling up in all that expensive storage media might as well be stored in vaults on the moon for all the good it is doing the corporations.

  14. susan says:

    I guess in general I am wary of notions that aspire to “explain it all” and your choice of wording (Database of Intentions) sets off that alarm in my head. Examine your premises very, very carefully

  15. manuel says:

    Hello, if I hear of chickens me badly the bird flu to have we of China get now the Chinese to have from the USA google gotten. Unfortunately neither of them gives a vaccine, unfortunate or for. I would like to mention still one the bird flu to have we soon in the grasp

  16. techno says:

    I may use Google to look up every time, because I know how and where it appears on the results list

  17. Melanie says:

    Hello, Great to see you on 60 min. last night speak on the Corporate culture at Google- really enjoyed your persepectives on the future of search -keep up the great work.

  18. berker says:

    Google is a mighty tool. So they made the price. Lokk a the other searchtools. I my website statistics I earn my traffic 99% from google.

  19. Very interesting post…the database of intentions is a great idea ;-)

  20. Ipod guide says:

    Marketers would jump on this like vultures on carrion, and that would encourage the database managers to clamp down on the information and auction it off. This widens the information (and thus power) gap between marketers and users.

  21. John Bags says:

    Yes, very interesting post. I have 90% traffic from Google.

  22. Volker says:

    Currently in the german TV we see telecasts of the dominant influence of Google.

  23. Schwarzenbek says:

    Google is the number one in the matter of searching machines in Germany. Simply everybody uses it

  24. I get all my visitors from google.
    Thanks for it.

  25. But I hope page rank updates will be more.

  26. Marcel says:

    there you are right perfectly

  27. Doesn’t building a Database of Intentions violate user rights? Marketers would jump on this like vultures on carrion, and that would encourage the database managers to clamp down on the information and auction it off. This widens the information (and thus power) gap between marketers and users.

  28. The google search is the number one however all tries by PAGE ranking front seats to reserve. Sides only for the google search machines are built are zugemuellt there, not the InterNet

  29. pet trixie says:

    Hi John,
    this widens the information gap between marketers and users. Examine your premises very, very carefully and very trixie.

  30. Oldude59 says:

    What I find most interesting in the original article was the cultural anthropological focus stated. The tecnology and who controls or not relates simply for me the mechanics of memes. The reality is that I’m glad you’ve done the work you have and the idea of tracking, managing such an important subject I found here.

    Keep it goin – If the book is available I buy it.

  31. Good luck with the book, and please let us know when will it can be read online. At least a short abstract.
    Concerning the Database of intentions, this actually is what google does, by incorporating the history of all your search activity, chats, calendar, websites, gmail, and all the google services (including the local history, on the personal computer). What would then be the result? Well, it can be a huge “thing” that knows everyone, that can serve someone exactly what he is looking for (based on the previous searches and history), so… we’ll just sit and relax, search and order things…
    I am still wondering though if there could be someone using this database… with some evil intentions. What if this comes up?

  32. Susi says:

    Google ist die Nummer eins unter den Suchmaschinen und vergibt auch die Page Ranking, wonach eine Seite bewertet wird. Ob es nun nur an den Keywords und backlinks liegt wie eine Seite bewertet wird ist ein Geheimnis von Google

  33. Corporate Information per Jackie Cohen ” most companies’ data now doubles in volume ANNUALLY”. Some of it really great data of potential long-term. Legislation like Sarbanes-Oxley (SOX) was lead to deliberate obfuscation. Removal of search functionality, new layers of password access. In my experience the information piling up in all that expensive storage media might as well be stored in vaults on the moon for all the good it is doing the corporations.

  34. Mastercrash says:

    Google is the number 1. Over 90% from my traffic are from google

  35. Robert says:

    The traffic on my websites is 95% from Google and the rest dividing Yahoo and MSN.

  36. Hi John,
    this widens the information gap between marketers and users. Examine your premises very, very carefully and very trixie.

  37. timm says:

    Hi John,

    some interesting thoughts …
    but how google handle the pages of chinese sites for the chinese authorities ?
    Maybe is the slogan from google “filter information to make money” and not “don’t be evil” …

    regards
    timm

  38. Tom says:

    Very interesting percents about Google.
    My experience last month: 6% Yahoo and Google 87%.

  39. Logodesign says:

    I am getting about 92% of my visitors by google. i would prefer when there would be more other strong engines.

  40. shaggy says:

    Another question is, who drives what. If I see that Britney Spears is on top of the list, it gets me curious too, so it’s the chicken before the egg question. What people get curiouis about is certainly driven by outside (Marketing) influences, and reveals more about the marketing engine rather than the intrinsic individual.link eklesite eklechat
    It’s just an electronic tally of what’s popular, plain and simple. Far from intention, which is self-driven.

  41. Lastminute says:

    great information and really scary the omnipotent google awareness! I am getting about 80% of my visitors by google. i would prefer when there would be more other strong engines to have fair competion and more traffic allover ;-)

  42. nivid says:

    there should be regular updates on this post as it still holds good,

  43. Sheri says:

    Okay – I’m a little slow. Just getting to reading “The Search”. Absolutely fascinating that I can find who is linking to my side by just typing in link: and then my website.

  44. L says:

    So many previous commenters have said “95% of my traffic comes from Google”, etc, but happens to your traffic when Google decides to tweak its algorithms? Like you said in ‘Search’, John, the Google Dance is always a concern for small business owners (like the big feet shoe seller). What does Google care if your website listing doesn’t show up in a results list, if you’re not spending money for paid placement or paid inclusion? What’s to stop them from making your site unreachable by the masses? (Albeit a rhetorical question)….NOTHING!
    Google has far too much control over what people see on the internet, and for that, to me, their corporate motto fails. Google IS evil.

  45. Toni Hambilton says:

    hi John,
    My question is not about Google, but about your MT install. I notice you have the latest version (3.34) and have FastCGI implemented. I’m wondering if you followed the instructions from Six Apart and if you had any errors, specifically with the StyleCatcher and Widget plugins. The missing Google API was another error I had listed several times in my site’s error file.
    Just curious. I’ve had so many problems that I nearly gave up. I’ve ended up disabling the offending plugins and the errors have quietened, but that isn’t the ideal scenario.

  46. daso4 says:

    My experiences: over 90% visitors via google, but I wish there were more engines.

  47. i gues livesearch will get some fooks soon. now it has %2 in my web sites

  48. in general I am wary of this, its a big information gap between marketers and users.

    I rely on search engines to provide me with certain links, rather than bookmarking them.

    Great Article, lots of information

  49. fatih says:

    I may use Google to look up every time, because I know how and where it appears on the results list

  50. Dr. Anon Y. Mous says:

    Are you people ASLEEP! George Santayana said [what] in the 19th century about “forgetting the past”? Go to my web site address link:

    relinkz dot com/googlesecrets

    for more information on “Google Secrets”.

    [Special Linking Instructions to the Internet Impaired] Copy & paste the link into your browser’s URL address field, change the word “dot” to a . period, take out all spaces, then hit [Enter] on your keyboard. The rest is academic…

  51. taking over intent,

    taking over education,

    taking over keywords,

    etc.

    Immortality in Paradigmality, accelerating

    rrh

  52. NosyJoe says:

    Hi John,

    I am from Europe but had bought your book ‘The Search’, I won’t lie you, just read the half of the book so far… quite busy! but your book in one way or another was my inspiration to keep on thinking of and working further on the search arena … amazing, no more comments… how from such a small start up Google managed to turn into a huge worldwide recognized brand …

    I am currently part of a very exciting and interesting small social search engine called http://NosyJoe.com :-) still in closed beta but if you have a couple of free minutes, someday, could you please stop by for a moment and take a quick look at it and give your professional opinion about at info at nosyjoe.com, I’d appreciate. It has recently been mentioned in New York Times at: http://www.nytimes.com/2007/06/24/business/yourmoney/24digi.html

  53. chim says:

    Great social search engine! We have also started a search engine on Gina-Wild.Com

    The engine is presented by the german moviestar
    Gina Wild

    Please take also a quick look at it and give your professional opinion. Thank you very much. Greetings from Germany.

  54. Speaking of intentions, check out this blog if you want to integrate an autocomplete application to your site/blog that displays contextual ads as visitors perform search queries…

    http://www.predictad.blogspot.com/

  55. rollo says:

    Very interesting post. We have 70% traffic from Google in Germany and 20% from Linkmaps. Our Website ist rollo-laden.de

  56. nmw says:

    can you tell how much of that traffic comes (for example) from marktplatz-mittelstand.de? (is that part of the remaining 10%?) What do you mean when you say “Linkmaps”?

  57. Toner says:

    I am currently part of a very exciting and interesting small social search engine called Toner :-) still in closed beta but if you have a couple of free minutes, someday, could you please stop by for a moment and take a quick look at it and give your professional opinion about at info at nosyjoe.com, I’d appreciate.

  58. Rollo says:

    I am from Europe but had bought your book ‘The Search’, I won’t lie you, just read the half of the book so rollo quite busy! but your book in one way or another was my inspiration to keep on thinking of and working further on the search arena … amazing, no more comments…

  59. george says:

    Very interesting percents about Google.
    My experience last month: 6% Yahoo and Google 87%.

  60. nadia says:

    I am getting about 92% of my visitors by google. i would prefer when there would be more other strong engines.

  61. sergej says:

    meine seite-also nicht hier schimpen ja-wenn nicht erwünscht löschen

  62. SEO says:

    Search is HUGE!

  63. John, I recently read your book ‘The Search’ and I found it very powerful and intersting. I would say it is a good primer for anyone interested in Internet Business. It also depicts an interesting journey about how search is pushing technology towards the dream of artificial intelligence, while thousands of small businesses thrive and die by the whims of search engine algorithms.

  64. Tommy Bren. says:

    We run casino-gambling-174.com and 97% of alle traffic commes from Google. What a mess, we are overdependent on Google.

  65. Just finished your book “Google” and found it fascinating. Many of your 2005/6 comments and predictions have indeed come to fruition. Our newest Tivo forced us to install an update that apparently is nothing more than a refined search based behavioral ad targeting upgrade.

    Thanks for the in depth analysis and behind the scenes look at a company and industry that has unarguably shifted the business model on which successful companies and industries have been run for 100’s of years.

    Drew

  66. proxy sites says:

    Doesn’t building a Database of Intentions violate user rights?

    Marketers would jump on this like vultures on carrion, and that would encourage the database managers to clamp down on the information and auction it off.

    This widens the information (and thus power) gap between marketers and users.

  67. John,

    My group is doing research on studying databases of intentions, aiming to find out to what extent they contain knowledge about common human goals. We found that while users express a broad variety of goals in search query logs, the goals are topically biased compared to other knowledge bases of human intentions. The following article was published at this years K-CAP’09 conference – it might be of interest to readers of your blog:

    M. Strohmaier, M. Kroell, Studying Databases of Intentions: Do Search Query Logs Capture Knowledge about Common Human Goals?, The Fifth International Conference on Knowledge Capture (K-CAP’09), Sep 1-4, Redondo Beach, California, USA, 2009.

    http://kmi.tugraz.at/staff/markus/documents/2009_K-CAP_Goal_Acquisition_from_Search_Query_Logs.pdf

    Markus

  68. Sebastian says:

    Thanks for the great article about search engines. I use the new german rock-music search engine.
    greetings from germany
    sebastian

  69. Gah John the (mostly interesting) comments on this post would do wonders with spam removal.

    Interesting idea, just found your blog, will try to pop by more.

  70. Tony says:

    I read your blog and just by reading what you said I have a very different view on the internet and the core value of it and how much of a difference it has made to peoples lives.

    Thank you for opening my eyes.

  71. “Marketers would jump on this like vultures on carrion”

    I don’t understand these privacy fanatics. So what if a marketer knows I’ve got three kids and sends me ads about people movers? I’m a big boy. I can deal with this. It’s better than sending me ads about make-up or sanitary protection.

    My really private stuff I keep to myself. I don’t care if Google deduces my demographic profile.

    There are much more important things to worry about than whether Google knows that I like red wine.

  72. I loved your statement
    “The aggregate results of every search ever entered, every result list ever tendered, and every path taken as a result. It lives in many places, but three or four places in particular hold a massive amount of this data (ie MSN, Google, and Yahoo)”

    It is so very true in every way. I was speaking with a group of my co-worker’s the other day about how most of my knowldge has come from google, yahoo, etc.
    I would honestly be lost without those tools that are available to us now days.

    GREAT ARTICLE :)

  73. Nailstk says:

    very good, thank you

  74. good post! keep on going.

  75. Laismasters says:

    Any way to know when yoy are finishing your book and have it ready for us to take a sneak peak?

  76. marc says:

    The Database of Intentions is simply… see more

    http://www.livecams3000.com

  77. fred says:

    nice and it is really consider as a product…
    Articwook Bookmarks

  78. youtube klicks kaufenasdasd says:

    To succeed in life,
    you need two things: ignorance and confidence.

    case study

  79. Samantha Smith says:

    Amazing website for write my essay http://write-my-essay-4me.com/ services

  80. Shamoon Bahader says:

    Anybody can qualify
    to receive an iHerb coupon and once you have a coupon, it is valid forever
    and you can redeem it anytime afterwards– no matter how much time has elapsed
    since your original purchase.

    http://skandinavisk-dynekompagni.dk/7-plaider