The Search Papers: Defining Intent

I've just finished reading A Taxonomy of Web Search by Andrei Broder, written largely while the author was CTO of Alta Vista (and using AV query data), and published after he moved to IBM Research in 2001.

The paper has a trove of references to other papers, which is good for my work, and it has a singular thesis: that all web searches are not equal. Broder sets out to dispel the notion that all searches are "informational" in nature. He instead maintains that many are "transactional" or "navigational" in nature. These two seemingly obvious categories are in fact relatively new to the academic field of Information Retrieval (IR), which developed largely in the context of large islands of data (ie, in the 70s/80s), rather than in the web era.

What I like about this paper is the use of the word "intent" - which over the years I've come to use quite a bit (see my last column on video advertising over the internet, in which I rant once again on "intent over content", or my post on The Database of Intentions). Intent is behind every kind of search, Broder says, but "there is no assumption ... that this intent can be inferred with any certitude from the query." Ay, there's the rub....To get to that intent, Broder employed a short survey on the site.

A few fun facts from Broder's analysis of response and related log data:
- nearly 15% of searchers wish for "a good collection of links on a subject" as opposed to "a good document."
- 12% of queries in the log data used were sexual in nature
- nearly 25% of searchers were looking for "a specific website that I already had in mind."
- An estimated 36% of searchers were looking for transactional information - what Broder calls "the intent to perform some web-mediated activity."

Broder concludes that the next generation of search engines will need to take into account this new taxonomy of intent - transactions, navigation, as well as informational. Given that this paper was published in late 2001, it's interesting to see how the major engines already are on that path - with Yahoo's focus on shopping being one of the best examples.

• Posted by John Battelle at 11:48 AM
• Permalink
• Comments (6)
• TrackBacks (2)

"eBay Is Search!"

A very kind reader (I'll buy the drinks next time) has forwarded me eBay's recent analyst day presentation (from earlier in the month). This reader's comment: "The focus on search was VERY new...analysts were trying to figure out the significance." Indeed, one of the slides declares: "eBay is Search!"

A couple of things come to mind when reviewing the slides (there were more than 300,covering the entire business, both US and int'l). First, eBay knows that the faster and more relevant they can make their internal search, the better their margins. They understand that they have a marketplace of people with very specific intents, looking for very specific things. The easier eBay can make it for folks to find what they want (whether it's a bid, an item, or a comparison), the better their bottom line looks. Toward this end, eBay has built its own internal search engine called Voyager, which is optimized for eBay users. One of the slides in the analyst presentation boasts: "We are world class at analyzing our user base."

Second and possibly more important is the role of search in customer acquisition. eBay has a world class IT solution in place to monitor tens of thousands of paid keywords across the web, each with its own P&L and analytics. I can't confirm this, but I would not be surprised if eBay is Google's largest customer, something that probably makes both companies uncomfortable, because each can analyze the other's data and mine it for competitive edge. In the presentation eBay also notes the power of natural or algorithmic search (the "pure" results) - the company says it is revamping its entire site to optimize for natural search. Now that's quite a statement. Again, major thanks to my source for this information.

• Posted by John Battelle at 10:51 AM
• Permalink
• Comments (6)

First Look at USBancorp Online Ad Projections: Next Year Projected to Match Best of Boom

Among the analysts still covering the internet after the great wipe out of 2001, Safa Rashtchy, of Piper Jaffray, has received the most notice as an early and ardent supporter of search. His company will soon come out with a new report forecasting online advertising revenues (Safa is credited with pegging the paid search segment of the market at a widely reported figure of nearly $7 billion by 2007). In his recent newsletter, which summarizes a conference Piper hosted on online advertising, Safa predicts revenues will sharply accelerate, to more than $15 billion by 2008. Also, he predicts that online ad revenues will match 2000's number of $8.1 billion - the height of the boom - by next year. My guess: he's wrong - when the counting's done, 2004 will beat 2000 by a comfortable margin.

• Posted by John Battelle at 8:50 PM
• Permalink
• Comments (0)

A Couple of Cultural Commentaries

I'm offline most of today, but enough folks have passed me these two sites as interesting cultural outgrowths of search that I'll post them here for your review. The first, GoogleRace, plugs into the Google API and ranks each candidate by keyword searches. Some of the tops searches are "large penis" and "George Bush is going to lose". (Thanks Kenny..)

Googlehouse, on the other hand, is more understated. This is an attempt to make a cultural commentary through images - the site polls the Google Image search database and pulls up images that fit various parts of an imaginary house. Click around a bit, if you've got the time. This kind of stuff usually pushes my MEGO button, but...give it a looksee. (Thanks Tim...)

And another note. Thanks to this blog, many readers are now starting to ping me with interesting stuff, tidbits, even insights for my book. I really appreciate and encourage this. I'm at jbat@battellemedia.com, keep those cards and letters coming.

• Posted by John Battelle at 8:13 AM
• Permalink
• Comments (0)
• TrackBacks (2)

The Search Papers: Europe Vs. U.S. Search Patterns

So I printed out three papers suggested by Gary Price in this post. I read the third one first, and didn't find it earth shattering, though there were a few interesting tidbits. The paper is titled: "U.S. Versus European Web Searching Trends" by Amanda Spink and Bernard Jansen (Penn St. Univ) and Seda Ozmutlu & Huseyin C. Ozmutlu (Uludag University). Basic conclusions: US searchers tend to use fewer words in queries, and tended to have shorter search sessions overall. Also, European users tend to look at more query results, compared with US searchers, who were vieweing fewer results per query. (This buttresses the stereotype that US citizens are more impatient and less deliberative than their European counterparts).
Also consistent with stereotype was a comparison of general topic categories searched for by each group. For US searchers, the #1 topic, with nearly 25% of the overall searches, was "Commerce, travel, employment, or economy." That category was # 3 for European searchers, with only 12.3% of the searches. European's #1 category was "People Places and Things." Also, it seems that Europe (recall this was in 2001) was still on a learning curve for tech, as the #2 search category was "Computers or the Internet." That term was #4 for the US during the same period. Also telling: European searchers were more than 4 times more likley to look for for "Performing or Fine Arts" than US users, and not surprisingly, "Sex or Pornography" was two places higher on the European list, coming in at #4.
The study goes on to conclude, though not very forcefully, that there are noticeable differences between US and European searchers, but the authors don't claim it's necessarily a cultural thing, it may well be the distinction in the engines themselves, as much as anything. This study left me wanting more, and happy they have continued this kind of work. (I'll be reviewing this latest find soon.)

• Posted by John Battelle at 2:16 PM
• Permalink
• Comments (0)

Looksmart Launches FindArticles...So Far I've Found A Lot of Press Releases

Looksmart today announced a major update to FindArticles, a great idea whose execution so far I can't quite endorse. The service - which has its own tab on the Looksmart site - has "over 3.5 million articles from over 700 publications." While I've not explored it fully, a search for various folks who I know are subjects of major magazine pieces turns up a boatload of BusinessWire-type press releases. Not sure how that matches up with expectations of "articles from 700 publications..." but I hope they clean it up, because it could be a great service.

• Posted by John Battelle at 9:35 AM
• Permalink
• Comments (1)

Yellow Pages - All Growth Is Digital

A new study says the Yellow Pages will continue to grow but...there's a catch. I've taken to quoting the current size of the Yellow Pages market - $26 billion or so - as proof there's a lot of growth left in search. After all, if non-local search is about $2-3 billion this year, and is poised to undermine the local Yellow Pages market due to its nascent push into local search, that's a rather large market to grow into. But a study by the Kelsey Group, made public by eMarketer today, says that the Yellow Pages themselves will also grow in the next five years, to $36 billion. However, a full 23% of that 2008 revenue - about $10 billion - will be "digital directory" Yellow Pages - yup, paid local search.
OK, so the math is: Yellow Pages is a $26 billion biz now, growing to $36 billion by 2008, a $10 billion increase. Local Yellow Pages "Digital Directory" is - well not much now (as far as I can tell it's not even in the current $26 billion number), but it will be $10 billion by 2008. Given that the local search piece of the business is quite small today, and the online yellow pages is probably no more than $300 million, if that, what this study really seems to say is that the lion's share of ALL growth in Yellow Pages will be digital. In other words, there will NO growth in the Yellow Pages market, if it weren't for online search. Now that I can buy into.

• Posted by John Battelle at 8:00 AM
• Permalink
• Comments (2)

More on Same...

The Boston Globe also has a "worm turns" piece here, and one of the key sources in the piece takes offense at how his quotes were used...Another source (Dave Winer) also feels wronged... I find this kind of journalism, where reporters try so hard to make quotes fit into a pre-concieved notion of how a story should play, the most irritating thing in business reporting today. In this case, it's not the Google Is Wonderful Look At All The Lava Lamps angle, it's the opposite: Google Is Making Enemies And Is Too Big For Its Britches. Neither is right. Ugh.

• Posted by John Battelle at 12:53 PM
• Permalink
• Comments (0)

Must...Put...Google...On...The Cover

Fortune this week proves that the worm can turn in the mainstream media's coverage of all stories, even one that for years has proven my predictions wrong. At least 18 months ago I was cluck-clucking to the communications honchos at Google that they should "beware the backlash." They were getting too much good press, and at some point the media always wakes up and eats its young. But Google enjoyed the longest free ride I've seen in recent history - even when those same PR honchos essentially went dark and refused to give anyone much access. The same story kept getting written, again, and again, and again....
This Fortune piece isn't a hit, but it does rehash the negative bits with at least as much ardor as the positive ones. The piece sets up like this: At Fortune we did some *real* reporting (the implication being that all the stories before were stage managed affairs), and we found out that the company that everyone's been lauding for the past three years is...complicated, contradictory, and not exactly perfect. Not a rocket science conclusion, but it manages to make the company seem a bit more human. The piece states some very old stuff as fresh (that the company has recently grown arrogant - this is new?), has tidbits of news known only to a very few insiders (that Bill Joy and Google flirted but eventually did not come to terms), and a couple off the record sources who are also investors saying stuff like: "Google has a lot of momentum, but its current position is probably not defensible." (Yahoo holds 5% of Google...).
The piece is a fine round up of where things stand, but I can't help feeling a bit empty - so much of the story was stuff we already knew, but had to be in there because of Fortune's large readership, not all of which could be assumed to be avid followers of all things Google. That's the problem with mainstream media coverage - it has to speak to everyone. The story lacked an analysis of business models, of the industry, and any deep discussion of the larger phenomenon Google represents. And the negative bits, while something of a first, had a twinge of gossip and/or sour grapes to them. Overall, I'm not sure this piece moved my view of Google one way or another. How about you all?

• Posted by John Battelle at 10:17 AM
• Permalink
• Comments (1)
• TrackBacks (1)

Monoculture, Innovation, and the Ivory Tower: The Search Papers

I'm starting to read more academic papers, research presented by various professors and the like (including some folks who are technologists working in the industry). One of the things I find fascinating about the search business is how quickly it's turned from an academic pursuit - with all the implications of open, non-commercial sharing of findings - to one driven by clusters of high-powered nerds hitched to a particular corporation's R&D machine. I've been asking around on this idea and found most landlocked geeks agree - Nutch aside, a good percentage of search research has by and large been silo'd - many of the best minds are at Google, Microsoft, Yahoo, and a few others. I'll wager that between the 500 or so top engineers at these corporations, there ain't a hell of a lot of sharing going on. Robust peer review between bare-knuckle competitors? Prolly not.
But it was not always so. Recall that both Yahoo and Google came out of the Stanford CS department, for the most part. Same for Excite (Joe and Graham). Lycos was midwived by CMU, and out of Berkeley came Inktomi. Anyway, you get the picture. A lot of innovation came out of the publish-and-peer-review culture of the university setting, and many of the folks who drove that culture are suited up, so to speak, in one corporate silo or another.
In any case, before they joined up, many of them wrote wonderful academic papers they shared with all in the name of progress (some still do). And there are still plenty of great academic researchers banging away on the database of intentions, though certainly they don't benefit from owning their own slice of it like the majors do (many borrow data from the majors and perform analysis on that). So as I work on the book, I'll be posting reviews of some of the papers I read - the interesting bits, so to speak. I'll title each entry "The Search Papers: Cute Name Here" for ease of use, or more likely, as a clear caveat that discussion of academic Mumbo-J will follow. The first will be out in the next day or so, a wonderfully predictable little study (found via Gary Price, of course) comparing European and US search patterns from 2001 FAST and Excite (pre-Chapter 11) data. Hope you like it.

• Posted by John Battelle at 8:15 PM
• Permalink
• Comments (3)
• TrackBacks (1)

This Month's Column Is Up

Here's the 2.0 link, but you have to be a subscriber to get past the first page. So here it is as a pdf file (caveat: it will download when you click). I'm still working on putting my Biz 2.0 columns on this site in a more reader-friendly fashion...This month's points to the idea (not really new now, but it felt more so six weeks ago when I wrote it) that the geeks behind the best innovations on the net are starting to get excited again over interesting problems that have Very Cool solutions, after two or so years of nuclear winter. Boing Boing points to another columnist who's recently seen the same trend.

• Posted by John Battelle at 7:21 AM
• Permalink
• Comments (1)
• TrackBacks (1)

Wharton States the Obvious, Hopes For The Press

OK, this is just too much. A major, respected school of business issuing what amounts to a puff piece/PR gloss on Google, quoting professors from various branches of the school (legal, marketing, etc), who manage to say just about nothing new. They don't even answer the question raised by the article's headline ("What is Google Worth?") What on earth are they doing? Why, riding the Google PR machine, of course. "Hey, here's an idea! Let's round up all our professors, have them say smart-sounding things about Google, then take credit for regurgitating old ideas!"
Hurray for bandwagonism. I'd wager the folks at Google are a bit embarrassed by this.
This piece at Cnet shows the school has been paying attention to the company for some time. Come on UPenn - if you are going to pick up the discussion, add to it at least, especially as an academic institution.

• Posted by John Battelle at 10:28 AM
• Permalink
• Comments (0)

Lanier (yes, Jaron) on Software, Biology, Complexity, and ...

I am a subscriber to John Brockman's Edge, which is sometimes (well, for me as a rank average mind, more than sometimes) insufferable for its - well, its intellectual preening (it's basically conversations with folks who are scary smart) and its steadfast unwillingness to make itself reasonably approachable (its last missive, which comes by email, weighed in at 8900 words and seems entirely unedited, for example). But, at times I will plow through the thing, and on a recent occasion, I did just that when along came an interview with none other than Jaron Lanier, Wired staple way back in the age and a constant reminder of how fun and freaky those days were. (more below)

Jaron's interview is entitled "WHY GORDIAN SOFTWARE HAS CONVINCED ME TO BELIEVE IN THE REALITY OF CATS AND APPLES." See, back when I was a fresh faced ME at Wired, I suffered this kind of stuff earnestly, and really focused and tried to understand what the hell folks like Lanier were talking about. The world hung in the balance, I felt, and it was my job to translate. I haven't had that urge for a while...

But this piece, dense though it is, touches on some basic realities behind search - which as most engineers in the field will tell you, is one of the most difficult software problems of all time. Lanier references the halcyon days of programming when Papert could make a perfect little Logo program, or Ivan Sutherland could revolutionize the field of graphics (and all of computing, really) with a little program called Sketchpad. (Makes me think of the gorgeous contours of early Wordperfect compared to that Kilimanjaro of shit now known as Word...) Lanier points to how most mourn that loss of simplicity. It feels a lot like how engineers in search talk about the first crawlers, the first search engines, PageRank back when it was not muddy. Lanier suggests we got it all wrong the first time we tried to create computer science, and we need a new approach to software, based on something he calls "phenotropic investigation." I cannot grok it to the point of making it make sense to you, but I can say this - it makes my Spidey senses tingle. The basic idea is similar to what Kevin Kelly and others have been all up about - a return to more biological approaches to solving problems. Here's an excerpt:

"My engineering concern is to try to think about how to build large systems out of modules that don't suffer as terribly from protocol breakdown as existing designs do. The goal is to have all of the components in the system connect to each other by recognizing and interpreting each other as patterns rather than as followers of a protocol that is vulnerable to catastrophic failures."

OK. The piece is on the Edge site if you want to read it. Why do you care? Well, it does mention Google. Sure, it's one of these pieces that makes my head hurt. But...I did find it interesting when he said this:

"Of all the things you can spend a lot of money on, the only things you expect to fail frequently are software and medicine. That's not a coincidence, since they are the two most complex technologies we try to make as a society. "
Funny, then, that I maintain two blogs, one on the impact of search (one of the biggest software problems in the world ), and one on health (Tonic, not yet fully public). I guess am a sucker for punishment. Onwards...

• Posted by John Battelle at 8:46 PM
• Permalink
• Comments (0)

Carat Strengthens Search Buying Department

Adweek reports Carat Interactive has appointed Ron Belanger vice president of search engine marketing.
Excerpt: "In the newly created role, he will guide the strategic vision of the interactive shop's 10-person search-engine marketing practice, which handles paid placement, paid inclusion and search-engine optimization for clients like Hyatt Hotels. He will report to Carat Interactive evp, managing director Alan Osetek and will be based at the Boston headquarters."
This is one more sign that Search has come of age in the media world. (via MarketingWonk)

• Posted by John Battelle at 10:21 AM
• Permalink
• Comments (0)

Cato Weighs in on Google-As-Public-Utility Meme

Upshot: they don't like the idea much. Nor do I, but it's not because I hold an unfettered libertarian worldview. I thought the meme of "let's regulate Google like a public utility because it has too much power" was a short lived one, and I'm not sure why Cato brought it up again, but, there you have it. The piece has a lot of links in it and therefore provides a pretty good overview of the "Is Google Too Powerful" concept. Plus the piece even quotes one of my columns, though without direct attribution (it does point to my article hosted and mirrored on some other site...hey...wait a minute! But I guess that's one way to get my stuff out there beyond the walls of the Time Inc. enclave....) It's nice to know folks are reading...

• Posted by John Battelle at 8:47 AM
• Permalink
• Comments (1)

Thoughts on a Day With Yahoo and Feedster

Today I was offline most of the day, as I was working on the book. I met with key folks at Yahoo Search and Scott Rafer , the reasonably new CEO of Feedster (Feedster is a search engine which trolls only RSS feeds. What's an RSS Feed? Read this). Lots of discussion about the present, past and future of search. I left Yahoo impressed with where the company is and how the folks I spoke to approached the problem/opportunity of search. Yahoo has the longest history and the richest resource set of any of the current search players, and they certainly have got serious search religion, something that definitely waxed and waned over the past five or so years. Much talk of the lessons of Google, the looming competition with Microsoft, and good discourse on the best ways forward w/r/t search solutions. Ninj Srinivasan, employee #5 and Editor of Yahoo, has been there for nearly 9 years and is still stoked to come to work every day. That says a lot to me. Funny side note: while I was there, 1600 Yahoo employees and various friends went out into the parking lots and broke the world record for simultaneous yodeling. I'm not kidding...

Meanwhile, Scott at Feedster reminded me that PageRank was created back in a time when making links was pretty hard to do, and therefore a scarce resource, so one could reasonably trust that the links were authoritative. But, he supports the idea (as do others, see here and here) that blogs have muddied the waters for PageRank to the point of diminishing returns. I am certainly not one with the chops to judge, but I do wonder how this meme is playing out now that it's been in the world for a while? It's not like anyone has come up with anything demonstrably better....
In a related note, Scott mentioned an intriguing development at Feedster that he calls Feedpaper. From his site: "Feedpaper is a dynamic newspaper constructed from RSS feeds around a particular topic." They are playing around with the idea on the site, for examples, check out this one on the Dean campaign and this one on the recent Longhorn developer conference (Longhorn is the new version of Microsoft's interface, due sometime later this decade...). Scott's thesis is that algorithmic search engines do a pretty poor job of aggregating this kind of content for readers interested in a very specific "island" of information. Watch this space, it could get interesting.

• Posted by John Battelle at 9:50 PM
• Permalink
• Comments (0)

Singingfish Bought by AOL

When I was at AdTech way back in June I met with Karen Howe, CEO of Singingfish, an audio/video search engine. I believed then and still believe now that video and audio files will become integral to the grammar of the web, but first we have to solve major search and copyright issues. In any case, I enjoyed meeting her, but left feeling like they had a significant uphill battle - a very small company in a land of giants.

Well, no more. I was messing around on Google news earlier this afternoon and somehow I came across a very odd link from the Puget Sound Business Journal about Singingfish being sold to AOL. Google's summary had details: the excerpt read "has acquired Seattle-based Singingfish Inc., which makes a search engine that scours the Web for audio and video files. Terms of the deal were not disclosed. ... " but when I clicked on the link, it took me to a story with the headline "MagnaDrive raises $3.5 million ". Turns out the Puget Sound Journal had jumped the gun on a embargoed story, and Google News grabbed it before they could swap it out. The news officially breaks tonight at midnight, but ... you saw it here first!

As to what it all means - I find it a bit out of character for AOL to acquire a search engine - and even more so one of this stripe. AOL has formidable serving and personalization technology in house, but it's never really been seen as a leader in search. It outsources that to Google - though it is true that Google does not do multimedia search (Alta Vista and AlltheWeb do). Is this a bid to change all that? Stay tuned, I'll keep you posted.

UPDATE: USA Today has a story on the sale now...

• Posted by John Battelle at 8:04 PM
• Permalink
• Comments (3)

On Bad Approaches To Search

• Posted by John Battelle at 4:25 PM
• Permalink
• Comments (1)

The NYT, Yesterday's News, and Plausible Deniability

In this post on his blog, Dan Gillmor of the SJMN points to the contradiction between "company executives and others" who claimed MSFT approached Google about a buyout (see my earlier post and comments here) and Gates' very clear denial of same earlier this week. Dan points out that Gates is the CEO of a public company, so he can't very well lie about something so material to his stock price. Because of this he implies that Google insiders - such as the VCs who backed the company - are the likely sources of the story.

I'm not so sure. The sourcing in the Times piece seems intentionally non-specific:

"According to company executives and others briefed on the discussions, Microsoft - desperate to capture a slice of the popular and ad-generating search business - approached Google within the last two months to discuss options, including the possibility of a takeover."

Which company does "company executives" modify, MSFT - which is closer to the sourcing, or Google? Who knows?! My guess is the Times kept it vague on purpose, to protect its sources. (more via link below)

Gillmor points out that this kind of press manipulation - where folks float trial balloons to pump a stock, or hype a product, or in this case, to burnish Google's pre-IPO valuation or test an acquisition via the press - is quite common in the Valley. I'd add that the practice is far more widespread in DC, where they ought to change the name of the New York Times to "Senior Administration Officials' Trial Balloon Daily". But I'm not sure I see the same smoke Dan does.

Taking a page from DC, what Gates probably has is plausible deniability. Here's one scenario: Acting on their own, his "people" - say senior execs from MSN on a fishing expedition - contacted folks at Google, Kleiner, Sequoia, or all three, and most likely, folks on the Google side took a meeting. It'd be irresponsible not to. The meetings were inconclusive, and Gates was never told about them. Even if there was not a meeting, MSFT officials know they could test Google's openess to an acquisition simply by leaking an imaginary meeting to the press (or, vice versa - Google insiders could have tested MSFT's openess the same way). In any case, someone (could have been either side or both) tipped the Times, and the story got out. Then Gates was asked by a reporter if he was in talks with Google, and he denied it because his people either never told him about it, or, if he asked them, denied to him they ever discussed acquisition. Believe me, fudging the truth to the CEO - particularly if it covers your ass - is a widespread practice in every company. I imagine it's reached epic proportions at a behemoth like MSFT.

The more troubling issue is whether the tipster(s) - who had their own interests at heart, as do all tipsters - "tricked" the Times into doing a story that was mostly air. But it's only troubling in that it points to the Times' possible lack of judgment about sources, rather than the sources themselves. Should, as Gillmor suggests, the Times call out the lie? That'd be great, I'd love to see that happen - with all sorts of stories. I've found the Times, and most news organizations, extremely short on institutional memory. It'd be a great service to readers if there was some way to institutionalize notation of earlier stories that have been contradicted by new facts (they'd have to print a new edition of the Times every so often just on the topic of the Bush administration). But it's simply not characteristic of newspapers, which still live in the world of RIGHT NOW, where RIGHT NOW is understood to be the point at which type hits paper. Yesterday's news, is, well, yesterday's news. (It is quite typical of blogs, I'd add, to update when new info comes to light). Perhaps this is a feature that Times Digital could take up.

In the end, whatever Google and MSFT do is a story now, even if what they're doing is using the press to communicate. I am sure the Times had credible sources claiming the acquisition talks happened. I'm equally sure that what's in all parties best interests is often not the Truth. Like Dan, I wish we'd get closer to it than we currently do via the mainstream press, but also like Dan, I believe that here in the blogosphere, for those of us who care enough, we can get pretty darn close ourselves.

• Posted by John Battelle at 11:43 AM
• Permalink
• Comments (0)

MSFT jumps into News

Internet News reports:

"Software giant Microsoft is testing its answer to Google's popular news aggregator and search site. "MSN Newsbot", on MSN UK, France, Spain and Italy, signals at least one of Microsoft's intentions as it seeks to build out its own search technology."

This is one area where MSFT has some serious prior chops - MSNBC has been thinking about news online for a long time. Should be interesting to see how it shakes out in this market.

• Posted by John Battelle at 9:55 AM
• Permalink
• Comments (0)

Another Good SE Relationship Chart

This one from Search-This, a SEO/SEM house. The buttons, when clicked, activate colored arrows that chart who supplies who with what...ie Inktomi powers MSN, Google powers AOL, etc. (Thanks to Josh Quittner for the reference...)

• Posted by John Battelle at 12:21 PM
• Permalink
• Comments (0)

Google Code Jam Winner

Once a year Google holds a contest for hardcore search-related coders. This year, the Europeans swept, interestingly. But no mention of what they actually did (as I recall, last year they asked for new features, and the winner made a local search app that Google ended up incorporating into Google Labs)? I'll look around...Aha...Read Slashdot threads here to find out more on the problems the Google coders were solving...MarketingWonk has a round up here of the PR and Marketing implications...

• Posted by John Battelle at 10:43 PM
• Permalink
• Comments (0)

Why We Wish Some Public Information Would Remain Bound To Atoms (The Public Privacy Dilemma)

In the past few months I've gotten a fair number of similar email threads forwarded my way by friends who know I'm writing about search. By the time they've gotten to me, the emails have wound their way fairly well through the six-degrees-of-separation web, with scores if not hundreds of souls cc'd, forwarded, and attached. The subject line usually blares something along the lines of "I can't believe they can do this!" and "Oh My God, Did You Know?"

Here's a sample email, with identifying information deleted:

-------------------
Subject: This is hard to believe, but true, I tried it.

Google has implemented a new feature wherein you can type someone's
telephone number into the search bar and hit enter and then you will be
given a map to their house.
Before forwarding this, I tested it by typing my telephone number in
google.com. My phone number came up, and when I clicked on the MapQuest link, it actually mapped out where I live. Quite scary.
Think about it--if a child, single person, ANYONE gives out his/her phone
number, someone can actually now look it up to find out where he/she
lives. The safety issues are obvious, and alarming. This is not a hoax; Mapquest will put a star on your house on your street.
---------------

I understand the initial reaction of many to this feature (which is not that new). My God, They Know Where I Live! But this fear of a such a simple thing - a reverse directory lookup - bears further contemplation. Fact is, reverse directories are not illegal. But they are also not widely available – usually only cops and reporters had access to them. No more. (more via link below)

Digital information is metastasizing our culture (I use the word neutrally, though I’d argue the author of the email would take the negative interpretation). As it spreads, unexpected challenges arise, and I'm interested in how newly digital information challenges presumptive and rarely voiced social norms. Google's reverse directory lookup illustrates a particularly discomforting expression of this phenomenon, which for now I'll call it the Public Privacy Dilemma.

Our society was built on the enlightened and somewhat thrilling idea of the public’s right to know. Our government is meant to operate more or less in the open. The same is true of our courts: unless a judge determines otherwise, every divorce, murder, felony, misdemeanor, and parking ticket is open to public scrutiny.

But while it’s comforting to know that we, the public, have the right to review this information, it’s also comforting to us to know that we very rarely do. After all, regardless of your prurient desire to know whether your new co-worker has a messy divorce or a DUI in his otherwise well-appointed closet, most of us will not waste an afternoon down in the basement of our county courthouse to find out. The very fact that it’s so much trouble to find such information has, in effect, muted that information. Unless office gossip precedes our new partner in cubicle land, we don’t even think of such questions when introduced.

But what if it were as easy as typing his name into Google? Oh, whoops, I forgot, in many cases, it already is.

And herein lies the rub: What do we do when information we know, by law, should be public, becomes, well….Really Public? As in, First Page of Links When You’re Googled public? Can we take control back (by building high-ranking websites titled “I’m Really Not So Bad, I Swear!")? Should we legislate away the digital, draw the line of what’s “public” at atoms – information on paper, stored in a musty clerk’s office? In fact, the Florida Supreme Court is considering just that question.

Search engines like Google hold up a mirror to our culture, and remind us of both the laws and the mores to which we’ve become accustomed. We’re fine with folks knowing our phone number – we know it’s pretty much public record. But the act of using technology to connect that number to our address, our home, the place we keep most sacred – that’s somehow out of bounds. I’d be interested in your comments on this thought. I find it fascinating the Database of Intentions is forcing us to once again confront one of the most significant and difficult issues a democracy can face – the balance between a citizen’s right to privacy and the public’s right to know.

• Posted by John Battelle at 9:28 PM
• Permalink
• Comments (3)

A Robust Market Ecology

When Google flutters its wings, a typhoon may result in the AdWords ecology - those hundreds of thousands of advertisers who depend on Google for sales via the company's paid listings. So points out CNet's Stephanie Olsen in a good overview of advertiser reaction to Google's most recent shift in its AdWords technology. The complications Olsen reports point to a larger story: the increasing complexity of this shifting market ecology. The question then becomes, can any one company maintain control of this? I don't think so, you need robust competition; the recent defection of Paul Ryan (former CTO of Overture) to MSN will help insure robust competitors for years to come.

• Posted by John Battelle at 10:43 AM
• Permalink
• Comments (0)

Search Drives First-Ever Quarterly Profits at MSN

This has been in the works for sometime, but MSFT formally announced its earnings yesterday, and broke out the MSN unit's numbers for the first time. Headline: It's all about paid search. The highlights (from the MSFT IR site): MSN had 50% quarter-to-quarter advertising revenue growth, total revenue totaled $491 million in the first quarter compared to $427 million in the prior year’s first quarter. MSN Subscription revenue declined $17 million or 6% reflecting a decrease in the number of subscribers. MSN Network services revenue grew $81 million or 51% as a result of growth in paid search and strong general advertising sales across all geographic regions.

• Posted by John Battelle at 6:33 AM
• Permalink
• Comments (0)

Ask Searches For What's Next In Search

Ask Chief Steve Berkowitz (caveat: we ran in the same circles at IDG) gives one of his first interviews (to the CC Times) since being formally named CEO. Steve's a good guy and he has quite a job - being #4 in a three-horse race ain't fun. But he lays out his plans and makes his case in the interview.

Samples:

Q: From whom are you gaining market share?

A: There's Google, Microsoft, Yahoo and AOL. I'm sure we're picking up a little from each. We grew our queries by 33 percent in (the third quarter). We believe the market grew at 10 percent. That means we're gaining share. Being a small player, that's pretty significant for us. I don't think the other people are noticing it. Google is focused on Yahoo, and MSN is focused on Google. They're all focused on each other.

and...

Q: (What's next for consumer search...)

A:....The next generation of search is going to be about getting underneath the needs of the user. One of the interesting things we're testing in research is if someone types in the word "AF," do you mean Air Force or do you mean Abercrombie & Fitch? We're developing technology that will get underneath that and realize that most people mean Abercrombie. Or if I know you're searching about upstate New York, and then you type in the word "apple," you're really not looking for Apple, the computer company. I can tell by the five searches you did previously that you're really looking for an apple orchard, so maybe I can skew your results towards that. It's getting into understanding the behavior of both the session you're in, as well as what the masses are saying are better results. It's about integrating all these different things back into the user experience, so what you get back is a much more perceived, personalized result.

• Posted by John Battelle at 2:25 PM
• Permalink
• Comments (1)

The Database of Intentions

So nothing really new in the news today, I wanted to take a graf or two and explain what I mean by The Database of Intentions, referred to in this post. That way I can use it again and again and just link the phrase to this post. Hey, we love the web, Ted Nelson lives....

The Database of Intentions is an idea central to the book I've been working on for the past year or so, which is tentatively titled "The Search: Business and Culture in the Age of Google" (Penguin/Putnam/Portfolio 2004). As with many in this industry, it all started with the Macintosh. Back in the mid 80s I was an undergraduate in Cultural Antropology, and I had a class - taught by the late Jim Deetz,which focused on the idea of material culture - basically, interpreting the artifacts of everyday life. It took the tools of archaeology - usually taught only in the context of civilizations long dead - and merged them with the tools of Cultural Anthropology, which interpreted living cultures. He encouraged us to see all things modified by man as expressions of culture, and therefore as keys to understanding culture itself. I began to see language, writing, and most everyday things in a new light - as reflecting the culture which created them, and fraught with all kinds of intent, contreversies, politics, relationships. It was a way to pick up current culture and hold it in your hand, make sense of it, read it.
(more via link below)

At the same time I was making extra money beta testing some software on a brand spanking new Mac, vintage 1984. Anthropology and technology merged, and I became convinced that the Mac represented mankind's most sophisticated and important artifact ever - a representation of the plastic mind made visible. (Yeah, college - exhaaaaale - wasn't it great!).

Anyway, the idea that a graphical user interface and, later, a network connecting many GUIs, could provide a medium between many minds drove much of my fascination with reporting on technology, from MacWeek to Wired to The Standard to now. The "Mac as the greatest artifact" meme became one of my standard riffs, from discussions with potential writers at Wired, to discussions with partners at The Standard. The idea that we could better understand ourselves by looking at how we employ technology was and remains the driving force of my work as a journalist.

This is all a long-winded way of saying, I've now come to the conclusion that humankind has created a far more fascinating and important artifact, one that surpasses the Macintosh (and its badly drawn descendant, Windows). And before you roll your eyes and say "Oh God, not the Internet...", no, it's not the Internet. It's something that is a product of the Internet, what I call the Database of Intentions.

The Database of Intentions is simply this: The aggregate results of every search ever entered, every result list ever tendered, and every path taken as a result. It lives in many places, but three or four places in particular hold a massive amount of this data (ie MSN, Google, and Yahoo). This information represents, in aggregate form, a place holder for the intentions of humankind - a massive database of desires, needs, wants, and likes that can be discovered, supoenaed, archived, tracked, and exploited to all sorts of ends. Such a beast has never before existed in the history of culture, but is almost guaranteed to grow exponentially from this day forward. This artifact can tell us extraordinary things about who we are and what we want as a culture. And it has the potential to be abused in equally extraordinary fashion.

Once I grokked this idea (late 2001/early 2002), my head began to hurt. This was A Big Idea, one that certainly was not new (I edited this piece in 1995), but my comprehension of it was new; and it explained the recent surge in paid listings as a successful advertising vehicle - the first truly robust commercial exploitation of the Database of Intentions.

So I decided to focus on this idea in book form. I started looking around for folks who understood these ideas better than I did, and I found many - an entire industry of people devoted to search , and a subset of academics, writers, entreprenuers and visionaries engaged in the exploration of this idea and its implications. The goal of my book, then, is to tell the story of this idea, and how it drove the rise of computing, the internet (and the bubble), and where it might be going. Clearly Google will play an important role, but I'm not out to write yet another boring and opportunistic book about today's hot company. I'm just starting the writing, and reporting continues apace. And if you've managed to read all the way down to this point, I hope you will join the conversation.

• Posted by John Battelle at 11:09 AM
• Permalink
• Comments (67)
• TrackBacks (18)

Online Ad Sales Booming (The Standard)

It gives me something of a thrill to reference The Standard in an article about booming online ad sales, and give credit to Matt McAllister, who runs Infoworld's site and took over thestandard.com as a sidelight, as IDG was about to shut it down along with all of IDG.net. The archive is still not up, but Matt vows it will be, and the stories are all headline retreads from other IDG publications, but, there's still a pulse there. Also, it's really poignant to see contextual ads on the site, after all the dreaming I did of CRM doing - far too expensively - what contextual ads essentially can do now. Thanks Matt!

• Posted by John Battelle at 10:43 AM
• Permalink
• Comments (2)
• TrackBacks (2)

Diller, Cheshire Cat of Local Search

Via MarketingWonk, I saw this short blurb in Crain's NY business which points to Barry Diller's increasing show of muscle in the local search market. He's got Citysearch and various other localized online businesses, and Yahoo, MSN and Google are all hot for the opportunity to extend their advertising networks into the local market. Diller seemed to be at his Cheshire'd best on an earnings call Tuesday.

• Posted by John Battelle at 9:15 AM
• Permalink
• Comments (0)

Watch What You Do While Online: Universities (Including Mine) Log Students' Net Usage...Govt. and RIAA Take Notice

Eye opening Salon piece on Universities' practice of logging the net usage of their student populations. It notes that these practices are under review as the Patriot Act and RIAA are subpoenaing the logs, which many universities kept as a matter of course (why? who knows). This is another example of the power of the Database of Intentions (a term that is central to my book, and I promise, I'll explain at some point). Interestingly, the Patriot Act may well be responsible for the widespread loss of this valuable resource (see excerpts).

Excerpts: "At the University of California at Berkeley, the everyday Web-surfing habits of students are regularly watched and recorded. Berkeley's Systems and Network Security group uses a program called BRO -- named after the infamous fascist icon from George Orwell's "1984" -- that keeps logs of every IP address students visit on the Internet from the campus network.

Cliff Frost, UC-Berkeley's director of communication and network services, says that "this practice is under review right now," because the campus community feels it interferes with academic freedom. He expects that the university will continue to keep logs but will discard them after a month or two. "I'd love to keep that data forever," he adds, "if there weren't the threats of subpoenas for vile purposes." "

And...

" The only way to defend student privacy against USA-PATRIOT subpoenas, says University of Michigan public policy professor Virginia Rezmierski, is for university IT departments to stop saving their logs. You can't subpoena information that doesn't exist. Rezmierski is the lead author of a 2001 National Science Foundation study of network monitoring and logging practices on college campuses.

"I don't think this study made people very happy when it came out," she says. "A lot of our findings were very disturbing." She describes interviewing a college systems administrator for the study who told her that he had singled out one student and periodically logged everything he did on his computer "because [the student] was really competent with network operations and he seemed a suspicious type." "

• Posted by John Battelle at 1