John Battelle's Searchblog

I've been complaining that nearly no search engines surface real time data (for now, that's Twitter, but Facebook is coming soon enough, and there will be tons more). In fact, I complained to Microsoft about this well before the launch of Bing, and then complained some more when Twitter results were not surfaced in initial beta versions of the service. Man, I'm grumpy lately, eh?

Well, that's changing. Sort of. From a Bing blog post today:

There has been much discussion of real-time search and the premium on immediacy of data that has been created primarily by Twitter. We’ve been watching this phenomenon with great interest, and listening carefully to what consumers really want in this space. Today we’re unveiling an initial foray into integrating more real time data into our search results, starting with some of the more prominent and prolific Twitterers from a variety of spheres. This includes Tweets from folks from our own search technology and business sphere like Danny Sullivan or Kara Swisher as well as those from spheres of more general consumer appeal like Al Gore or Ryan Seacrest. Starting later today, when you search for these folks names in association with Twitter, you’ll see their latest Tweets come up in real time on Bing’s search results.

Oh boy! I wonder if maybe...I'm one of those folks? Sigh. No such luck. Although, to be honest, I can't seem to make it work for anyone, including Danny and Kara. Maybe it's not working yet in my area.

In any case, what DOES come up is my and everyone else I tested's Twitter account, at least when I add "Twitter" to the query. That's a major step forward from where Bing was even at launch. That said, there is NO reason to make folks put the word "Twitter" into the query. None. That is a failed use case. Commit, or don't commit, but don't ask users to specify Twitter to know what someone might be saying in real time. Better to indicate that the query has real time results, and offer them if a searcher wants them. Or figure out some other clever UI solution. Real time is here to stay, may as well design to it, and not ask users to do it for you.

After all, with the whole Websquared thing, we'll soon be leaving real time trails all over the globe, and we may well want them surfaced by our favorite search engine, no?

But good on ya, Microsoft, for dipping your toe into the water. Google, your ball.

UPDATE: It works now. I'm one of the chosen ones! Oh joy!

There's quite a wonderful authorial kerfuffle happening between Chris Anderson, whose recent book "Free" has been the target both of plagiarism charges (from Wikipedia, of all places, oh the wonderful irony, one might think Chris actually planted the whole damn thing...) and Malcolm Gladwell, who never met a clever anecdote he couldn't convert into a well turned (and dammingly entertaining) book of his own.  

I won't go into the whole thing, because, honestly, I just don't have the, er, free time.

However, I do find it noteworthy that Chris's much-linked to riposte to Malcolm's initial evisceration comes on Wired.com, where, shock of all shocks, advertising is prominently featured. Free, of course, doesn't come without a business model.

Last week I wrote a post in which I opined a bit about Facebook search. In it I wrote:

Facebook is way more than its newsfeed, and its search play is key to proving that value, and extending it....No doubt building Facebook search today is akin to building Google ten years ago - bigger, most likely, in terms of data, algorithmic, and platform challenges.

If only I had waited a few days, I could have pointed to Fred's piece in Wired, out this week. He profiles the ongoing feud between the King of Search, Google, and the upstart, Facebook. In his piece, he writes:

For the last decade or so, the Web has been defined by Google's algorithms—rigorous and efficient equations that parse practically every byte of online activity to build a dispassionate atlas of the online world. Facebook CEO Mark Zuckerberg envisions a more personalized, humanized Web, where our network of friends, colleagues, peers, and family is our primary source of information, just as it is offline. In Zuckerberg's vision, users will query this "social graph" to find a doctor, the best camera, or someone to hire—rather than tapping the cold mathematics of a Google search. It is a complete rethinking of how we navigate the online world, one that places Facebook right at the center. In other words, right where Google is now.

I agree that of all the contenders out there right now (including Twitter), Facebook has the most data, position, and potential to upset Google's dominance of the web. But I disagree with one premise of the piece, which is that Facebook's proprietary approach to the data it stores presents a blind spot to Google that gives Facebook a competitive edge. Fred writes:

Together, this data comprises a mammoth amount of activity, almost a second Internet. By Facebook's estimates, every month users share 4 billion pieces of information—news stories, status updates, birthday wishes, and so on. They also upload 850 million photos and 8 million videos. But anyone wanting to access that stuff must go through Facebook; the social network treats it all as proprietary data, largely shielding it from Google's crawlers. Except for the mostly cursory information that users choose to make public, what happens on Facebook's servers stays on Facebook's servers. That represents a massive and fast-growing blind spot for Google, whose long-stated goal is to "organize the world's information."

I think it's a major strategic mistake to not offer this information to Google (and anyone else that wants to crawl it.) In fact, I'd argue that the right thing to do is to make just about everything possible available to Google to crawl, then sit back and watch while Google struggles with whether or not to "organize it and make it universally available." A regular damned if you do, damned if you don't scenario, that....

For an example of what I mean, look no further than Twitter. That service makes every single tweet available as a crawlable resource. And Google certainly is crawling Twitter pages, but the key thing to watch is whether the service is surfacing "superfresh" results when the query merits it. So far, the answer is a definitive NO.

Why?

Well, perhaps I'm being cynical, but I think it's because Google doesn't want to push massive value and traffic to Twitter without a business deal in place where it gets to monetize those real time results.

Is that "organizing the world's information and making it universally available?" Well, no. At least, not yet.

By making all its information available to Google's crawlers (and fixing its terrible URL structure in the process), Facebook could shine an awfully bright light on this interesting conflict in interest.

I'm finding Mashable, an FM site, on a tear lately. These headlines over the past two days caught my attention:

Google vs. Bing Battle Heating Up: Is Google Scared?

Google to Launch a Twitter Search Engine?

Social Media Goo: Cadbury Campaign Going Viral

Apparently I am not alone in noticing Mashable's ascendance.

Much buzz over the past few days about Facebook's plans to let folks (and, ahem, brands) claim their namespaces on Facebook. IE, Starting this weekend, I should be able to claim www.facebook.com/johnbattelle, just like I already "own" www.twitter.com/johnbattelle (sort of).

Anil Dash has a very funny send up of all this in a future forward timeline satire here. His point is - why is everyone falling all over themeselves to get their vanity URL on Facebook - or Twitter, or anywhere else for that matter - when the web is an open place and anyone can get their own URL, after all.

Well, yes and no. I've been complaining about Facebook's terrible link structure for a long time. We all spend time there, and create and share value there, but up till this weekend, it's been very difficult to point folks to places *inside* Facebook from places *outside* Facebook. The future of the web is ecosystemic - it's not about being in one place - this blog, that Twitter feed, or that Facebook page, it's about the ability to be anywhere, depending on the context and the moment. Sewing it all together is critical, and this move should make Facebook that much easier to incorporate into an ongoing, web wide conversation. I hope.

It had to happen, and it has. Twitter's unbelievable growth numbers have flatlined, or even gone down, if you look at Quantcast (the site is not Quantified).  

This was predictable, given all the media hype and new folks, and the very real newbie problem I outlined in this post last month.

I predict Twitter will address this issue, and growth will resume, but at a more moderate and sustainable pace. But this is a very clear sign that Twitter, which made the cover of Time magazine last week, is on the other, less happy side of a traditional hype cycle.

As a reminder, here's what I said in a post just a month ago, noting the incredible growth of Twitter:

I think this is both Twitter's most important and dangerous phase of its young life. The retention problem must be addressed, and quickly. In my previous post about Twitter adding value to new users, I suggested Twitter incorporate some structure around its suggested users feature.

But with an inflection like this, I think it's time to swallow hard and embrace some serious social media jujitsu. In short, Twitter should integrate Facebook Connect in its signup process, and offer it as a feature for current users.

At the CM Summit earlier this week, Terry Kawaja debuted his parody "Mad Avenue Blues", set to Don McLean's American Pie. It's now up on YouTube. It's a brilliant send up of our industry. You'll want to watch it a few times to get all the jokes.

Read this: Twitter = YouTube. Then read this: Visits to Twitter Search Soar. Discuss.

TC notes the extraordinary growth Twitter has seen since its initial inflection. This is a growth pattern I have never seen in terms of speed - not in the nearly 25 years I've been watching this industry.

I think this is both Twitter's most important and dangerous phase of its young life. The retention problem must be addressed, and quickly. In my previous post about Twitter adding value to new users, I suggested Twitter incorporate some structure around its suggested users feature.

But with an inflection like this, I think it's time to swallow hard and embrace some serious social media jujitsu. In short, Twitter should integrate Facebook Connect in its signup process, and offer it as a feature for current users.

If it were to do that, then every new user who is also on Facebook (and who is not, at this point?) would be able to instantly follow Facebook friends who are on Twitter. Integrate this with a Groups feature, and you're proving value immediately.   

Talking with Whrrl CEO Jeff Holden yesterday, I mentioned this idea (I have also been bouncing it off a number of folks over the past month or so). He countered that such a move might add too much value to Facebook, but I disagree. Facebook has won the first round of the social graph game. Twitter has far more to gain by leveraging Facebook right now.

And after all, they can always turn the feature off it they want to.

What do you think?

Gian Fulgoni, Executive Chair of Comscore, has an interesting analysis of what's happening in paid search lately. It's germane to my earlier posts about paid search share sliding and Google's decision to allow trademark ad bidding.

In his post, Gian notes that overall search queries are up dramatically (68% over two years) but:

if one looks at the number of paid clicks, the growth rate is a lower 18%, which raises the question: why have paid clicks grown 3x slower than the total number of queries?

Gian answers:

The reason appears to be that the ad coverage (i.e. the percent of search results pages with a paid ad) has dropped from 64% to 51% of searches.

Here's where it gets interesting. Why has ad coverage dropped? Gian has two hypotheses. First, search engines are getting better to reduce less relevant advertisers from the mix. But the second reason points to a more important potential breakdown in the AdWords model:

An analysis of comScore data shows that search queries are actually getting longer and that as searchers become more experienced they are using more words per search query. And this apparently reduces the likelihood that an advertiser has bid to have his/her ad included in the results page from these longer queries, due to paid search advertising strategies that limit ad coverage, such as Exact Match, Negative Match, and bid management software campaign optimization.

In short, our queries are getting closer to real conversation, real natural language, and Google's algorithms are having a harder time keeping up - matching advertiser demand to our increasingly complex queries.

As Gian said, fascinating.

Also worth noting, my pal Chas's analysis of what the decline in paid search means for brand advertising.

In its often overly clever way, Google became a brand television advertiser this month. Here's the ad:

Now, why do I say backed into? Well, this video was created by some Google employees in Japan (so it's not an effort by the main company, see?!), and it was an promotion to show off how cool Chrome was (not designed to be an ad, see?!). It was released on the web first (see, not debuted on silly old school TV!), and when the ad got some pickup, Google decided to run it on its fledgling Google TV Ads service, the sole remaining attempt by the company to do Adsensify old media (see, we're not really doing a traditional media buy!). But none of this really matters. At all. Because at the end of the day, consumers watching TV are going to see this ad, and judge it as that, an ad. That means the company, no matter how cleverly it's thought itself into this execution, has to consider itself a brand advertiser, and act like one as well. Question is, can it do that?

First two headines from IWantMedia today:

Here are the actual stories:

Google May Add Twitter-Like Features

Twitter to Expand Search Functionality

Yesterday the Senate held hearings about the sorry state of the news biz, and Marissa Mayer from Google, Arianna Huffington from the Huffpo, and various others held forth.

This in context of a building chorus of press voices saying that 1/ Google is stealing what is rightfully ours and 2/consumers must pay for content on the web (Murdoch, Belo). And in the context of a scramble to figure out whether or not the new big screen Kindle is a good thing, or a bad thing for newspapers (Murdoch is not a fan, the NYT is.)

I can't imagine a more interesting time to be in the news business. Note I didn't say "fun." But interesting, yes, very interesting.

More links: Forbes chief Spanfeller says Google is stealing $60million from Forbes.com, Danny of SEL responds, and link to coverage of AP's saber rattling (last month) along the same lines.

I missed this, but after some detective work with sources inside Facebook, I've confirmed that Facebook now lets third party applications create live links inside Facebook. This has been one of my principal complaints about how Facebook interacts with the "rest of the web" - and now it's resolved.  

For example, I use the Twitter application on Facebook to update my Facebook status. Everything I tweet ends up as a Facebook status update. There's certainly no love lost between these two companies, and I've pointed out in the past that links in my Tweets are not live in Facebook, so my Facebook friends can't click on them and see what I'm talking about.

As of sometime a month or so ago, Facebook now makes those links live. As this populates the service, there's a huge search opportunity (all those links can be crawled, FaceRank can be calculated, etc...)

I think this is huge for the company, and is a major step toward an "off domain" strategy which will let it truly embrace and extend the web. Now, if I could only Tweet from inside Facebook....

I've spent a bit of time going back in time lately, at least as far as Twitter is concerned. In short, I created a new account, as if I had never used the service before.

Why? Well, as Twitter hits inflection, it struck me that there was something really, really important that had to happen, in terms of how the service works. As millions of new users try the service, it's crucial that they find something useful when they arrive. If they don't, well, they'll leave.

And leaving they are, if this report from Nielsen is to be believed. Widely picked up last week in the Twitterverse, the report does the math and finds that 60 percent of those who try Twitter abandon the service within a month. That means no matter how steep the inflection, Twitter will soon burn through its available fuel (new user attention) and could fail to hit escape velocity (where escape velocity = a scaled platform at the level of Facebook, Google, or Yahoo).

That got me thinking. What do new users do when they first log into a service like, say, Facebook? Why, they search, of course.

For old friends, for the names of their colleges or high schools, for any kind of social connection that might make sense of the very large universe that is Facebook.

So when Twitter integrated search last week, it was, as I said, a very big deal.

But to my mind, it's not enough.

To explain my point, let me go back to the experience I recently had of creating a new account - going back in time, so to speak, and pretending to be a newbie to Twitter. The service is very easy to sign up for (see the screen shot at left). Once you pass this screen, you can check to see if

your friends are on the service. This is a pretty standard email database lookup, and I have no idea how many folks go through it. I don't have email at any of those services (at least, none with any real contacts), so I passed. (I'd be interested in how many folks do use this service, and how many hit the button to skip this step. If it's a high percentage that use this step, I'd also be interested in what

the experience is like in terms of making Twitter more useful, but I'll have to be blind to it for

this post. I think my conclusions will be valid in any case....).

Next comes the step that I find most interesting, and in its current iteration, most frustrating. This is where the new user gets a

list of folks that Twitter suggests he or she might follow. It's a pretty random list of interesting folks, including (as I write this) John McCain, Fred Durst, Chris Anderson, Oprah, John Legend, and so on. It changes from day to day, but anyone who's ever made it onto the list reports that their followers skyrocket - sometimes by an order of magnitude.

Why? Well, turns out most newbies to Twitter simply hit "follow all" and end up with the list of twenty or so suggested Tweeters as their first set of folks they are following.

Therein lies the problem. Ah, the dinner bell is ringing, when I come back, I'll explain why, and suggest a better way. I'm sure many have already thought about this, but I never claimed to be original, just persistent. And...I really want Twitter to get

escape velocity...because every time a rocket makes it out of the Valley and into the Rest of The World, it feels like the work we all do is worth it.

(Back from Dinner). So why is following twenty or so interesting people a problem? Well, while I am sure these folks are chosen for their general interest and lively tweets (for more, see Twitter's blog post on suggested users), it turns out that it's simply not very

compelling, in the main, to watch these guys tweet. It's certainly not as addictive as finding an old friend on Facebook, for example. It's neat, but it's not going to get folks to come back, over and over again.

What *is* interesting, or could be, is watching folks tweet who you care about. Perhaps they are friends, or family, or leaders in your line of work, or entertainers you love. For whatever reason, they are *your* leaders, and finding them, at least during the sign up process, is entirely too hard.

But it doesn't have to be that way. It strikes me that a few more structured steps in the sign up process could really pay significant dividends for Twitter. Perhaps a "follow wizard" that asks a few questions, and makes suggestions based on input

from the new user. Let us drill down by category: Business:Technology:Internet, or Health:Diseases:Cancer. The ontology isn't very complicated - mapping users to it is a bit more complex, but not impossible.

And encourage folks to put in the names of their friends via search - that's magic when you find a friend who's already on Twitter, and might act as a sherpa of sorts.

There are already a lot of third party services that help users find folks worth following, but new users are never going to find them in their initial interaction with Twitter. incorporating this kind of a service into a newbie's initial experience - even if it's very, very simple - could pay huge benefits in turning around that 60% abandonment number, and soon.

In short, you never get a second chance to make a first impression, and right now, Twitter's initial impression does not add enough value. But with a few tweaks, it most certainly could.

Just wrote a long-ish post on FM over at the FM blog....from it:

One of the greatest challenges we face here at FM is answering a very simple question: What kind of a business are you?

When a reporter, partner, or colleague asks me this question, it's usually followed by a deep intake of breath on my part, because my answer often runs for quite some time. Federated Media wasn't built to answer easy questions, and our business isn't easy to define. Quite purposefully, we have built our business in the center of some swiftly changing currents at the nexus of media, marketing, and technology.

But if you don't take the time to define your brand publicly, others will. Over the past few months FM has been called a social media network, a blog network, a vertical ad network, an advertising rep firm, a media services firm, and even a new kind of advertising agency.

The truth is, FM incorporates aspects of all these businesses. What we are not, however, is any one of them alone.

Hence, our problem (I prefer opportunity, being eternally an optimist...)

So for the record, let me state what we believe FM *is*: FM is a media company, founded on an evolving definition of the word "media."

Over and over I've predicted that Google will be forced to act like a publisher, because there's only so much demand that can be harvested, and sooner or later, Google's core revenue-generating customers - that'd be marketers - will demand some help creating supply.

Supply means branding, and branding happens in the magical world of publishing. Here are two additional Google initiatives that point the company toward that world:

Google launches Digg-like feature

A Cnet piece giving an overiew of Google's attempt to curate value from the wisdom of the masses. Called What's Popular.

and...

Eric Schmidt on Google's New Plan for the News

Never seen this site before, but the woman who writes is managed to get into a party where she talked to Eric Schmidt. From the piece:

I asked if the rumors I’d heard, that Google was changing its mind about getting involved with creating original content, were true. No, he responded, quite convincingly, they’re not. Google is not a content company, and is not going in that direction, he explained. But Google does have plans for a solution. In about six months, the company will roll out a system that will bring high-quality news content to users without them actively looking for it. Under this latest iteration of advanced search, users will be automatically served the kind of news that interests them just by calling up Google’s page. The latest algorithms apply ever more sophisticated filtering – based on search words, user choices, purchases, a whole host of cues – to determine what the reader is looking for without knowing they’re looking for it. And on this basis, Google believes it will be able to sell premium ads against premium content.

If this is true (sending a note now to ask), it's a big step.

I nearly re-upped my subscription to the online version of the WSJ this evening, so as to read this piece: Facebook Opens Site To Developers Of Services.

But I found the text here - also on the WSJ site. Genie's outta the bottle. From it:

The announcement, expected Monday, means developers can build services that access the photos, videos, notes and comments users upload to Facebook, with users' permission. That's a big change for the social-networking site, which has exercised tight control over the look and feel of its service and how developers can interact with it.

What I cannot figure out is whether this means Facebook is going to solve its linking problem. I've complained about it before, but the big issue with Facebook, to my mind, is that it does not play well with others when it comes to linking on the open web. In fact, it's damn hard to even link *inside* Facebook, never mind sending links *from* other services (IE Twitter) into your Facebook status feed (you can't).

If Facebook fixes this, it's a game changer. If folks can create a useful service that acts and looks like Twitter, and works both inside and outside Facebook, and if these kinds of services can make money on their own terms (and not be subject to the whims of Facebook's current TOS, which are terrifying), well, that's a very big deal.

Yesterday's news about Yahoo's layoffs was well received by Wall Street (which seems to love layoffs in every sector except its own), and part of the optimism about Yahoo's future seems to lay in folks expecting Yahoo and Microsoft to finally get around to doing a search deal. I've written over and over that I think the two should do this, but as time goes by and the machine at Microsoft continues to iterate on its own internal search play, I find it harder and harder to see how such a deal actually gets done, at least when it comes to organic search.

Now, I predicted in January this deal would get done, of course, so I kind of have a dog in this fight. But recall how I predicted it would go down:

"Microsoft will gain at least five points of search share in 2009, perhaps as much as 10. This is a rather radical prediction, I know, but hear me out. I think Redmond is tired of losing in this game, and after trying nearly every trick in the book, Microsoft will start to spend real money to grow share (IE, buying distribution), while at the same time listening to the advice of thoughtful folks who want to help the company improve the product. However, search share is half the game, as we know. The second half is monetization, and Microsoft will continue to struggle here, unless it manages to buy Yahoo's search business. Which it won't, because....

6. Yahoo and AOL will merge.

7. However, in the second half of the year, Microsoft will buy its search monetization from the combined company."

That's some pretty damn specific predictions, now that I think about it, and it depends on a lot of stuff happening that is out of Microsoft's control (AOL merging with Yahoo!) but I think the idea of combining search monetization efforts still makes sense. Yahoo has tons of distribution. Microsoft has tons of money. Both have a common enemy. We shall see.

Meanwhile, Bartz's bluntness is still pretty damn provocative. Here's a quote you have to love if you work in product management at Yahoo:

"We sort of had one product management person for every three engineers, so we had a lot of people running around and telling people what to do, but nobody was doing anything," Bartz said.

Yow!

One of the principal things nearly anyone does on Google.com is a vanity search: We ask the question: What do people see when they put my name into Google?  

Today, Google is announcing, for the first time, that anyone can change what is seen. (The initial launch is US only).

This, to be clear, is a Very Big Deal.

Joe Kraus, one of the founders of Excite and founder of JotSpot, is now at Google, and this new feature is his baby. I spoke to him today when he sent me a note about the launch. I immediately called him back, because, as I said, I see this as a Very Big Deal.

Why? Well, Google has always been predicated on being a neutral black box. You, as a solitary entity, could not influence the results that Google provided (though of course a very large industry has emerged that attempts to do just that). But this launch changes the game, in a few very, very interesting ways.

First, and most obvious, this is Google leveraging its might in search to get more people to sign up for Google profiles. I shouldn't have to explain why this is important, given the competition from Facebook and Twitter, but trust me, it's really important that Google 1. know who you are and 2. compel you to have ongoing relationship with the company.

Second, this move creates, for the first time ever, a new signal that is directly controlled by an individual but changes what *everyone else* will see in results. True, for now, the results are at the bottom of the first page of results, but that doesn't mean it won't move up once Google learns enough to make it truly useful.

Third, this is Google putting a human, community-driven face on itself. It's Google saying "Hey search user! We want to listen and respond to you!" This is a very good thing for the company, and how it plays its hand from now forward is going to be very, very interesting.

Fortunately for Google, the man with the hand is Kraus, who is a master poker player (yes, I've lost to him) and a generally good guy to boot.

There are many questions to be asked about this new service, but the first one that came to my mind is this: Who ranks first for any given name? There are a lot of Joe Smiths, for example, and even more than one John Battelle's, despite the relative uniqueness of that name (and even more if you count dead folks on the roles of Ancestry.com).

Kraus explained that the initial signal for which profiles would be shown (four will be shown, with a "more" button) will be based on completeness of the Google Profile. AHA! Another motivation to give Google more info on you!

What if there are like 200 John Smiths, and they all have complete profiles? What signals will determine which get into the top four, and which gets the coveted top spot? Kraus said he didn't have a good answer for that yet, but one signal will certainly be clickthrough rates (like it is for AdSense), and they will be learning and iterating over time.

Google is also doing a US promotion to encourage folks to set up a profile - when you "Google Me" (literally, "me"), you get an ad (see image at left). Again, this is something of a first for Google, or at least unusual, as there are other AdSense advertisers for the term "me" who are not getting placement at the top - Google is taking it for its own promotion.

This all reminds me of the ending of my book. Which of course is my favorite part. In the epilogue, the final paragraph reads:

What does it mean, I wondered, to become immortal through words pressed in clay – or, as was the case here, through words formed in bits and transferred over the web? Is that not what every person longs for – what Odysseus chose over Kalypso’s nameless immortality – to die, but to be known forever? And does not search offer the same immortal imprint - is not existing forever in the indexes of Google and others the modern day equivalent of carving our stories into stone? For anyone who has ever written his own name into a search box and anxiously awaited the results, I believe the answer is yes.