When Google Ad Planner came out back in June, I immediately thought of Comscore – and I was not alone. Many in the marketing industry thought that Google’s product would be a “Comscore killer,” and when I noted as much in my coverage, Gian Fulgoni, Comscore’s chair, shot back in a comment to my post:
Hi John: Before celebrating the availability of these products from Google, I think it would be prudent for web site operators to compare their site traffic numbers as obtained from their server logs (or Google Analytics for that matter) with the unique visitor numbers that Google is now publishing through Google Trends and Ad Planner. I think they will be astonished at how much lower Google now says their traffic is.
I asked Gian to elaborate, and published the resulting interview here.
But until now, we’ve not had the data to back Gian’s claim. I asked him if he could provide it, and to his credit, he did. The story it tells is certainly not what one might expect. (Of course, the data is from Comscore, so it must be taken as such, but remember, Comscore is a public company that stakes its reputation and its market value on data, so my gut tells me that Gian is not trying to pull the wool over anyone’s eyes.)
A bit of background: Anyone paying attention has noticed that publishers, by and large, believe Comscore’s panel-based measurement system grossly underestimates traffic and unique visitors. As a publisher myself (FM represents more than 160 middle to large sized sites, including this one), I’ve been one of the most visible such complainants. And that list is not short. In fact, Comscore and Nielsen are both working with the Interactive Advertising Bureau (I am a board member) on an audit of their practices to verify their methodologies. (Comscore notes that it believes the issue of cookie deletion can cause significant inflation in unique visitors, for more see this release.)
Given this, the world expected that Google, with its unparalleled access to web-wide data, would validate publishers’ concerns and show that Comscore’s numbers were significantly under-reporting reality.
Turns out, the reverse is true. Gian provided me data comparing Google Ad Planner and ComScore data in two cases. First, for a large sample of 20,163 sites, his shop compared reporting on monthly uniques between the two services. Secondly, Gian pulled out 5,398 sites that are part of the Google Adsense ad network, and ran the same comparison.
The results are pasted in these two charts (provided to me by Comscore):
What to make of the numbers? First off, it’s quite interesting to see that Comscore measures, on average, a significantly higher number of uniques across all types of sites. Comscore’s numbers are three to three and a half times higher, according to Comscore.
Secondly, for sites that are using Google’s Adsense network, the undercounting is not as dramatic (that’s the second chart.). As Comscore’s charts note, there seems to be a “significant bias in Google Ad Planner data” toward “sites that carry more ad impressions from Google.”
In short: If you were a media planner using Google Ad Planner, and you were looking for larger sites, you would be led to sites that are running Google AdSense, on average, over sites that do not. Net net: This data indicates that Google Ad Planner pushes ad dollars to Google sites over non-Google sites. This makes sense – Google has data on Google users, after all. So that data might naturally bias toward Google-related sites.
But as I said in my coverage: “Such a tool must be neutral and not bias advertisers toward buying on Google properties or those that have Google ads, which of course is going to be a perceived bias in any case. Such is the price of being Very Big.”
So far, not so good on this measure. As Gian and Comscore have long pointed out to me, it takes more than raw data to make for good measurement. Ideally, you weight your data with a lot more knowledge of its context – what kind of machine is creating it (work or home? Man or woman? etc.). While Google once blended Comscore demographic data into its ad network, Comscore confirmed to me that this is no longer the case. And while it is subject to endless criticism, Comscore does have a lot more practice at this game than does Google. At least for now.
This data once again raises the question, long asked, of how Google is measuring in the first place. Most believe Google must be leaning heavily on its Toolbar data (see TC for more here and Danny here), and this data does nothing to counter that argument. The strong bias toward Google network sites is suspicious – one can imagine that folks who might install the Google toolbar are clearly already biased toward visiting Google-related sites, for example.
But Google will not acknowledge any use of the Toolbar. Instead it said in its announcement: “Google Ad Planner combines information from a variety of sources, such as aggregated Google search data, opt-in anonymous Google Analytics data, opt-in external consumer panel data, and other third-party market research.”
As I pointed out earlier, I don’t think such coyness can stand. I’ve pinged folks at Google to get a response on this, and as soon as I do, I’ll update this post.
UPDATE: Google has provided a statement to me:
We take the objectivity of Google Ad Planner very seriously in providing advertisers and publishers with a better understanding into online audiences. While we don’t comment specifically on our data collection methods, Google Ad Planner in no way treats AdSense sites differently than non-AdSense sites.