Brian Morrisey scratches an itch for anyone who’s ever doubted industry forecasts:
The problem: these figures are used by companies to justify strategies to investors and the press. Yet nobody asks whether they’re valid. For fun, I decided to hunt down an Internet ad forecast from years ago to see how the gurus did. Sure enough, Jupiter projected in August 1999 robust Internet ad growth, enough to reach $11.5 billion in 2003. Guess where it ended up? $7.3 billion, according to the IAB.
Brian was smart enough to pick a time (1999) when, given the dot com collapse, it’s a pretty sure bet that five year forecasts would be wrong. I wonder, however, how wrong they’ve been over time? It’d be really interesting to see a scorecard of sorts, a spreadsheet of two columns, with forecasted and actual numbers over the past ten years. Anyone seen this?
If there isn’t one, shall we make it?
I think its a great idea to make one. At a minimum would give the folks that use this data some sort of reliability factor to incorporate into their planning.
Assumptions of CPC, CPMs, overall market would be good to know.
I do see one issue. A lot of this research is individual driven (though within a firm). And these individuals do not keep their jobs for too long and hence are we measuring a company or a particular analyst.
I think the most valuable outcome from the forecast vs. actuals comparison would be some kind of variance correction to apply to new forecasts and some kind of accuracy rating for the forecasters.
We used to have this kind of thing with a friend who was notorious for exaggerating stories: divide by 2.5, subtract the date with the waitress.
Yes, make it!! I’ve said for a long time it’d be very interesting to compare the ‘prophecies’ from the ‘prophets’ to find out if they really can predict the future, or are they just false prophets making money by prognosticating.
My bet? They’re not very good at it. If they were, Forrester, Jupiter, et al would be screaming from the rooftops, “We called it! See, we were right. We were spot on!” They don’t use that logic nearly at all in their marketing. So that means they’re not right most of the time is my guess. And like James says, over time you’d probably be able to track how wrong they always are.
I think it would be a very interesting thing to make.
But your desire to have that sort of information raises a problem/issue that I have with media in general.. both “old” media as well as “new” media.
Newpapers, broadcast television news, and their ilk do a very poor job of contextualizing and situationalizing (is that a word?) current stories/predictions/analyses within a historical stream of similar stories/predictions/analyses.
When the media only focuses on the current situation and does not explain or contextualize it, we the media consumers lose the ability to really understand the longer term implications of the information that we hear. If we don’t know where something has “come from”, our ability to ponder where it is “going” goes away.
This would not be such a bad thing, were it not for the fact that new media fails at this as well. My ongoing example is Google. For years and years now, I have wanted to be able to use Google to search for the “context” to a current news item, fact, story, prediction, etc. I’ve wanted to be able to use Google to quickly and easily piece together all the information that I need to create exactly what you’re asking for right now: A ten-year retrospective and scorecard.
Google has hundreds of thousands of servers and data mining algorithms and similar technologies, that could be put to use to deliver this sort of search experience.
Instead, Google only gives me the ability to find 1..10 web pages (or YouTube videos) at a time, with no sense of their interconnectedness, context, history, or relationship.
If I want to assemble that scorecard, Google gives me no help. I have to manually input dozens and dozens of queries, hoping that each new query is the “right” query to give me the next piece of the puzzle, to put all the necessary information together.
So old media has failed to deliver this sort of information, but new media has failed as well.
Those are some interesting stats by Brian Morrisey, and it would certainly be interesting to see a chart like you reference.
A bit off topic with the point of the post, but……If you’re using these types of projections for business planning purposes, if you’re an 800 pound gorilla (or even a 100 pound gorilla), it probably matters if these projections are ‘off’. However, if you’re a small entrepreneur, there are enough internet opportunities out there that it doesn’t matter if internet advertising is growing at 50% or 5% per year.
I’ve been hoping to see this for a long time. I’d like to see most of our modern “prophets” put to the test … when someone says ___ will happen, make a note of who said it, what they said would happen, and by what time. Then, when the time comes, were they right, wrong, or somewhere in between?
Note: I emphatically want to see this for “prophets” like Katie Couric, Nancy Pelosi, Mark Levin, etc. In other words, when politicians, pundits, journalists, and the like claim something will happen (especially when trying to influence legislation), let’s hold them accountable to their claims.
Bob Cringely spoke about this kind of setup back in ’05.
Sadly … I see two things happening. Either the “grading service” doesn’t stay neutral enough, or the “prophets” get better at being vague enough to ensure they’re right.
Great idea, John. I’d love to see this as well.
*subscribes*
🙂
This would not be such a bad thing, were it not for the fact that new media fails at this as well. My ongoing example is Google. For years and years now, I have wanted to be able to use Google to search for the “context” to a current news item, fact, story, prediction, etc. I’ve wanted to be able to use Google to quickly and easily piece together all the information that I need to create exactly what you’re asking for right now: A ten-year retrospective and scorecard.