Economist: Egalitarian Engines, and Thoughts on Transparency

Interesting article (thanks Cyril) in the Economist about the effect of search engines on traffic distribution.

…there is a widespread belief among computer, social and political scientists that search engines create a vicious circle that amplifies the dominance of established and already popular websites. Pages returned by search engines are more likely to be discovered and consequently linked to by others.

Not so, according to a controversial new paper that has recently appeared on

arXiv, an online collection of physics and related papers.

It took some searching (arXiv has a terrible search engine) but I found the referred piece here.

What I find intersting and important are the Economist’s final conclusions. From the article:



The paper, which was posted on arXiv for comment, has now come under attack. Matthew Hindman, a political scientist at Arizona State University, says that the data used in the research are pretty shoddy. Moreover, he says, the discrepancy between the model and the real world does not necessarily come from the role of the search engine.

Whether Dr Fortunato’s thesis stands the test of time remains to be seen. That it is tested must be a good thing.

I can’t agree more. One of the things which is most frustrating about search, to me and to many, is the lack of transparency, and the lack of knowledge about how an increasingly convoluted ranking scheme actually works. Of course, Yahoo and Google can’t publish their entire ranking scheme. But some kind of guideposting should be done.

This is even more true in the AdWords/Overture world, where real money is at stake, every minute of every day. I think this will come to a head sooner rather than later. For now it’s all well and good to let Google determine its own profit margins by optimizing AdWords and AdSense behind the curtains of darkness. But that can’t stand forever. There is too much opportunity to use that lack of transparency to ill ends – ie to bury competitors which are surfacing now by paying more than market prices to ensure that publishers stay with Google, for example. I am not suggesting that is happening, just that we have no way of knowing if it ever were to happen.

Toward that end, Seth’s Root Markets is written up by ClickZ here…