Every so often an idea comes along that has the potential to change the game. When it does, you find yourself saying – “Sheesh, of course that was going to happen. Why didn’t I predict it?” Well, I didn’t predict this happening, but here it is, happening anyway.
In short, Alexa, an Amazon-owned search company started by Bruce Gilliat and Brewster Kahle (and the spider that fuels the Internet Archive), is going to offer its index up to anyone who wants it. Alexa has about 5 billion documents in its index – about 100 terabytes of data. It’s best known for its toolbar-based traffic and site stats, which are much debated and, regardless, much used across the web.
OK, step back, and think about that. Anyone can use Alexa’s index, to build anything. But wait, there’s more. Much more.
Anyone can also use Alexa’s servers and processing power to mine its index to discover things – perhaps, to outsource the crawl needed to create a vertical search engine, for example. Or maybe to build new kinds of search engines entirely, or …well, whatever creative folks can dream up. And then, anyone can run that new service on Alexa’s (er…Amazon’s) platform, should they wish.
It’s all done via web services. It’s all integrated with Amazon’s fabled web services platform. And there’s no licensing fees. Just “consumption fees” which, at my first glance, seem pretty reasonable. (“Consumption” meaning consuming processor cycles, or storage, or bandwidth).
The fees? One dollar per CPU hour consumed. $1 per gig of storage used. $1 per 50 gigs of data processed. $1 per gig of data uploaded (if you are putting your new service up on their platform).
In other words, Alexa and Amazon are turning the index inside out, and offering it as a web service that anyone can mashup to their hearts content. Entrepreneurs can use Alexa’s crawl, Alexa’s processors, Alexa’s server farm….the whole nine yards.
Does this change the game? Because I was embargoed and could not really talk to anyone about this, I have not had a chance to talk to folks who are smarter than me about this. So my analysis is limited to my imagination. And that itself is limited by the pricing structure – I do not know if using this service will be cheaper for developers and entrepreneurs than rolling their own. But I can only imagine that indeed it is, or Amazon would not be doing this.
So what has been a jealously guarded secret – the contents of the entire index – is now available to anyone who wants it (of course, this assumes Alexa’s index is comparable to the big guys – honestly, I have no idea). The costs are modest – a few thousand bucks to process the entire web, Gilliat told me. How might that change the game? You guys are smarter than me – what do you think?
I am quite sure this means that Yahoo and Google will have to stare hard at their own (somewhat limited) search services and APIs, and think what they might do to compete, that much is certain. And if this starts to gain traction, all of a sudden, Amazon is a major search player, right next to Yahoo, Google, MSN, and IAC. A9+Alexa+web services= hmmmm….
Again, what do you think? Will this be like A9, a groundbreaking development that fails to get traction with a wider audience? Or might this just start something?
Wired News (not up yet) and the WSJ (free link) were also briefed on this news.