Monoculture, Innovation, and the Ivory Tower: The Search Papers

I’m starting to read more academic papers, research presented by various professors and the like (including some folks who are technologists working in the industry). One of the things I find fascinating about the search business is how quickly it’s turned from an academic pursuit – with all the implications of open, non-commercial sharing of findings – to one driven by clusters of high-powered nerds hitched to a particular corporation’s R&D machine. I’ve been asking around on this idea and found most landlocked geeks agree – Nutch aside, a good percentage of search research has by and large been silo’d – many of the best minds are at Google, Microsoft, Yahoo, and a few others. I’ll wager that between the 500 or so top engineers at these corporations, there ain’t a hell of a lot of sharing going on. Robust peer review between bare-knuckle competitors? Prolly not.
But it was not always so. Recall that both Yahoo and Google came out of the Stanford CS department, for the most part. Same for Excite (Joe and Graham). Lycos was midwived by CMU, and out of Berkeley came Inktomi. Anyway, you get the picture. A lot of innovation came out of the publish-and-peer-review culture of the university setting, and many of the folks who drove that culture are suited up, so to speak, in one corporate silo or another.
In any case, before they joined up, many of them wrote wonderful academic papers they shared with all in the name of progress (some still do). And there are still plenty of great academic researchers banging away on the database of intentions, though certainly they don’t benefit from owning their own slice of it like the majors do (many borrow data from the majors and perform analysis on that). So as I work on the book, I’ll be posting reviews of some of the papers I read – the interesting bits, so to speak. I’ll title each entry “The Search Papers: Cute Name Here” for ease of use, or more likely, as a clear caveat that discussion of academic Mumbo-J will follow. The first will be out in the next day or so, a wonderfully predictable little study (found via Gary Price, of course) comparing European and US search patterns from 2001 FAST and Excite (pre-Chapter 11) data. Hope you like it.