Today marks the first of what I hope will become a regular series of Q&A interviews on Searchblog. The format is simple - I send an opening question to the luminary in question via email, they respond, and we go from there. First up is Gary Flake, a veteran of Overture, Yahoo and now Microsoft's vaunted research labs (he's founder and director of the new "Live Labs.") Gary and I have known each other since I first began work on the book, and he's always had a refreshingly frank outlook. I expected that to be tempered by a year at the world's largest (and oft-criticized) software company, but I was wrong. If anything, Gary has become more outspoken. I've bolded the really juicy bits, but see for yourself....
You've been at Microsoft for nearly a year now. Are you satisfied with the pace of development on the core search product?
Broadly speaking, yes, but there is always room for improvement. We've been laying a foundation for the past year, trying to solve the hardest problems first. Frankly, we've left some low-lying fruit hanging on the vine, so to speak. On the good side, we have a rock-solid 64-bit backend architecture (probably the only one in the industry) that can scale almost indefinitely on multiple dimensions. Our relevance framework, built with machine learning technology from Microsoft Research, is now picking up steam and is getting close to parity with the competition. We even have some real product differentiation that we've just launched with virtual earth maps, search macros, and the image search experience. But to be honest, we pretty much blew it with the GUI for the past year. Why? There's no good reason, really. The truth is we've had so much going on over the past year that it was simply more fun to focus on the core issues first, which is a mistake typical of engineers. I think the whole industry has been in a bit of a rut with respect to the user experience, and we are more guilty than most, but the good news is that there is a lot of room for improvement and we are now in pretty good shape to experiment more.
Say more about 64 bit architecture. Why is it different? What does it allow you to do with search you can't with 32 bits? How easy or hard might it be for Google and others to migrate to a similar architecture? And can you tell us a bit more about Microsoft Research? Lastly, if you could - can you give me a specific example about what you did not like about your old UI?
"64 bits" refers to the native address space of the CPUs that we use in our architecture. Having the entire architecture be 64 bit carries several implications that are quite subtle and mostly of interest only to engineers, but I'll take a shot at highlighting some of the important differences.
First, all search architectures tier their data stores so that the data can reside on relatively slow disks or fast RAM. For performance purposes, you want the most frequently accessed data to be on the fastest store (RAM). A 64 bit system can have vastly more RAM than a 32 bit system, which means that we can have a higher ratio of RAM to disk space (per CPU), versus others in the industry. Second, we've only scratched the surface on the possibilities for using advanced hyperlink analysis for web relevance. There are a class of algorithms (like Pagerank) that work well in either 32 or 64 bit systems (technically, because the algorithms can be realized via a linear sweep of hyperlinks). However, there is an even richer class of algorithms that can only be efficiently built on a 64 bit system because you essentially have to have a significant part of the web stored close to a single CPU. So, 64 bit systems pave the way for entirely new forms of relevance that look at how pages relate to one another. Finally, with the bigger address space, we may also see 64 bit systems critical to realizing more sophisticated relevance algorithms that analyze a result set in the aggregate as a final step. In this scenario, you may have a huge amount of data in memory, which 64 bit systems can do just fine. So, in short, it's a real big deal and it will be more important as the entire industry matures.
Google and others can certainly start to migrate towards 64 bit. In fact, everyone will have to at some point. However, there is a huge cost to make the jump, in terms of hardware, porting existing code, but especially in revamping algorithms to make the most use of the new resources. We took our medicine early because we could (we didn't have a legacy system as baggage). This is an example of us taking a longer-term view, in the sense that we increased our startup costs, but are now have a more strategic technology platform.
Microsoft Research is possibly the greatest computer science research institute in the world. MSR was also one of the key reasons why I came to MS. MS basically offered me the role of figuring out how to make MSN and MSR even more tightly connected, and the idea of being able to fully leverage MSR towards web search gave me goose bumps. Internally, we talk about what we are working on as being historic, in the sense that we are defining a computer science research agenda that could possibly influence an entire generation of computer scientists. Without MSR as part of our story, we wouldn't stand a chance at such lofty goals.
What didn't I like about our old UI? Where do I begin? There is so much to choose from ... The fonts inconsistantly rendored and were too big. The damn search box was too small. It made no sense to have redundant buttons, links, and menus -- yet we had some of each. Even the size of the page (in terms of raw HTML and CSS) was entirely too large. Our margins weren't even consistant from browser to browser. So, to be blunt, it sucked.
But I need to also be clear on a couple of related points: none of us had any illusion that this was a pretty UI. We, including the folks who coded it up, hated it. Our team is small and growing and has always had more engineering talent than anything else. And like I said, there were just so many other problems that were more interesting to solve first. So, in the end, we are guilty of indulging our engineering passions first, which is a very human mistake to have made. This situation is also evidence that the executive team has a lot of trust in us. Of course they disliked the UI as much as us, but in the end they let the team decide the priorities - which was the right call for a variety of reasons.
Now, thankfully, we are in a much better place with the UI. In particular, we just launched a new search experience on live.com. And while it is not perfect, I think it shows more UI experimentation than just about anywhere. In the end that is the important thing: not that we nail the UI or any piece on our first try -- but that we have a platform, framework, and philosophy that are all in support of trying new things so that we can progressively improve and learn.
Your take is quite refreshing, to be honest, it doesn't sound like Microsoft. Do you sense the culture there is shifting? Or are you ever frustrated?
"Yes" on both counts, but as an answer to your second question, this says more about me than it does about Microsoft (I have a grumpy old man in me yearning to be free). I don't think MS is particularly clear in how it communicates things to the outside world. I also think that MS often makes mistakes by trying -- wait for it -- too hard. Specifically, instead of doing one thing and nailing it, we'll sometimes do six slightly duplicative things in parallel -- each being slightly below critical mass -- which can create confusion inside and outside of MS until we've had time to sort things out better. So, yes, I get frustrated at times. So it goes. For me, frustration is a biological signal that means I should try harder, which is a good and helpful thing.
I think your first question, about our company culture, is very important, so please forgive the length of this next part.
Let's talk about people as a preface to culture, because I want to get the "evil" thing out of the way. I know the reputation that MS has with some communities. I was hacking Linux in the early nineties, I have a four digit Slashdot userid, and I've personally written over 100,000 lines of open source code. I say all of this to qualify my opinion. Being new to MS, the biggest surprise for me was the people. I knew they were smart. I knew they were driven. I knew they were competitive. But I had no idea they would also collectively tend towards kindness, openness, self-criticism, passion, righteousness, and even uncertainty. These are great people -- from the executive team down to the rank and file -- these are simply wonderful people in every way.
As for culture, I think a change has been underway within MS, but it's not the one that you may be thinking of. This isn't a change about core values, which have been pretty steady as far as I can tell. I think the change is actually a reaffirmation -- an increase in confidence so to speak -- of old core values. Let me explain.
People from Redmond often speak of a "platform" while in the valley they speak of an "ecosystem". Here's the surprise: both groups are talking about the same thing. To MS, Windows is a platform because it fosters a virtuous cycle in two parts: developers come to the platform because it has the most users; users come to the platform because it has the most software.
In the valley, everyone talks about Web 2.0ish things that all come down to creating something of an ecosystem where people and companies of all sizes can be both producers and consumers. In both cases, we see something of an indirect network effect, where the aggregate value -- the whole -- is greater than the sum of the parts.
Here's what's new: many folks within MS are now realizing that a lot of the stuff that's real about Web 2.0 -- not the fluffy buzzwords but the fundamental and important parts -- have actually been part of MS's core vision and mission for *decades*. Do you believe in the importance of ecosystems? Windows is the biggest one ever, that generates vastly more wealth for third parties than for MS. Do you value the long tail? MS has done more for bringing computing to the masses (both people and small businesses) than any other company. Is the democratization of tools important to you? Desktop publishing and simplified development flourish because of tools like Office and Visual Basic. What about network effects? We've created several that have reaped incalculable value to society.
Okay, now I know I am sounding like a corporate drone and I am well aware that for every example above, there are plenty of people in the valley that will bitch about my characterization. When you assume evil, then an ecosystem looks less than well-intentioned. When you assume goodness, then an ecosystem looks like benevolence. The truth is more complex and vastly more interesting.
So look at the facts: MS was pushing Web 2.0 values over a decade ago. It's the realization that this is the case that's new for MS. Our vision for where we've been, what we know how to do, and where we want to go is now much more crisp and consistent than it has ever been before. We know that helping people to be more successful -- seeking win-win dynamics -- is a major part of what we do. It's not new, but clearly connecting the dots between our past and our future is very new and, frankly, downright exciting.
I'm sure all of you have questions for Gary as well. Post them in comments, and maybe we can lure him into the comments thread!
