Once upon a time when search was new, Google came along and put the whole darn Internet in RAM. This was an astonishing (and expensive) feat of engineering at the time – one that gave Google a significant competitive moat. Twenty years ago, very few companies had the know how or the resources to keep an up-to-date copy of the entire web in expensive, super fast silicon. Google’s ability to do so allowed it unprecedented flexibility and speed in its product, and that product won the search crown, building a trillion-dollar market cap along the way.
Since then compute, storage, and engineering costs have declined in a kind of reverse version of Moore’s Law. Pretty much anyone with a bit of funding and some basic Internet crawling skills can stand up a web index – but there’s been no reason to do so. For 15 or so years one of the biggest clichés in venture circles was “no one will ever fund another search engine.” (A second cliché? “No one’s ever said “Just Bing it.”)
Read More