There are precious few companies in the tech world that are willing to stick their necks out and “do the right thing,” and even fewer who both operate at Internet scale and enjoy Wall Street’s unabashed fandom.
In fact, I can only think of one: Cloudflare. And today, the $65 billion public company* announced a new policy that has the potential to tilt the balance of the Internet back toward the little guys. Starting this morning, Cloudflare will automatically block AI crawlers from copying the content of every website the company protects. And it’s doing it for free.
“We’re changing the rules of the internet across all of Cloudflare,” Matthew Prince, Cloudflare CEO, told The New York Times. “If you’re a robot, now you have to go on the toll road in order to get the content of all of these publishers.”
Cloudflare’s scale – it powers more than 20 percent of the Internet – and its financial profile has consistently allowed it to reimagine how the Web can and should work. Without fear or favor, the company has rolled out services that reset the economics and data practices governing our day to day interactions with the Internet. Most of these services fall into the “dull but important” category, things like domain registration, the Domain Name System, or peering and streaming networks. Each move has been marked by a unmistakable philosophy of what might best be called “De-shittification” – a counter narrative to Cory Doctorow’s now famous observation that the Internet is getting worse and worse, mainly due to large tech companies’ extractive, rent-seeking practices.
Seven years ago, in Cloudflare and the Art of Breaching Moats, I wrote: “With every one of these steps, Cloudflare is doing two things: First, it’s refusing to view the Internet as property to be cornered, as real estate where infrastructure owners can camp out and collect rent. And secondly, Cloudflare is actively exercising a core philosophy which can be honestly described as embracing the best (and most earnest) values of Internet 1.0: The web should be open, freely accessible, and an equal playing field upon which anyone can frolic.”
Fast forward to the present: Generative AI is breaking the Internet as we know it. As I wrote in my recent piece about Bill Gross and ProRata, the economic bargain that powered “the open web” – we let the big guys crawl our sites, they send us traffic in return – is rendered moot within walled gardens like OpenAI, Microsoft CoPilot, and Google Gemini. But those AI chatbots all rely on fresh data from the open web – and until now, they’ve simply taken it. Cloudflare’s new policy means that if the wish to continue the practice, they’ll have to start paying for the privilege.
Blocking AI crawlers by default is only the first step toward building a new economic contract for the web. Step two is creating a marketplace that allows buyers (AI companies) and sellers (websites) to set prices for a fair exchange of value. Cloudflare has already built a version of such a market, currently in private beta. It follows that there’s a step three, and that’s where things get tricky. As I pointed out in that ProRata piece, if AI companies are going to have to pay the piper, they’re going to need new revenue streams beyond their current model of charging a small percentage of users for subscriptions. That’s where advertising comes in – but advertising models for AI chat have yet to develop at any significant scale.
I’ve long predicted that advertising will become the core business model driving AI. Now that Cloudflare has thrown down the gauntlet and is forcing AI companies to pay for their source material, the timeline for the shift to advertising will most likely accelerate. That’s good news for entrepreneurs like Gross, who has developed an advertising solution purpose built for AI chat. But it may spell trouble for behemoths like Google and OpenAI, who so far have been cautious about introducing advertising into their services.
It’s too early to know if Cloudflare’s latest gambit will change the Internet for the better, but one thing is certain: it’s hard to imagine it getting any worse, particularly for the millions of companies, news organizations, and individual creators whose work fuels the ambitions and profits of today’s tech industry. As I wrote in that Moats piece:
“The world needs more Cloudflares, if only to remind us that it’s possible to move past the exhaustingly brutalist architecture we’ve managed to build around ourselves. Perhaps in fact we can trust ourselves to do what’s right for more than just us, more than just our company, more than just our shareholders. Perhaps our industry can dream to reach just a bit further, and imagine we are agents of larger purpose; and that, if we practice enough, we might earn the right to become what we’ve always imagined we could be, over these so many years: A force for good.”
—
*Caveat: I have been a proud investor in Cloudflare for the past six years
One thought on “CloudFlare To The AI Industry: Pay Up!”