Key Takeaways:
Cloudflare is forcing AI companies to choose between search indexing and model training on ad-supported websites, a shift that could reshape how $200 billion in digital content is accessed and monetized.
Cloudflare will block mixed-use web crawlers from ad-supported pages by default starting Sept. 15, 2026, forcing AI companies to separate search indexing from model training or pay publishers for access. The policy targets crawlers that blend traditional search, AI agent use, and model training into a single bot.
"The vast majority of our customers want AI to engage with their content," Stephanie Cohen, chief strategy officer at Cloudflare, said. "However, for those who rely on advertising and subscriptions, the challenge is distinct: they want to remain discoverable without being forced to give their work away for free."
The default applies to new Cloudflare customers, new sites set up by existing customers, and all existing free users, with the option to adjust settings through their dashboards. Cloudflare is also evolving its Pay Per Crawl experiment into a broader Pay Per Use model, where publishers receive compensation when their content appears in AI search results or when an agent purchases premium information. Ceramic.ai and You.com are the first partners in the program. Cloudflare's data shows more than 50% of AI crawler traffic involves re-fetching pages that have not changed, wasting bandwidth for both publishers and AI companies.
The move addresses a structural breakdown in the web's economic model. AI answer engines extract information and deliver summaries without sending users to the original source. Anthropic crawls 11,122 pages for every single referral it sends back, according to Adexchanger data, while AI chatbot referrals drive roughly 96% less traffic than traditional search and users click cited sources only about 1% of the time. Publishers have lost 20% to 90% of their traffic and revenue over the past year, creating urgency for a new compensation framework.
Cloudflare specifically called out Google as having access to about twice as much information as other AI companies because the search giant makes it difficult for customers to remain discoverable without being used for AI training. Google offers a bot called Google Extended that lets site owners opt out of AI use, but its flagship Googlebot crawls for Search including AI features like AI Overviews and AI Mode. "Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge," Cloudflare co-founder and CEO Matthew Prince said in announcing the policy.
The infrastructure provider is not alone in targeting the content licensing opportunity. TollBit, ProRata, and Microsoft have all moved into the space, each with a different approach to who controls the terms. Really Simple Licensing is pushing for an open standard before any single company locks in the rules. Nearly 70% of publishers expect AI licensing deals to generate at least some revenue over the next three years, according to Press Gazette, though most see it as a minor source today.
Cloudflare plans to launch an Attribution Business Insights dashboard showing how AI bots access content, where that content is cited, and how much human traffic different AI platforms return. The tool would make answer engine optimization — the practice of making content findable by AI rather than just by search engines — measurable for the first time.
The risks of Cloudflare's approach are significant. The company could potentially control the identification of agents, the permissions layer, usage measurement, and payment infrastructure, concentrating power in a single intermediary. Attribution also remains unsolved: an AI-generated answer may combine dozens of sources or paraphrase an original idea without displaying a citation. Paid access could also entrench existing advantages for well-funded AI platforms while making the web more expensive for startups, researchers, and open-source developers.
Cloudflare trades at roughly 45 times forward earnings, reflecting the market's expectation that its infrastructure position translates into new revenue streams. The company's Pay Per Use model, if widely adopted, could add a recurring revenue line tied to AI content consumption — a business model no cloud infrastructure provider has yet proven at scale.
This article is for informational purposes only and does not constitute investment advice.
