CONTEUDO:
Cloudflare has issued a definitive ultimatum to the AI industry: separate web crawlers used for traditional search from those deployed for AI training and agentic services. Starting September 15, 2026, the company will automatically block “mixed-use” crawlers from accessing any webpages that host advertisements, effectively forcing AI companies to compensate publishers for their data.
The Impact of New Default Settings
Under this new policy, crawlers that conflate search, AI agent activity, and model training will be restricted by default unless site owners manually override the settings. Cloudflare confirmed these changes will apply to all new customers, newly created sites for existing users, and the entirety of their existing free user base.
This strategic shift directly challenges how AI model providers harvest web content. While Cloudflare acknowledges that most website owners desire search engine discoverability, they are increasingly wary of having their intellectual property exploited for free by AI developers.
Addressing the Google Monopoly
Cloudflare explicitly targeted the “world’s largest search engine”—a clear jab at Google—accusing it of accessing roughly “2x more information” than other AI entities. The company argues that Google creates an environment where site owners are forced to sacrifice their data for AI training simply to remain visible in search results.
Google has previously contested this framing, pointing to Google Extended, a tool allowing publishers to opt out of having their content used for training Gemini Apps or the Vertex API without impacting Google Search indexing. Despite this, Googlebot continues to crawl for both standard search and AI-integrated features like AI Overviews.
A Sustainable Web Ecosystem
“Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge,” stated Cloudflare co-founder and CEO Matthew Prince. His comments follow recent data confirming bots have surpassed human traffic online—a milestone reached significantly earlier than industry projections.
“Cloudflare’s new tools and partnerships give website owners increased visibility and commercial opportunities and benefit AI companies that have bots with clear and transparent intent,” Prince added. The company aims to incentivize AI developers to distinguish between search-only bots and those intended for data scraping.
Evolution of the “Pay Per Use” Model
While Cloudflare continues to offer infrastructure to help users launch their own AI systems, it is simultaneously expanding its suite of defensive tools for publishers. The previously introduced “Pay Per Crawl” marketplace is now evolving into a “Pay Per Use” model, enabling publishers to monetize their content based on the value generated, rather than just the act of fetching data.
This transition also addresses technical inefficiencies, as Cloudflare data indicates that over 50% of AI crawl traffic is wasted on re-fetching unchanged pages, consuming unnecessary bandwidth and compute resources for publishers.
Cloudflare is launching this initiative with initial partners Ceramic.ai and You.com. When publishers opt in, they receive compensation when their content is utilized in Ceramic’s AI search results or accessed by You.com for premium content delivery. Cloudflare expects other AI companies to adopt and customize this monetization framework moving forward.
About the Author
