Cloudflare is not a household name for most business operators, but its position in the internet's infrastructure makes this announcement significant. The company sits between web servers and the public internet for roughly 20% of websites globally. When it changes a default setting, that setting applies across millions of sites simultaneously.
The specific change is this: from September 15, 2026, any new website using Cloudflare, and any existing free-tier site that does not opt out, will automatically block bots that scrape content for AI training or AI agent use on pages that carry advertising. Search crawlers, the kind that make your site appear in Google results, are still allowed.
The numbers behind this decision are striking. Anthropic's crawler made around 73,000 page requests for every single human visit it referred back to publishers. OpenAI's ratio stood at roughly 1,700 to one. Google's traditional search crawler, by contrast, sends back one human visitor for every 14 pages it reads. The old web economy depended on that return traffic. AI companies have been taking the content and sending almost nothing back.
The scale of the problem has been growing fast. AI scraper traffic grew nearly 600% across 2025 alone. Automated traffic is now expanding eight times faster than human traffic online. One study found that when Google shows an AI summary at the top of search results, users clicked through to any website only 8% of the time, roughly half the rate without a summary. Publishers across all sectors, not just news, are seeing their web traffic fall.
If your business runs a website that depends on visitors to generate ad revenue or subscription sign-ups, that traffic decline is money leaving your business. The industries most affected so far are media and publishing, but the same mechanics apply to any company that publishes content to attract customers: insurers with advice portals, manufacturers with product documentation, travel companies with destination guides, retailers with buying guides.
Cloudflare is also launching a new payment model called Pay Per Use. The previous version, Pay Per Crawl, charged AI companies each time they fetched a page. The new version pays site owners when their content actually appears inside an AI-generated answer. This is a meaningful shift. A single page might get crawled once and then cited in thousands of AI answers, or crawled constantly and never cited at all. Paying per outcome rather than per visit is a more honest way to price the value of content.
Two partners have signed on so far: Ceramic.ai and You.com. The system is early-stage and the dollar amounts have not been disclosed. But the direction matters. It is the first concrete attempt to build a payment layer between AI companies and the web they depend on.
The most consequential pressure is aimed at Google. Google's main crawler both indexes pages for traditional search and collects data to train Gemini and power features like AI Overviews. There is no way for a site owner to appear in AI-powered search results without also feeding Google's AI training. A separate opt-out crawler exists, but it removes you from AI search features entirely. Cloudflare's new defaults mean that sites blocking AI training will also block Googlebot, which is a serious enough consequence to force Google to reconsider how it separates those two functions.
The honest caveat is that Cloudflare controls one fifth of the web, not all of it. Large AI companies have the resources to find other sources, negotiate direct licensing deals, or route around individual infrastructure providers. The robots.txt protocol, the traditional way sites signal crawling preferences, is already ignored by roughly 30% of AI scrapes. Infrastructure-level blocking is harder to circumvent, but it is not a complete solution.
What this signals is a direction, not a finished outcome. The era of free, unlimited web scraping for AI training is ending. The replacement, a permission-based, payment-linked model, is being built in real time. Business operators with content-heavy web presences should start thinking now about which bots they want to allow, and whether the new payment models are worth engaging with.