Cloudflare handles traffic for roughly 20% of all websites globally. When it changes its default settings, the effect is not theoretical.
On July 1, Cloudflare replaced its single AI-blocking switch with three separate controls: Search, Agent, and Training. Search bots index pages to feed results later. Agent bots read pages in real time on behalf of someone waiting for an answer. Training bots pull content to build or improve AI models. The new controls went live immediately for all customers, including the free tier.
The deadline that matters is September 15. From that date, any page with advertising will block agent and training bots by default. The rule applies to all new domains, all new sites set up by existing customers, and every existing customer on the free plan who does not actively change their settings before then.
The reasoning Cloudflare uses is clear. An advertisement on a page signals that the site owner built it for human eyes, expecting that human to see the ad. A search bot that sends readers back to the page is a fair exchange. A bot that reads the page and hands a summary to someone else cuts the publisher out of that deal entirely.
For anyone running AI tools in their business, this is the practical problem. Research agents that check competitor pricing, monitoring tools that scan supplier announcements, customer service systems that pull product specifications from manufacturer sites: all of these are classified as agent bots. The classification is based on behaviour, not on what the operator calls the tool. If it reads a live page and hands the result to a person, it is an agent.
Ad-supported pages are exactly the pages these tools want to reach, because that is where news, pricing information, reviews, and product coverage live. The blocks operate at the network level. Unlike the old robots.txt system, which was essentially a request a bot could choose to ignore, Cloudflare's blocks cannot be bypassed by changing how the bot identifies itself.
A complication lands on Google specifically. Googlebot blends search indexing with AI training in a single crawler, which means a site that blocks training bots also blocks Googlebot. Publishers wanting to keep their Google search ranking and block AI training face a genuine conflict, not a theoretical one. Bing and Apple's crawler present the same problem.
The payment angle is worth watching. Cloudflare is evolving a feature called Pay Per Use, where publishers are paid not when their page is fetched but when their content actually appears in an AI answer. Two early partners are Ceramic.ai and You.com. More than half of AI crawler traffic currently goes to re-fetching pages that have not changed since the last visit, so both sides have an interest in making the system more efficient.
The weakness in the whole arrangement is that the three categories rely on AI companies honestly declaring what their bots are doing. A company that wants to run a training crawl without being classified as a training crawl has an obvious reason to mislabel it, and the announcement does not explain what stops that.
The practical steps are different depending on which side of this you are on. If your business uses AI tools that read live web pages, check whether those tools will still reach the pages they need after September 15. The failure mode is not a clear error message. It is an AI tool that still works but gives answers built from a narrower slice of the web, with the most relevant ad-supported pages missing.
If your business owns a website with ads, check your Cloudflare plan tier. Free-tier customers are moved to the new defaults automatically on September 15 unless they change their settings first. Deciding whether to block training bots means accepting that Googlebot may be blocked too, which affects your visibility in Google search results.
The broader shift here is from a web where access was free and unlimited by default to one where access to commercial content is increasingly negotiated. That process is happening across lawsuits, licensing deals, and now infrastructure defaults. September 15 is just the next hard date on that calendar.