Cloudflare Begins Blocking AI Crawlers by Default
Cloudflare Introduces New AI Traffic Rules That Separate Search, Training and Agent Crawlers as Publishers Seek Greater Control Over Their Content.
Cloudflare is beginning a major change to how artificial-intelligence crawlers access websites, introducing stricter controls aimed at separating legitimate search activity from AI training and automated agents. From September 15, 2026, new domains on Cloudflare will have AI training and agent crawlers blocked by default on pages that display advertisements, while search crawlers will generally remain allowed.
The move reflects growing tension between website publishers and AI companies over the use of online content. Cloudflare argues that traditional search crawlers can send visitors back to publishers, while AI training crawlers may collect large amounts of material without providing equivalent traffic or compensation. The company is therefore giving website owners more control over which categories of automated systems can access their pages.
Cloudflare’s new system divides AI-related traffic into three broad categories: Search, Agent and Training. Search crawlers are designed to index information for search results, agent crawlers access information to perform tasks for users, while training crawlers collect material that may be used to develop or improve AI models. Website owners can configure these categories separately rather than relying on a single setting that blocks all AI bots.
The policy could also affect mixed-use crawlers that perform more than one function. Cloudflare says such crawlers will be governed by the most restrictive applicable rule. This means that a crawler used for both search and AI training could potentially face restrictions when a website owner chooses to block training activity. The change adds another layer to the increasingly complicated relationship between search engines, AI systems and online publishers.
Cloudflare’s decision is part of a broader shift toward treating access to web content as something publishers should be able to control and, in some cases, monetize. The company has also promoted mechanisms for AI companies to compensate publishers for content access. As AI-generated answers become more common, the new rules could intensify negotiations over who owns, controls and benefits from the information powering these systems.
