AI Crawl Control

Pay Per Crawl advanced configuration

You can now configure advanced Pay Per Crawl settings for your zone, including: Disable Pay Per Crawl by URI pattern using Configuration Rules to offer free access to specific pages while charging for others. Dynamic pricing by having your origin return a crawler-price response header, or by using a Cloudflare Worker to set prices based on request properties. When dynamic pricing is enabled, Pay…

Introducing Redirects for AI Training

Cloudflare's network now supports redirecting verified AI training crawlers to canonical URLs when they request deprecated or duplicate pages. When enabled via AI Crawl Control > Quick Actions, AI training crawlers that request a page with a canonical tag pointing elsewhere receive a 301 redirect to the canonical version. Humans, search engine crawlers, and AI Search agents continue to see the or…

Tools to prepare your site for the agentic Internet

AI Crawl Control now includes new tools to help you prepare your site for the agentic Internet—a web where AI agents are first-class citizens that discover and interact with content differently than human visitors. Content Format insights The Metrics tab now includes a Content Format chart showing what content types AI systems request versus what your origin serves. Understanding these patterns h…

Advanced WAF customization for AI Crawl Control blocks

AI Crawl Control now supports extending the underlying WAF rule with custom modifications. Any changes you make directly in the WAF custom rules editor — such as adding path-based exceptions, extra user agents, or additional expression clauses — are preserved when you update crawler actions in AI Crawl Control. If the WAF rule expression has been modified in a way AI Crawl Control cannot parse, a…

Analytics enhancements

AI Crawl Control metrics have been enhanced with new views, improved filtering, and better data visualization. Path pattern grouping In the Metrics tab > Most popular paths table, use the new Patterns tab that groups requests by URI pattern (/blog/*, /api/v1/*, /docs/*) to identify which site areas crawlers target most. Refer to the screenshot above. Enhanced referral analytics Destination patter…

New reference documentation

New reference documentation is now available for AI Crawl Control: GraphQL API reference — Query examples for crawler requests, top paths, referral traffic, and data transfer. Includes key filters for detection IDs, user agents, and referrer domains. Bot reference — Detection IDs and user agents for major AI crawlers from OpenAI, Anthropic, Google, Meta, and others. Worker templates — Deploy the…

AI Crawl Control Read Only role now available

Account administrators can now assign the AI Crawl Control Read Only role to provide read-only access to AI Crawl Control at the domain level. Users with this role can view the Overview, Crawlers, Metrics, Robots.txt, and Settings tabs but cannot modify crawler actions or settings. This role is specific for AI Crawl Control. You still require correct permissions to access other areas / features o…

New AI Crawl Control Overview tab

The Overview tab is now the default view in AI Crawl Control. The previous default view with controls for individual AI crawlers is available in the Crawlers tab. What's new Executive summary — Monitor total requests, volume change, most common status code, most popular path, and high-volume activity Operator grouping — Track crawlers by their operating companies (OpenAI, Microsoft, Google, ByteD…

Pay Per Crawl (Private beta) - Discovery API, custom pricing, and advanced configuration

Pay Per Crawl is introducing enhancements for both AI crawler operators and site owners, focusing on programmatic discovery, flexible pricing models, and granular configuration control. For AI crawler operators Discovery API A new authenticated API endpoint allows verified crawlers to programmatically discover domains participating in Pay Per Crawl. Crawlers can use this to build optimized crawl…

Crawler drilldowns with extended actions menu

AI Crawl Control now supports per-crawler drilldowns with an extended actions menu and status code analytics. Drill down into Metrics, Cloudflare Radar, and Security Analytics, or export crawler data for use in WAF custom rules, Redirect Rules, and robots.txt files. What's new Status code distribution chart The Metrics tab includes a status code distribution chart showing HTTP response codes (2xx…

New Robots.txt tab for tracking crawler compliance

AI Crawl Control now includes a Robots.txt tab that provides insights into how AI crawlers interact with your robots.txt files. What's new The Robots.txt tab allows you to: Monitor the health status of robots.txt files across all your hostnames, including HTTP status codes, and identify hostnames that need a robots.txt file. Track the total number of requests to each robots.txt file, with breakdo…

Enhanced AI Crawl Control metrics with new drilldowns and filters

AI Crawl Control now provides enhanced metrics and CSV data exports to help you better understand AI crawler activity across your sites. What's new Track crawler requests over time Visualize crawler activity patterns over time, and group data by different dimensions: By Crawler — Track activity from individual AI crawlers (GPTBot, ClaudeBot, Bytespider) By Category — Analyze crawler purpose or ty…

Enhanced crawler insights and custom 402 responses

We improved AI crawler management with detailed analytics and introduced custom HTTP 402 responses for blocked crawlers. AI Audit has been renamed to AI Crawl Control and is now generally available. Enhanced Crawlers tab: View total allowed and blocked requests for each AI crawler Trend charts show crawler activity over your selected time range per crawler Custom block responses (paid plans): You…

Introducing Pay Per Crawl (private beta)

We are introducing a new feature of AI Crawl Control — Pay Per Crawl. Pay Per Crawl enables site owners to require payment from AI crawlers every time the crawlers access their content, thereby fostering a fairer Internet by enabling site owners to control and monetize how their content gets used by AI. For Site Owners: Set pricing and select which crawlers to charge for content access Manage pay…

AI Crawl Control refresh

We redesigned the AI Crawl Control dashboard to provide more intuitive and granular control over AI crawlers. From the new AI Crawlers tab: block specific AI crawlers. From the new Metrics tab: view AI Crawl Control metrics. To get started, explore: Manage AI crawlers. Analyze AI traffic.

AI Crawl Control

Every site on Cloudflare now has access to AI Audit, which summarizes the crawling behavior of popular and known AI services. You can use this data to: Understand how and how often crawlers access your site (and which content is the most popular). Block specific AI bots accessing your site. Use Cloudflare to enforce your robots.txt policy via an automatic WAF rule. To get started, explore AI audi…