>_toolkitbox

~/ / AI articles / Do AI bots follow robots.txt? Cloudflare has started measuring, and how to check your own logs

Do AI bots follow robots.txt? Cloudflare has started measuring, and how to check your own logs

Cloudflare announced on 16 October 2026 that it is introducing a system to continuously evaluate whether Verified bots honor site preferences, such as robots.txt and HTTP 429 responses. This article summarizes the announcement, recalls what robots.txt can and cannot do, and lets you check your own access log and write a robots.txt for AI crawlers.

Published: · Checked against the official sources on 2026-10-24. This article summarizes them and does not replace them.

What Cloudflare announced

  • The post, dated 16 October 2026, says Cloudflare is introducing a system to continuously evaluate whether Verified bots honor site preferences. The system is used internally for now, and Cloudflare plans to make it available to all customers, who will choose the trust threshold that decides how Verified bot traffic is handled.
  • robots.txt is described as the primary way to communicate site owner preferences. Isolated violations may not matter, and the system looks for repeated patterns, in line with RFC 9309.
  • For HTTP 429, the system checks whether a bot slows down, pauses or keeps the same pace after it receives the response. Slowing down or pausing raises trust, and repeatedly ignoring the signal lowers it.
  • The output is a trust band, a range that shows how consistently a bot respects site preferences. Cloudflare says scores will be shared publicly on Radar in the future, and the post gives no numeric scale or thresholds.
  • The post says Perplexity was delisted from the Verified bot list in 2025 after it allegedly accessed opted-out resources and tried to get around blocks.

A related pilot with OpenAI

  • On 15 October 2026 Cloudflare described a research pilot with OpenAI that began in July 2026 with a cohort of about 1% of Cloudflare's network. Cloudflare supplies signals to OpenAI's search crawler, and OpenAI evaluates and ranks the pages itself.
  • The post says this research uses metadata about the web for search and discovery, rather than sharing page content for model training, and that a signal about a page does not grant permission to access it.
  • It points to Cloudflare's AI Crawl Control for managing how AI systems access a site.

What robots.txt can and cannot do

  • It is not access control. RFC 9309 says the rules are not a form of access authorization, and anyone can still request a disallowed URL.
  • It controls crawling, not indexing. Google says a page blocked by robots.txt can still appear in search results, without a description.
  • A /robots.txt that answers with a 4xx status is treated as unavailable, so a crawler may access any resource. A 5xx, or a server that cannot be reached, is treated as unreachable, so a crawler must assume everything is disallowed.
  • Fetchers that act for a person are a special case. OpenAI says that because ChatGPT-User actions are initiated by a user, robots.txt rules may not apply. Perplexity says Perplexity-User generally ignores robots.txt rules because a user requested the fetch. Meta says meta-externalfetcher may bypass robots.txt rules because it performs fetches that a user requested.

See which AI crawlers really visit

Your access log shows what actually arrives. The analyzer below reads an nginx or Apache log in your browser and counts requests from AI crawlers by crawler and by purpose, with unique IP addresses, status codes, whether the crawler fetched robots.txt, and the most requested paths. Compare, for each crawler, whether it fetched robots.txt and which status codes it received, including 429.

A User-Agent is only a claim, so the counts are requests that say they are that crawler. To check that a request is real, compare its source IP address with the ranges the operator publishes. The tool does not look up IP addresses.

Try it: AI Bot Log Analyzer

Paste a log to see which AI crawlers visited.

A User-Agent is a claim, not proof. Anyone can send the same string.

To check that a request is real, compare its source IP address with the ranges the operator publishes. This tool does not look up IP addresses.

Google-Extended and Applebot-Extended never appear in logs: they are control names for robots.txt, and their requests come from Googlebot and Applebot.

Reads the Apache and nginx combined format, JSON lines with common field names, and lists of User-Agent strings.

The crawler list was checked against each operator's documentation on 2026-10-15.

Everything runs in your browser. The log is not uploaded.

Write your preferences

AI companies run several kinds of crawler: some collect training data, some build an index for AI search, and some fetch a page when a person asks. The generator below keeps these apart, so you can disallow training without leaving AI search results, and it builds a robots.txt from the crawler names that each operator documents.

Try it: AI Crawler robots.txt Generator

Presets

Training crawlers

Collect content for model training. Blocking them does not remove you from search or AI answers.

AI search crawlers

Build the index behind AI search answers. Blocking them can stop your pages being cited.

User-triggered fetchers

Load a page only when a person asks. The vendors say robots.txt may not apply.

Names checked against each vendor's documentation on 2026-10-15.

robots.txt

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: CCBot
Disallow: /

robots.txt is a request that well-behaved crawlers follow. It does not stop a bot that ignores it; use a firewall for that.

Sources