~/tools / AI Bot Log Analyzer
AI Bot Log Analyzer — Count AI crawler visits in your access logs
Paste a web server access log to see which AI crawlers visited, what they fetched and how your server answered. Works with nginx and Apache logs and JSON lines. Nothing leaves your browser.
Paste a log to see which AI crawlers visited.
A User-Agent is a claim, not proof. Anyone can send the same string.
To check that a request is real, compare its source IP address with the ranges the operator publishes. This tool does not look up IP addresses.
Google-Extended and Applebot-Extended never appear in logs: they are control names for robots.txt, and their requests come from Googlebot and Applebot.
Reads the Apache and nginx combined format, JSON lines with common field names, and lists of User-Agent strings.
The crawler list was checked against each operator's documentation on 2026-10-15.
Everything runs in your browser. The log is not uploaded.
AI companies run several kinds of crawler: some collect training data, some build an index for AI search, and some fetch a page when a person asks. robots.txt can treat them differently, but only your logs show what actually arrives. This tool reads an access log, finds the requests whose User-Agent names a crawler from the AI Crawler robots.txt Generator's list, and counts them by crawler and by purpose, with unique IP addresses, status codes, whether they fetched robots.txt, and the most requested paths. A User-Agent is only a claim, so the counts are requests that say they are that crawler.
Use Cases
- Seeing whether the crawlers you blocked in robots.txt are still arriving
- Checking that AI search crawlers can fetch your pages without errors
- Spotting a bot-like User-Agent that is not on the AI list
Usage
- Paste log lines, or choose a log file. The file is read in your browser.
- Read the table: one row per crawler, with its purpose, requests, unique IPs and status codes.
- Look through the other bot-like User-Agents for names you do not know.
Three kinds of AI crawler
| Purpose | What it does | Examples | robots.txt |
|---|---|---|---|
| Training | Collects web content to train models | GPTBot, ClaudeBot, meta-externalagent, CCBot | The operators say they follow it |
| AI search | Builds an index used to answer questions | OAI-SearchBot, Claude-SearchBot, PerplexityBot, meta-webindexer | The operators say they follow it |
| User-triggered | Fetches a page when a person asks | ChatGPT-User, Claude-User, Perplexity-User, meta-externalfetcher | OpenAI, Perplexity and Meta say it may not apply; Anthropic says Claude-User follows it |
Blocking one kind does not block the others. For example, stopping a training crawler does not remove you from AI search, which has its own crawler.
Reading the table
- Requests and unique IPs: many requests from one address and a few from many addresses mean different things, so look at both.
- Status codes: for a search crawler, 4xx and 5xx answers mean your pages could not be fetched, so they cannot appear in answers.
- robots.txt column: a crawler that fetches /robots.txt is at least reading your rules. RFC 9309 lets crawlers cache the file but says they should not use a copy older than 24 hours unless it cannot be reached, so a crawler that never asks may be ignoring the rules.
- Other bot-like User-Agents: names that are not on the AI list. They are a place to look for crawlers you have not heard of.
A User-Agent is a claim
Anyone can send the same User-Agent string, so a count by name is a count of requests that say they are that crawler. To check one, compare its source IP address with what the operator publishes.
- OpenAI, Perplexity, Anthropic and Common Crawl publish IP ranges for their crawlers. Anthropic notes that blocking by IP may not reliably opt you out and recommends robots.txt.
- Google describes two checks: a reverse DNS lookup of the IP followed by a forward lookup that returns the same address, or a match against its published IP lists.
- Apple publishes IP prefixes for Applebot and also describes reverse DNS checks.
- The Meta crawler documentation does not list IP ranges, and it says to allow the User-Agent strings or the IP addresses.
What a log cannot show
- Google-Extended has no User-Agent string of its own, and Applebot-Extended does not crawl web pages. Both only control how data collected by Googlebot and Applebot is used, so they never appear as names in a log.
- Requests that were answered by a CDN cache, or stopped before your server, are missing from your origin log.
Sources (checked 2026-10-23): OpenAI: Overview of OpenAI crawlers / Anthropic: Does Anthropic crawl data from the web? / Perplexity: Perplexity crawlers / Meta: Web crawlers / Common Crawl: CCBot / Apple: About Applebot / Google: Google's common crawlers / Google: Verifying Googlebot / RFC 9309: Robots Exclusion Protocol
FAQ
Which log formats work?
The Apache and nginx combined format, one JSON object per line with common field names such as http_user_agent, and plain lists of User-Agent strings, one per line. Lines it cannot read are counted and reported.
Does a match prove the visit was really from that company?
No. A User-Agent can be set by anyone, so a request that says it is GPTBot may come from someone else. Several operators publish the IP ranges of their crawlers, for example OpenAI, Perplexity, Anthropic and Common Crawl, so you can check the source address of a request. The Meta crawler documentation does not list any. This tool does not check IP addresses.
Why don't Google-Extended and Applebot-Extended appear?
Both are control names for robots.txt, not crawlers. Google says Google-Extended has no separate User-Agent string, and Apple says Applebot-Extended does not crawl web pages. The requests come from Googlebot and Applebot, so they cannot be told apart in a log.
What do the purposes mean?
Training means collecting data to train models. Search means building an index for AI search answers. User means a fetch triggered by a person's request; some operators state that these fetches may not follow robots.txt.
Why is my count lower than in another tool?
A log only contains what reached your server. Requests answered from a CDN cache, or blocked before they arrive, may never show up in your origin log.
Is the log uploaded?
No. Reading and counting happen in your browser. Lines beyond 200,000 are not counted so that the page stays responsive.