AI bot traffic analytics
Know which AI bots read your site. Page by page, day by day.
AI engines can only cite what their crawlers have fetched. AutoSEO reads your server logs, CDN or log files, recognizes 27 AI and search crawlers — from GPTBot and ClaudeBot to PerplexityBot and OAI-SearchBot — verifies them against published IP ranges, and shows which pages they visit and how your site responds.
Why it matters
No crawl, no citation. Every AI answer starts with a fetch.
Before an AI engine can quote your pricing page, its crawler has to fetch it — for training, for a search index or live when a user asks. Server logs are the only place that shows whether that happened, and most analytics tools filter bots out entirely.
- 01
Training, search and user fetches differ
GPTBot collects training data, OAI-SearchBot builds a search index and ChatGPT-User fetches pages live for a user. Each tells you something different.
- 02
Spoofed bots distort the picture
Anyone can send a GPTBot user agent. Checking the IP address against the operator's published ranges separates real crawlers from scrapers.
- 03
Errors cost citations
A crawler that hits 404s, redirect chains or server errors on key pages leaves with nothing to cite. Status codes per bot show where to fix.
- Crawlers recognized
- 27
- AI, search and SEO bots
- Operators verified
- 6
- Against published IP ranges
- Live connectors
- 3
- Cloudflare, Akamai, server logs
- Log file uploads
- 1 GB
- Plain or gzip, per file
What you get
Every AI crawler visit, verified and broken down.
27 crawlers, grouped by purpose
Bots from OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple, Meta, Amazon and more, labeled as training, search index, user-triggered or SEO tool.
IP verification
Requests from OpenAI, Anthropic, Perplexity, Google, Microsoft and Apple crawlers are checked against their published IP ranges, and spoofed visits are counted separately.
Visits per day and per bot
Daily crawler visits, unique URLs and trends against the previous period, plus when each bot was last seen.
Crawled pages
Which URLs each crawler fetched and how often, so you can see whether your pricing, product and docs pages get read.
Response performance
Status codes per crawler — success, redirects, client and server errors — to catch pages that fail only for bots.
Logs, CDN or API
Upload nginx, Apache, Cloudflare, Akamai or NDJSON logs, stream from a Cloudflare Worker, Logpush or Akamai DataStream 2, or push NDJSON from any backend.
How it works
From raw logs to a crawler report in three steps.
- 1 Step 1
Choose how data arrives
Upload a log file, deploy the Cloudflare Worker on any plan, use Logpush or Akamai DataStream 2, or ship logs with nginx, Vector or Fluent Bit.
- 2 Step 2
AutoSEO filters and verifies
Only crawler requests are kept. Each is matched to a bot, checked against published IP ranges where available and deduplicated.
- 3 Step 3
Spot gaps and fix them
See which bots skip important pages or run into errors, then run a crawlability check to find out why.
FAQ
Frequently asked questions
What is AI bot traffic?
AI bot traffic is the requests AI crawlers make to your website: training crawlers like GPTBot and ClaudeBot, search crawlers like OAI-SearchBot and PerplexityBot, and user-triggered fetchers like ChatGPT-User that load a page while answering a question. It shows up in server logs, not in normal web analytics.
How do I see if ChatGPT crawls my website?
Look for GPTBot, OAI-SearchBot and ChatGPT-User in your server or CDN logs. AutoSEO does this for you: upload a log file or connect Cloudflare, Akamai or a log shipper, and it lists every visit by these crawlers with the pages they fetched and the status codes they got.
Which AI crawlers does AutoSEO recognize?
27 user agents, including GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Googlebot, GoogleOther, Bingbot, Applebot, Meta-ExternalAgent, Bytespider, Amazonbot, DuckAssistBot, MistralAI-User and CCBot. SEO crawlers such as AhrefsBot and SemrushBot are shown for comparison.
How does bot verification work?
OpenAI, Anthropic, Perplexity, Google, Microsoft and Apple publish the IP ranges their crawlers use. AutoSEO loads these lists and checks the IP address of each request, so a visit that claims to be GPTBot from an unknown address is marked as unverified.
Does the Cloudflare Worker slow down my site?
The Worker passes every response through unchanged and reports bot visits in the background. On Cloudflare Enterprise you can use Logpush instead, which sends logs without touching requests at all.
Can I use AutoSEO without CDN access?
Yes. Upload access logs from nginx, Apache or other servers as plain or gzip files of up to 1 GB, or push NDJSON to the ingest API with a project token. Fastly and AWS CloudFront connectors are listed as coming soon.
Is AI bot traffic analytics free?
Bot traffic analytics is part of the open-source AutoSEO app and free to self-host with every feature. AutoSEO Cloud gives you a managed workspace for $50 per month with $10 of AI and data usage included. Log uploads and the Cloudflare and Akamai connectors cost nothing extra.
Related features
Works better together. One platform, one dataset.
- AI crawlability checkCheck whether AI crawlers can reach and read your site, with fixes you can paste in.
- AI traffic analyticsSee how many visitors ChatGPT, Perplexity and Gemini send you, and what they convert.
- Site auditCrawl your site with 29 technical SEO checks, Lighthouse scores and run comparisons.
- AI citation trackingSee which pages AI engines cite for your prompts, and where you need to be listed.
See which AI crawlers read your site
Start with an AutoSEO Cloud workspace for $50/month, or self-host the open-source edition for free.