Bot traffic integrations
Track AI crawlers where they reach your site. At the CDN or in your logs.
Connect Cloudflare, Akamai or your server logs and AutoSEO records every visit from GPTBot, ClaudeBot, PerplexityBot, Google-Extended and other AI crawlers — which pages they request, how often and which status codes they get back.
Bot traffic integrations
5 integrations
- CloudflareStream AI crawler visits with a Cloudflare Worker or Logpush.
- AkamaiSend DataStream 2 logs to your AutoSEO HTTPS endpoint.
- Server logs / APIPush NDJSON from nginx, Apache or any backend, or upload log files.
- FastlyComing soonReal-time log streaming from Fastly.
- AWS CloudFrontComing soonReal-time logs from Amazon CloudFront distributions.
What you get
See what AI engines read before they answer.
Every AI crawler, by purpose
Visits are grouped by crawler, company and purpose — model training, search indexing or fetches for a user's question — so you can tell GPTBot from ChatGPT-User.
Spoofed bots flagged
Requests are checked against the IP ranges OpenAI, Anthropic, Perplexity, Google, Microsoft and Apple publish. The lists refresh daily.
Pages and errors per crawler
See the URLs each crawler requests and the ones that fail with 4xx or 5xx. Repeated failures turn into prioritized tasks.
Streamed or uploaded
Stream from a Cloudflare Worker, Logpush or Akamai DataStream 2, push NDJSON from any backend, or upload log files up to 1 GB.
FAQ
Frequently asked questions
How do I track AI crawlers on my website?
Connect a log source: a Cloudflare Worker or Logpush, Akamai DataStream 2, or the server-log API for nginx, Apache or any backend. AutoSEO identifies AI crawlers by user agent, verifies them by IP range and stores only crawler requests.
Does the Cloudflare integration work on the free plan?
Yes. The Cloudflare Worker works on every plan; Logpush requires an Enterprise plan. The Worker passes responses through unchanged and reports bot visits in the background.
Can I upload server log files instead?
Yes. Upload access logs up to 1 GB, gzip included, in nginx or Apache combined format, W3C/IIS format, Cloudflare or Akamai JSON, or NDJSON. For continuous data, push NDJSON to the ingest API; examples for nginx, Vector and Fluent Bit are in the app.
Which AI crawlers does AutoSEO recognize?
Among others GPTBot, OAI-SearchBot and ChatGPT-User from OpenAI, ClaudeBot and Claude-User from Anthropic, PerplexityBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Bytespider, Amazonbot and CCBot — plus Googlebot, Bingbot and common SEO crawlers.
Are Fastly and AWS CloudFront supported?
Both connectors are coming soon. Until then, send their logs through the server-log API or upload log files.
Does AutoSEO store my human visitors' requests?
No. Lines from regular visitors are ignored; only requests from known crawlers are saved, and duplicate lines are skipped.
Is AI bot tracking free?
Yes. All bot traffic connectors are part of the open-source app and free to self-host. AutoSEO Cloud includes them in a managed workspace for $50 per month, with $10 of AI and data usage included.
All integrations
Connect the tools you already use
- CMS integrationsPublish AI-optimized drafts to WordPress, Webflow and Framer, as drafts or live.
- Analytics integrationsMeasure visits, conversions and revenue from AI platforms in GA4, Matomo or Piwik PRO.
- Search Console integrationsTurn Google Search Console and Bing queries into AI prompts, tasks and opportunities.
- Attribution integrationsTie orders, leads and deals to AI search — from your shop, CRM, forms and surveys.
- Project management integrationsPush evidence-backed SEO and GEO tasks to Jira, Linear, Asana, ClickUp and more.
- Data & reporting integrationsGet AutoSEO data into BI tools, spreadsheets, scripts and AI agents.
Know which AI crawlers read your pages
Start with an AutoSEO Cloud workspace for $50/month, or self-host the open-source edition for free.