Bot traffic integrations
Cloudflare integration
Deploy a small Cloudflare Worker — or point Logpush at AutoSEO on Enterprise — and see which AI crawlers visit your pages, how often and which status codes they get.
Overview
What you get with the Cloudflare integration
- A Worker for every Cloudflare plan, Logpush for Enterprise zones
- Real-time visits from 27 known AI, search and SEO crawlers, including GPTBot, ClaudeBot and PerplexityBot
- IP verification against the ranges crawler operators publish, with spoofed requests flagged
- Crawled pages, visits per day and response status for every bot
- Responses pass through unchanged — reporting runs in the background
- An ingest token that is stored only as a hash and can be rotated anytime
Use cases
What you can do with Cloudflare and AutoSEO
See which AI crawlers read your site
Find out whether GPTBot, ClaudeBot, PerplexityBot and Google-Extended actually fetch your pages — and how often.
Separate real bots from fakes
Each request is checked against the operator's published IP ranges, so spoofed user agents don't inflate your numbers.
Catch crawl errors early
Status codes per request show when AI crawlers hit redirects or errors instead of your content.
Confirm new pages get crawled
After you publish or update content, check that AI crawlers pick it up in the crawled pages list.
Setup
Connect Cloudflare in four steps, on any plan.
- 1 Step 1
Generate an ingest token
In your project, open Analytics → Bot Traffic → Sync → Cloudflare and generate a token. It's shown only once.
- 2 Step 2
Deploy the Worker
Download the Worker script — it contains no secrets — and deploy it with Wrangler or the Cloudflare dashboard. Store the token as the secret AUTOSEO_TOKEN.
- 3 Step 3
Add a route
Route your site through the Worker with a Workers Route such as example.com/*. Responses are passed through unchanged.
- 4 Step 4
Watch bot visits arrive
Crawler requests are reported as they happen. On Enterprise, a Logpush job to AutoSEO can replace the Worker.
FAQ
Frequently asked questions
How do I track AI crawlers on Cloudflare?
Generate an ingest token in AutoSEO, deploy the AutoSEO Worker with that token as a secret and add a route for your site. Every request from a known AI or search crawler is then reported to AutoSEO in real time.
Does the Worker change my responses?
No. It forwards each request to your origin and returns the response unchanged. Only requests from known crawler user agents trigger a background report, and a failed report never breaks the page.
Do I need a Cloudflare Enterprise plan?
No. The Worker runs on every Cloudflare plan; its requests count toward your normal Workers usage. Logpush is an alternative for Enterprise zones that sends HTTP request logs as compressed NDJSON, and only crawler requests are stored.
Which crawlers does AutoSEO recognize?
27 AI, search and SEO crawlers, including GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, Googlebot, Bingbot, Applebot-Extended, Meta-ExternalAgent, Bytespider and CCBot. All other traffic is ignored.
What data does the Worker send to AutoSEO?
One line per crawler request: timestamp, IP address, method, host, path, status code, user agent and response size. Requests from human visitors aren't reported.
What if my site isn't on Cloudflare?
AutoSEO also accepts Akamai DataStream 2, an NDJSON ingest API for nginx, Apache or any backend, and manual log file uploads of up to 1 GB.
Bot traffic integrations
More integrations in this category
- AkamaiSend DataStream 2 logs to your AutoSEO HTTPS endpoint.
- Server logs / APIPush NDJSON from nginx, Apache or any backend, or upload log files.
- FastlyComing soonReal-time log streaming from Fastly.
- AWS CloudFrontComing soonReal-time logs from Amazon CloudFront distributions.
See which AI crawlers read your site
Start with an AutoSEO Cloud workspace for $50/month, or self-host the open-source edition for free.