Bot traffic integrations
Server logs integration
Push access logs from nginx, Apache or any backend to the AutoSEO ingest API — or upload a log file — and see which AI crawlers read your pages, how often and with which status codes.
Overview
What you get with server log ingestion
- An NDJSON ingest API with a bearer token, gzip bodies detected automatically
- Log file uploads up to 1 GB: nginx, Apache, Cloudflare, Akamai, NDJSON or IIS, plain or .gz
- Ready-made configs for nginx, Vector and Fluent Bit
- Visits from 27 known AI, search and SEO crawlers; other lines are ignored
- IP verification against published crawler ranges, with spoofed requests flagged
- Duplicate requests skipped, so re-sending a log does no harm
Use cases
What you can do with your server logs and AutoSEO
Track AI crawlers without a CDN
Hosting on your own servers or a platform without log streaming? Your access logs are enough.
Analyze history in one upload
Upload last month's access log to see which AI crawlers visited before you set up live ingestion.
Tell AI crawler types apart
See training crawlers like GPTBot, search crawlers like OAI-SearchBot and user-triggered fetchers like ChatGPT-User separately.
Find errors served to bots
Status codes per request show which pages AI crawlers can't fetch, so you can fix them before they drop out of answers.
Setup
Start in four steps, live or with a file.
- 1 Step 1
Generate an ingest token
In your project, open Analytics → Bot Traffic → Sync → Server Logs / API and generate a token. It's shown only once.
- 2 Step 2
Write JSON access logs
Use the nginx log format from the Sync tab, or send your own NDJSON with timestamp, user_agent and path — plus ip and status for verification and error tracking.
- 3 Step 3
Ship them to AutoSEO
POST the lines with Authorization: Bearer and your token, or use the Vector and Fluent Bit configs that forward only crawler requests.
- 4 Step 4
Or upload a log file
Click Upload Logs in Bot Traffic, pick the format or let AutoSEO detect it, and upload a file of up to 1 GB.
FAQ
Frequently asked questions
How do I find AI crawlers in my server logs?
Upload an access log to AutoSEO or push your log lines to its ingest API. AutoSEO recognizes 27 AI, search and SEO crawlers by user agent, verifies them against their published IP ranges and shows visits per bot, page and status code.
Which log formats does AutoSEO support?
nginx and Apache combined, common and vhost_combined, Cloudflare Logpush JSON, Akamai DataStream 2 JSON, NDJSON and W3C extended logs from IIS, with a heuristic fallback for custom formats. Uploads can be plain or gzip-compressed.
What are the limits of the ingest API?
Up to 16 MB and 20,000 lines per request and 1,200 requests per minute per token. Larger log files can be uploaded in the app, up to 1 GB each.
Does AutoSEO store my full access logs?
No. Only requests from known crawlers are kept; lines from human visitors are ignored. Duplicates with the same bot, timestamp, IP and path are skipped.
How is the ingest API secured?
Every request needs your project's ingest token, sent as a bearer token. AutoSEO stores only a hash of it, and you can rotate or revoke it at any time.
Bot traffic integrations
More integrations in this category
- CloudflareStream AI crawler visits with a Cloudflare Worker or Logpush.
- AkamaiSend DataStream 2 logs to your AutoSEO HTTPS endpoint.
- FastlyComing soonReal-time log streaming from Fastly.
- AWS CloudFrontComing soonReal-time logs from Amazon CloudFront distributions.
See which AI crawlers read your site
Start with an AutoSEO Cloud workspace for $50/month, or self-host the open-source edition for free.