Bot traffic integrations
Akamai integration
Point an Akamai DataStream 2 stream at your AutoSEO endpoint and see which AI crawlers request your pages, which status codes they get and whether they're genuine.
Overview
What you get with the Akamai integration
- DataStream 2 delivery to a custom HTTPS endpoint in JSON, gzip allowed
- Authentication with a bearer token in a custom header — stored only as a hash
- Visits from 27 known AI, search and SEO crawlers; all other lines are ignored
- IP verification against the ranges crawler operators publish, with spoofed requests flagged
- Crawled pages, visits per day and response status for every bot
- Data appears within minutes of activating the stream
Use cases
What you can do with Akamai and AutoSEO
See AI crawlers at the edge
Your CDN sees every request, including the ones cached at the edge that never reach your origin logs.
Spot fake crawlers
Requests whose user agent claims to be GPTBot or ClaudeBot but come from outside the published IP ranges are flagged as spoofed.
Catch errors served to bots
Status codes per request show when AI crawlers get redirects, 4xx or 5xx responses instead of your content.
Setup
Connect Akamai in four steps, with DataStream 2.
- 1 Step 1
Generate an ingest token
In your project, open Analytics → Bot Traffic → Sync → Akamai and generate a token. It's shown only once.
- 2 Step 2
Create a DataStream 2 stream
In Akamai Control Center, create a stream for your property with the JSON log format and the fields reqTimeSec, cliIP, reqHost, reqMethod, reqPath, queryStr, statusCode, UA and totalBytes.
- 3 Step 3
Set the Custom HTTPS destination
Use the endpoint URL from AutoSEO, no authentication, and a custom header Authorization: Bearer with your token. Gzip compression is allowed.
- 4 Step 4
Activate the stream
Crawler visits appear in Bot Traffic within a few minutes.
FAQ
Frequently asked questions
How do I track AI crawlers on Akamai?
Create a DataStream 2 stream with a Custom HTTPS destination that points to your AutoSEO ingest endpoint and carries your ingest token in the Authorization header. AutoSEO keeps only requests from known crawlers and shows them under Bot Traffic.
Which DataStream 2 fields does AutoSEO need?
reqTimeSec, cliIP, reqHost, reqMethod, reqPath, queryStr, statusCode, UA and totalBytes, in the JSON log format. The client IP is needed to verify crawlers against their published IP ranges.
Does AutoSEO store all of my Akamai logs?
No. Lines from human visitors and unknown bots are ignored, and duplicate crawler requests are skipped. Only visits from the 27 known AI, search and SEO crawlers are stored.
How is the Akamai ingest endpoint secured?
Every request needs your project's ingest token, which AutoSEO stores only as a hash. You can rotate or revoke the token at any time; the old one stops working immediately.
Bot traffic integrations
More integrations in this category
- CloudflareStream AI crawler visits with a Cloudflare Worker or Logpush.
- Server logs / APIPush NDJSON from nginx, Apache or any backend, or upload log files.
- FastlyComing soonReal-time log streaming from Fastly.
- AWS CloudFrontComing soonReal-time logs from Amazon CloudFront distributions.
See which AI crawlers reach your edge
Start with an AutoSEO Cloud workspace for $50/month, or self-host the open-source edition for free.