AI crawlability
Test your site's AI crawlability. Then fix what blocks the bots.
AutoSEO checks your robots.txt against 27 AI and search crawlers, requests your pages with their real user agents, inspects the raw HTML they receive and validates your llms.txt. You get a score from 0 to 100, findings ranked by severity and fixes you can paste straight into your site.
Why it matters
Blocked pages can't be cited. Most blocks are accidental.
A broad Disallow rule, a firewall that challenges unknown bots or a page that only renders with JavaScript can keep AI crawlers out without anyone noticing. It looks like weak AI visibility, but the cause is technical.
- 01
Most AI crawlers don't run JavaScript
They read the raw HTML. If your content only appears after client-side rendering, they see an empty app shell.
- 02
robots.txt rules interact
Wildcard groups, per-bot groups and crawl delays combine in ways that are hard to read by hand — especially across 27 user agents.
- 03
CDNs treat bots differently
Bot protection can return a challenge to GPTBot while browsers get a normal page. Only a request with the crawler's user agent reveals it.
- Crawlers tested
- 27
- Against your robots.txt
- Check categories
- 9
- robots.txt to response time
- Point score
- 100
- Weighted by crawler purpose
- Self-hosted
- $0
- MIT licensed, every feature
What the check covers
Nine categories, one score, fixes included.
robots.txt per AI crawler
Allowed, partly blocked or blocked for every AI search, assistant, training and SEO crawler — with the exact rule and line that decides it.
Bot access vs. browser
Your pages requested with each crawler's real user agent and compared with a browser request, to catch firewall blocks, challenges and errors.
Raw-HTML rendering
Whether the HTML crawlers receive contains your content or just an app shell, with the frameworks detected.
llms.txt validation and generator
Checks /llms.txt and /llms-full.txt against the llmstxt.org format and drafts a file from your site structure — from a template or written by AI.
Meta robots, sitemaps, schema
Meta robots and X-Robots-Tag directives, sitemap discovery, structured data, canonicals, time to first byte and HTML weight.
Schedules and alerts
Run the check weekly or monthly and get an in-app and email alert when the score drops.
How it works
From domain to fix list in about a minute.
- 1 Step 1
Start a check
Your homepage is always checked. Add key pages like pricing, product or docs, or let AutoSEO sample pages from your sitemap.
- 2 Step 2
Read the score and findings
The score weighs AI search and user-triggered crawlers three times as much as training crawlers. Findings run from critical to passed.
- 3 Step 3
Paste the fix and re-check
Findings come with ready-made snippets, such as a robots.txt group that re-allows blocked crawlers and keeps your existing exclusions.
FAQ
Frequently asked questions
What is AI crawlability?
AI crawlability describes whether the crawlers behind ChatGPT, Claude, Perplexity, Gemini and other AI engines can access and read your pages. It depends on robots.txt rules, how your server and CDN respond to bot user agents, and whether your content is in the HTML without JavaScript.
How do I check if GPTBot can crawl my site?
Run a crawlability check in AutoSEO. It reads your robots.txt for GPTBot, OAI-SearchBot, ChatGPT-User and 24 other crawlers, then requests your pages with their real user agents. You see per bot whether access is allowed, partly blocked or blocked, and why.
Should I block AI training crawlers?
That's a business decision, and AutoSEO treats it as one. Training crawlers such as GPTBot, ClaudeBot and Google-Extended are reported separately from search and user-triggered crawlers: blocking training bots is a warning, blocking AI search bots is critical, because it keeps you out of live answers.
What is llms.txt and do I need one?
llms.txt is a proposed Markdown file at your domain root that gives language models a short summary of your site and links to its key pages, following the llmstxt.org format. It's optional and counts for 10 of the 100 points. AutoSEO validates an existing file or drafts one for you to review and publish.
How is the crawlability score calculated?
The score adds up nine weighted categories: robots.txt access 30, bot HTTP access 20, server-side rendering 15, llms.txt 10, meta robots 10, sitemap 5, structured data 5, canonical 3, and response time and page weight 2. SEO tool crawlers are shown but not scored.
How is this different from a site audit?
The site audit crawls your whole site with 29 technical SEO checks and Lighthouse. The crawlability check focuses on AI crawlers: per-bot robots.txt rules, responses to bot user agents, raw-HTML rendering and llms.txt. Use both.
Is the AI crawlability check free?
The crawlability check is part of the open-source AutoSEO app and free to self-host with every feature. AutoSEO Cloud gives you a managed workspace for $50 per month with $10 of AI and data usage included. AI-written llms.txt drafts can also run on your own Claude Code or Codex through a local agent.
Related features
Works better together. One platform, one dataset.
- AI bot trafficSee which AI crawlers read your site, which pages they fetch and how your server responds.
- Site auditCrawl your site with 29 technical SEO checks, Lighthouse scores and run comparisons.
- AI SEO tasksEvidence-backed GEO and SEO tasks, prioritized and pushed to Jira, Linear or Asana.
- AI citation trackingSee which pages AI engines cite for your prompts, and where you need to be listed.
Find out what AI crawlers see on your site
Start with an AutoSEO Cloud workspace for $50/month, or self-host the open-source edition for free.