- Home
- Artificial intelligence
- AI Crawler Access Checker
AI Crawler Access Checker
Find out which AI crawlers can read your pages — and which rule actually decides.
Your robots.txt is fetched live and parsed according to RFC 9309, the standard that governs robots exclusion. Group precedence and longest-match rules are applied exactly as a compliant crawler would apply them. No usage limit.
What this tool is for
A growing share of searches never reaches a list of blue links. People ask ChatGPT, Perplexity or Google's AI Overviews, and get an answer with a handful of citations. Being among those citations requires one thing first: the crawler must be allowed to read you.
That permission lives in robots.txt, a file most sites set up years ago and never revisited. AI crawlers did not exist then. Many sites therefore sit in a state nobody chose — either wide open when the owner assumed protection, or silently blocked by a rule written for a different purpose.
This tool reads your robots.txt the way a compliant crawler does and gives a verdict per bot: allowed or blocked, and by which line. It separates the crawlers that train models from those that fetch pages to cite them, because that distinction is usually what matters and is almost always conflated.
How to read your results
The verdict per bot is the headline. Allowed means the crawler may read your pages. Blocked means it should not — should, because robots.txt is a convention that well-behaved crawlers honour, not a wall.
The deciding rule matters more than the verdict. It shows the exact line that applies, and that is where surprises live. A crawler that finds its own name in the file applies that group and ignores the wildcard group entirely — even if the wildcard group is more permissive and even if the named group is empty.
The training versus citation split deserves a deliberate decision. GPTBot and Google-Extended feed model training. OAI-SearchBot and PerplexityBot index for citation, sending traffic back. ClaudeBot has separate identities for each. Blocking all of them to "protect the content" also removes you from the answers.
The absence of robots.txt is not a problem in itself. No file means everything is allowed, which for a site that lives on visibility is usually the right outcome. Add a file when you want to say no to something specific — not out of habit.
The most common mistakes
Believing robots.txt hides a page. It does the opposite of what people
expect. A blocked URL can still appear in results, without a snippet, if other sites link
to it — the crawler cannot read the page to discover a noindex directive. To keep a page
out of the index, allow crawling and serve noindex. Blocking guarantees the
directive is never seen.
Writing a named group and expecting the wildcard to still apply. This is
the single most common mistake in the file. Under RFC 9309, one group applies and only
one. A User-agent: GPTBot section with a single narrow rule cancels every
permission written under User-agent: *.
Blocking training and losing citations by accident. The bot names are close enough to be confused, and the documentation moves. Check with the tool rather than from memory — the roster of AI crawlers has changed several times a year since 2023.
Assuming a block erases what was already collected. robots.txt governs future access. Content gathered before the rule existed stays where it is. Blocking today is a decision about tomorrow.
Forgetting that robots.txt is public. Anyone can read yours, including
the paths you tried to hide by listing them. A Disallow: /admin-secret/ line
is a signpost, not a lock.
Related tools
- Robots.txt Tester Test whether a URL is allowed or blocked, and by which rule
- llms.txt Generator Build the llms.txt file that tells AI assistants what your site is about
- Googlebot Simulator See a page exactly as a search engine crawler sees it
- Meta Tags Analyzer Title, description, Open Graph and indexing directives of a page
Frequently asked questions
Should I let AI crawlers read my site?
It depends on what you sell. If your traffic comes from being found, being quoted in AI answers is the new visibility and blocking is self-harm. If your content is the product — paid archives, a database, original photography — blocking makes sense. There is no universally right answer, only a decision that should be deliberate rather than accidental.
What is the difference between GPTBot and OAI-SearchBot?
GPTBot collects content that may be used to train future models. OAI-SearchBot indexes pages so ChatGPT can cite them when answering a question, with a link back to you. ChatGPT-User fetches a page in real time because a user asked about it. Blocking all three is a common mistake: many sites want to refuse training while still being cited.
Does blocking Google-Extended hurt my Google rankings?
No. Google-Extended controls whether your content feeds Gemini and grounds AI answers. It is separate from Googlebot, which handles Search indexing. Blocking Google-Extended leaves classic rankings untouched — but it can remove you from AI Overviews, which increasingly sit above the traditional results.
Why does a rule seem ignored?
Almost always because of group precedence. A crawler that finds its own name in robots.txt applies that block and completely ignores the wildcard group. So a site with Allow under User-agent: * and a bare User-agent: GPTBot group with nothing useful in it does not do what its author expected. This tool shows which group actually applies.
Is robots.txt legally binding?
No. It is a convention, not a technical barrier and not a contract. Major operators state that they honour it, and evidence suggests they largely do, but a crawler that ignores robots.txt breaks no law by doing so. If you need enforcement rather than cooperation, block at server or firewall level.
How quickly does a change take effect?
Crawlers cache robots.txt, typically for up to twenty-four hours. A change is therefore not instant. More importantly, blocking a crawler today does not remove content already collected: robots.txt governs future access, not past copies.