- Home
- Files and configuration
- Robots.txt Tester
Robots.txt Tester
Test whether a URL is crawlable — and see exactly which rule makes the decision.
Your robots.txt is fetched live and evaluated according to RFC 9309, including group selection and longest-match precedence — the same algorithm a compliant crawler uses. Google retired its own tester in December 2023. No usage limit.
What this tool is for
robots.txt looks like the simplest file on a website, and it is the one whose behaviour is most consistently misread. Two rules governing the same path do not resolve in reading order. A crawler that finds its own name ignores the general rules entirely. And blocking a page does not remove it from search results.
The consequences run in both directions, and both are expensive. A rule intended to hide an admin area that quietly blocks an entire product catalogue. Or a file everyone assumes is protecting something that has permitted access all along.
This tester applies the real algorithm to real URLs and reports which line decided. Since Google removed its own tester from Search Console at the end of 2023, there is no first-party way to answer the question — and it remains worth answering before a deployment rather than after.
How to read your results
The verdict per URL is allowed or blocked for the crawler you selected. Comparing all crawlers at once is the fastest way to spot a rule that treats one bot differently from the others without anyone intending it.
The deciding rule is the useful part. It shows the exact line that applied and why it beat the others — almost always because it matched a longer portion of the path, not because it appeared first.
The selected group matters just as much. Only one User-agent
group applies to a given crawler: the most specific one that names it. Everything written
under User-agent: * is then irrelevant for that crawler, however permissive
it may be.
Wildcards behave in a specific way. * matches any sequence
of characters; $ anchors the end of the URL. Disallow: /*.pdf$
blocks PDFs; Disallow: /*.pdf without the anchor also blocks
/guide.pdf.html.
The most common mistakes
Using robots.txt to hide sensitive pages. The file is public and lists
every path you named. Disallow: /internal-backups/ is a directory listing for
anyone curious. Use authentication for anything that genuinely must not be reached.
Blocking a page to remove it from search. It produces the opposite result: the URL can remain listed with no description, and the crawler can never read the noindex tag that would have removed it. Allow crawling, serve noindex, then block later if you still want to.
Blocking CSS and JavaScript. A habit from a decade ago. Google renders pages now; blocking the assets prevents it from seeing the layout, which affects mobile usability assessment. Let them through.
Adding a named group without repeating the general rules. The single most common defect in the file. Naming a crawler cancels the wildcard group for it entirely — everything you want that crawler to obey must be restated inside its own group.
Deploying a staging robots.txt to production. Staging environments
block everything, and that file gets copied over. A Disallow: / in
production removes the whole site from search, and it can persist for weeks before
anyone connects the drop in traffic to a two-line file.
Related tools
- AI Crawler Access Checker Check whether GPTBot, ClaudeBot and PerplexityBot can read your pages
- Googlebot Simulator See a page exactly as a search engine crawler sees it
- llms.txt Generator Build the llms.txt file that tells AI assistants what your site is about
- Meta Tags Analyzer Title, description, Open Graph and indexing directives of a page
Frequently asked questions
Which rule wins when two rules match?
The most specific one — the longest matching path — regardless of the order they are written in. This surprises almost everyone, because people assume the file is read top to bottom like a firewall. Under RFC 9309, Disallow: /admin/ followed by Allow: /admin/public/ allows the second path, and reversing the two lines changes nothing.
Does robots.txt keep a page out of Google?
No, and this is the most consequential misunderstanding in the file. A blocked URL can still be listed, without a description, if other pages link to it — the crawler cannot fetch the page to discover a noindex directive. To remove a page from the index you must allow crawling and serve noindex.
Why does Google Search Console no longer have a tester?
Google retired the robots.txt tester from Search Console in December 2023, replacing it with a simpler report that shows the fetched file but does not test individual URLs against it. That is why standalone testers matter again.
Is robots.txt case-sensitive?
The paths are, the user-agent names are not. /Admin/ and /admin/ are different paths, but Googlebot and googlebot are the same crawler. Mixed-case URLs are a frequent source of rules that appear to be ignored.
Where must the file be?
At the root of the host, and nowhere else: https://example.com/robots.txt. A file in a subdirectory is never read. Each subdomain needs its own — blog.example.com does not inherit the rules of example.com.
What happens if the file returns a 500 error?
Google treats a persistent server error as a signal to stop crawling the site entirely, on the assumption that the file might forbid everything. A missing file, by contrast, means everything is allowed. A broken robots.txt is far more dangerous than an absent one.