Direct answer: This is a robots.txt checker that runs entirely in your browser. Paste a file, enter a path, and it applies RFC 9309's group selection and longest-match rules to give GPTBot, OAI-SearchBot, PerplexityBot, Bytespider and ClaudeBot separate allow/deny verdicts, showing the line that decided each one. It makes no requests, cannot see your live file, and cannot prove that AI will cite you.
The tool exists because of one sentence in acceptance meetings: "We already allow AI." That sentence cannot be checked. The same file can produce opposite answers for different bots; Allow does not simply win; a missing file is not an explicit allow. Making those three facts into line-by-line verdicts is more useful than another paragraph of explanation.
1. Pick the group: User-agent values match as a case-insensitive prefix of the crawler name; longest wins, * only as fallback.
2. Pick the rule: the longest matching path wins, and Allow wins a tie.
3. Nothing matches = allowed; no file at all = absent, which is not an explicit allow.
The tool: paste → enter a path → check five bots
Calculating…
| Crawler | Verdict | Group used | Winning rule | Line |
|---|
The verdict applies to this one path only; another path or another bot can give a different answer.
Change the path to /download/public/manual.pdf and the * group flips from "not allowed" to "allowed": the longer Allow: /download/public/ wins. Switch the user agent to GPTBot/1.2 and it flips back, because its dedicated group blocks the whole site and a dedicated group beats *.
How the verdict is computed
| Step | Rule | Common mistake |
|---|---|---|
| 1 Group | Take the product token from the crawler name (e.g. GPTBot/1.2), prefix-match it case-insensitively against each User-agent value, keep the longest; only fall back to * | "A * line covers every bot" only when no more specific group exists |
| 2 Rule | Inside that group set, the longest matching Allow/Disallow path wins; equal length goes to Allow | It is not last-one-wins, and Allow is not automatically first |
| 3 No rule | No rule matches the path → allowed | An "Allow only" file does not implicitly block everything else |
| 4 No group | No specific group and no * → unrestricted | Same outcome as an allow-all *, but different evidence |
| 5 Empty | Disallow: with no value → no restriction; empty file → absent | Empty values are often read as "block everything" |
Wildcards: * matches any sequence; a trailing $ anchors the end, so /*.php$ does not match /a.php?x=1.
Five common patterns and what they actually mean
| Pattern | Actual meaning | What to record |
|---|---|---|
User-agent: *Allow: / | No restriction for crawlers without a dedicated group | allowed (wildcard) |
User-agent: *Disallow: / | Site-wide block for those crawlers | disallowed (wildcard) |
Disallow: /admin/Allow: /admin/public | Admin blocked, one subtree reopened | Test per path, not just the homepage |
User-agent: GPTBotDisallow: / | GPTBot only; everyone else follows * or default | Name the bot and the verdict, never "AI allowed" |
| No robots.txt | No declaration; commonly treated as unrestricted | absent, and ask whether it was intentional |
Why line-level checking matters
In our November 2026 scan of 30 equipment manufacturers' websites, 25 homepages could be judged: 17 allowed all five bots, 1 blocked all five, and 7 had no robots.txt; another 4 homepages could not be identified and 1 robots.txt returned 502. This is not a random sample and cannot be generalised to the industry, but it shows one thing clearly: having a robots.txt is not the same as having a clear policy, and absent must be recorded separately from disallowed.
The same batch also contained a site whose homepage was allowed while a product path was blocked for all five bots (see the 15 product-page pilot). Testing the homepage alone would have missed it entirely. Making the path a required input is a deliberate choice in this tool.
Five pitfalls
- Thinking Allow reopens another bot.
Allowonly competes withDisallowinside the same group; it does not let a bot back in past a dedicated block. - Thinking order decides. Rules are compared by path length, not position. Moving an
Allowup changes nothing. - Treating Sitemap as a rule. It is a pointer; deleting it will not lift a block.
- Writing absent as allow. Crawlers usually default to unrestricted when the file is missing, but that is a default, not a permission.
- Forgetting query strings and
$. A URL with parameters is a different path, and the anchor changes what matches.
Boundaries and limits
- No fetching: the tool only processes pasted text; it will not retrieve your live robots.txt. For production evidence, fetch it yourself and keep the time, UA and raw file.
- Not a crawler: it applies the text rules of RFC 9309 and does not model every vendor-specific alias or implementation detail.
- No encoding normalisation: paths and rules are compared literally, so
%20and a space differ. RFC 9309 expects both sides to percent-encode consistently, which most sites do. - No citation forecast: allowed means crawl permission only. Storage and citation are decided by each product's policy and are not promised here.
- Not a replacement for acceptance: for clauses, evidence and recheck scope see the 12 acceptance clauses; for scoring see the 20-check self-audit.
Tests and recalculation
The matching logic lives in js/robots-matcher.js with no dependencies and no network access; scripts/test_robots_matcher.js covers 30 assertion blocks, including group selection, longest match, Allow winning ties, * and $, empty rules, BOM/CRLF, merged * groups, path case and encoding differences. Run it with:
node scripts/test_robots_matcher.js
The page and the tests call the same functions, so there is no separate "article version" of the logic.
Sources and definitions
- RFC 9309, Robots Exclusion Protocol, section 2.2 on groups and rule matching: rfc-editor.org.
- Google Search Central, "How Google interprets the robots.txt specification": developers.google.com.
- Sample observations cited here: 30-company robots scan; 15 product entry pages.