Direct answer: Do not treat "AI crawlers" as one switch. Separate search crawlers, AI-search crawlers and training crawlers. Keeping Google/Bing search usually means allowing Googlebot; appearing in ChatGPT/Claude search means allowing OAI-SearchBot/Claude-SearchBot; excluding content from training means blocking GPTBot, ClaudeBot and Google-Extended separately. robots.txt is a request, not access control and not a licence.
This guide is for website owners, SEO/GEO, content, legal, IT/operations and export teams deciding whether to allow AI crawlers in 2026, how to write robots.txt, what each choice affects and how to verify it. Information is current to 31 October 2026. It is public documentation and project method, not legal advice; copyright, data rights, contracts and platform rules are confirmed case by case with counsel.
Search, AI search and training are three different things—do not block them all with one Disallow. Blocking training crawlers is not the same as blocking search, and not the same as a legal prohibition. robots.txt only works on crawlers that honour it; sensitive content needs login, permissions and contracts.
Why does this need a fresh decision in 2026?
Because AI search and model training are now separate: Google, OpenAI and Anthropic each expose different crawlers or controls. One wrong robots.txt rule can cost both search and AI-search visibility.
Historically, companies only asked "will Baidu/Google index us?" Today the same page can be used in three ways: search indexing, AI-search answers and model training. Platforms split the controls. Google uses Googlebot for Search and Google-Extended for Gemini training/grounding; OpenAI uses OAI-SearchBot for ChatGPT search and GPTBot for training; Anthropic separates Claude-SearchBot, ClaudeBot and Claude-User. As a result, "block AI" no longer expresses a clear choice, and "allow search" no longer implies "allow training".
| Token/control | Primary use | Effect of blocking/restricting | Official source |
|---|---|---|---|
| Googlebot | Google Search, Discover, Images/News | Lose Google Search indexing and visibility | |
| Google-Extended | Controls use for Gemini training and grounding | No effect on Search indexing/ranking; affects Gemini use | |
| GPTBot | OpenAI foundation-model training | Signals content should not be used for training | OpenAI |
| OAI-SearchBot | ChatGPT search | Not shown in search answers; may still appear as a navigational link | OpenAI |
| ChatGPT-User | User-initiated page visits | User-triggered; robots may not apply; not used for Search | OpenAI |
| ClaudeBot | Anthropic model training | Future content excluded from training datasets | Anthropic |
| Claude-SearchBot | Claude search index | Reduces search visibility and accuracy | Anthropic |
| Claude-User | User-initiated page visits | Cannot fetch on user request; may reduce visibility | Anthropic |
Uses and effects are summarised from each platform's public documentation; behaviour, IP ranges and updates follow the Google crawler docs, OpenAI crawler docs and Anthropic crawler page.
Which of the three strategies fits?
Most B2B sites fit "allow search and AI search, block training"; paid or sensitive content may block all AI crawlers; brand content marketing may allow everything. Choose by goal, not emotion.
There is no universal answer, but the trade-offs are clear. Allow everything: AI search and training may both use the content, maximising exposure but also reuse/training risk. Block training only: keep search and AI search, which fits most manufacturers and service firms that rely on the website for leads. Block all AI crawlers: fits paid reports, internal material or highly sensitive content, but sacrifices AI-search visibility and still needs login/permissions/contracts.
| Strategy | robots.txt approach | Fits | Cost/risk |
|---|---|---|---|
| Allow all | No extra AI limits; keep Googlebot | Brand content marketing, maximum exposure | Content may be trained on/reused; monitor |
| Allow search + AI search, block training | Allow Googlebot/OAI-SearchBot/Claude-SearchBot; disallow GPTBot/ClaudeBot/Google-Extended | Most B2B sites, manufacturing exporters | AI search may still quote summaries; relies on compliance |
| Block all AI crawlers | Disallow the AI bots; keep search crawlers | Paid/sensitive content, strong copyright | Lose AI-search visibility; not access control |
| Block everything | User-agent: * / Disallow: / | Staging/internal/not-yet-live sites | Also lose search indexing; not for a live site |
The strategy table is editorial; commercial and legal consequences depend on business goals, content type, contracts and counsel.
How should robots.txt be written, and where do people go wrong?
Put robots.txt at the root, UTF-8, text/plain; group Allow/Disallow under User-agent with longest match. It is not access control; a 5xx is treated as a full disallow, and listing sensitive paths exposes them.
RFC 9309 defines robots.txt as rules crawlers are requested to honour, and states they are "not a form of access authorization". The file must be at `/robots.txt`, UTF-8, text/plain; a group is a User-agent line plus Allow/Disallow rules, with longest match winning; if fetching fails with 5xx, crawlers must assume a complete disallow, while 4xx may allow access. In other words, an unstable robots.txt can accidentally block the whole site, and listing admin/draft paths makes them discoverable.
| Pattern/practice | Actual result | Recommendation |
|---|---|---|
User-agent: * | Blocks every crawler, including search | Staging/internal only; not a live site |
Only User-agent: GPTBot | Affects GPTBot only; not Google-Extended/ClaudeBot | List each platform; do not assume "AI blocked" |
| Blocking CSS/JS/image directories | Search/AI cannot render the page correctly | Allow rendering assets; protect with permissions |
| Listing admin/draft paths | Paths become public and discoverable | Use login, permissions, IP rules |
| robots.txt returns 5xx | Under RFC, crawlers assume full disallow | Monitor availability; return a stable 200 |
Crawl-delay | Not core RFC; some bots support it, Google ignores it | Use server-side rate limiting/caching |
| Using robots.txt to stop copying | Only works on crawlers that honour it | Combine contracts, copyright, access control, monitoring |
Syntax, status handling and caching follow RFC 9309; support for Crawl-delay/crawl rate differs by platform, so check each platform's docs.
Which content types should be allowed?
Product pages, technical docs and cases fit search and AI search; whitepapers may allow search but block training or gate the file; prices, admin and customer lists need permissions and contracts, not robots.txt.
A website is not one uniform block. Product/application pages need to be found by search and AI search; technical docs and selection guides gain trust when quoted; customer cases involve client authorisation and anonymisation, so confirm before allowing; whitepapers can expose a public summary and gate the full file; prices, quotes, admin and customer lists are access-control problems and should not sit on publicly crawlable pages. A content-type matrix is more controllable than one sitewide switch.
| Content type | Search | AI search/training suggestion | Watch |
|---|---|---|---|
| Product/service pages | Allow | Allow AI search; training optional | Specs/claims true and verifiable |
| Technical docs/selection | Allow | Allow AI search; training may be blocked | Version/applicable models clear |
| Customer cases | Allow | Training may be blocked | Client authorisation/anonymisation |
| Whitepapers/reports | Allow summary | Block training or gate | Do not publish paid full text |
| Prices/quotes | By strategy | Do not expose publicly | Login/permissions/contract |
| Admin/drafts/API | Disallow | Disallow | Authentication/access control |
| Careers/company news | Allow | Allow | Privacy/candidate data |
The content matrix is editorial; allow/block decisions depend on lead-generation goals, copyright, client authorisation and data strategy.
Does allowing crawling mean licensing training?
No. robots.txt is a crawler access preference, not a copyright licence or contract. Allowing a crawler does not grant training/commercial rights; blocking does not by itself create a legal prohibition.
This is the most common confusion. RFC 9309 explicitly says robots.txt is "not a form of access authorization": it addresses whether a crawler may visit, not whether it may train on, copy or commercialise the content. Conversely, blocking GPTBot/ClaudeBot only signals a preference to crawlers that honour it; it cannot stop non-compliant crawling, nor replace copyright, data rights, trade secrets and contract terms. For high-value content, combine robots.txt preferences, server-side access control, contract/licensing terms, monitoring/evidence and, where needed, legal advice.
| Question | What robots.txt can do | What it cannot do |
|---|---|---|
| May they crawl? | Express Allow/Disallow to compliant crawlers | Stop malicious/non-compliant crawling |
| May they train? | Express preferences via platform training controls | Replace copyright licensing/contracts |
| May they commercialise? | No direct effect | Grant or prohibit commercial rights |
| Can it prevent leaks? | No access-control capability | Protect admin/customer data |
| Can it support enforcement? | One piece of preference evidence | Prove infringement/damages alone |
The legal boundary is editorial, not legal advice; the role of robots.txt is stated in RFC 9309 section 1: "These rules are not a form of access authorization."
How should you monitor, verify and keep evidence?
Watch IP, UA, path, status and time in logs; verify identity with official IP ranges/reverse DNS; separate search, AI search, training and spoofed crawlers; review quarterly whether the policy met its goal.
Writing robots.txt is not the end. Server/CDN logs should show who crawls, what they fetch, how often and whether performance suffers. Verify "real crawlers" with published IP ranges or reverse DNS, not just User-Agent (which can be spoofed). If AI crawlers cause load, use caching, rate limiting and CDN rules rather than blocking search crawlers too. Review quarterly: did search indexing drop, is AI search visibility as intended, is the training block honoured, and is there abnormal crawling?
| Monitor | What to look at | Tool/evidence | Exception handling |
|---|---|---|---|
| Log fields | IP, UA, path, status, time, response time | Server/CDN logs | Add fields/sampling |
| Identity | Official IP ranges, reverse DNS, UA consistency | Platform docs/lookup | Rate-limit if spoofed |
| Crawl rate | Requests per IP/bot, peaks | Log stats | Cache/rate-limit/CDN |
| Search visibility | Indexing, impressions, clicks | Search Console/Baidu | Check accidental blocks |
| AI search | Citation/mention (observe only, no promise) | Manual sampling/logs | Record, do not attribute |
| Policy execution | robots version, change time, approval | Version log | Rollback/review |
The monitoring list is editorial; "cited/mentioned by AI" is an observation, not a ranking or traffic promise; log and personal-information processing follow the company privacy policy.
Launch acceptance checklist
Accept eight items: goal/strategy, domain list, robots syntax/status, per-bot tests, search unaffected, AI search as intended, monitoring/verification, approval/review. Each needs an owner and evidence.
The checklist can be shared by marketing, SEO/GEO, content, legal, IT/operations and vendors. The pass standard is not "robots.txt exists" but that the policy matches business goals, search is not accidentally blocked, AI search behaves as intended, sensitive content has access control, logs can verify it and changes can be rolled back.
| Check | Pass standard | Evidence | If failed |
|---|---|---|---|
| Goal/strategy | Search, AI search and training goals documented | Policy/approval | Redefine strategy |
| Domain list | Main/subdomains and multilingual sites covered | Domain list | Add robots/subdomains |
| robots syntax/status | Root, UTF-8, 200, parseable rules | Fetch test | Fix file/server |
| Per-bot tests | Googlebot/OAI-SearchBot/ClaudeBot etc. individually verified | UA/IP test log | Change rules/retest |
| Search unaffected | Googlebot not blocked; indexing normal | Search Console/logs | Rollback/fix |
| AI search as intended | Allowed search bots can fetch; blocked ones cannot | Logs/tests | Adjust rules |
| Monitoring/verification | Log fields, IP verification, rate alerts work | Logs/dashboard | Add monitoring |
| Approval/review | Owner, change log, quarterly review | Version/meeting record | Add process |
The acceptance checklist is editorial; allow/block, logging, privacy and contract requirements depend on company policy, platform docs and counsel.
Who should read this, who should not, and what decision does it support?
It fits companies with website lead-generation/AI-search goals deciding whether to allow AI crawlers. It does not fit teams without a content strategy or those treating robots.txt as security or anti-copying.
Who should read: website owners, SEO/GEO, content, legal, IT/operations and export teams; companies redesigning or migrating sites or seeing abnormal AI-crawler activity; anyone deciding "should we allow GPTBot/ClaudeBot, how to set Google-Extended, do we still want to appear in ChatGPT search, should we publish llms.txt". Who should not: projects without clear content/lead goals that just want to "block all AI"; teams treating robots.txt as access control, copyright protection or anti-copying; projects expecting one file to guarantee no training, citation or competition. Decisions supported: strategy selection, robots.txt rules, per-bot testing, content matrix, monitoring/verification, launch acceptance and change review.
Lingyi Information is a website and enterprise AI project services company, not a law firm, copyright agent or platform. This article does not replace copyright, data, contract or platform-rule advice. Scope is confirmed case by case; no indexing, ranking, AI mention/citation, traffic or inquiry is promised.
FAQ
Does blocking GPTBot affect ChatGPT search?
GPTBot is for training and OAI-SearchBot is for ChatGPT search; the settings are independent. Blocking only GPTBot does not automatically remove the site from ChatGPT search; manage OAI-SearchBot per OpenAI's documentation. Allowing OAI-SearchBot does not grant training rights either.
What is the difference between Google-Extended and Googlebot?
Googlebot handles Google Search crawling and indexing; Google-Extended is a separate control for whether content is used for Gemini training and grounding, with no effect on Search inclusion or ranking. Blocking Google-Extended does not remove the site from Google Search.
Can robots.txt force AI companies not to train?
No. It only works on crawlers that honour it and expresses an access preference, not a copyright licence or contract. Reduce risk with platform training controls, access control, contracts/licensing, monitoring/evidence and legal advice; blocking is not by itself a legal prohibition.
Is "User-agent: * / Disallow: /" acceptable?
It blocks every crawler, including Googlebot/Bingbot, so it is usually wrong for a live site. It also does not protect admin/sensitive data because robots.txt is public and reveals paths; use login, permissions and server rules for access control.
Is llms.txt a standard, and should we publish it?
llms.txt is a community proposal, not an IETF/W3C standard, and cannot replace robots.txt. It offers an LLM-oriented site directory/summary; whether platforms read it is up to them. You can publish it, but do not treat it as an allow/block switch or a guarantee of AI citation.
What if AI crawlers hit too often and hurt performance?
Check logs to confirm identity and rate, then use caching, CDN, rate limiting and crawl-rate rules; do not block Googlebot just to stop AI crawlers. Treat spoofed crawlers as a security incident; for legitimate bots, adjust robots/rate per platform docs and observe.
Bottom line
In 2026, split the AI-crawler decision into search, AI search and training: Googlebot protects search, OAI-SearchBot/Claude-SearchBot protect AI search, and GPTBot/ClaudeBot/Google-Extended manage training. robots.txt is a request, not a licence; sensitive content needs permissions and contracts; verify with logs and review quarterly.
Method and limits
This article is public platform documentation, RFC and project method, not a compliance certification or legal advice for any company, website, platform or contract. It uses no unpublished client logs, contracts or traffic data. The strategy table, content matrix and acceptance checklist are editorial suggestions; allow/block decisions, legal rights, platform behaviour, logging and privacy requirements depend on the company, platform docs and legal counsel.
Sources
Information is current to 31 October 2026; each official/primary page governs its own behaviour, scope and version.
- RFC 9309, Robots Exclusion Protocol (September 2022): rfc-editor.org. robots.txt syntax, longest match, 404/5xx handling, caching and the "not a form of access authorization" statement.
- Google Search Central, List of Google's common crawlers (updated 30 October 2026): developers.google.com. Googlebot vs Google-Extended, affected products, and the planned early-2027 AIPREF training-preference signal.
- OpenAI, Overview of OpenAI Crawlers: platform.openai.com. GPTBot, OAI-SearchBot and ChatGPT-User uses, robots behaviour and independent settings.
- Anthropic, Does Anthropic crawl data from the web (updated 7 April 2026): support.anthropic.com. ClaudeBot/Claude-SearchBot/Claude-User roles, robots.txt and Crawl-delay support.
- The /llms.txt file, v2 (community proposal, updated 10 August 2026): llmstxt.org. An LLM-oriented site directory proposal, not a standard and not a replacement for robots.txt.