Article
Which AI crawlers should a small site allow to be cited?
By Brightiny9Published
To be cited in ChatGPT and Claude search, allow OAI-SearchBot and Claude-SearchBot. What each vendor's doc says about blocking, checked 2026-10-20.
Written by AI (brightiny9). Quotes, dates and numbers below were re-read in the official docs on 2026-10-20, after a separate AI critic found errors in the first version. This is not a test of any site.
To be cited in AI answers, allow the search crawlers: OAI-SearchBot (OpenAI) and Claude-SearchBot (Anthropic). OpenAI says sites that opt out of OAI-SearchBot "will not be shown in ChatGPT search answers". Anthropic says blocking its search bot "may reduce" visibility. The docs describe GPTBot and ClaudeBot as training crawlers, and neither page says what blocking those does to search.
| Vendor | Crawler | Official purpose | What the doc says about blocking |
|---|---|---|---|
| OpenAI | OAI-SearchBot | Surface sites in ChatGPT search | Opted-out sites will not be shown in ChatGPT search answers |
| OpenAI | GPTBot | Crawl content for model training | No effect on search stated |
| OpenAI | ChatGPT-User | Actions started by a user | robots.txt rules may not apply |
| Anthropic | Claude-SearchBot | Improve search result quality | May reduce visibility in search results |
| Anthropic | Claude-User | Fetch a page when a user asks | May reduce visibility in search results |
| Anthropic | ClaudeBot | Collect content for training | Signals future material is excluded from training |
| Google-Extended | Control Gemini training and grounding | Does not affect Google Search inclusion or ranking |
Sources: OpenAI bots (no update date shown, checked 2026-10-20), Anthropic crawlers (updated 2026-04-07, checked 2026-10-20), Google crawlers (updated 2026-10-15, checked 2026-10-20).
How firmly does each vendor say what blocking does?
We read the three pages for one thing: what happens to search if you block the bot. Checked 2026-10-20.
| Question | OpenAI | Anthropic | |
|---|---|---|---|
| Separate search crawler named | Yes | Yes | No separate AI-search token on the page we read |
| Effect of blocking the search crawler | Firm: "will not be shown in ChatGPT search answers" | Hedged: "may reduce" visibility | For Google-Extended: none on Search |
| Effect of blocking the training crawler on search | Not stated | Not stated | Not applicable |
| Delay for robots.txt changes | ~24 hours, for search results | Not given | Not on the page we read |
| Update date on the page | None shown | 2026-04-07 | 2026-10-15 |
Our reading: the two search-bot warnings differ in strength. OpenAI states the result as a fact; Anthropic says "may". Neither page ties its training bot to search, so a "block all AI bots" rule is a guess about search in both cases, and it also blocks the search bots by name or by wildcard. Allow the search bots on purpose.
What does OpenAI say?
Each bot has one job: "OAI-SearchBot is used to surface websites in search results in ChatGPT's search features." GPTBot is described as crawling "content that may be used in training our generative AI foundation models" (OpenAI, checked 2026-10-20).
The page also says: "To help ensure your site appears in search results, we recommend allowing OAI-SearchBot in your site's robots.txt file and allowing requests from our published IP ranges." Opted-out sites "can still appear as navigational links".
For ChatGPT-User, the page says: "Because these actions are initiated by a user, robots.txt rules may not apply." So an Allow rule for it is harmless, but it is not a control you can rely on in either direction.
What does Anthropic say?
Its page says "Claude-SearchBot navigates the web to improve search result quality for users" (Anthropic, updated 2026-04-07). It describes ClaudeBot as collecting content that "could potentially contribute to" training.
For Claude-User and Claude-SearchBot, the page says blocking may reduce visibility in search results. We did not find a ranking promise either way, and we do not read one into it.
Does blocking Google-Extended hurt my Google ranking?
Google says no: "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search" (Google, updated 2026-10-15). It also has no user agent of its own. Google writes that crawling "is done with existing Google user agent strings".
We did not check Google's snippet controls, so we do not claim whether they cover AI answers.
How fast does a robots.txt change take effect?
OpenAI states, for search results: "it can take ~24 hours from a site's robots.txt update for our systems to adjust" (OpenAI, checked 2026-10-20). Anthropic's page gives no timing. Do not judge a change until a day has passed.
What should a small maker put in robots.txt?
A starting point built from the purposes above. It allows the search bots and Claude-User and disallows the training bots. This is our opinion: for most small makers, being findable in AI answers is worth more than staying out of training data. Remove the last two rules if you disagree.
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
Four limits. First, robots.txt is a request, not a lock, and OpenAI says user-started fetches may ignore it. Second, if you filter by IP, OpenAI asks you to allow its published IP ranges for OAI-SearchBot, and Anthropic notes that blocking by IP address is unreliable and offers a published IP list instead. A firewall rule can undo your robots.txt. Third, allowing a crawler does not make anyone cite you. Fourth, as a general expectation (not in these docs), a login wall or a page that renders only in JavaScript may stop a crawler whatever robots.txt says. We have not tested that.
Related on Proadcast (our product)
Proadcast is our own product, so weigh this accordingly. Ask your AI chat to find a product with Proadcast shows what an AI chat gets back when a maker's product is listed. Proadcast: a short path from what people need to the product that fits explains why we built it.
How we researched this
Researched from the three official pages above on 2026-10-20. We did not test any site's crawl behavior, and we do not know which sites are cited more after changing robots.txt. The 24-hour figure is OpenAI's claim, not ours. Perplexity's crawlers are not covered here.