AI crawlers: who they are and what to allow
By Michael Nefedov, noetio · Updated 21 October 2026
AI crawlers are bots from AI companies that read your site for one of three jobs: training a model, building a search index, or fetching a page because a user asked. Each job needs a different robots.txt rule.
Short answer
- Our call, not the owners': for AI search visibility but no training, block GPTBot, ClaudeBot and CCBot. Allow the search and user bots.
- User-triggered bots (ChatGPT-User, Perplexity-User, Amzn-User, Meta-ExternalFetcher) may ignore robots.txt. The owners say so.
- Allowing a search bot lets you be shown. No owner page we read says it makes you rank higher.
The main AI crawlers
Each bot does one of three jobs: training, search or a user-triggered fetch. Every row comes from the owner's own page, read on 21 October 2026: OpenAI, Anthropic, Perplexity, Google, Common Crawl, Apple, Meta and Amazon. Google-Extended is a special case: Google says it covers training and grounding in Gemini Apps, so blocking it can cut your content from Gemini answers.
| Owner and bot | Purpose | robots.txt |
|---|---|---|
| OpenAI GPTBot | Training. Content that may be used to train foundation models (OpenAI) | Disallow = do not use for training |
| OpenAI OAI-SearchBot | Search. Shows sites in ChatGPT search (OpenAI) | Opt out and the site “will not be shown in ChatGPT search answers, though can still appear as navigational links” |
| OpenAI ChatGPT-User | User-triggered. A page visit when a user asks ChatGPT (OpenAI) | “May not apply” |
| Anthropic ClaudeBot | Training. Web content that could contribute to training (Anthropic) | Disallow = exclude future material from training |
| Anthropic Claude-SearchBot | Search. Improves relevance of search results (Anthropic) | Block = no indexing for search |
| Anthropic Claude-User | User-triggered. Supports Claude users (Anthropic) | Block = no retrieval for a user query |
| Perplexity PerplexityBot | Search. Surfaces and links sites in Perplexity results. “Not used to crawl content for AI foundation models” (Perplexity) | Perplexity recommends allowing it |
| Perplexity Perplexity-User | User-triggered. Visits a page to answer a user (Perplexity) | “Generally ignores” robots.txt |
| Google Google-Extended | A control token for training future Gemini models and for grounding in Gemini Apps and Vertex AI. Not a ranking signal (Google) | Token only. No separate user-agent string |
| Google Googlebot | Search. Google lists it as a common crawler (Google). Google says AI Overviews and AI Mode have no additional requirements beyond Search (Google) | “Always respect” it for automatic crawls |
| Common Crawl CCBot | Builds an open repository of crawl data anyone can use (Common Crawl) | User-agent: CCBot + Disallow: / blocks it |
| Apple Applebot | Search in Spotlight, Siri and Safari. Its data may also be used to train Apple foundation models (Apple) | Apple says it respects standard robots.txt directives in general search crawls aimed at Applebot. “Enabling Applebot in robots.txt allows website content to appear in search results” |
| Apple Applebot-Extended | Secondary user agent that controls whether Applebot data is used to train generative models. Apple: it “does not crawl webpages” (Apple) | Disallow = opt out of training. Pages can still be in search results. To keep content out of Siri and Search answers, Apple says to use the nosnippet meta tag |
| Meta Meta-ExternalAgent | Crawls the web for uses such as training foundation AI models or improving products by indexing content directly (Meta) | Meta says to add a disallow for the crawler in robots.txt |
| Meta Meta-WebIndexer | Search. Navigates the web to improve Meta AI search result quality (Meta) | Meta: allowing it “helps us cite and link to your content in Meta AI's responses” |
| Meta Meta-ExternalFetcher | User-triggered. Fetches individual links at a user's request (Meta) | Meta: “this crawler may bypass robots.txt rules” |
| Amazon Amazonbot | Improves Amazon products and services. “May be used to train Amazon AI models” (Amazon) | Amazon says it respects the Robots Exclusion Protocol. Amazon also lists a noarchive robots meta tag as “do not use the page for model training” |
| Amazon Amzn-SearchBot | Search experiences such as Alexa. Does not crawl for generative AI training (Amazon) | Allowing it makes your content eligible for those experiences |
| Amazon Amzn-User | User-triggered. May fetch live information to answer Alexa queries. Does not crawl for training (Amazon) | “May not follow all robots.txt directives” |
Not covered: ByteDance's Bytespider (we found no ByteDance page that documents it) and any owner not named above. Google also lists Google-CloudVertexBot; it affects crawls that site owners request for building Vertex AI Agents and, Google says, has no effect on Google Search. Detail pages, which cover one owner each and do not repeat this table: GPTBot, OAI-SearchBot and ChatGPT-User and ClaudeBot, Claude-User and Claude-SearchBot.
User-triggered bots may ignore robots.txt
When a person asks an AI product to open a page, the owner may treat the fetch as the person's request, not a crawl. Four owners say this in their own words. OpenAI says robots.txt rules “may not apply” to ChatGPT-User. Perplexity says Perplexity-User “generally ignores” them. Meta says Meta-ExternalFetcher “may bypass” them. Amazon says Amzn-User “may not follow all robots.txt directives”. Anthropic's page does not say this about Claude-User: it says its bots honor robots.txt.
Recommended robots.txt: AI search yes, training no
This file blocks the training and open-dataset bots and allows the search and user bots. It uses only names from the table. It leaves out Google-Extended on purpose (see the first note).
# Training and open datasets: no
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
# AI search and user-triggered fetches: yes
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
# Everyone else, including Googlebot and Bingbot
User-agent: *
Allow: /
Sitemap: https://yoursite.com/sitemap.xmlRead these four notes before you paste it.
- Google-Extended is not in the file. Google says it controls use of your content for training future Gemini models and for grounding (content from the Google Search index given to the model at prompt time) in Gemini Apps and Grounding with Google Search on Vertex AI. It does not affect Google Search. If you want no Gemini training and are fine with losing Gemini grounding, add
User-agent: Google-ExtendedandDisallow: /. - Private folders. A crawler uses the
*group only when no group matches its own token (RFC 9309, section 2.2.1). If your*group hasDisallow: /admin/, copy that line into each Allow group. - CCBot and the choice to block. Common Crawl says its data is open to everyone and does not say who trains on it. We block it because open data can be used for anything. If you want to be in training data, change the three Disallow lines to Allow.
- Apple, Meta and Amazon are not in this file. To opt out of training for those three, add the tokens their pages name. Amazon says Amazonbot may be used to train its AI models, and lists Amzn-SearchBot as its search bot. Paste this:
User-agent: Applebot-Extended
Disallow: /
User-agent: Meta-ExternalAgent
Disallow: /
User-agent: Amazonbot
Disallow: /Does this help you show up in AI answers? It leaves the search bots free to read your pages. Nothing in the docs we read says that allowing a bot makes you more likely to be cited. For what the data says, read SEO vs GEO and how to rank in AI Overviews.
How to check your logs for AI crawlers
Count the bots in your raw access logs, then check status codes and IPs. If you can get logs from your host or CDN, run this:
awk 'BEGIN{n=split("gptbot oai-searchbot chatgpt-user claudebot claude-searchbot claude-user perplexitybot perplexity-user ccbot",b," ")} {l=tolower($0); for(i=1;i<=n;i++) if(index(l,b[i])) c[b[i]]++} END{for(k in c) print c[k], k}' access.log | sort -rnThis counts each request once per bot, whatever the case. A plain grep -o would count GPTBot and PerplexityBot twice per request, because their user-agent strings contain the name twice (for example GPTBot/1.4; +https://openai.com/gptbot).
To see which pages one bot asked for (the 7th field is the path in the common log format):
grep -i 'OAI-SearchBot' access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20- 1Count requests per bot. A search bot with no requests has not visited, or your logs do not show it. Check the status codes next.
- 2Check the status codes. A 403 or 429 can come from a firewall or rate limit, which robots.txt does not control.
- 3Compare IPs to the owner's published list. Do not trust the name alone.
- 4Look at which URLs they fetch.
- 5Check /robots.txt itself returns 200 and plain text.
What we ran: on 21 October 2026, curl requests to getcompassseo.com with the GPTBot, OAI-SearchBot and ChatGPT-User strings from OpenAI's page, and a ClaudeBot-style string (ClaudeBot/1.0; +claudebot@anthropic.com), all returned HTTP 200. Our robots.txt is User-agent: * with Allow: /.
What to do next
Decide on training first, then on search. Check your logs after you change robots.txt. OpenAI and Amazon each say a change can take about 24 hours to show, and Meta says up to 24 hours. To measure whether AI answers mention you, see the LLM SEO tools. For the llms.txt question, read whether llms.txt is worth your time.
Sources
All read on 21 October 2026.
- Overview of OpenAI Crawlers, OpenAI, https://developers.openai.com/api/docs/bots
- Does Anthropic crawl data from the web, and how can site owners block the crawler?, Anthropic Privacy Center, page dated 7 April 2026
- Perplexity Crawlers, Perplexity Documentation, https://docs.perplexity.ai/guides/bots
- Google's common crawlers, Google Search Central, last updated 15 October 2026
- Overview of Google crawlers and fetchers, Google Search Central, last updated 12 June 2026
- AI features and your website, Google Search Central, last updated 10 December 2025
- CCBot, Common Crawl, https://commoncrawl.org/ccbot
- About Applebot, Apple Support, https://support.apple.com/en-us/119829
- Web Crawlers, Meta for Developers, https://developers.facebook.com/docs/sharing/webmasters/web-crawlers. The English text is quoted; the page shows in German for some locations.
- About Amazonbot, Amazon Developer, https://developer.amazon.com/amazonbot
- RFC 9309, Robots Exclusion Protocol, IETF, September 2022
Questions
A training crawler collects content for model training (GPTBot, ClaudeBot). A search crawler indexes pages so an AI search product can show them (OAI-SearchBot, Claude-SearchBot, PerplexityBot). You can allow one and block the other.
No. User-triggered fetchers are the main exception, as the table shows. Also, robots.txt is a request, not a lock. A bot that ignores it will not be stopped by it. To block it, use your firewall.
No. Google says Google-Extended does not affect inclusion in Google Search and is not a ranking signal. For AI Overviews, Google's AI features page says the normal Search rules apply, so use Googlebot controls such as nosnippet.
Compare the request IP with the owner's published list. OpenAI, Perplexity, Anthropic, Amazon and Common Crawl publish IP lists. Anyone can copy a user-agent name.
It may. Apple says Applebot data may help train its foundation models. The table shows the two opt-outs: Applebot-Extended for training, and the nosnippet meta tag for Siri and Search answers.
We found no evidence that you do. See our llms.txt guide for the sources.
More guides
- ClaudeBot: What It Is and How to Block or Allow It
- GPTBot vs OAI-SearchBot vs ChatGPT-User: What to Allow
- llms.txt: What It Is and Whether You Need One (2026)
- How to Rank in AI Overviews: What Google Says (2026)
- How to Get Cited by ChatGPT: What Is Documented (2026)
Turn your next topic into a useful page
Compass reads your website, helps you plan relevant topics and creates sourced article drafts. Review and approve the work, then send it to a connected CMS or export it.
Written by Michael Nefedov, founder of noetio. Updated 21 October 2026.