AI crawlers list: what each bot does and which to allow

Which AI crawlers should you know about?

This table lists the crawlers documented by each company, read at each company’s own pages in early November 2026. Bots get added and renamed, so check the source if a decision depends on one line.

The names in robots.txt are not case-sensitive, and the versions in the user-agent string change over time, so match on the name only. Our guide to LLM SEO explains how each main assistant uses its bots.

Why does the job matter more than the company?

Because blocking the wrong job has a very different cost.

Training bots gather content that may help build future models. Blocking one does not remove you from today’s answers. OpenAI writes that its settings are independent, so you can allow OAI-SearchBot and disallow GPTBot. It is a business and rights decision, with no clear right answer.

Search bots build the index that answers draw from. Blocking one has an immediate cost. OpenAI says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links. Anthropic says disabling Claude-SearchBot may reduce your visibility and accuracy in user search results.

User fetchers visit one page because a person’s question calls for it. They are not crawling the web. OpenAI says robots.txt rules may not apply to ChatGPT-User because a person started the action. Perplexity says its fetcher generally ignores them, and Amazon and Meta say much the same of theirs. Anthropic differs: it says blocking Claude-User prevents it from retrieving your content for a user’s query.

Cloudflare’s published data shows why the split matters. In an August 2025 post, it reported that training accounted for nearly 80% of crawling by AI bots on its network, with ClaudeBot and GPTBot together making up nearly half of the crawling it saw in one week. So most AI crawling comes from the job that gives the least back to the site. It is one network’s view at one moment, but it explains why many sites block training and keep search.

Which bots should a business allow?

There is no universal answer, but there is a sensible starting point for each kind of site. This is our recommendation, not a rule from any of these companies.

If customers need to find you, the search bots are the ones that matter most. Our guide on how to get cited by ChatGPT goes through what else is needed.

What details do the companies add about their bots?

The table can only hold so much. These notes come from the same documentation, and a few of them change what you might do.

• Anthropic. Restricting ClaudeBot signals that future material from your site should be excluded from its training datasets. That wording covers future material, so it says nothing about what was collected before.
• Perplexity. It says PerplexityBot is not used to crawl content for AI foundation models. Each of its two bots has its own IP ranges, so match each name to its own list.
• Apple. Disallowing Applebot-Extended opts you out of training, but Applebot may still crawl, and your content stays discoverable through Spotlight, Siri and Safari. Apple also says pages marked isAccessibleForFree: false can appear in search but will not be used as context for its AI output.
• Amazon. If robots.txt does not mention Amzn-SearchBot but allows other search bots, it follows the rules given to those bots. Amazon also treats a page-level noarchive tag as “do not use the page for model training”.
• Meta. FacebookExternalHit fetches link previews when someone shares your page on Facebook, Instagram or Messenger, so blocking it can spoil those previews. Meta-ExternalAds crawls for advertising and business products.
• Google. Google-CloudVertexBot only affects crawls that site owners request for building Vertex AI Agents, and it has no effect on Google Search.

Which robots.txt lines can you copy?

Each block goes at yourdomain.com/robots.txt. Read the warning that follows them before you paste anything.

Allow search and user bots, block training. A common choice for those who want to appear in answers but not supply training data:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Amazonbot
User-agent: Meta-ExternalAgent
User-agent: CCBot
Disallow: /

User-agent: *
Disallow: /private/

Sitemap: https://www.example.com/sitemap.xml

Allow everything, keep private folders out. Bots with no group of their own follow the “*” group:

User-agent: *
Disallow: /private/
Disallow: /cart/

Sitemap: https://www.example.com/sitemap.xml

Block all the AI bots but keep ordinary search. Think before using this one, because it removes you from most AI answers outside Google:

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Google-CloudVertexBot
User-agent: Applebot-Extended
User-agent: Amazonbot
User-agent: Amzn-SearchBot
User-agent: Amzn-User
User-agent: Meta-ExternalAgent
User-agent: Meta-WebIndexer
User-agent: Meta-ExternalFetcher
User-agent: CCBot
Disallow: /

User-agent: *
Disallow: /private/

Sitemap: https://www.example.com/sitemap.xml

This covers the AI names in the table above except OAI-AdsBot, which only visits pages you submit as ads. Googlebot, Bingbot and Applebot stay out of the list because they also run ordinary search. The user fetchers marked “May not apply”, “May bypass it” or “Generally ignores it” may come anyway, so only a firewall rule stops those for certain.

Keep one folder out of training only. Useful if you sell research or courses and are happy to be read everywhere else. Note that the shared rule is repeated, for the reason given just below:

User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /research/
Disallow: /private/

User-agent: *
Disallow: /private/

Three traps with any of these:

• A bot with its own group ignores the shared group. If you give OAI-SearchBot its own group that only says “Allow: /”, it will also ignore the “Disallow: /private/” you set for all bots. Repeat any shared rules in each group.
• Changes are not instant. OpenAI and Meta each mention about 24 hours. Amazon says its bots may use a cached copy of robots.txt from the last 30 days.
• robots.txt is a request. Well-behaved bots follow it, but it does not protect anything. Use a login or a firewall for anything truly private.

Can you slow a bot down instead of blocking it?

If your concern is server load and not the use of your content, blocking may be more than you need, but there is no common way to slow every bot.

User-agent: ClaudeBot
Crawl-delay: 10
Disallow: /private/

The Disallow line is your shared rule repeated, because this group replaces the shared group for that bot. Leave it out and ClaudeBot may start reading a folder you meant to keep private. Anthropic says it respects Crawl-delay where appropriate. Amazon says its bots do not support it, and others do not mention it, so check each company’s page. A rate limit at your firewall works on any bot, but use it carefully: a search bot that keeps getting 429 replies may read too little of your site. Limit the heavy training bots first and leave the search bots alone. OpenAI adds one detail that may reassure you: if a site allows both of its main bots, it may use the results of one crawl for both jobs.

What exactly is Google-Extended?

It is the most misunderstood line on the list. Google-Extended has no user agent of its own and never shows up in your logs. It is a token in robots.txt. Google’s crawler documentation says it manages whether content Google crawls may be used for training future generations of Gemini models and for grounding in Gemini Apps and on Vertex AI.

Google states that it does not affect a site’s inclusion in Google Search and is not used as a ranking signal. It also does not take you out of AI Overviews. For that, Search Console has had a separate Search generative AI control since August 31, 2026, as our guide to AI Overviews and AI Mode explains. Blocking Googlebot instead would remove you from Search itself, which is rarely what anyone wants. Google also notes that it intends to support AIPREF, a shared signal for training preferences, in early 2027, so this line may change.

Why might a bot be blocked even when robots.txt allows it?

Because robots.txt and your firewall are separate systems, and the firewall wins. When we tested the MWY checker on 36 business websites in seven countries on November 3, 2026 (a hand-picked sample, not a market figure), 4 refused the checker itself, with errors 403 and 429, so we could not test them. Of the 32 we could read, 4 refused requests sent under AI bot names, while only 1 blocked an AI search bot in robots.txt. A refusal to our request is only a signal, since real bots come from other network addresses, but it tells you where to look.

Common sources:

• Cloudflare. On July 1, 2025 Cloudflare announced it was changing its default to block AI crawlers unless they pay. Its AI Crawl Control, formerly AI Audit, is available on all plans and sets allow or block rules per crawler. Check it if your site runs on Cloudflare, even if nobody on your team remembers turning anything on.
• Your host. Many hosts run bot protection or mod_security rules. Ask support which of the bots on this page are refused.
• WordPress security plugins. Some block unfamiliar user agents by default.
• Rate limits. A bot that gets a 429 reply keeps being turned away on a busy site, without anyone having decided that.

Perplexity’s documentation gives sample firewall rules that match its user agents to its IP ranges, a useful pattern for any bot you want to let through.

How can you tell a real bot from a fake one?

Anyone can write “GPTBot” in a user-agent string, and scrapers do. Common Crawl says it is aware of crawlers falsely identifying themselves as CCBot. So a name in your log proves nothing by itself. Check the address it came from:

To see which bots visit you, search your server’s access log for the bot names. Most hosts offer the raw logs in the control panel. On Cloudflare, AI Crawl Control shows it without any log files. Look at the response codes too: 200 means the bot received the page, while 403 and 429 mean it was turned away.

What the MWY checker tests about crawlers

The MWY checker reads your robots.txt and reports whether each AI search bot is allowed, including through the “*” group. It lists any training bots you block as your own choice and not an error. It also asks your site for a page under the names of the ChatGPT, Claude and Perplexity bots to spot a firewall refusal, and marks that result as a signal, because the real bots come from their own networks.

It does not read your server logs or tell you how often a bot visits, and it cannot verify a bot’s IP addresses for you. It also does not tell you whether to block training, which is your call. Allowing every bot makes you eligible, nothing more: no setting on this page promises that an assistant will use your site. For more on the training question, see our methodology page.