LLM SEO: how ChatGPT, Gemini, Claude and Perplexity find your pages

What does “LLM SEO” mean?

A large language model answers from two places. The first is what it learned in training: a huge collection of text gathered up to a cutoff date, then frozen. The second is what it looks up while answering: when an assistant searches the web, it fetches a handful of pages, reads them and writes its answer from them, usually with links.

For a business, the second place matters most. Prices, opening hours, a new service, this year’s reviews: none of it is in a model trained a year ago. When someone asks ChatGPT for a plumber in their town or the best accounting software for a five-person firm, the assistant searches, and the pages it can find and read in those few seconds are the pages it can quote.

LLM SEO is the work of making sure your pages are among them. Part of it is the same as classic search engine optimization: a page that is indexed, loads fast and has a clear title. Part of it is new, because the assistants use their own bots and their own indexes, and each one can be blocked separately. A site that never appears in AI answers is not always losing a contest of quality. Sometimes it is simply closed to the bots, and nobody has noticed.

Training bots, search bots and user fetchers

Each AI company runs more than one bot, and each bot has its own name in robots.txt. They do different jobs, and the difference decides what blocking one of them costs you.

• Training bots collect pages that may be used to train future models. Blocking them keeps your future content out of training data. It does not remove you from search answers today.
• Search bots build the index an assistant searches when it answers. Blocking them keeps your pages out of that assistant’s answers.
• User fetchers open a page when a person asks the assistant about it, or pastes its address. They act for one user, and some companies say robots.txt may not apply to them.

Here is how the companies name them in their own documentation, as of November 2026.

The settings are independent. A site can block GPTBot and ClaudeBot to keep its content out of training while leaving OAI-SearchBot, Claude-SearchBot and PerplexityBot free to read it for search. That is a legitimate choice, and a common one.

How each assistant finds pages

ChatGPT. OpenAI’s search bot is OAI-SearchBot. OpenAI’s documentation says that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links, and that a change to robots.txt can take about 24 hours to take effect. GPTBot, the training bot, has no effect on search.

Gemini, AI Overviews and AI Mode. Google’s AI features in Search use the Google Search index. Google states that there are no extra requirements to appear in AI Overviews or AI Mode: a page must be indexed and eligible to show in Search with a snippet, and no special schema or AI text file is needed. Google-Extended is a separate control. It decides whether Google may use your content to train future Gemini models and to ground answers in the Gemini apps and on Vertex AI, and Google says it does not affect inclusion or ranking in Google Search.

Claude. Anthropic’s search bot is Claude-SearchBot. Anthropic writes that blocking it prevents your content from being indexed for search, which may reduce your site’s visibility and accuracy in user search results, and that blocking Claude-User may reduce visibility when a user asks Claude to look something up. All three of Anthropic’s bots honor robots.txt.

Perplexity. PerplexityBot is built to surface and link websites in Perplexity’s search results, and Perplexity says it is not used to crawl content for AI foundation models. Perplexity recommends allowing it in robots.txt. Perplexity-User fetches pages when a person asks a question, and Perplexity says it generally ignores robots.txt because a user asked for the page.

Microsoft Copilot answers from the Bing index, so the usual Bing basics apply: a page Bingbot can crawl and index. Bing Webmaster Tools now shows a report of how often pages are cited in Copilot and Bing’s AI answers.

What the bots see when they open a page

A person sees your page after the browser has run its scripts. An AI search bot sees what your server sends before any script runs, and for many modern sites those are two very different pages.

In December 2024 Vercel published an analysis of AI crawler traffic across its network and found that none of the major AI crawlers it studied rendered JavaScript. The bots of OpenAI, Anthropic, Perplexity and others fetched JavaScript files without running them. The exceptions were Gemini, which uses Googlebot’s infrastructure and can render pages, and Applebot.

That has three plain consequences.

• Text loaded by JavaScript after the page opens is invisible. A React or Vue app that builds the page in the browser can look complete to a visitor and send a nearly empty shell to a bot.
• Text inside images is invisible. A banner that says “Same-day repairs, licensed since 2009” is a picture to a bot, not a sentence.
• Content behind clicks may be missing. Tabs, accordions and “load more” buttons that fetch their text only when clicked never deliver it to a bot.

You can see what a bot sees in a minute: open the page, choose “View page source” in your browser, and search for a sentence from the middle of the page. If it is in the source, the bots can read it. If it only appears in the browser’s developer tools, they cannot.

If the text is missing, the fix depends on how the site is built. WordPress, Shopify, Squarespace, Wix and Webflow usually send the main text in the HTML; the gaps there tend to be sliders, tabs and page-builder blocks that load their text later, and those can be swapped for static blocks. A site built as a single-page app in React, Vue or Angular needs server-side rendering or pre-rendered pages, which frameworks such as Next.js and Nuxt provide. That is a job for a developer, and when the page source is empty it is the change that matters most.

robots.txt lines that close a site without anyone noticing

robots.txt is a short file at the root of your site that tells bots what they may read. It has been an internet standard since 2022 (RFC 9309). A single line in it can hide a site from every AI assistant, and nobody visiting the site would ever notice.

The four patterns that do it most often:

• A blanket block left over from development. “User-agent: *“ followed by “Disallow: /“ closes the site to every bot that has no group of its own. It is common on sites built on a staging server and launched without cleaning the file.
• A copied list of AI bots. Lists of “AI bots to block” circulate widely, and some include OAI-SearchBot, Claude-SearchBot and PerplexityBot alongside the training bots. Pasting one blocks AI search, not just training.
• A bot group that forgets the shared rules. Under the standard, a bot that finds a group with its own name follows only that group and skips the “User-agent: *“ group. Adding “User-agent: OAI-SearchBot” with “Allow: /“ therefore also opens your admin and cart pages to it, unless you copy the shared Disallow lines into its group.
• A robots.txt that answers with a server error. Google treats a robots.txt that answers with a server error (a 5xx code) as if the whole site were closed.

Open yourdomain.com/robots.txt in a browser now. If you see any of the four, that is the first thing to fix, before any content work.

Firewalls and bot protection

robots.txt is a request. A firewall is a wall. Many sites allow AI bots in robots.txt while their security layer answers those same bots with a challenge page, an error or nothing at all.

The usual sources are bot protection at the network provider and security rules at the host. On July 1, 2025, Cloudflare announced it was changing its default to block AI crawlers unless they pay creators for their content, and its dashboard has had a setting to block AI bots for some time. Hosts with their own protection against attacks may block unfamiliar bots by default too. In each case the site owner rarely sees anything, because the block only hits bots. Bot protection is not a mistake in itself: it keeps out scrapers and attacks. The point is to decide which bots you want, rather than let a default decide for you.

What to do:

• In Cloudflare, open the bot settings of your domain and check whether AI bots or AI crawlers are blocked. If you want AI search to read your site, allow the search bots, or allow verified bots.
• If your host runs its own firewall, ask it in writing whether OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot and Bingbot can reach your pages.
• Look at your server logs, if you can, for those names. A bot that appears with only error codes is being refused.

Once a bot can read the page, what makes it useful

Access is the floor. Once the bots can read your pages, the question becomes whether a page helps the assistant answer, and that is where content and structure come in.

• A clear title and one H1 per page that say what the page is about, in the words buyers use.
• The answer near the top of the page, in plain sentences, with the numbers: what it costs, how long it takes, where you work.
• Structured data that states who you are: Organization or LocalBusiness schema with your name, address, phone and logo, written the same way everywhere.
• Dates on articles, so a model can tell whether a fact is still current.

Research on what makes content more likely to be used in AI answers points the same way: adding sources, quotations and statistics helped, stuffing keywords did not. That research is covered in our guide to generative engine optimization.

A 15-minute check of your own site

Most of the points above can be checked by hand, without tools.

Fifteen minutes is enough to find the problems that close a site completely. The firewall is the one most owners cannot check alone, because the block only shows to bots.

What the MWY checker tests here

The MWY checker reads your site the way the AI search bots do: raw HTML, no JavaScript, robots.txt obeyed. Of its 24 items, the access group covers this guide. It checks whether robots.txt lets each AI search bot in, whether you block the training bots (shown as your choice, not an error), how much of your home page text is in the HTML, how fast your server answers, whether there is a sitemap, whether pages are hidden from search, and whether HTTPS and redirects are clean.

It also asks your site for a page under the names of the ChatGPT, Claude and Perplexity bots, to see whether a firewall refuses them. That item is marked as a signal, not a certainty: the real bots come from their own network addresses, which no outside tool can use. You can see every item on a real site in the sample report, and how each one is weighted on the methodology page.

What the checker does not do is look inside an AI company’s index or promise that an assistant will name your business. Nobody outside those companies can see which pages they hold or how they choose among them. What can be checked is whether your site lets them in and gives them text to read, and that is the part a site owner can fix this week.