GPTBot, OAI-SearchBot and ClaudeBot: what to allow in your store’s robots.txt
Quick answer
AI crawlers split into three jobs: training models, feeding a search engine and opening a page at a user’s request. For a store that wants to be recommended, the usual approach is to allow the search bots and decide separately whether to allow training. According to OpenAI, blocking GPTBot doesn’t remove you from ChatGPT search, but blocking OAI-SearchBot does.
AI crawlers are not all the same. Each provider uses different bots to train models, to build its search engine and to open a page when a user asks for it at that moment. If you lump them together, you either block more than you meant to or let through exactly the opposite.
The three jobs of an AI crawler
- Training: collects content to improve foundation models.
- Search: builds the index that answers with links to sources come from.
- User action: opens a specific URL because someone asked for it in that conversation.
What each bot does
| Provider | Bot | What it is for |
|---|---|---|
| OpenAI | GPTBot | Model training. Respects robots.txt. |
| OpenAI | OAI-SearchBot | ChatGPT search. Respects robots.txt. |
| OpenAI | ChatGPT-User | User-initiated actions. According to OpenAI, robots.txt rules may not apply. |
| Anthropic | ClaudeBot, Claude-SearchBot, Claude-User | Training, search and user actions, controlled separately according to its documentation. |
| Perplexity | PerplexityBot, Perplexity-User | Search index and user actions. Perplexity says the latter usually ignores robots.txt. |
| Googlebot, Google-Extended | The first crawls for the search engine, including its AI features. The second is a control over the use of content in Gemini and doesn’t affect your ranking in Google Search. |
Names and versions change. Before touching anything, confirm each bot in its provider’s documentation.
What suits an online store
If your goal is to be recommended, the sensible thing is to allow the search bots: OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot. Whether someone trains models on your content is a separate business decision, and there is no single answer.
The key point, according to OpenAI’s documentation, is that GPTBot and OAI-SearchBot are managed separately. Blocking GPTBot doesn’t remove you from ChatGPT search, but blocking OAI-SearchBot does leave you out of its answers.
Option A: allow everything
You don’t need to add any rules. If your robots.txt doesn’t block anything, all of those bots can get in.
Option B: allow search and not training
Keep your current rules and add these blocks:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: OAI-SearchBot
Allow: /Where to edit the robots.txt
- Shopify: with the theme’s
robots.txt.liquidtemplate. Only touch what you understand. - WooCommerce and other WordPress sites: with the file editor of your SEO plugin, if your hosting allows it, or by uploading the file to the root.
- PrestaShop: the
robots.txtfile in the root, which can be regenerated from the traffic and SEO section.
Common mistakes
- An inherited Disallow: / from a test environment that was copied to production.
- A firewall or anti-bot protection that blocks crawlers even though the robots.txt allows them. OpenAI recommends also allowing its published IP ranges.
- Old user-agent lists copied from 2023 articles. Names change.
- Believing that allowing access is enough. It is necessary, but it doesn’t guarantee you will be cited.
How to check it works
- Open
yourstore.com/robots.txtand confirm the rules say what you want. - Check the server logs and count each bot’s visits separately.
- Check that the visits are real by comparing the IP with the ranges each provider publishes.
- Wait. OpenAI says its systems can take around 24 hours to reflect a change.
- Ask 5 to 10 buying questions in ChatGPT and note whether you appear. You have the method in how to measure your visibility in ChatGPT.
Bear in mind that robots.txt only regulates access. What decides whether you are recommended is your content, your product data and the mentions you have outside your website.
Frequently asked questions
Does blocking GPTBot take me out of ChatGPT?
Not necessarily. According to OpenAI, GPTBot is the control for training and OAI-SearchBot the one for ChatGPT search. They are independent.
Should I block ClaudeBot or Google-Extended?
It is a business decision about the use of your content for training models. In the case of Google-Extended, Google says it doesn’t affect your inclusion or ranking in Google Search.
Does ChatGPT-User respect robots.txt?
OpenAI says that, because these are user-initiated actions, robots.txt rules may not apply. OAI-SearchBot is the one used to manage search.
Does llms.txt replace robots.txt?
No. Robots.txt regulates access and llms.txt is an optional guide. You have more detail in llms.txt for ecommerce.
Sources consulted
Want to know how your store is doing?
Give us your website and within 48 hours you get your five biggest opportunities on Google, whether AIs cite you today and a comparison with your main competitor.
Get my free audit
Leave a Reply