Which AI crawlers visit a PrestaShop shop?
Short answer: those of the major AI companies, which identify themselves by their user agent. Some collect pages to train models; others read them to answer a user or feed an AI search engine.
| Crawler (token) | Company | Role | Obeys robots.txt (per the company) | Published IP list |
|---|---|---|---|---|
GPTBot | OpenAI | Collects content to train models | Yes | Yes |
OAI-SearchBot | OpenAI | ChatGPT search (sites shown in answers) | Yes | Yes |
ChatGPT-User | OpenAI | Opens a page when a user asks | “may not apply” | Yes |
ClaudeBot | Anthropic | Collects content to train models | Yes (and Crawl-delay) | Yes (shared list) |
Claude-SearchBot / Claude-User | Anthropic | Search / page opened for a user | Yes | Yes (shared list) |
PerplexityBot | Perplexity | Perplexity search results, not training | Yes | Yes |
Perplexity-User | Perplexity | Page opened for a user | Generally no | Yes |
CCBot | Common Crawl | Open web archive, reusable by third parties | robots.txt block documented | Yes |
Bytespider | ByteDance | Crawler | Not verified | No official list found |
Google-Extended | robots.txt token only (use by Gemini), no separate crawler | It is a robots.txt token | Not applicable |
This table is based on each company’s documentation (links at the end of the page), checked in November 2026. For Bytespider we found no current official documentation: treat it as a crawler that may ignore robots.txt.
Does blocking AI crawlers hurt Google SEO?
Short answer: no, as long as Googlebot is allowed. Google states that Google-Extended does not affect a site’s inclusion in Google Search and is not a ranking signal. GPTBot, ClaudeBot and CCBot have nothing to do with Google.
The real trade-off is visibility in AI assistants. OpenAI says sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers. Anthropic says blocking Claude-SearchBot or Claude-User may reduce your visibility in searches made with Claude.
| Your goal | Block | Allow |
|---|---|---|
| Stay out of AI training, stay visible everywhere | GPTBot, ClaudeBot, CCBot, Bytespider, Google-Extended token | Googlebot, Bingbot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, Claude-User |
| Save server resources, no AI visibility | Training + AI assistants and search | Googlebot, Bingbot |
| Be cited by AI assistants | Nothing, or training only | All declared crawlers |
Method 1: robots.txt, a polite request
Short answer: add one User-agent group with Disallow: / per crawler. Crawlers that follow the standard stop crawling; the others carry on.
# AI training crawlers: please do not crawl
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Bytespider
Disallow: /
# Google token for use by Gemini models (no effect on Google Search)
User-agent: Google-Extended
Disallow: /- The standard (RFC 9309) says crawler names are matched case-insensitively and that a crawler should not use a cached copy for more than 24 hours: the effect is not immediate.
- PrestaShop pitfall: the “Generate robots.txt file” button (Shop Parameters › Traffic & SEO) rewrites the whole file. Keep your lines aside and add them back after each generation.
- robots.txt protects nothing: it is public and a crawler can ignore it. The standard itself says these rules are not a form of access authorization.
Method 2: a real user-agent block (.htaccess or nginx)
Short answer: the server answers “403 Forbidden” to any visitor whose user agent contains the crawler’s name. The crawler no longer gets your pages, whether it obeys robots.txt or not.
Apache (.htaccess)
Put this block above the # ~~start~~ line: PrestaShop keeps whatever sits outside its own section when it regenerates the file.
# AI crawlers refused (put this ABOVE the "# ~~start~~" line)
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (GPTBot|ClaudeBot|CCBot|Bytespider) [NC]
RewriteRule .* - [F,L]
</IfModule>nginx
# Inside the server { } block (nginx)
if ($http_user_agent ~* "(GPTBot|ClaudeBot|CCBot|Bytespider)") {
return 403;
}Limits of this method:
- A user agent is self-declared: a crawler posing as a browser gets through. Conversely, anyone can claim to be “Googlebot”, which is why you verify the IP (next section).
- The list has to be updated by hand as new crawlers appear.
- A typo in
.htaccesscan take the shop offline: keep a copy and test straight away. - Test with
curl -I -A "GPTBot" https://your-shop.tld/: you should get403, and200with a browser user agent.
How can you be sure you never block Googlebot?
Short answer: do not trust the user agent alone. Google gives two methods: reverse then forward DNS, or matching the IP against the ranges it publishes as JSON.
- Run a reverse DNS lookup on the IP (
hostcommand). - Check that the name ends in
googlebot.com,google.comorgoogleusercontent.com. - Run a forward DNS lookup on that name and check that it returns the same IP.
host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.
host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1Careful with country blocking: Google says Googlebot’s default IP addresses appear to be based in the USA. A shop that blocks countries without a Google white list can cut off its own search traffic.
Block AI crawlers in one click with Block IPs
Short answer: in Block IPs, under Shop › User agents, two buttons add AI training crawlers and AI assistants to the blocked list, separately. Recognised Google crawlers are never blocked.
| Feature | Detail |
|---|---|
| Training crawlers in one click | 25 user agents: GPTBot, ClaudeBot, anthropic-ai, CCBot, Bytespider, meta-externalagent, cohere-ai, Diffbot… |
| AI assistants and search in one click | 12 user agents: ChatGPT-User, OAI-SearchBot, PerplexityBot, Perplexity-User, Claude-User, Claude-SearchBot, MistralAI-User, DuckAssistBot… |
| Matching | Case-insensitive, on part of the user agent; you can add or remove any name. |
| Response | 403 message or redirect, your choice. On Apache, an option blocks before PrestaShop even starts, through .htaccess, to save server resources. |
| Google protected | White list of recognised Google crawlers by their IP ranges, ahead of every rule, with a fallback copy shipped in the module if the host blocks the download. |
| Scraping networks | Detects one user agent used by hundreds of IPs, banned in one click. |
From version 5.6, the module also verifies trusted crawlers against each company’s official IP lists (Bing, Apple, DuckDuckGo, OpenAI, Perplexity, Anthropic, Mistral): a genuine search or assistant crawler gets through even if its country is blocked, and a fake user agent gains nothing. Training crawlers have a separate switch, off by default, and a crawler you ban stays refused.
Steps
- Open Block IPs › Shop › User agents.
- Click “Block them” in the AI training robots box.
- Only block AI assistants if you do not want to appear in their answers.
- Choose the response (403 or redirect) and, on Apache, optionally turn on blocking before PrestaShop.
- Follow blocks by reason and the Top 5 bots in the Overview.
See also the Block IPs module page and our guide on fake customer accounts.