← Stef Development

Block AI crawlers on a PrestaShop shop without hurting Google

Are GPTBot, ClaudeBot or CCBot copying your product pages? Here is the difference between a robots.txt request and a real block, the effect on your SEO, and how to keep Googlebot safe.

Updated on November 4, 2026 · Mustapha EL ATRASSI, Stef Development

Key points

  • Two families: training crawlers (GPTBot, ClaudeBot, CCBot…) and AI assistant and AI search bots (OAI-SearchBot, ChatGPT-User, PerplexityBot…). Blocking them has different effects.
  • robots.txt is a request, not a lock: the standard (RFC 9309) says its rules are not a form of access authorization, and Google notes they cannot enforce crawler behaviour.
  • Blocking GPTBot, ClaudeBot or Google-Extended does not change Google Search; Google says so for Google-Extended. Blocking OAI-SearchBot or PerplexityBot does remove you from those engines.
  • Real blocking: refuse by user agent in .htaccess, nginx or a module, with a 403 response.
  • Googlebot: never block it by mistake. Verify it by reverse DNS or Google’s published IP ranges, especially if you block countries.

Which AI crawlers visit a PrestaShop shop?

Short answer: those of the major AI companies, which identify themselves by their user agent. Some collect pages to train models; others read them to answer a user or feed an AI search engine.

Crawler (token)CompanyRoleObeys robots.txt (per the company)Published IP list
GPTBotOpenAICollects content to train modelsYesYes
OAI-SearchBotOpenAIChatGPT search (sites shown in answers)YesYes
ChatGPT-UserOpenAIOpens a page when a user asks“may not apply”Yes
ClaudeBotAnthropicCollects content to train modelsYes (and Crawl-delay)Yes (shared list)
Claude-SearchBot / Claude-UserAnthropicSearch / page opened for a userYesYes (shared list)
PerplexityBotPerplexityPerplexity search results, not trainingYesYes
Perplexity-UserPerplexityPage opened for a userGenerally noYes
CCBotCommon CrawlOpen web archive, reusable by third partiesrobots.txt block documentedYes
BytespiderByteDanceCrawlerNot verifiedNo official list found
Google-ExtendedGooglerobots.txt token only (use by Gemini), no separate crawlerIt is a robots.txt tokenNot applicable

This table is based on each company’s documentation (links at the end of the page), checked in November 2026. For Bytespider we found no current official documentation: treat it as a crawler that may ignore robots.txt.

Does blocking AI crawlers hurt Google SEO?

Short answer: no, as long as Googlebot is allowed. Google states that Google-Extended does not affect a site’s inclusion in Google Search and is not a ranking signal. GPTBot, ClaudeBot and CCBot have nothing to do with Google.

The real trade-off is visibility in AI assistants. OpenAI says sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers. Anthropic says blocking Claude-SearchBot or Claude-User may reduce your visibility in searches made with Claude.

Your goalBlockAllow
Stay out of AI training, stay visible everywhereGPTBot, ClaudeBot, CCBot, Bytespider, Google-Extended tokenGooglebot, Bingbot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, Claude-User
Save server resources, no AI visibilityTraining + AI assistants and searchGooglebot, Bingbot
Be cited by AI assistantsNothing, or training onlyAll declared crawlers
Never block a generic token such as “Google” or “bot”: you may block Googlebot, AdsBot (Google Ads) or payment and shipping services that call your shop.

Method 1: robots.txt, a polite request

Short answer: add one User-agent group with Disallow: / per crawler. Crawlers that follow the standard stop crawling; the others carry on.

# AI training crawlers: please do not crawl
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

# Google token for use by Gemini models (no effect on Google Search)
User-agent: Google-Extended
Disallow: /
  • The standard (RFC 9309) says crawler names are matched case-insensitively and that a crawler should not use a cached copy for more than 24 hours: the effect is not immediate.
  • PrestaShop pitfall: the “Generate robots.txt file” button (Shop Parameters › Traffic & SEO) rewrites the whole file. Keep your lines aside and add them back after each generation.
  • robots.txt protects nothing: it is public and a crawler can ignore it. The standard itself says these rules are not a form of access authorization.

Method 2: a real user-agent block (.htaccess or nginx)

Short answer: the server answers “403 Forbidden” to any visitor whose user agent contains the crawler’s name. The crawler no longer gets your pages, whether it obeys robots.txt or not.

Apache (.htaccess)

Put this block above the # ~~start~~ line: PrestaShop keeps whatever sits outside its own section when it regenerates the file.

# AI crawlers refused (put this ABOVE the "# ~~start~~" line)
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (GPTBot|ClaudeBot|CCBot|Bytespider) [NC]
RewriteRule .* - [F,L]
</IfModule>

nginx

# Inside the server { } block (nginx)
if ($http_user_agent ~* "(GPTBot|ClaudeBot|CCBot|Bytespider)") {
    return 403;
}

Limits of this method:

  • A user agent is self-declared: a crawler posing as a browser gets through. Conversely, anyone can claim to be “Googlebot”, which is why you verify the IP (next section).
  • The list has to be updated by hand as new crawlers appear.
  • A typo in .htaccess can take the shop offline: keep a copy and test straight away.
  • Test with curl -I -A "GPTBot" https://your-shop.tld/: you should get 403, and 200 with a browser user agent.

How can you be sure you never block Googlebot?

Short answer: do not trust the user agent alone. Google gives two methods: reverse then forward DNS, or matching the IP against the ranges it publishes as JSON.

  1. Run a reverse DNS lookup on the IP (host command).
  2. Check that the name ends in googlebot.com, google.com or googleusercontent.com.
  3. Run a forward DNS lookup on that name and check that it returns the same IP.
host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.

host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1

Careful with country blocking: Google says Googlebot’s default IP addresses appear to be based in the USA. A shop that blocks countries without a Google white list can cut off its own search traffic.

Block AI crawlers in one click with Block IPs

Short answer: in Block IPs, under Shop › User agents, two buttons add AI training crawlers and AI assistants to the blocked list, separately. Recognised Google crawlers are never blocked.

FeatureDetail
Training crawlers in one click25 user agents: GPTBot, ClaudeBot, anthropic-ai, CCBot, Bytespider, meta-externalagent, cohere-ai, Diffbot…
AI assistants and search in one click12 user agents: ChatGPT-User, OAI-SearchBot, PerplexityBot, Perplexity-User, Claude-User, Claude-SearchBot, MistralAI-User, DuckAssistBot…
MatchingCase-insensitive, on part of the user agent; you can add or remove any name.
Response403 message or redirect, your choice. On Apache, an option blocks before PrestaShop even starts, through .htaccess, to save server resources.
Google protectedWhite list of recognised Google crawlers by their IP ranges, ahead of every rule, with a fallback copy shipped in the module if the host blocks the download.
Scraping networksDetects one user agent used by hundreds of IPs, banned in one click.

From version 5.6, the module also verifies trusted crawlers against each company’s official IP lists (Bing, Apple, DuckDuckGo, OpenAI, Perplexity, Anthropic, Mistral): a genuine search or assistant crawler gets through even if its country is blocked, and a fake user agent gains nothing. Training crawlers have a separate switch, off by default, and a crawler you ban stays refused.

Steps

  1. Open Block IPs › Shop › User agents.
  2. Click “Block them” in the AI training robots box.
  3. Only block AI assistants if you do not want to appear in their answers.
  4. Choose the response (403 or redirect) and, on Apache, optionally turn on blocking before PrestaShop.
  5. Follow blocks by reason and the Top 5 bots in the Overview.

See also the Block IPs module page and our guide on fake customer accounts.

Block AI crawlers in one click

Block IPs blocks AI training crawlers and AI assistants separately, and never blocks recognised Google crawlers. Firewall, country blocking and two-factor authentication included (€69.99 excl. VAT on PrestaShop Addons). See the module overview.

See Block IPs on PrestaShop Addons

Frequently asked questions

How do I block GPTBot on PrestaShop?

Add “User-agent: GPTBot” then “Disallow: /” to robots.txt: OpenAI says GPTBot follows it. For an enforced refusal, return a 403 to user agents containing “GPTBot” through .htaccess, nginx or a module such as Block IPs.

Will blocking AI crawlers lower my Google rankings?

No, if Googlebot stays allowed. Google states that Google-Extended does not affect inclusion in Google Search and is not a ranking signal. Blocking OAI-SearchBot, however, removes you from ChatGPT search answers.

Is robots.txt enough to block AI crawlers?

No. It is a request that reputable crawlers follow, but RFC 9309 says it is not access authorization. Some bots acting for a user, such as Perplexity-User, generally ignore it.

What is the difference between GPTBot, OAI-SearchBot and ChatGPT-User?

GPTBot collects content for training, OAI-SearchBot indexes for ChatGPT search, ChatGPT-User opens a page when a user asks. OpenAI lets you control each one separately.

How do I check that a visitor really is Googlebot?

Run a reverse DNS lookup on its IP (the name must end in googlebot.com, google.com or googleusercontent.com), then a forward lookup that must return the same IP. Or match the IP against Google’s published JSON ranges.

Can I block training crawlers and keep ChatGPT and Perplexity?

Yes. Block GPTBot, ClaudeBot, CCBot and Bytespider, and allow OAI-SearchBot, ChatGPT-User, PerplexityBot and Claude-SearchBot. Block IPs splits these two groups into two buttons.

Official sources

This guide is for information only. Test every change (login, sign-up, checkout) and keep a backup before editing .htaccess or robots.txt.