GPTBot, ClaudeBot & Friends: Which AI Crawlers to Allow or Block

Sometime in the last year, a wave of well-meaning advice told website owners to block AI bots in robots.txt. Plenty of local businesses did — and quietly erased themselves from ChatGPT, Claude, and Perplexity in the process. The advice wasn't wrong when it was written. It's wrong now, because the bots split in two.

Training bots vs. search bots — the only distinction that matters

Every major AI company now runs (at least) two kinds of crawler, and they do opposite jobs for you:

• Training crawlers collect content to train future models. You get no link, no credit, no visitor. OpenAI's is GPTBot; Anthropic's is ClaudeBot; Google's training opt-out is Google-Extended.
• Search & retrieval crawlers fetch pages so the AI can answer a user's question right now — with a citation and often a clickthrough. OpenAI's is OAI-SearchBot (plus ChatGPT-User for live fetches); Anthropic's is Claude-SearchBot; Perplexity's is PerplexityBot.

Block the training bots and you've made an IP decision. Block the search bots and you've made a marketing decision — the same one as blocking Googlebot in 2010.

What the crawl data actually shows

Crawl-log analyses in 2025–26 put numbers on the trade. GPTBot crawls roughly 1,255 pages for every referral visit AI products send back; ClaudeBot's ratio runs into the tens of thousands to one. Training crawls are, bluntly, a bad trade in raw traffic terms — you donate content and bandwidth to a model that may never send anyone back. Search crawls are the opposite: the visitors they do send convert around 4x better than average organic visitors, because they arrive already recommended.

So what should a local business do?

For most service businesses — plumbers, dentists, law firms, restaurants — the calculus is simple: allow everything. Your website content is marketing material, not proprietary IP. You want it in training data (that's the model's long-term memory recommending you for years) and you want it in live search. The "protect my content" argument makes sense for publishers who sell content; it makes little sense for businesses whose content exists to attract customers.

If you do care about training use — say you publish original research or paid guides — the defensible middle is: block training bots, allow search bots. In robots.txt terms: disallow GPTBot, ClaudeBot, and Google-Extended; explicitly allow OAI-SearchBot, Claude-SearchBot, ChatGPT-User, and PerplexityBot.

Three mistakes we keep seeing

1. Blanket blocks. A single "block all AI" rule (or an overzealous security plugin/CDN setting) that catches the search bots too. This is the silent visibility killer of 2026.

2. Thinking it protects Google rankings. AI crawlers are completely separate from Googlebot. Blocking or allowing them has zero effect on your Google search position.

3. Set-and-forget. New bots appear constantly and names change. Audit your robots.txt a couple of times a year.

Check your site in ten seconds

Our free AI Visibility Checker tests exactly which AI crawlers your robots.txt allows or blocks — along with your schema, llms.txt, and answer structure — and tells you what it means in plain English. If you need to rewrite the rules, the robots.txt generator will build a correct file for the policy you choose. And if you're deciding what policy to want, start from this question: would you rather your content be protected, or be recommended?