Why this matters for your business at all
If your site runs on custom code, you have a robots.txt file - even if you've never touched it. That file already contains (or doesn't contain) directives for every AI bot trying to reach your site, and the decision to leave it empty, block everything, or block selectively affects two opposing things at once: whether your content enters an AI model's training set without your say, and whether you show up at all when someone asks ChatGPT, Perplexity, or Gemini a question your business could answer. 44.2% of ChatGPT's citations come from the first 30% of a document, so even when a bot does crawl you, structure decides whether it ever reaches the answer (source: searchengineland.com). Blanket-blocking the wrong bot removes you from that game entirely, before your content is even evaluated.
Three kinds of bots, not one - even within the same company
OpenAI's own documentation makes a point most site owners miss: a single company runs several bots with separate jobs, and each is controlled independently in robots.txt.
- GPTBot - OpenAI's training crawler. Blocking it keeps your content out of future model training sets, but does not remove you from ChatGPT search or live answers.
- OAI-SearchBot - the bot that decides search visibility in ChatGPT. This is the one to block if you don't want to be cited in answers - and the one to leave open if that's your goal.
- ChatGPT-User - only hits your site when a specific user asks ChatGPT to open a link. Per OpenAI's own documentation, this is a user-initiated action, so robots.txt rules may not necessarily apply to it (source: developers.openai.com).
Anthropic is built the same way: ClaudeBot for future model training, Claude-SearchBot for its search index, and Claude-User for live retrieval triggered by a user's request inside Claude. Per Anthropic's own documentation, blocking ClaudeBot alone signals that your future content should be excluded from training - but it doesn't touch the other two bots, each of which needs its own separate block rule (source: support.claude.com).
What blocking actually does - and what happens when a "block" doesn't work
Here's the real complication: a robots.txt block is an agreed-upon request, not a lock. The bot has to choose to honor it. In August 2025, Cloudflare published a report finding that Perplexity kept scraping content from sites that had explicitly blocked it in robots.txt and at the server level - using IP addresses not published as belonging to it, and a generic browser signature impersonating Chrome, to work around the block. In Cloudflare's parallel test, OpenAI's crawler stopped scraping the moment it hit the robots.txt directive - the opposite behavior entirely. Following the findings, Cloudflare removed Perplexity from its list of Verified Bots (source: blog.cloudflare.com). The practical takeaway: robots.txt works for bots that intend to honor it, but it isn't a substitute for a server- or CDN-level block once there's evidence of systematic disregard.
How many sites actually block - and why it's not one number
Per public tracking by Dark Visitors, as of early July 2026 about 20% of top websites blocked ClaudeBot in robots.txt (source: darkvisitors.com). The rate varies sharply by sector - news and media sites block at far higher rates than ecommerce sites, because they have more to lose from unauthorized training and depend less on traffic from AI engines. There's no single "right default" for every business - there's a question every site owner has to answer for themselves: is my content training-sensitive, and do I depend on AI citations as a traffic channel.
The cost of a blanket block
Blocking "just to be safe" isn't free. A study by researchers at Rutgers and Wharton (Zhao & Berman) that tracked news publishers who imposed blanket AI-bot blocks found a measurable traffic decline - in the study's more recent version, about a 7% traffic drop for large publishers that blocked, compared to those that didn't (source: ppc.land). In other words: a block meant as "protection" also costs you real visibility, which is why the decision calls for judgment based on your industry and your goal - not a default of Disallow: / applied across the board.
How May I actually approaches this
On the projects we build, we don't start from "block everything" or "allow everything" - we map the relevant bots by name (not by company), and decide per bot based on intent: training bots get blocked by default for clients who are sensitive to that, search and citation bots stay open for clients who want GEO/AEO visibility, and bots that ignore the directive (like Perplexity's documented case) get handled at the CDN level, not through robots.txt alone. That's the difference between a robots.txt file someone filled in once and forgot, and a business decision that gets revisited as the data changes.
Not sure what's right for your business?
We audit your site's robots.txt and AI visibility, and set up a bot policy that matches both your stance on model training and your GEO/AEO goals - not a one-size-fits-all template.
Get started