How to

robots.txt for AI Bots: Complete 2026 Guide

AI crawlers now rival search bots in volume. This guide shows you the exact robots.txt directives to block or allow GPTBot, ClaudeBot, and PerplexityBot — protecting content while staying visible in AI search.

By Daniel MercerPublished May 22, 20266 min read

Controlling AI crawlers starts in one file: robots.txt. By adding the correct User-agent directives for GPTBot, ClaudeBot, and PerplexityBot, you decide whether large language models train on your content or cite it in answers — without touching a single line of server code.

What Is robots.txt and Why Do AI Bots Read It?#

robots.txt is a plain-text file hosted at the root of your domain (https://yourdomain.com/robots.txt) that tells crawlers which URLs they may or may not access. It follows the Robots Exclusion Protocol, a de facto standard originally designed for search engines. Major AI labs — OpenAI, Anthropic, Perplexity, and others — have committed to honoring robots.txt directives for their training and retrieval crawlers, making it your first line of defense and your primary opt-in mechanism.

How Does robots.txt Differ From Meta Robots Tags?

A robots.txt file operates at the crawl level: it stops a bot from fetching a URL entirely. A <meta name="robots"> tag or X-Robots-Tag header operates at the indexing level: the bot fetches the page but respects instructions like noindex or noai. For AI bots you usually want crawl-level blocking (robots.txt) when you want zero content exposure, and index-level tags when you're fine with crawling but not training.

Which AI Bots Have Official User-Agent Strings in 2026?#

Every major AI crawler publishes a canonical user-agent string. Use these exactly — they are case-sensitive in most implementations.

Bot	User-agent string	Operator
OpenAI training crawler	`GPTBot`	OpenAI
OpenAI browse/retrieval	`ChatGPT-User`	OpenAI
Anthropic training crawler	`ClaudeBot`	Anthropic
Anthropic browse/retrieval	`Claude-User`	Anthropic
Perplexity retrieval	`PerplexityBot`	Perplexity AI
Google extended crawl	`Google-Extended`	Google
Common Crawl	`CCBot`	Common Crawl
Cohere training	`cohere-ai`	Cohere
Meta AI crawler	`FacebookBot`	Meta
Apple AI crawler	`Applebot-Extended`	Apple

Note: User-agent strings change. Always verify against each operator's official documentation before deploying to production.

How Do You Block All AI Bots With robots.txt?#

To block every listed AI crawler from your entire site, add one Disallow block per user-agent:

User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-User
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: cohere-ai
Disallow: /

User-agent: FacebookBot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

Place these blocks above your general User-agent: * rule so they're parsed independently. Never rely on a catch-all User-agent: * to block AI bots unless you've verified those bots don't have a more specific rule in your file — specificity wins.

How Do You Allow AI Bots but Block Training?#

Some AI crawlers distinguish between a training crawler and a retrieval (browse) crawler. OpenAI's ChatGPT-User fetches pages to answer user questions in real time; GPTBot fetches pages to train future models. If you want citations in ChatGPT answers without contributing training data, allow the retrieval agent and block the training agent:

# Block training crawler
User-agent: GPTBot
Disallow: /

# Allow retrieval crawler (citations in ChatGPT)
User-agent: ChatGPT-User
Allow: /

Apply the same logic for Anthropic: block ClaudeBot, allow Claude-User.

How Do You Partially Block AI Crawlers by Directory?#

You don't have to choose all-or-nothing. Disallow specific paths while leaving the rest open:

User-agent: GPTBot
Disallow: /members/
Disallow: /premium-content/
Disallow: /api/
Allow: /blog/
Allow: /

This keeps your free blog accessible for AI citations while protecting paywalled or proprietary directories. The order of Allow and Disallow rules within a block matters — most parsers apply the longest matching path, so be explicit.

What Is Google-Extended and Should You Block It?#

Google-Extended is Google's dedicated token for controlling whether your content trains Gemini models and Google's AI Overviews feature. It is separate from Googlebot. Blocking Google-Extended does not affect your Google Search rankings or crawl budget — Googlebot continues to index your pages normally. If you block Google-Extended, your content may appear less frequently as a source in AI Overviews.

# Block Gemini/AI Overviews training only
User-agent: Google-Extended
Disallow: /

# Googlebot is unaffected — rankings preserved
User-agent: Googlebot
Allow: /

How Do You Verify Your robots.txt Is Working?#

Verification has three layers:

Syntax check — Paste your file into Google Search Console's robots.txt Tester or a standalone validator. Look for parse errors and conflicting rules.
Log file analysis — Filter your server access logs for the AI bot user-agent strings. If a bot you blocked still appears, it may be spoofing its user-agent (robots.txt only binds compliant crawlers).
Third-party audit — Tools like the SeoChatAI robots generator let you build, preview, and validate your AI-bot directives before deployment, reducing configuration errors.

What Happens If an AI Bot Ignores robots.txt?#

robots.txt is not legally enforceable on its own — it is a protocol, not a firewall. Compliant operators like OpenAI, Anthropic, and Perplexity have publicly committed to honoring it, but there is no technical mechanism preventing a non-compliant crawler from ignoring the file. For stronger protection consider:

Rate-limiting by IP range — AI labs publish their crawler IP ranges; block these at the CDN or WAF layer.
Cloudflare's AI bots block — Cloudflare's "Bot Fight Mode" includes a one-click AI crawler block as of 2024.
Terms of Service — Explicit ToS language prohibiting scraping adds a legal layer on top of robots.txt.
llms.txt — A newer convention (see our guide on what is llms.txt) that tells AI systems what content you do and don't want used in inference.

robots.txt Template: Balanced Configuration for 2026#

The following template blocks all training crawlers, allows all retrieval crawlers, protects private directories, and preserves normal search engine crawling:

# ==========================================
# AI Training Crawlers — Blocked
# ==========================================
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: cohere-ai
Disallow: /

User-agent: Applebot-Extended
Disallow: /

# ==========================================
# AI Retrieval Crawlers — Allowed
# ==========================================
User-agent: ChatGPT-User
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

# ==========================================
# Search Engines — Full Access
# ==========================================
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

# ==========================================
# Global Fallback
# ==========================================
User-agent: *
Disallow: /private/
Disallow: /api/
Disallow: /members/

Sitemap: https://yourdomain.com/sitemap.xml

You can generate a customized version of this file using the SeoChatAI robots generator, which keeps the user-agent list current and validates your syntax automatically.

Common Mistakes to Avoid#

Merging AI bot rules into User-agent: * — This applies to every bot including Googlebot. Always use named agents.
Using Disallow: with no path — An empty Disallow: means "allow everything," not "block everything." Use Disallow: / to block the full site.
Forgetting ChatGPT-User when blocking GPTBot — They are separate agents. Blocking one does not block the other.
Not adding a Sitemap: directive — AI retrieval bots use sitemaps to discover content efficiently. Include it.
Caching stale files — If your CDN caches robots.txt, new directives won't propagate immediately. Set a short TTL (≤1 hour) on this file.

Key Takeaways#

Training crawlers and retrieval crawlers use different user-agent strings — control them separately.
Google-Extended blocks Gemini training without touching Google Search rankings.
robots.txt is the fastest, lowest-risk way to control AI content access — deploy it before any other solution.
Pair robots.txt with log monitoring and IP-level blocking for full coverage against non-compliant bots.

robots.txt for AI Bots: Complete 2026 Guide — illustration 1

Try our free robots.txt generator

Read article

Share this articleX LinkedIn

Published May 22, 2026

Frequently asked questions

How do I block GPTBot in robots.txt?

Add a dedicated block: `User-agent: GPTBot` on one line, then `Disallow: /` on the next. This stops OpenAI's training crawler from accessing any page on your site. To also block ChatGPT's browsing/retrieval crawler, add a separate block for `ChatGPT-User`.

Does blocking AI bots in robots.txt hurt my Google rankings?

No — as long as you target specific AI bot user-agents and leave `Googlebot` unrestricted. Blocking `Google-Extended` only affects Gemini model training and AI Overviews sourcing; it does not affect your organic search ranking or crawl budget.

What is the difference between GPTBot and ChatGPT-User?

`GPTBot` is OpenAI's crawler for collecting training data for future models. `ChatGPT-User` is OpenAI's retrieval crawler that fetches pages in real time to answer user questions. You can block one without blocking the other, giving you granular control over training versus citation.

Does PerplexityBot respect robots.txt?

Perplexity AI has publicly committed to honoring robots.txt directives. Add `User-agent: PerplexityBot` with the appropriate `Allow` or `Disallow` directive. Note that like all robots.txt rules, compliance is voluntary — monitor your server logs to confirm it is being respected.

Can I block AI crawlers from some pages but not others?

Yes. Use path-specific `Disallow` directives within each bot's block — for example, `Disallow: /premium/` blocks only that directory while leaving the rest of your site accessible. Combine with `Allow:` rules for exceptions within a blocked path.

Is robots.txt enough to stop AI bots from scraping my site?

robots.txt only binds compliant crawlers. For stronger protection, combine it with IP-range blocking at your CDN or WAF, Cloudflare's AI bot block feature, and explicit Terms of Service language. robots.txt is the necessary first step, not the complete solution.

How often should I update my robots.txt for new AI bots?

Review your robots.txt at least quarterly. New AI labs launch crawlers regularly, and existing operators sometimes change or add user-agent strings. Subscribing to each operator's developer changelog and auditing your server logs monthly helps you catch new bots before they accumulate significant crawl activity.