Fondamentaux

The Anthropic Claude Bot Family: 5 Crawlers Explained

Anthropic operates five distinct web crawlers under the Claude brand. Knowing which bot does what lets you make smarter allow/block decisions in robots.txt and protect server resources.

Par Daniel MercerPublié le 30 mai 20265 min de lecture

Anthropic operates five distinct web crawlers tied to its Claude AI platform. Each bot has a specific job — from training data collection to real-time search grounding — and each sends a different user-agent string. Identifying them correctly lets you make precise allow/block decisions instead of blanket rules that hurt legitimate traffic.

What Are the Anthropic Claude Crawlers?#

The Claude bot family is a set of HTTP crawlers maintained by Anthropic to power different capabilities of its Claude large language model (LLM) products. They share a common IP range (documented in Anthropic's published AS number) and respect robots.txt when correctly configured. The five crawlers are:

ClaudeBot — the primary training-data crawler
Claude-Web — real-time web search grounding for Claude.ai
anthropic-ai — a legacy user-agent string, now largely superseded
Claude-SearchBot — focused retrieval for Claude's search features
CCBot (via partnership) — used in early data pipelines; still appears in older logs

How Does ClaudeBot Work?#

ClaudeBot is Anthropic's main broad-web crawler. It fetches publicly accessible pages to build and refresh training datasets for Claude models. It identifies itself with the user-agent string ClaudeBot/1.0 and should include a link to Anthropic's crawler documentation in the User-agent header comment.

Key behaviours to know:

Respects Crawl-delay directives when set
Follows Disallow rules in robots.txt
Typically operates on a slower crawl cadence than Googlebot

How Does Claude-Web Work?#

Claude-Web is a live-retrieval bot. When a Claude.ai user triggers a web-search action, Claude-Web fetches the target URL in near real-time to ground the response in current information. Unlike ClaudeBot, it is not building a long-term index — it is answering a single query. Blocking it stops Claude from citing your content in live answers, which may or may not suit your goals.

What Is the anthropic-ai User-Agent?#

anthropic-ai is a legacy identifier used in early Anthropic crawl infrastructure. You may still see it in server logs on pages that were indexed before Anthropic standardised its bot naming. It behaves identically to ClaudeBot in terms of robots.txt compliance and should be treated the same way for access-control purposes.

What Is Claude-SearchBot?#

Claude-SearchBot supports Anthropic's retrieval-augmented generation (RAG) pipeline. It crawls specific URLs — often triggered by a user query rather than a scheduled crawl — to pull structured content into Claude's context window. Its crawl pattern is more targeted than ClaudeBot: expect fewer, more intentional requests rather than broad spidering.

What Is CCBot and How Does It Relate to Claude?#

CCBot is the crawler run by Common Crawl, an open-access web archive that Anthropic (like many AI labs) has historically licensed data from. CCBot is not operated by Anthropic, but blocking it has downstream effects on Claude training data if Anthropic sources from those snapshots. CCBot has its own robots.txt compliance policy and its own IP ranges, separate from Anthropic infrastructure.

How Do You Identify Claude Bots in Server Logs?#

Look for these user-agent substrings in your access logs:

ClaudeBot
Claude-Web
anthropic-ai
Claude-SearchBot
CCBot

Verify ownership by doing a reverse DNS lookup on the source IP and confirming it resolves to an Anthropic-owned subnet. Spoofed user-agents are common; IP verification is the authoritative check.

How Do You Block or Allow Claude Bots in robots.txt?#

Each bot requires its own User-agent directive because they serve different purposes. A targeted configuration looks like this:

# Block training crawl only
User-agent: ClaudeBot
Disallow: /

# Allow live-search grounding
User-agent: Claude-Web
Allow: /

# Block legacy identifier
User-agent: anthropic-ai
Disallow: /

This approach lets you opt out of training data collection while keeping your content eligible for citation in live Claude answers — a distinction that matters as AI search surfaces grow.

Audit tip: Use SeoChatAI to scan your site's current robots.txt for gaps in AI-crawler coverage. The audit flags missing or conflicting directives across all major bot families, including Claude's five crawlers.

Why Does Distinguishing Between Claude Crawlers Matter for SEO?#

Blocking all Claude bots with a single wildcard rule (User-agent: *) can inadvertently suppress your content from AI-generated answers in Claude.ai, which is increasingly a traffic source. Conversely, allowing everything without review means your training-data opt-out preferences are not enforced. Precise per-bot rules give you both protection and visibility.

For sites auditing their full AI-crawler posture — across Claude, GPTBot, Google-Extended, PerplexityBot, and others — SeoChatAI's crawler audit generates a prioritised fix list without requiring manual log parsing.

Summary: The 5 Claude Crawlers at a Glance#

Bot	Purpose	Real-time?	Training?
ClaudeBot	Broad training crawl	No	Yes
Claude-Web	Live search grounding	Yes	No
anthropic-ai	Legacy training crawler	No	Yes
Claude-SearchBot	RAG retrieval	Near real-time	Partial
CCBot	Common Crawl archive	No	Indirect

Understanding this table is the foundation of any AI-search readiness strategy. Each row requires a deliberate allow-or-block decision, not a default.

Partager cet articleX LinkedIn

Publié le 30 mai 2026

Questions fréquentes

What user-agent string does ClaudeBot use?

ClaudeBot identifies itself with the user-agent string `ClaudeBot/1.0`. You can block or allow it specifically in robots.txt using `User-agent: ClaudeBot`. Always verify the source IP resolves to Anthropic infrastructure before trusting the user-agent string alone, since it can be spoofed.

How do I stop Anthropic from using my content for Claude training?

Add `User-agent: ClaudeBot` followed by `Disallow: /` to your robots.txt file. Also add a matching rule for `anthropic-ai` to cover the legacy identifier. This signals to Anthropic's training crawlers that your content is off-limits, though it does not affect content already crawled.

What is the difference between ClaudeBot and Claude-Web?

ClaudeBot is a scheduled training crawler that builds datasets for Claude model development. Claude-Web is a real-time retrieval bot triggered when a Claude.ai user performs a web search. They serve different functions and should be configured separately in robots.txt if you want granular control.

Does blocking CCBot affect Claude AI?

Indirectly, yes. CCBot is run by Common Crawl, not Anthropic, but Anthropic has historically licensed Common Crawl snapshots for training data. Blocking CCBot may reduce your content's presence in future Claude training sets sourced from those archives, though the effect is indirect and not guaranteed.

Can I allow Claude-Web but block ClaudeBot?

Yes. Because each crawler has its own user-agent string, you can write separate robots.txt rules — `Disallow: /` under `ClaudeBot` and `Allow: /` under `Claude-Web`. This lets your pages appear in Claude's live search answers while opting out of long-term training data collection.

How do I verify a Claude bot is really from Anthropic?

Perform a reverse DNS lookup on the IP address in your server log. A legitimate Anthropic crawler will resolve to a hostname within Anthropic's published autonomous system. User-agent strings alone are unreliable because any client can spoof them — IP verification is the authoritative method.

What is Claude-SearchBot used for?

Claude-SearchBot supports Anthropic's retrieval-augmented generation pipeline. It fetches specific URLs in response to user queries to pull live, structured content into Claude's context window. Its crawl pattern is targeted rather than broad — you'll see fewer but more intentional requests compared to ClaudeBot.