The Anthropic Claude Bot Family: 5 Crawlers Explained

Anthropic operates five distinct web crawlers tied to its Claude AI platform. Each bot has a specific job — from training data collection to real-time search grounding — and each sends a different user-agent string. Identifying them correctly lets you make precise allow/block decisions instead of blanket rules that hurt legitimate traffic.

What Are the Anthropic Claude Crawlers?#

The Claude bot family is a set of HTTP crawlers maintained by Anthropic to power different capabilities of its Claude large language model (LLM) products. They share a common IP range (documented in Anthropic's published AS number) and respect robots.txt when correctly configured. The five crawlers are:

1. ClaudeBot — the primary training-data crawler
2. Claude-Web — real-time web search grounding for Claude.ai
3. anthropic-ai — a legacy user-agent string, now largely superseded
4. Claude-SearchBot — focused retrieval for Claude's search features
5. CCBot (via partnership) — used in early data pipelines; still appears in older logs

How Does ClaudeBot Work?#

ClaudeBot is Anthropic's main broad-web crawler. It fetches publicly accessible pages to build and refresh training datasets for Claude models. It identifies itself with the user-agent string ClaudeBot/1.0 and should include a link to Anthropic's crawler documentation in the User-agent header comment.

Key behaviours to know:

• Respects Crawl-delay directives when set
• Follows Disallow rules in robots.txt
• Typically operates on a slower crawl cadence than Googlebot

How Does Claude-Web Work?#

Claude-Web is a live-retrieval bot. When a Claude.ai user triggers a web-search action, Claude-Web fetches the target URL in near real-time to ground the response in current information. Unlike ClaudeBot, it is not building a long-term index — it is answering a single query. Blocking it stops Claude from citing your content in live answers, which may or may not suit your goals.

What Is the anthropic-ai User-Agent?#

anthropic-ai is a legacy identifier used in early Anthropic crawl infrastructure. You may still see it in server logs on pages that were indexed before Anthropic standardised its bot naming. It behaves identically to ClaudeBot in terms of robots.txt compliance and should be treated the same way for access-control purposes.

What Is Claude-SearchBot?#

Claude-SearchBot supports Anthropic's retrieval-augmented generation (RAG) pipeline. It crawls specific URLs — often triggered by a user query rather than a scheduled crawl — to pull structured content into Claude's context window. Its crawl pattern is more targeted than ClaudeBot: expect fewer, more intentional requests rather than broad spidering.

What Is CCBot and How Does It Relate to Claude?#

CCBot is the crawler run by Common Crawl, an open-access web archive that Anthropic (like many AI labs) has historically licensed data from. CCBot is not operated by Anthropic, but blocking it has downstream effects on Claude training data if Anthropic sources from those snapshots. CCBot has its own robots.txt compliance policy and its own IP ranges, separate from Anthropic infrastructure.

How Do You Identify Claude Bots in Server Logs?#

Look for these user-agent substrings in your access logs:

ClaudeBot
Claude-Web
anthropic-ai
Claude-SearchBot
CCBot

Verify ownership by doing a reverse DNS lookup on the source IP and confirming it resolves to an Anthropic-owned subnet. Spoofed user-agents are common; IP verification is the authoritative check.

How Do You Block or Allow Claude Bots in robots.txt?#

Each bot requires its own User-agent directive because they serve different purposes. A targeted configuration looks like this:

# Block training crawl only
User-agent: ClaudeBot
Disallow: /

# Allow live-search grounding
User-agent: Claude-Web
Allow: /

# Block legacy identifier
User-agent: anthropic-ai
Disallow: /

This approach lets you opt out of training data collection while keeping your content eligible for citation in live Claude answers — a distinction that matters as AI search surfaces grow.

Audit tip: Use SeoChatAI to scan your site's current robots.txt for gaps in AI-crawler coverage. The audit flags missing or conflicting directives across all major bot families, including Claude's five crawlers.

Why Does Distinguishing Between Claude Crawlers Matter for SEO?#

Blocking all Claude bots with a single wildcard rule (User-agent: *) can inadvertently suppress your content from AI-generated answers in Claude.ai, which is increasingly a traffic source. Conversely, allowing everything without review means your training-data opt-out preferences are not enforced. Precise per-bot rules give you both protection and visibility.

For sites auditing their full AI-crawler posture — across Claude, GPTBot, Google-Extended, PerplexityBot, and others — SeoChatAI's crawler audit generates a prioritised fix list without requiring manual log parsing.

Summary: The 5 Claude Crawlers at a Glance#

Understanding this table is the foundation of any AI-search readiness strategy. Each row requires a deliberate allow-or-block decision, not a default.