ClaudeBot is Anthropic's primary web crawler that gathers content for Anthropic's products, including the Claude assistant. The Anthropic crawler family includes several user-agent strings — ClaudeBot, Claude-Web, Claude-SearchBot, Claude-User, and anthropic-ai — each used for slightly different purposes across the Claude product surface.
ClaudeBot is the main training-data crawler, similar in role to OpenAI's GPTBot. Claude-SearchBot is used for Claude's search-augmented answer generation. Claude-User is the on-demand fetcher that fires when a Claude user asks the assistant to read a specific URL.
Anthropic publishes its bot list at https://anthropic.com/claudebot and is one of the most transparent crawler operators in the AI industry. The bots respect robots.txt directives; blocking ClaudeBot removes you from Anthropic training data, while blocking Claude-SearchBot removes you from search-augmented Claude answer candidacy.
Anthropic is the most explicit adopter of the llms.txt standard, treating it as a primary source-prioritization signal. A site that allows the Claude bot family AND ships a clean llms.txt is in the top tier of Claude-citation candidates as of 2026. The combination is the highest-leverage single Anthropic-targeted AEO investment available to SaaS sites today.