ClaudeBot
Anthropic's crawler for both Claude training data and Claude's live web-fetch tool.
Definition
ClaudeBot is Anthropic's web crawler. It serves dual duty: collecting training data and powering Claude's live web-fetch and web-search tools (used in Claude.ai and Claude Code). Allowing it makes your content quotable when Claude searches the web during a conversation.
When to use
Allow ClaudeBot in robots.txt if you want Claude to be able to fetch and cite your content. Blocking it removes you from both training and live retrieval.
See also
- Claude Code — Anthropic's official terminal-based coding agent, with MCP support and a plugin ecosystem.
- robots.txt — Plain text file at /robots.txt that tells crawlers which paths they may or may not fetch.
- AI crawler — Web crawler operated by an AI company — GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended.