Anthropic documents ClaudeBot training opt-out via robots.txt

What happened

Anthropic publishes help-center documentation describing the three web robots it operates and how site owners can control them. The training-relevant control: blocking ClaudeBot in robots.txt "signals that the site's future materials should be excluded from our AI model training datasets." The page separately documents Claude-User (fetches pages when a Claude user asks a question) and Claude-SearchBot (indexes content for search-result quality), each with its own opt-out trade-off. Anthropic also supports the non-standard Crawl-delay directive to slow crawls, and warns that blocking by IP address is not a reliable substitute for robots.txt. The page is dated 7 April 2026.

Why it may matter to you

For a creator with a website, this is Anthropic's own stated opt-out: a ClaudeBot disallow rule in robots.txt. Blocking only ClaudeBot keeps the site eligible for Claude's search-grounded answers and user-initiated fetches, which is the middle ground most site owners want. Note the limitation Anthropic states itself: the opt-out covers "future materials" — it does not describe any mechanism for removing content already collected.

What is not yet known

The documentation does not say how long it takes for an opt-out to propagate into training datasets, nor does it offer a way to verify exclusion. As with all robots.txt mechanisms, compliance is voluntary on the company's part.