OpenAI, Google, Microsoft and Anthropic back shared robots.txt AI-training opt-out signal

What happened

On 30 October 2026, OpenAI, Google, Microsoft and Anthropic all announced support for a single shared robots.txt policy token, usage_ai-training, based on an Internet Engineering Task Force (IETF) draft, that lets a website express one preference about foundational generative-AI training across providers instead of maintaining separate rules for each company's crawlers.

Timelines differ

Anthropic says it has already started implementing support. OpenAI says it will honour the signal "in the coming months" and that it will not replace its existing robots.txt controls. Google says it aims to support the control in 2027 and will interpret an opt-out via the shared token as disallowing Google-Extended unless Google-Extended is explicitly specified. Microsoft says its new training and grounding preferences take effect on 1 February 2027, and adds a separate AI-grounding preference; both default to Allow when no preference is published.

What it does and does not do

The signal records a preference, not a price: Microsoft states that neither allowing nor refusing training guarantees inclusion, traffic or payment, and that refusing grounding reduces brand visibility in AI experiences. The draft is an Internet-Draft dated 30 October 2026, not yet a final IETF standard, and a robots.txt file that does not use the token expresses no preference at all.