OpenAI publishes crawler controls: GPTBot training opt-out via robots.txt

What happened

OpenAI maintains public documentation describing its web crawlers and how site owners can control them. The key control for creators: disallowing GPTBot in robots.txt signals that a site's content should not be used in training OpenAI's generative AI foundation models. The controls are independent — a site owner can allow OAI-SearchBot (so the site still appears in ChatGPT search results) while disallowing GPTBot (opting out of training). OpenAI also documents ChatGPT-User, a user-triggered fetcher for which it says robots.txt rules may not apply because the fetch is initiated by a user.

Why it may matter to you

For a creator with a website, this is the concrete, followable opt-out OpenAI offers: add a GPTBot disallow rule to robots.txt. It does not remove content already absorbed into an existing model, and it does not stop user-initiated fetches (ChatGPT-User), but it is the mechanism OpenAI itself points to for future training exclusion — and it can be combined with keeping search visibility via OAI-SearchBot.

What is not yet known

The documentation does not say how quickly an opt-out takes effect for training pipelines, nor does it provide a way to verify that previously crawled content was excluded. Compliance with robots.txt is voluntary on OpenAI's part; there is no independent audit mechanism described.