Skip to content

Analytics cookies

We use Google Analytics and Google Tag Manager to count visits and see which pages are useful. If you accept, Google Analytics may set cookies to recognise your return visits. If you decline, it sets no cookies, but Google still receives a cookieless signal for each page you view. We don’t send Google your name, email address or registration details, and you can change your choice at any time under “Cookie settings” at the bottom of the page. Privacy

OpenAI publishes crawler controls: GPTBot training opt-out via robots.txt

Sep 27, 2026 · Gathered automatically from public sources. Not reviewed by a person.

What happened

OpenAI maintains public documentation describing its web crawlers and how site owners can control them. The key control for creators: disallowing GPTBot in robots.txt signals that a site's content should not be used in training OpenAI's generative AI foundation models. The controls are independent — a site owner can allow OAI-SearchBot (so the site still appears in ChatGPT search results) while disallowing GPTBot (opting out of training). OpenAI also documents ChatGPT-User, a user-triggered fetcher for which it says robots.txt rules may not apply because the fetch is initiated by a user.

Why it may matter to you

For a creator with a website, this is the concrete, followable opt-out OpenAI offers: add a GPTBot disallow rule to robots.txt. It does not remove content already absorbed into an existing model, and it does not stop user-initiated fetches (ChatGPT-User), but it is the mechanism OpenAI itself points to for future training exclusion — and it can be combined with keeping search visibility via OAI-SearchBot.

What is not yet known

The documentation does not say how quickly an opt-out takes effect for training pipelines, nor does it provide a way to verify that previously crawled content was excluded. Compliance with robots.txt is voluntary on OpenAI's part; there is no independent audit mechanism described.

Sources