ABC tells Australian parliament robots.txt opt-out is bypassed by OpenAI and Anthropic

Kate Gilchrist, the ABC's head of content and legal operations, told Australia's Joint Select Committee on Artificial Intelligence on Oct 6 that the copyright system is adequate for AI and that companies can negotiate licences with rightsholders, rejecting an opt-out model.

Per the report, Gilchrist told lawmakers that robots.txt "do not crawl" signals have been documented as bypassed by OpenAI and Anthropic despite public claims that both companies respect them, citing research by the licensing startup TollBit and subsequent reporting by Business Insider. The ABC itself added a block on OpenAI's crawler to its robots.txt in 2023. Gilchrist argued rightsholders cannot feasibly maintain opt-out signals across every scraping channel, so an opt-out-based regime would not work.

OpenAI chief strategy officer Jason Kwon testified at the same hearing. The report also notes the ABC has selected Anthropic's Claude as an enterprise AI pilot while arguing Anthropic's products may have been trained on its content without permission. All bypass claims are the ABC's testimony citing third-party research, not adjudicated facts.