Websites may soon be able to tell several AI companies “don’t train on this” with one line instead of one rule per company. OpenAI, Google, Microsoft and Anthropic have each said they will honour a shared opt-out signal placed in a site’s robots.txt file.
None of this is switched on everywhere yet. Anthropic says it has already started supporting the signal, OpenAI has promised it in the “coming months”, and Google and Microsoft are both aiming for 2027.
One token instead of a rule per crawler
Today a publisher has to name each company’s crawler separately in robots.txt, for instance Google-Extended for Google’s AI training. The proposal adds a shared “policy token” that expresses a broad preference, and any participating company would read it as an instruction for its own crawlers.
Google’s Search Central post explains the plan in detail. If a site uses the token to opt out of training and does not mention Google-Extended, Google plans to read that as a Google-Extended block. Rules aimed at a single company keep working alongside it, so a site can opt out broadly and still make an exception.
The idea comes from Kevin Kelley, a standards lead at Anthropic, and is being developed in the AIPREF working group of the Internet Engineering Task Force. Google describes it as a draft proposal while talks on the wider vocabulary continue.
What each company has said
According to a report from The Verge, Microsoft will go further and let publishers separate AI training from AI grounding, the step where a model is pointed at a trusted source to answer a question. Sites will still appear in Bing results whatever they choose.
Meta’s communications head Andy Stone also welcomed a common signal, but the report says he did not commit Meta to honouring it.
The signal is voluntary. Robots.txt has always worked on trust, and an opt-out added now does nothing about content already collected into existing datasets. Publishers who want a stricter block today still need the company-specific rules.
