One Robots.txt Line to Opt Out of AI Training: How __usage_ai-training Works — and Why It Changes Nothing Yet

For years, opting out of AI training meant writing a separate rule for every company: GPTBot for OpenAI, ClaudeBot for Anthropic, Google-Extended for Google, and Microsoft's NoArchive and NoCache tags. On 30 October 2026, as reported, the big AI companies announced a shared alternative: one robots.txt group keyed by the policy token __usage_ai-training, expressing a single training preference to every participating provider at once. The mechanism is defined in an Internet-Draft by K. Kelley of Anthropic, dated 30 October 2026 and expiring 3 May 2027. The idea is genuinely simpler. The catch: the signal is voluntary, not a standard, training-only, honoured on each company's own timeline, and live nowhere everywhere today. This article compares the old per-crawler opt-outs with the new shared token, and keeps every caveat in place.

Illustration generated for INORYVA — not a real person, product, or place.

How the new signal works

The draft defines one policy token, __usage_ai-training, used inside an ordinary robots.txt group. A site writes a User-Agent line naming the token, then Disallow and Allow lines as usual. The draft's own example: User-Agent: __usage_ai-training, Disallow: /, Allow: /press-kit/ — a preference against AI training for everything except the press kit. No new directive names and no new grammar are introduced; the existing RFC 9309 path language and precedence rules are unchanged. The draft is explicit that "this policy token is not a directive for crawling behavior" and that usage preferences and crawler rules are "independent and orthogonal". In plain terms: the token says what a site prefers about training; it does not block any crawler from visiting, and it does not affect search indexing.

Two details matter for anyone implementing it. First, a wildcard group — User-Agent: * — must not be interpreted as matching the token. If no group names __usage_ai-training, no preference has been expressed at all, whatever the rest of the robots.txt says. Second, the draft defines "AI Training" narrowly: "using an asset to modify the learned parameters of a generative AI model". Training a model on a page is covered. Quoting a page in an AI-written answer is a different activity, and this token does not govern it.

The old way: one rule per company

Before this week, a publisher who wanted out of training had to maintain a list. OpenAI's GPTBot, Anthropic's ClaudeBot and Google's Google-Extended each needed their own robots.txt group, and Microsoft's approach ran through NoArchive and NoCache tags that were doing double duty. Each rule had its own syntax quirks, its own documentation page, and its own update history — and missing one company meant that company's training pipeline kept its default. The shared token replaces that list with one group, for the companies that participate. That is the whole improvement, and it is a real one for small publishers who never had a lawyer or an SEO team maintaining four separate opt-outs.

What has not changed is the enforcement model. Robots.txt has always been a voluntary convention: well-behaved crawlers follow it, and nothing in the file itself stops a scraper that ignores it. The new token inherits that model exactly. It is a machine-readable request, not a technical barrier and not a legal right. A company that chooses not to participate is not bound by it, and a scraper that ignores robots.txt altogether will ignore this too.

Who honours it, and when — with the sourcing stated honestly

OpenAI's announcement, read directly, says: "OpenAI will honor a common signal in robots.txt... This voluntary effort makes it easier for publishers and website operators to communicate the same preference to multiple participating providers at once." OpenAI adds that it aims "to implement the signal in the coming months" — no date — and that the new signal "will not replace the controls we already provide through robots.txt", which "will continue to apply and be respected". The announcement draws on the IETF AI Preferences working group, and OpenAI says it is "joining other participants in the working group in today's announcement". The 30 October date for the announcement comes from secondary coverage, and is stated here as reported.

The other companies' positions could not all be read directly, and this article does not pretend otherwise. As reported in consistent secondary coverage: Anthropic has already started implementing support for the draft, with no completion date given, per the draft author's message to the IETF mailing list; Microsoft honours the new preferences as it crawls from 1 February 2027, and extends the shared training preference to page level with a meta tag or HTTP header; Google aims to support the new control in 2027, and plans to interpret use of the shared token without an explicit mention of Google-Extended as disallowing Google-Extended. Microsoft's Bing blog post was blocked for direct reading and Google's post was not read directly, so every Microsoft and Google detail in this article is attributed as reported, not primary-verified. Meta expressed support but has not committed to implementing, according to a single secondary source — so Meta is not listed here as a supporter.

Illustration generated for INORYVA — not a real person, product, or place.

Microsoft's extra: a separate grounding control

Microsoft's reported plan goes one step further than the shared token, and the distinction matters. Alongside the training preference, Microsoft is reported to be adding a separate page-level AI Grounding preference — Allow, Disallow, or DisallowExtended — where DisallowExtended limits grounding to Microsoft's own experiences and excludes third-party grounding services. Grounding is the practice of an AI system quoting or drawing on retrieved pages when answering; it is not training. Both Microsoft preferences default to Allow, and neither changes eligibility for traditional Bing search results. As reported, NoArchive and NoCache tags stop being read as training and grounding signals after 1 March 2027, with temporary transition preferences opening in Bing Webmaster Tools on 15 December 2026 and expiring 30 September 2027. All of this is secondary-reported, not read on Microsoft's own blog, and a publisher relying on it should confirm it in Bing's own documentation before acting.

Six caveats that must stay attached

First, voluntary. The token binds only companies that choose to follow it. Second, not a standard. The document is an individual Internet-Draft — a working document, valid for a maximum of six months, "inappropriate to use as reference material or to cite other than as 'work in progress'" — with no IETF endorsement and no formal standing, expiring 3 May 2027 unless revised or adopted. Third, training-only. It does not block crawling, search indexing, or AI answers that quote pages. Fourth, differing timelines. Anthropic is implementing already with no date, OpenAI says "coming months", Microsoft starts 1 February 2027, Google says 2027 — nothing is live everywhere today. Fifth, no retroactive effect. Preferences apply as in the robots.txt in effect when the asset was retrieved; the token does not reach back to content already collected or already trained on. Sixth, the wildcard does not count: User-Agent: * expresses no training preference.

The practical advice is narrow. If you run a site, add the token group alongside — not instead of — your existing per-crawler rules, because the old controls remain respected and the new one is not yet live everywhere. And if you are a reader rather than a publisher, the honest summary is this: the industry has agreed on a shared way to say no. It has not yet agreed on when, or in every case whether, no means no.

Illustration generated for INORYVA — not a real person, product, or place.

Key Takeaways

• One shared token, __usage_ai-training, replaces the old list of per-company opt-outs (GPTBot, ClaudeBot, Google-Extended, NoArchive/NoCache) — one robots.txt group reaches every participating provider at once.
• The mechanism is defined in an IETF Internet-Draft dated 30 October 2026, expiring 3 May 2027: an ordinary robots.txt group, no new grammar, and explicitly not a crawling directive. User-Agent: * does not express a preference.
• OpenAI will honour it and aims to implement it "in the coming months"; the old controls stay respected. Anthropic has started implementing (no date, as reported); Microsoft from 1 February 2027 and Google in 2027 (both as reported, not primary-verified).
• Microsoft's reported extra is a separate page-level grounding control (Allow / Disallow / DisallowExtended), defaulting to Allow — grounding is not training, and neither affects Bing search eligibility.
• Six caveats: voluntary, not a standard, training-only, differing timelines, no retroactive effect, wildcard does not count. Keep existing opt-outs in place; add the token alongside them.

Conclusion

A shared opt-out beats four separate ones, and a voluntary convention beats none. But a draft that expires in six months, honoured on four different timelines, with no effect on content already collected, is a beginning, not a settlement. Publishers should use it — alongside the old rules, not instead of them — and watch what each company actually ships. The test of this token will not be the announcement. It will be the first site that checks whether its preference was honoured, and finds out.

Sources

• IETF Internet-Draft draft-kelley-aipref-training-preference-signal-00, "Machine-Readable Training Preference Signal" (K. Kelley, Anthropic; 30 Oct 2026; expires 3 May 2027; token definition, wildcard rule, training definition, draft status): https://datatracker.ietf.org/doc/draft-kelley-aipref-training-preference-signal/
• OpenAI, "Making it easier for websites to express AI training preferences" (voluntary effort; "coming months"; existing controls remain respected): https://openai.com/index/helping-websites-express-ai-training-preferences/
• Microsoft/Bing, "Your content, your choice: clearer preferences for training and grounding" (30 Oct 2026; 1 Feb 2027 start, page-level grounding control, NoArchive/NoCache end 1 Mar 2027 — as reported in secondary coverage by Search Engine Land, PPC Land and others; Bing blog blocked for direct reading; no URL verified for this article)
• Google, "Supporting a shared AI training control for websites" (30 Oct 2026; 2027 support, Google-Extended interpretation — as reported in secondary coverage by PPC Land, Unite.ai and others; post not read directly; no URL verified for this article)