🔍 Read the full analysis: The Coverage And Limits Of The Shared AI Training Opt-Out on ThorstenMeyerAI.com
Get business pricing on monitors, keyboards and dev gear
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
OpenAI, Google, Microsoft and Anthropic have said they will support a shared robots.txt signal for publishers who do not want their content used to train generative AI models. The signal is voluntary, adoption dates differ, and it does not block crawling, remove content from existing models or cover every use of web content in AI answers.
OpenAI, Google, Microsoft and Anthropic said on October 30 that they will honor a shared robots.txt signal asking companies not to use a website’s content for generative AI model training. The proposal gives publishers one rule intended to replace multiple company-specific instructions, but it remains voluntary and does not block other uses of a site’s content.
The signal appears in an Internet-Draft titled “Machine-Readable Training Preference Signal.” Its proposed robots.txt entry is a user-agent group for __usage_ai-training followed by Disallow: /. A publisher adding it would express a preference against training use to companies that choose to recognize the signal. The draft was published on October 30 by Kevin Kelley of Anthropic.
Companies have given different implementation timelines. Anthropic says it has already started implementing support, though its crawler documentation still described company-specific controls as of October 31. OpenAI aims to implement the signal “in the coming months.” Google says it aims to support the control in 2027, with its crawler documentation specifying “early 2027.” Microsoft says it will honor it from February 1, 2027, as it crawls websites.
Microsoft also announced a separate page-level preference for AI Grounding—the use of web content to inform answers in Bing, Copilot and Microsoft 365. That is distinct from the shared training preference. Microsoft says both new preferences take effect February 1, 2027, and that its existing NoArchive and NoCache tags will stop counting as preferences for training or grounding after March 1, 2027. The company says the new choices do not determine inclusion in search results, traffic or payment.
Infographic 01 · Who backs it, and when
One line in robots.txt, four companies, four timelines.
A shared signal against AI training. Only Anthropic says it has started; the others name dates or months. The standard itself is still a draft.
- 30 Oct 2026Draft published, four companies announce supportdoneIndividual Internet-Draft by Kevin Kelley (Anthropic)
- 30 Oct 2026Anthropic: “has already started implementing support”startedIts standards lead on the IETF list: “It’ll take us some time to add support across our services.”
- Coming monthsOpenAI aims to implement itplanExisting GPTBot and OAI-SearchBot rules keep applying
- 16 + 19 Nov 2026Working group meets at IETF 127, San FranciscoopenOn 21 Oct a chair said no consensus has been declared yet
- 1 Feb 2027Microsoft honors the new preferences as it crawlsplanTraining in robots.txt or per page; grounding per page
- 1 Mar 2027Bing stops reading NoArchive / NoCache as AI preferencesdeadlinePublish the new settings before this date
- 2027Google plans support; its docs say “early 2027”planA “no” maps to Google-Extended unless you set that token yourself
- 3 May 2027Draft version 00 expires unless updatedInternet-Drafts are valid for at most six months
Binding? No, voluntary
- OpenAI calls it a “voluntary effort”.
- The vocabulary draft does not “ensure that preferences are followed”.
- The draft “has no formal standing in the IETF standards process”.
- Private agreements may override it, the draft says.
Who else?
- Meta supports a common signal, but did not say whether it will honor it.
- Apple, Amazon, xAI, Perplexity: no statement in the sources reviewed for this article.
- The draft’s acknowledgments list contributors “for identification only”, not supporters.
Sources: IETF draft, 30 Oct 2026 · Anthropic on the IETF list · AIPREF minutes, 21 Oct 2026 · Google, 30 Oct 2026 · Google crawler docs · OpenAI, 30 Oct 2026 · Microsoft Bing, Oct 2026 · Meta (Andy Stone on X). Company plans are company statements.
Powered by Thorsten Meyer AI
One Rule, Different AI Uses
The proposal could reduce the work required for publishers to communicate a training preference. Until now, site operators have needed to account for separate crawler names and controls, including GPTBot for OpenAI, Google-Extended for Google and ClaudeBot for Anthropic. A shared signal offers a common instruction, but only for companies that adopt it and for the use it names.
That distinction matters because training is not the same as retrieval or AI answers. A site can ask that its content not be used to train models without necessarily preventing a service from crawling pages or using current web content to answer a question. Microsoft’s separate grounding control makes this split explicit. Publishers will need to decide independently what they want to permit for training, AI-generated answers and search, rather than assuming one opt-out controls all three.
The signal also does not create a payment arrangement or guarantee that a publisher’s content will be excluded from systems that have already been trained. It communicates a preference; it is not a technical lock. That makes the announcement relevant to publishers seeking clearer controls, while leaving unresolved how much practical control the rule will provide when companies or crawlers do not follow it.
Products Worth Considering
Bose QuietComfort True Wireless Earbuds with Active Noise Cancellation, Black
Amazon Kindle Scribe Colorsoft, 64GB, 11-Inch Color Display with Premium Pen
DJI Mic 3 Wireless Microphone System (2 Transmitters, 1 Receiver, Charging Case)
As an affiliate, we earn on qualifying purchases.
The proposal is connected to the AI Preferences working group within the Internet Engineering Task Force, which develops internet standards. But this document is an individual Internet-Draft, not an adopted standard. The source material reports that the draft has no formal standing in the IETF standards process; at a working-group meeting on October 21, a chair said the group had not declared consensus on anything.
The companies’ announcements refer to the same draft, but their commitments are not identical in timing or scope. Google says that if a site uses the new signal and has no Google-Extended rule, it plans to interpret the signal as if the publisher had disallowed Google-Extended. Google also says Google-Extended does not affect Google Search. OpenAI says its existing crawler controls will continue to apply. Microsoft’s announcement adds a distinct control for grounding, while Anthropic says implementation across its services will take time.
Meta has not made the same implementation commitment. Communications chief Andy Stone said the company strongly supports publishers’ ability to express preferences and called a common signal an important step. That statement expresses support for the idea, but does not establish that Meta will honor the rule. The source account also found no statement on adoption from Apple, Amazon, xAI or Perplexity.
““has already started implementing support for this draft””
— Anthropic
Adoption and Enforcement Remain Open
It is not yet clear how widely the signal will be honored beyond the four companies that announced plans to support it. Meta’s statement was not an implementation commitment, and the source material reports no adoption announcement from several other AI companies. The rule is voluntary, so adding it cannot compel a company to comply.
The final scope and status of the proposal may also change. It is currently an Internet-Draft rather than an adopted IETF standard, and the working group has not declared consensus. Anthropic’s implementation remains in progress, while Google and OpenAI have described future targets rather than completed rollouts. The available announcements do not establish how each service will verify or handle a publisher’s preference across all products.
Publishers should also avoid treating the rule as a general ban on AI use. The shared signal concerns training; it does not, by itself, block crawling, govern live-web answers or remove material from models trained earlier. Microsoft has announced its own separate grounding preference, but other companies’ approaches to those uses are not specified here.
Watch the Rollout Through 2027
The next developments are whether the companies publish updated crawler documentation and put their stated plans into operation. Anthropic says implementation has begun; OpenAI has set a target of the coming months, while Google is aiming for 2027 and identifies early 2027 in its documentation. Microsoft’s stated start date is February 1, 2027.
Microsoft says publishers using NoArchive or NoCache should plan around March 1, 2027, when those tags will stop counting as its training or grounding preferences. Site operators can add the shared training rule if they want to express that preference, but the source recommends keeping existing company-specific rules in place until each provider has switched. Publishers will also need to decide separately whether to use Microsoft’s grounding control.
Further signs of progress will include any revision or formal consideration of the Internet-Draft by the IETF working group, and any clear commitment from Meta or other AI providers. Until then, the practical reach of the shared opt-out will depend on which services adopt it, when they do so and how they apply the distinction between training and other uses.
Key Questions
What is the shared AI training opt-out?
It is a proposed robots.txt signal, using the user-agent name __usage_ai-training and a Disallow rule, that tells participating companies a publisher does not want its content used for generative AI training.
Which companies have said they will honor it?
OpenAI, Google, Microsoft and Anthropic have announced plans to support the signal, with different timelines. Meta expressed support for publishers’ ability to state preferences but has not said it will honor this specific signal.
Does the signal stop AI companies from crawling a website?
No. The shared rule communicates a preference about training. It does not itself block crawling, and it does not automatically govern the use of live web pages in AI answers.
Does it remove a site’s content from models already trained?
No such removal is promised. The signal addresses a preference about training use and does not remove content from models that have already been trained.
Should publishers remove their existing crawler rules?
The source recommends keeping company-specific rules until each provider has implemented the shared signal. Microsoft has also set dates for its separate treatment of NoArchive and NoCache: those tags stop counting as training or grounding preferences after March 1, 2027.
Source: ThorstenMeyerAI.com
