The open web may soon get its first shared mechanism allowing website operators to tell several major AI companies at once whether their content may be used to train generative AI. OpenAI, Google, Microsoft and Anthropic are backing a new approach within robots.txt designed to replace today’s fragmented opt-out rules with a more unified system.
On 30 October 2026, Anthropic standards specialist Kevin Kelley submitted an Internet-Draft to the Internet Engineering Task Force proposing a new so-called policy token. Website operators could use it to indicate, for example, that content from their domain should not be used to train generative foundation models without having to maintain separate rules for every individual AI crawler. OpenAI has said it plans to support the shared signal in the coming months. Anthropic is already further along with implementation, while Google and Microsoft have also announced support.
At first glance, this looks like a small technical change to a file that most internet users will never see. In reality, it touches one of the central questions of the next phase of generative AI: who is allowed to use publicly accessible content for model training, and how can website owners express their preferences in a machine-readable way?
Until Now, Every AI Crawler Had to Be Managed Separately
The robots.txt file has existed for decades. Traditionally, it has been used to tell search engines and other automated crawlers which areas of a website they should or should not access.
The rise of generative AI made that simple model much more complicated.
OpenAI operates its own crawlers, as does Anthropic. Google uses different mechanisms for conventional search and AI-related uses. Microsoft, Meta, Perplexity and many others add further layers. Website owners who wanted to prevent certain content from being used for AI training therefore often had to identify individual bots and create separate rules for each of them.
The new approach is intended to reduce that complexity.
OpenAI describes the goal as a shared mechanism through which website operators can express the same training preference to several participating AI providers at once.
The New Entry Looks Surprisingly Simple
The proposed mechanism relies entirely on the existing robots.txt structure.
A website operator could, for example, use an entry such as:
User-Agent: __usage_ai-training
Disallow: /
This would express the preference that content from the website should not be used for generative AI training.
If a particular section should still be made available, that can also be defined:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
In this example, AI training would be disallowed across the site except for content under /press-kit/.
The proposal deliberately avoids introducing a completely new file format. Existing robots.txt path rules and priority mechanisms can continue to be used, which should make implementation easier for web servers, content management systems and existing parsers.
The Most Important Difference: Crawling and Training Are Separated
The technically most interesting point is also one of the easiest to miss.
The new policy token does not decide whether an AI crawler is technically allowed to visit a website.
It describes how content that has already been retrieved may be used.
That is a fundamental distinction.
A publisher may want OpenAI or Microsoft to continue accessing its website so that articles can appear in search or answer systems. At the same time, it may not want those same articles to be used as training material for future foundation models.
The new standard is designed to separate those two decisions.
The IETF proposal explicitly treats the signal as a preference governing content use rather than a mechanism for controlling network access. Existing crawler-specific rules therefore remain relevant alongside it.
robots.txt Is Taking on an Entirely New Role
Historically, robots.txt was primarily a crawling protocol.
A search engine could be told, for example, not to crawl or index a certain section of a website.
The new approach uses the same infrastructure for a different question:
Not just “May you access this page?” but also “What may you do with the content afterwards?”
That turns robots.txt from a purely technical crawling file into something closer to a machine-readable declaration of usage preferences.
The implications could be significant.
If the concept gains traction, further categories could eventually follow. Websites might one day express separate preferences for model training, AI search, retrieval, summarisation or other forms of automated processing.
For now, the draft is focused specifically on the ai-training use case.
It Is Not Yet a Final Internet Standard
Despite the high-profile companies involved, the current status needs to be understood carefully.
The proposal submitted on 30 October is still only an Internet-Draft.
Such drafts can be submitted by individuals and do not represent an approved IETF standard. The specification may still change substantially before any formal standardisation process is completed.
The large AI companies are also at different stages of implementation. OpenAI says it intends to support the signal in the coming months. Anthropic is already further along. Google and Microsoft have announced support, but complete implementation is expected to happen gradually.
A website owner who adds the new entry today should therefore not assume that every AI provider already respects it automatically.
The Mechanism Is Voluntary as Well
There is another important limitation.
robots.txt is not a technical access barrier.
A website can publish a preference there, but an external provider still has to choose to respect that preference.
The fact that OpenAI, Anthropic, Google and Microsoft support a shared mechanism makes it considerably more meaningful. But it does not create a universal prohibition that automatically applies to every crawler on the internet.
A company that does not participate in the standard could still technically retrieve content unless additional protections are in place.
The long-term value of the system therefore depends heavily on how widely it is adopted.
Meta Supports the Idea but Has Not Fully Committed
Meta’s position is particularly interesting.
Current reporting suggests that the company supports the general idea of a shared standard for expressing AI usage preferences. However, Meta has not yet made the same level of commitment to implementing this specific policy token across its own systems.
That already reveals the political dimension of technical standards.
A shared opt-out becomes more useful as more major model providers respect it. If important companies use separate rules or decline to support the common signal, publishers once again have to maintain multiple systems in parallel.
Google Already Has Its Own Controls
Google is not starting from scratch.
Website owners can already use Google-Extended to influence whether content may be used for certain generative AI systems without automatically removing that content from conventional Google Search.
The new shared policy token could eventually add a cross-industry layer on top of such company-specific mechanisms.
For website operators, that would be much easier to manage.
Instead of dealing separately with Google-Extended, GPTBot, ClaudeBot and other systems, a general training preference could potentially be defined once.
More specific rules could still remain alongside it.
Microsoft Wants to Treat Training and Grounding Separately
The distinction between training and so-called grounding is especially important.
Training means that content is used to influence a model’s parameters during development.
Grounding means that an already trained model retrieves current information from external sources in order to answer a specific user query.
Those two processes are frequently conflated in public debate.
A publisher may be perfectly comfortable saying: our articles must not be used to train the next foundation model, but they may be retrieved for current AI answers as long as the source is linked.
Microsoft has indicated that it wants to offer separate controls for such use cases.
This illustrates where the technical debate is heading: a simple binary choice between “AI allowed” and “AI forbidden” is unlikely to be sufficient in the long term.
For Publishers, This Is About More Than Copyright
The debate is often framed purely as a copyright issue.
For publishers, the economic dimension is at least as important.
Traditional search engines were part of a relatively clear exchange. They indexed content and sent users back to the original website through links.
Generative AI changes that model.
If an AI system summarises information directly and users no longer need to visit the original site, the economic value of producing that content may decline.
That raises several new questions.
Should an article be allowed to contribute to model training?
Can it be retrieved for a current AI-generated answer?
Must the original source be cited prominently?
Should the publisher be paid?
The proposed robots.txt mechanism answers only one part of that wider debate, but it could provide a shared technical foundation for it.
Corporate Websites Face the Same Strategic Question
This is not only relevant to media companies.
Businesses publish enormous volumes of valuable material on the open web.
Product documentation, technical articles, studies, knowledge bases, support pages and white papers can all potentially become training material.
Until now, many organisations have not made an explicit decision about whether they want that to happen.
A standardised AI-training signal could turn this into a normal part of web governance.
A company might decide, for example, that public marketing pages can be used for training, technical documentation should be excluded and press material can remain open.
The robots.txt file would suddenly become a strategic governance document.
Blocking Everything Could Have Downsides Too
A blanket opt-out is not automatically the best choice.
If AI assistants become an important gateway to the internet, companies may still want their content to appear in AI-generated answers.
Completely blocking all AI crawlers could eventually become similar to making a website invisible to search engines today.
That is why the separation between training and other uses matters so much.
A company may want to refuse model training while still allowing current content to be retrieved and cited in AI answers.
This is where traditional SEO begins to overlap with concepts such as AI visibility or Generative Engine Optimisation.
The question is no longer only how well a website ranks in Google.
It is also whether and how its content is discovered, understood and cited by AI assistants.
The New Standard Could Also Support Licensing Markets
Another possible consequence concerns content licensing.
If websites can express clearly and in machine-readable form that their material should not be used freely for training, that also creates a clearer basis for individual agreements.
A publisher could reject general training access while granting specific model providers licensed access in return for payment.
Such agreements already exist between AI companies and large media organisations.
A standardised opt-out could therefore indirectly accelerate the emergence of a commercial market for training data.
The ability to say no by default gives content owners a stronger position from which to negotiate a conditional yes.
The Core Conflict Is Shifting from Access to Usage Rights
This may be the most important development behind the proposal.
The early debate around AI crawlers focused heavily on whether bots should be allowed to access websites at all.
The next phase is more nuanced.
Content can be publicly accessible while still being subject to different usage preferences.
A person may read an article.
A search engine may index it.
An AI assistant may retrieve it for a current answer.
But that does not automatically mean the same article should be used to train the next generation of models.
The new approach is an attempt to separate these layers technically.
A Tiny Entry with Potentially Major Consequences
Technically, the proposal amounts to just a few lines in a robots.txt file.
Strategically, it could become much more significant.
For the first time, several of the most important Western AI companies are supporting a common mechanism through which website operators can express their training preferences.
It is not yet a binding internet standard. Not every provider supports it fully. And robots.txt remains a voluntarily respected preference rather than a technical copy-protection system.
Even so, the direction is changing.
Until now, publishers had to identify and manage new AI crawlers one by one.
In future, the logic could be reversed.
The website states once how its content may be used, and AI providers are expected to interpret that preference.
That gives an old and rather unglamorous file from the early web a completely new purpose.
robots.txt may no longer determine only who is allowed to visit a website.
It may increasingly determine what machines are allowed to do with the knowledge they find there.
