A shared robots.txt signal for AI training: what changed and what remains pending

The announcement of a shared signal for AI training

On October 30, 2026, OpenAI, Google, Microsoft, and Anthropic announced that they will respect a common signal in the robots.txt file. This initiative allows website operators to communicate to multiple providers, via a single rule, their preferences regarding whether their content can be used to train generative AI models.

Until now, editors had to maintain separate exclusion lists for each specific crawler, such as OpenAI's GPTBot or Google-Extended. The new signal aims to reduce this operational complexity by offering a unified policy token that participating providers will interpret similarly, simplifying the management of data usage permissions.

Technical mechanics: the __usage_ai-training token

The mechanism is based on an individual Internet draft titled 'Machine-Readable Training Preference Signal,' proposed by Kevin Kelley of Anthropic to the Internet Engineering Task Force (IETF). The proposal does not modify the existing syntax of robots.txt nor introduce new directives; instead, it uses ordinary user-agent groups.

The specific token defined for this purpose is __usage_ai-training. By associating this identifier with Allow or Disallow directives, the operator expresses a usage preference. Parsers that do not recognize this token will treat it as an unknown agent name, allowing existing systems to continue functioning without changes while the new functionality is integrated.

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

Provider participation and implementation timelines

Adoption of the standard varies by company. Anthropic has confirmed that it has already implemented support for this token. OpenAI plans to incorporate it in the coming months, although it has not specified a concrete date. Google and Microsoft plan to accept the signal in early 2027.

Google has indicated that if a site uses the policy token to opt out of training and does not explicitly mention Google-Extended, it will interpret this as if the editor had disabled that specific crawler. Microsoft, for its part, will offer additional controls allowing refusal of training or 'grounding' independently.

Standard limits and scope of preferences

It is fundamental to distinguish between crawl control and usage control. The __usage_ai-training signal expresses a preference regarding model training, but it does not by itself determine whether a crawler can visit a page for other purposes, such as search or real-time inference.

The IETF draft clarifies that private agreements may prevail over preferences expressed in robots.txt. Furthermore, the technical document underscores that this signal does not replace existing crawl controls. An administrator can, for example, block training via the common signal but allow access for generative search if they do not block specific search crawlers.

Considerations for editors and web operators

Editors wishing to use this signal must understand that its effectiveness depends on adoption by providers. Since the standard remains a draft at the IETF, it does not have the formal backing of the organization nor the status of an approved standard.

Operators are recommended to combine this signal with other protective measures, such as WAF rules or bot management, as content signals are preferences and not technical countermeasures against content appropriation. Additionally, it is important to verify the specific documentation of each provider to understand how they interpret the signal in relation to their own crawlers.

Keep exploring

Corrections and transparency

If you spot incorrect or outdated information, send us a correction with the page and source we should review.