Robots.txt AI Training Opt-Out: Google, Microsoft, OpenAI and Anthropic Back One Shared Signal

TL;DR: Google, Microsoft, OpenAI and Anthropic have backed a single robots.txt AI training opt-out. Instead of blocking each company’s crawler by name, a site can add a group for the shared policy token __usage_ai-training and say “Disallow” to refuse use of its pages for generative AI training. It’s still an individual IETF draft, nobody reads it yet, and start dates vary: Microsoft from February 1, 2027, Google in early 2027, OpenAI “in the coming months,” and Anthropic says it has started implementing. Important catch: a User-agent: * block does not count.

Until now, a robots.txt AI training opt-out meant keeping a growing list of rules for Google-Extended, GPTBot, ClaudeBot and many more. On October 30, 2026, four of the biggest AI developers said they will honor one shared instruction instead. Coverage from The Verge and a detailed breakdown by PPC Land followed through the weekend. Here’s how it works and what site owners should do.

What was announced

Each company published its own statement on the same day:

• Google said on its Search Central blog that it plans to support “a shared AI training control in robots.txt,” based on work from the IETF’s AI Preferences (AIPREF) working group.
• OpenAI said in a policy post that it “will honor a common signal in robots.txt” and aims “to implement the signal in the coming months.”
• Microsoft published a Bing blog post saying it will honor new training and grounding preferences from February 1, 2027, according to PPC Land and The Verge.
• Anthropic’s Kevin Kelley wrote the underlying draft and told the AIPREF mailing list that “Anthropic has already started implementing support for this draft.”

How the shared token works

The mechanism is defined in the IETF Internet-Draft draft-kelley-aipref-training-preference-signal-00. It adds no new directives. You write a normal robots.txt group, but the User-Agent line names a “policy token” instead of a crawler. The draft’s own example:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

According to the draft, that expresses a preference against AI training for the whole site except /press-kit/, where it expresses a preference in favor. Path matching follows the existing Robots Exclusion Protocol, RFC 9309.

The three rules most site owners will miss

1. Wildcards don’t count. The draft says groups for User-Agent: * “MUST NOT be interpreted as matching the policy token.” If the token never appears in your file, “no preference has been expressed.” A site that blocks every bot today has, under this draft, said nothing about AI training.
2. It’s about use, not crawling. The draft states the token “is not a directive for crawling behavior.” To stop a crawler fetching pages, you still need rules for that crawler’s own name. If you want a refresher on that difference, our explainer on what Google says robots.txt actually controls covers it.
3. Private deals win. The draft says private agreements may override preferences expressed this way, so a publisher with a licensing deal can still have that contract govern its relationship with a given AI company.

How each company will treat it

The Google catch: training and Gemini grounding are linked

Google’s blog says that when a site uses the shared token to opt out “(and Google-Extended is not explicitly specified), Google plans to interpret that as if the publisher had disallowed Google-Extended.” Google-Extended covers both Gemini training and grounding, so PPC Land’s reading is that a shared-token opt-out could also remove a site from grounded answers in Gemini Apps. Google itself notes that “websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps.” If you want to stay visible in Gemini while refusing training elsewhere, you would add an explicit Google-Extended allow group. Google Search rankings are not affected either way, the company says.

Microsoft splits training from grounding

Microsoft goes further, with a page-level training setting (for example <meta name="Bingbot" content="ai-training:disallow">) and a separate grounding preference with three values, according to PPC Land’s summary of the Bing post. One of them, DisallowExtended, keeps pages in Microsoft’s own AI answers but out of the grounding services it sells to third parties. Microsoft also warns that neither choice guarantees inclusion, traffic or payment.

What it doesn’t do

• It isn’t a finished standard. The IETF datatracker says the draft “is not endorsed by the IETF and has no formal standing.” It expires on May 3, 2027 unless revised.
• It isn’t retroactive. The draft ties preferences to the robots.txt file in effect when content was retrieved. It does not address content already collected or models already trained.
• It isn’t universal. Meta, Amazon and Apple are not among the four companies that committed, and the draft has no enforcement mechanism. OpenAI calls its participation a “voluntary effort.”
• It doesn’t cover answer-time retrieval like RAG, since that doesn’t change a model’s parameters. That’s why Microsoft built a separate grounding control.

What site owners should do now

1. Decide your policy first. Do you want to refuse training, stay cited in AI answers, or both? The answer differs by company.
2. Keep your existing rules. All four companies say current crawler controls continue to work. Don’t delete GPTBot or Google-Extended lines yet.
3. Add the shared group if you want a broad opt-out. A simple version is User-Agent: __usage_ai-training followed by Disallow: /. Add an explicit Google-Extended allow if you want Gemini grounding.
4. Check Microsoft’s calendar if you rely on NoArchive or NoCache, since those stop working as AI preferences after March 1, 2027.
5. Audit after any CDN or plugin change. Tools that auto-generate robots.txt can overwrite your rules.

The fight over training data is also playing out in court. See our coverage of Canadian publishers suing Anthropic and OpenAI’s change to live-internet training.

Bottom line

A shared robots.txt AI training opt-out is a real step toward simpler controls for publishers: one line instead of a dozen. But it’s a draft, it won’t be read before 2027 at Google and Microsoft, wildcard blocks don’t count, and Google ties it to Gemini grounding. Add the token if it fits your policy, keep your existing crawler rules, and watch for implementation updates from each company.

About Teck Hustlers Editorial Team

The Teck Hustlers Editorial Team covers technology, business, finance and gaming news for readers who want the story behind the headline. Every article is researched, written and edited by our in-house team, checked against company announcements, filings and reputable reporting, and updated when new facts emerge. We focus on what each development means for founders, professionals, investors and everyday users. Have a tip or a correction? Reach us through our contact page.