AI Training Opt-Out: One robots.txt Rule for Google, OpenAI and Microsoft
A new shared robots.txt token lets websites ask Google, OpenAI and Microsoft not to train AI on their content. Here is the exact rule, when each company will honour it and what it means for SEO.
Image: TeQBlogs
A new AI training opt-out lets website owners tell several AI companies, through one robots.txt rule, not to train AI models on their content. Google, OpenAI and Microsoft announced on 30 October 2026 that they will honour the shared signal, which comes from an IETF draft written at Anthropic. It is not fully live yet: Microsoft starts on 1 February 2027, Google aims for 2027 and OpenAI says “in the coming months”.
Quick summary
- The rule: a robots.txt group for the policy token
__usage_ai-training, for exampleUser-Agent: __usage_ai-trainingfollowed byDisallow: /. - What it controls: whether your content is used to train generative AI models. It does not block crawling or remove you from Google or Bing search results.
- Who supports it: Google, OpenAI and Microsoft have said they will honour it; Anthropic reportedly started implementing it first.
- Old rules still work: GPTBot, Google-Extended and other company-specific rules keep working alongside the new token.
- Bing extra: Microsoft also adds a separate page-level “AI Grounding” preference for answers in Copilot and Bing’s AI features.
What is the new AI training opt-out?
It is a shared “policy token” for robots.txt that expresses one preference, AI training, to every AI provider that supports it. Before this, site owners had to block or allow each company’s crawler separately, such as GPTBot for OpenAI or Google-Extended for Google.
Also read: 9 Useful Tools For SEO-Content Strategies to Grow Your Business
The mechanism is described in an individual IETF Internet-Draft, “draft-kelley-aipref-training-preference-signal”, published on 30 October 2026. It adds no new robots.txt commands. You simply write a normal group with __usage_ai-training in the User-Agent line, and the usual Allow and Disallow path rules apply. As an individual draft, it has no formal IETF standing yet and expires in May 2027 unless it is updated.
One detail catches people out: a wildcard group (User-Agent: *) does not count. The draft says wildcard groups must not be read as matching the token, so you have to name __usage_ai-training explicitly.
When will Google, OpenAI and Microsoft honour it?
None of the three has switched it on fully yet, so treat it as a preference you set now that takes effect over the coming months.
| Company | What it said (30 Oct 2026) | Timeline | Existing control |
|---|---|---|---|
| An opt-out with the token, without a Google-Extended rule, will be treated as disallowing Google-Extended | Aims to support it in 2027 | Google-Extended | |
| OpenAI | Will honour the common signal for foundational generative AI training | “In the coming months” | GPTBot and other crawler controls |
| Microsoft | Will honour AI Training in robots.txt and adds page-level AI Training and AI Grounding preferences | From 1 February 2027, as Bingbot recrawls | NoArchive / NoCache (being replaced) |
| Anthropic | The IETF draft was written at Anthropic | Reportedly already implementing | ClaudeBot rules |
How to add the AI training opt-out to robots.txt
Adding the rule takes two lines in your site’s robots.txt file. Here is how to do it on any site, with WordPress steps in brackets.
- Open your robots.txt file. It lives at yourdomain.com/robots.txt. (In WordPress with Yoast SEO, go to Yoast SEO > Tools > File editor.)
- Add a new group at the end, leaving your existing Googlebot, Bingbot and
*rules untouched. - To opt the whole site out of AI training, add:
User-Agent: __usage_ai-trainingDisallow: / - To opt out only one section, use a path instead, such as
Disallow: /premium/. - To opt out but allow one folder, add an Allow line, for example
Allow: /press-kit/. - Keep your company-specific rules (GPTBot, Google-Extended and others) until each provider confirms its support is live.
- Save and test. Open yourdomain.com/robots.txt in a browser and check the robots.txt report in Google Search Console for errors.
Remember that robots.txt is a request, not a lock. Responsible providers say they will follow it, but scrapers that ignore robots.txt need firewall or bot-management controls instead.
Does opting out of AI training hurt SEO?
No, not for normal search results. Microsoft says the new preferences do not change eligibility for Bing search results, and the token is separate from the Googlebot and Bingbot rules that control crawling and indexing.
AI answers are a different matter. Google notes that sites which allow Google-Extended can appear in responses and receive clicks via grounding in Gemini Apps. Microsoft warns that blocking grounding will significantly reduce visibility in AI-generated experiences. If AI citations matter to your traffic, think carefully before blocking more than training. Our digital marketing checklist for 2026 covers how AI answers fit into a wider search plan.
What is Microsoft’s AI Grounding preference?
AI Grounding is a separate, page-level setting from Microsoft that controls whether your pages can be used to inform AI-generated answers, as opposed to training models. It has three values: Allow (the default), DisallowExtended and Disallow.
- Allow: pages can inform AI answers. No action is needed.
- DisallowExtended: pages can inform answers only in Microsoft-operated experiences such as Copilot and Microsoft 365, not in grounding services Microsoft sells to third parties.
- Disallow: pages are not used to inform AI answers.
You set it with a meta tag such as <meta name="Bingbot" content="ai-grounding:disallow"> or the matching X-Robots-Tag header. There is no site-wide robots.txt version of the grounding setting.
If your site uses NoArchive or NoCache today, note the dates. A temporary grounding setting opens in Bing Webmaster Tools on 15 December 2026, the new preferences take effect from 1 February 2027, and after 1 March 2027 Microsoft will stop reading NoArchive and NoCache as training or grounding preferences.
Should Indian businesses and bloggers opt out?
It depends on what your content earns you. Use this as a starting point.
| Type of site | Suggested training setting | Why |
|---|---|---|
| Business or service website | Usually allow | Being known to AI assistants can help people find your brand |
| Blog that wants AI citations | Allow, or opt out of training only | Training and grounding are separate, so citations can continue |
| Paid courses, research or premium content | Disallow for those paths | Protects content that is your product |
| News or original reporting | Decide by policy; many publishers restrict training | Original work is what AI models most want |
If you are planning a wider search and AI-visibility strategy, Suprams’ SEO services page explains how these technical settings fit with content and rankings. For the basics of a site health check, read our post on why an SEO audit matters.
Frequently asked questions
What is __usage_ai-training in robots.txt?
It is a shared policy token that lets you state an AI training preference in robots.txt. A group with Disallow: / under it asks participating providers not to train on your site.
Does User-Agent: * block AI training?
No. The IETF draft says wildcard groups must not be treated as matching the training token, so you need a separate group naming __usage_ai-training.
Will opting out remove my site from Google or Bing search?
No. The token covers AI training, not crawling or indexing for search. Your Googlebot and Bingbot rules still control normal search.
Do I still need GPTBot and Google-Extended rules?
Keep them for now. Both companies say their existing controls continue to work, and neither has switched on the shared token yet.
Bottom line
The shared AI training opt-out turns a messy list of crawler rules into one clear preference that Google, OpenAI and Microsoft have promised to respect. Add it now if you want to restrict training, keep your existing bot rules until support goes live in 2027, and decide separately whether you want to appear in AI answers. For a related look at how platforms are changing access to content, see our report on Reddit’s login requirement.
Sources
- IETF Datatracker: draft-kelley-aipref-training-preference-signal
- Google Search Central Blog: Supporting a shared AI training control for websites (30 October 2026)
- OpenAI: Making it easier for websites to express AI training preferences (30 October 2026)
- Microsoft Bing Blogs: Your content, your choice (30 October 2026)
About The Author
