This article’s text was created with the help of AI systems and reviewed by the WIT ICT team, in accordance with Article 50 of the EU AI Act (Regulation (EU) 2024/1689) on the transparency of AI-generated content. The featured image was also generated using AI.
On 30 October, Microsoft announced that Bing will treat two uses of web content separately: training AI models, and using pages to build AI-generated answers, known as grounding. On the same day, OpenAI, Google and Microsoft said they would support a shared robots.txt signal that lets sites opt out of AI training. For a small or mid-sized business running a company website, an online shop or an industry blog, this turns a vague worry into a practical decision. Should your pages feed the models, appear in Copilot’s answers, both, or neither? The deadlines are a few months away, and the choice affects how well your content is protected and how visible you are.
Training versus grounding: what each one means
In Microsoft’s definition, training changes the learned parameters of a generative model. Your text becomes a small part of what the model “knows”, usually with no link back to you. Grounding happens when someone asks a question: the AI reads your page as a source for its answer, and often cites it.
For a business, those are very different deals. A company that publishes detailed technical guides or carefully written product pages may not want to donate that work to model training. The same company probably does want to be the cited source when a prospect asks Copilot for a reliable local IT partner. Until now, the tools available (the NoArchive and NoCache tags) couldn’t separate the two. Microsoft also states that indexing and ranking for traditional search results are not considered grounding.
Inside Bing’s new controls
There are two preferences, and they work at different levels:
- AI Training can be set site-wide or for specific folders in robots.txt, or on individual pages through a meta tag or HTTP header. A page-level setting overrides robots.txt.
- AI Grounding is page-level only and has three values: Allow (the default), Disallow, and DisallowExtended. DisallowExtended keeps grounding within Microsoft’s own products (Copilot, Bing’s AI answers, Microsoft 365) and excludes third-party grounding services such as API access through Foundry.
Note the defaults. If you publish nothing, both preferences count as Allow, so doing nothing means agreeing to everything. Microsoft also says these settings govern how content is used, not whether it’s crawled. Setting Allow won’t unblock a Bingbot you’ve already disallowed, and neither setting creates any licensing or payment arrangement.
One robots.txt line for several AI providers
The wider news is the industry-level signal. Opting out of training will use a conventional user agent, __usage_ai-training, which several providers have agreed to read. Standardisation is under discussion in the IETF’s AIPREF working group and isn’t final yet. The providers have announced different timelines:
- Microsoft will honour it from 1 February 2027;
- OpenAI aims to implement it within the next few months;
- Google expects to support it during 2027, treating it as a Google-Extended opt-out for sites that haven’t configured that token themselves.
The minimum rule to keep a whole site out of training is:
User-Agent: __usage_ai-training
Disallow: /
This states a preference. Providers that have signed up commit to respecting it, but nothing forces others to follow suit.
Key dates for your calendar
The timeline is short:
- 15 December 2026: Bing Webmaster Tools opens temporary grounding preferences for sites that currently rely on NoArchive or NoCache.
- 1 February 2027: the new preferences take effect as pages are crawled. Once Bing finds the new directives on a site, it ignores the old tags there.
- 1 March 2027: deadline to publish grounding preferences or record temporary ones. After that, NoArchive and NoCache are no longer read as AI preferences.
- 30 September 2027: temporary preferences expire, or earlier if a page is recrawled with its new directive.
If your site uses NoArchive to keep AI away and you do nothing before March, your pages will be treated as Allow for grounding.
Protecting content or being found: a business decision
No single setting suits everyone. An accountancy firm or consultancy that relies on new enquiries will probably want to be cited in Copilot’s answers, because there grounding works like a shop window. A training provider or publisher whose revenue depends on its material may see those texts as assets worth keeping out of training, at the very least.
Microsoft says plainly that Disallow and DisallowExtended reduce brand visibility in AI experiences. Allowing everything, though, guarantees neither traffic nor citations. For many smaller companies a middle path makes sense: opt out of training, keep grounding open on commercial and service pages, and be more careful with high-value sections such as member areas or course content. Whoever owns marketing should be involved in this call, not just whoever maintains the server.
Applying the settings on a WordPress site
On WordPress this is very doable, but it needs care. Your robots.txt may be a physical file in the site root, or a virtual one generated by WordPress or by an SEO plugin such as Rank Math, which lets you edit it from the dashboard. Find out which one is actually being served before you change anything, or your edits may never go live.
Page-level directives need either a meta tag in the page head, for example <meta name="Bingbot" content="ai-grounding:disallow">, or an X-Robots-Tag header set at server level. On larger sites, apply them by post type or category through a child theme or a small code snippet rather than page by page. Once they’re published, IndexNow can notify Bing of the changes and make a recrawl more likely, although it won’t guarantee when it happens.
A practical plan for the coming weeks
You can break the work into a few steps and avoid a last-minute rush:
- check whether the site already uses NoArchive, NoCache, or blocks AI crawlers such as GPTBot or Google-Extended;
- sort your sections into three groups: promote, protect, and neutral;
- choose training and grounding preferences for each group and write the decision down;
- make sure you can log in to Bing Webmaster Tools, which many businesses neglect in favour of Google Search Console;
- publish the directives, test them, and recheck after any major theme or plugin update.
These rules don’t settle the question of what web content is worth to AI companies. They do give site owners a readable way to tell AI systems what they may do with their pages. If you set a policy that matches your business goals, you’ll be in a better position than competitors who stay on the defaults without realising it.
Want to know more? Contact us for a free consultation.