OpenAI, Google, Microsoft and Anthropic to share AI opt-out token in robots.txt
Under the proposal, publishers place one policy token in robots.txt and each company's crawlers would respect it, the companies said.
OpenAI, Google, Microsoft and Anthropic said on October 30, 2026 that they will adopt a shared "policy token" signal in a website's robots.txt file that lets publishers opt out of AI scraping across multiple crawlers at once. The companies said they will restrict their AI crawlers from accessing a site when they encounter the token.
The signal is proposed by Anthropic standards technical lead Kevin Kelley as part of the Internet Engineering Task Force, according to theverge.com. It sits in the robots.txt file publishers already use to direct web crawlers and expresses AI training preferences to several crawlers at the same time.
Adoption timelines differ. Anthropic has already adopted the signal, Google and Microsoft plan to support it next year, and OpenAI said it will implement it in the coming months.
Microsoft said it will also offer extra granular controls that let publishers opt out of AI training or AI grounding separately.
Regardless of the preferences a site sets, Microsoft said websites will still appear in Bing search results.
Quick answers
What is the policy token in robots.txt?
It is a shared signal, proposed by Anthropic's Kevin Kelley within the Internet Engineering Task Force, that publishers place in robots.txt to express AI training preferences to multiple crawlers at once.
When will each company support the signal?
Anthropic has already adopted it, Google and Microsoft plan to support it next year, and OpenAI says it will implement it in the coming months.
Will opting out remove a site from Bing search results?
No. Microsoft said websites will still appear in Bing search results regardless of their preference settings.