The biggest names in AI are backing a new way for publishers to opt out of AI scraping. OpenAI, Google, Microsoft and Anthropic say they will adopt a shared “policy token” signal that lets publishers apply their AI training preferences across multiple AI companies at once — instead of setting them individually for each crawler.
The shared policy token was proposed by Kevin Kelley, a technical lead for standards at Anthropic, as part of the Internet Engineering Task Force (IETF). It lives inside a website’s robots.txt file, which has long told crawlers where they can and can’t go.
(Credits: The Verge)
AI Scraping Opt-Out: How the Policy Token Works
Currently, publishers must signal their AI training preferences separately to every AI company. The new standard changes that: all four companies say they will restrict their AI crawlers from accessing a site if they encounter the shared token.
Anthropic has already adopted the signal. Google and Microsoft plan to support it next year, while OpenAI says it will implement it in the “coming months.” Microsoft will go further with granular controls — letting publishers opt out of AI training or AI grounding (pointing a model to a verified source to reduce hallucinations) separately. Microsoft notes sites will still appear in Bing search results regardless of their preferences.
Why Publishers Want AI Crawler Controls
The announcement lands as publishers grapple with a flood of AI crawlers scraping their content to feed AI models. Others have fought back differently: the Really Simple Licensing (RSL) standard lets publishers charge crawlers, while Cloudflare now blocks AI crawlers by default.
A single universal signal could finally make “do not train” as easy to declare as “do not index” — a small line in robots.txt that every major lab respects.
Also Read: Wikipedia Tells Court: A Bad AI Copyright Ruling Could Threaten the Open Web (NYT vs OpenAI)
Also Read: Elvis Presley’s Music Owners Sue Anthropic for Copyright Infringement, Demand Nearly $50 Million
Source: The Verge AI — read the full story there.
