Google, Microsoft, Anthropic and OpenAI back new robots.txt standard to block AI scrapers
The standard does not block crawling but signals how content should be used, and Meta supports it without committing to implementation.
Rhea Lease · AI writer ·
0 primary · 1 independent · 4 commentary · 0 vendorImage courtesy of Fay Tograph · AI photographer, made with Grok image generation
The crawling standard that Google, Microsoft, Anthropic and OpenAI are backing would allow website operators to block all scrapers that collect training data for generative AI models with a single robots.txt entry, SiliconANGLE reported.[1]Because the major AI labs are aligning on this signal, publishers can now communicate training preferences across multiple companies at once, reducing administrative work.[1][2]
Each company is implementing the standard on its own timeline: Anthropic has already begun integration, OpenAI plans to implement it in the coming months, Microsoft will support it starting February 1, 2027, and Google aims for 2027 without specifying a month.[1][3][4][2]
The draft does not block crawling itself; it only signals how content should be used, meaning bots may still access pages, AI Weekly reported.[4][3]Google clarified that its existing crawler controls will remain in place alongside the new signal, and Microsoft says its new preferences will replace NoArchive and NoCache for training and grounding.[2]
Meta Platforms has expressed support for the idea but has not committed to implementation, SiliconANGLE reported.[1]
What people on X are saying, summarized by Grok
On X, some site owners welcomed one shared line as easier than blocking each lab's crawler, but the sharper reactions were skeptical. People called the signal voluntary and late, a sign rather than a lock, and said it would not undo past training, cover answers that quote a page, or stop scrapers that already ignore robots.txt. Others warned that opting out could reduce AI citations, and read Meta's statement as praise without a promise to comply.
Posts chosen and quoted by Grok
A few catches worth knowing:
- User-Agent: * doesn't count. The token needs its own group.
- It only covers training. A bot fetching your page to answer a prompt isn't covered. Microsoft added a separate "AI Grounding" preference for that.
- It's still an IETF draft, not a final standard.
- It's a signal, not a block. Scrapers that ignore robots.txt today will ignore this too.
OpenAI, Google, Microsoft and Anthropic will all respect one line in robots.txt that says don't train on this site. Anthropic already does. OpenAI in "the coming months", Google and Microsoft next year. Meta likes the idea and won't say if it'll follow it. Course it won't. So they've hoovered up the whole internet, and now they've agreed to start knocking. Thanks, lads. Bit late.
Written by Grok from posts on X, with post text as Grok returned it. Not reported or checked by us, and not checked against X. The story above does not rely on it.