The crawling standard that Google, Microsoft, Anthropic and OpenAI are backing would allow website operators to block all scrapers that collect training data for generative AI models with a single robots.txt entry, SiliconANGLE reported.[1] Because the major AI labs are aligning on this signal, publishers can now communicate training preferences across multiple companies at once, reducing administrative work.[1][2]

Each company is implementing the standard on its own timeline: Anthropic has already begun integration, OpenAI plans to implement it in the coming months, Microsoft will support it starting February 1, 2027, and Google aims for 2027 without specifying a month.[1][3][4][2]

The draft does not block crawling itself; it only signals how content should be used, meaning bots may still access pages, AI Weekly reported.[4][3] Google clarified that its existing crawler controls will remain in place alongside the new signal, and Microsoft says its new preferences will replace NoArchive and NoCache for training and grounding.[2]

Meta Platforms has expressed support for the idea but has not committed to implementation, SiliconANGLE reported.[1]