Independent reporting on artificial intelligence
Edition No. 19
AI, closely read
THE CONTEXTEER
Saturday
October 31, 2026
The daily editionSaturday, October 31, 2026
No. 19 · 13 stories · Archive
Models & Tools

Google, Microsoft, Anthropic and OpenAI back new robots.txt standard to block AI scrapers

The standard does not block crawling but signals how content should be used, and Meta supports it without committing to implementation.

The crawling standard that Google, Microsoft, Anthropic and OpenAI are backing would allow website operators to block all scrapers that collect training data for generative AI models with a single robots.txt entry, SiliconANGLE reported. Because the major AI labs are aligning on this signal, publishers can now communicate training preferences across multiple companies at once, reducing administrative work.

Read the full story
Cut-paper collage for Models & Tools: a vast library reading room at night, in tangerine, emerald and cobalt on bright paper.
Image courtesy of Fay Tograph · AI photographer, made with Grok image generation
  1. 01

    Intel shares fall 29% after weak fourth-quarter revenue forecast

    The drop cut the value of the U.S. government's 10% stake by about $12 billion, a setback for chief executive Lip-Bu Tan.

  2. 02

    Apple Intelligence to launch in China with iOS 27.3 in December

    The features run on Apple's own models, and users can also call on Alibaba's Qwen inside Siri, Writing Tools and Visual Intelligence, a setup China's internet regulator cleared in July.

  3. 03

    DoorDash says outside AI agents have placed only several hundred orders

    DoorDash's connector that lets outside AI tools order through its platform rolled out less than a month ago and is open only to a small number of "early beta partners."

  4. 04

    Amazon can borrow up to $80B against hardware it buys for data centres

    Amazon can draw on the money from the start of 2027 until the third quarter of 2028, and had a free cash outflow of $42.4B in the 12 months to September.

  5. 05

    Proton to interview some staff about AI use, denies layoff motive

    The assessments will feed a wider review that may change roles or continued employment, while Proton says it is hiring and denies using AI as an excuse for layoffs.

Today's edition has 6 stories, 5 briefs and 2 updates. 286 claims were checked against the cited sources; 2 stories were cut because they did not pass.