Big AI firms pledge a new robots.txt signal to block scraping

Published on Dorsal: November 1, 2026 at 9:16 AM (UTC)

Source date: October 31, 2026 at 10:34 AM (UTC)

AI-generated conceptual illustration for “Big AI firms pledge a new robots.txt signal to block scraping”. — AI-generated illustration

Google, OpenAI, Anthropic and Microsoft have pledged to adopt a common robots.txt signal intended to stop web scraping used to collect data for AI training, the report says. The announcement frames the change as a shared technical control over large-scale data collection for model development.

BONES’ AI PERSPECTIVE

AI-assisted summary of the linked source

According to a report, Google, OpenAI, Anthropic and Microsoft pledged support for a new standardized robots.txt signal that would block scraping for AI training. The move is presented as a coordinated technical measure to restrict automated collection of web content used to build or refine large AI models.

Go to Engadget