原标题：The biggest names in AI are backing a new way to opt out of scraping
据 The Verge 报道，OpenAI、Google、Microsoft 和 Anthropic 表示将采用一项新标准，让出版商可以一次性向多家 AI 公司声明拒绝被抓取的偏好，而无需逐家设置。该 "policy token" 信号由 Anthropic 标准技术负责人 Kevin Kelley 提出，提交至互联网工程任务组（IETF），将写入网站 robots.txt 文件中。
多家头部 AI 公司表态支持统一的抓取退出标准，读者可据此了解数据授权机制可能如何统一化。
本页的摘要、推荐理由与解读出自 AIsa Studio 的 AI 风向智能体，原文版权归原作者所有。
OpenAI, Google, Microsoft, and Anthropic say they will adopt a new standard that publishers can use to opt out of being scraped by AI, as reported earlier by Search Engine Roundtable. With this control, publishers can apply their AI training preferences across multiple AI companies, instead of setting them individually.
The shared “policy token” signal is proposed by Kevin Kelley, a technical lead for standards at Anthropic, as part of the Internet Engineering Task Force (IETF). It will live within a website’s robots.txt file, which has long signaled which web crawlers can and can’t scrape a site — whether for search indexing, archiving, or AI training. Currently, sites have to signal their AI training preferences separately to each AI company.
The announcement comes as publishers grapple with the continuing influx of AI crawlers scraping their sites and feeding their content into AI models. Some companies and organizations have attempted to combat unauthorized scraping, with the Really Simple Licensing (RSL) standard giving publishers the ability to charge crawlers that visit their sites, and Cloudflare blocking AI crawlers by default.
OpenAI, Google, Microsoft, and Anthropic say they will restrict their AI crawlers from accessing a site if they encounter the shared token. Meta communications head Andy Stone also signaled support for the new setting, but doesn’t say whether the company will honor it. “We believe creating a common signal that conveys such preferences to multiple providers at once is an important step,” Stone says. “And we look forward to working with other AI companies on the standards and their implementation across the industry.”
Microsoft says it will offer granular controls that will give publishers the ability to opt out of AI training or AI grounding, the process of pointing an AI model to a verified source of information to reduce hallucinations. Microsoft notes that websites will still appear in Bing’s search results regardless of their preferences. Google previously implemented similar controls that allow sites to control whether their content appears in AI search features or is used for AI training, without impacting visibility in search.
While Anthropic has already adopted the signal, Google and Microsoft plan to support it next year. OpenAI says it will implement it in the “coming months.”
Update, October 30th: Added a comment from Meta.
Follow topics and authors from this story to see more like this in your personalized homepage feed and to receive email updates.
原文版权归 The Verge · AI 及原作者所有，本站转载仅供阅读。如需删除，请联系 support@aisa.one。
Studio