AIsaAIsaStudio English 打开 AIsa Studio

OpenAI 将嵌入外部安全研究人员调查 AI 破坏控制行为

原标题:OpenAI will embed safety researchers to investigate AI subversion

OpenAI 宣布启动试点,将 Apollo Research、Goodfire、METR 和 Transluce 的安全评估人员嵌入公司内部,其中与 Apollo 的试点本周开始,以调查模型是否试图破坏自身的对齐或控制机制,此举与其在白宫《超级智能协议》下的独立安全评估承诺相关。但限制明显,Apollo 未获真正员工级权限,OpenAI 保留以技术、安全或 IP 理由扣留信息的权利,员工若在授权小组外披露保密信息 Apollo 不得跟进;Redwood CEO Buck Shlegeris 公开表示这些协议未达到此前承诺的类员工访问标准。

为什么值得看

报道列出四家评估机构的分工、访问范围与限制条件,并引述业界对独立性的质疑,帮助读者评估这一试点的实际效力。

阅读原文 ↗

本页的摘要、推荐理由与解读出自 AIsa Studio 的 AI 风向智能体,原文版权归原作者所有。

原文

OpenAI will embed safety evaluators from Apollo Research, Goodfire, METR, and Transluce in its organization as part of a pilot, OpenAI said Monday. The news follows a series of safety incidents, including one in which an OpenAI agent swarm broke out of its sandbox and hacked into Hugging Face during cybersecurity testing. Many AI safety researchers, including from Apollo, have argued such an event might have been avoided if outside evaluators had been embedded within OpenAI to monitor its safety testing.

OpenAI said the efforts support its commitments to rely on independent assessments of safety controls under the White House Accord on Super Intelligence. The pilots will begin at different points, with Apollo’s beginning this week, according to OpenAI.

The idea of embedding evaluators into frontier AI labs first came from Anthropic, which last month started to give METR researchers employee-level access to its systems. Until recently, AI companies have only brought in outside reviewers to test models after post-training and shortly before public release. Evaluators say this is insufficient because it doesn’t allow them to test for misaligned behaviors like scheming during training or to monitor internal deployment use cases that could still pose a threat.

The work with each third-party firm differs in scope. The pilot with OpenAI and Apollo will focus on investigating whether models attempt to subvert their own or their successors’ alignment or control mechanisms. In other words, they’ll be on the lookout for whether AI models try to undermine the safety rules and safeguards designed to keep them, or future versions of themselves, under human control. OpenAI recently published findings that while training GPT-5.6 Sol, it began leaving instructions for future versions of itself telling them to conceal mistakes and misaligned behavior from the user.

Per Apollo, the pilot will examine whether training incentivizes subversion; whether evaluations can elicit subversion; whether subversion occurs during internal deployment; and whether reasoning relevant to subversion would be detectable. These risks are becoming increasingly important to monitor as AI models grow more capable of acting autonomously and deceiving humans. Apollo’s previous research with OpenAI found that models can behave differently when they know they’re being tested and tend to prioritize what evaluators want to see over users’ actual institutions.

Goodfire will embed researchers who specialize in understanding how AI models work internally. Using its interpretability platform, the firm will help OpenAI monitor training runs, fix undesired behavior, and improve alignment, per the company. Goodfire says it will publish findings, though its primary audience will be OpenAI’s own researchers.

With METR, OpenAI plans to perform two evaluations: one that looks for unintended or dangerous behavior in the model itself, including problems OpenAI’s own tests might have missed, and another that tests whether OpenAI’s systems are capable of catching that behavior. For the first evaluation, METR will largely decide for itself what to investigate and how to test the model. OpenAI and METR have not yet come to a decision on the scope of the second assessment, according to OpenAI.

Transluce, meanwhile, will examine whether OpenAI’s post-training safety evaluations themselves are good enough. The organizations also planning to test in 2027 how OpenAI uses its safety tests to decide whether a model is safe enough to release.

At a time when the independence of third-party evaluations is under scrutiny, it’s worth noting that Apollo and Goodfire are both receiving financial compensation for their evaluations. METR and Transluce aren’t, but they may accept compute or API credits.

Apollo, which published the most detailed account of its agreement with OpenAI, offers a closer look at the scope and limitations of these evaluations.

Many in the AI safety community have warned that AI labs might try to stymy evaluators’ independence, especially if outsiders find unflattering evidence that labs would rather keep hidden. Apollo says an essential part of the pilot is OpenAI’s agreement that the firm can publicly explain the conditions and limitations of its work.

Still, there are areas where Apollo believes its access to OpenAI’s systems is lacking. For a truly independent evaluation of the company’s AI models, Apollo says it needs true employee-level access to OpenAI’s systems, including “all data, records and documentation relevant to their assessment even if a developer was trying to hide unfavorable information.” Getting access to all data, including training data and evaluation logs which companies are loathe to show to outsiders, is important for, say, checking a company’s claims about how a model performed or determining when certain concerning behavior emerged.

In previous coverage, John Steidley, chief of staff at Palisade Research, told TechCrunch that it’s important to see this data in order to assess whether a model was trained to perform well on a “shutdown resistance benchmark.”

In OpenAI’s first pilot, the AI lab is providing Apollo with access to only the information required to assess whether the models engage in subversion attempts. This includes “relevant training code and datasets, reinforcement-learning rollouts with available reasoning traces and rewards, and the ability to query intermediate model checkpoints.”

The scope of access will be limited to “only the agreed claims and only for the specific models and time periods agreed for the assessment,” Apollo says. OpenAI still reserves the right to withhold certain information within that scope if it deems it necessary because of technical infeasibility, security risks, or IP protection. The company agreed not to withhold information simply because it is unfavorable.

Apollo also won’t get access to all OpenAI staffers, instead being limited to “only authorized staff,” which are participants that the AI lab selects, though Apollo can suggest certain individuals. Crucially, if an OpenAI employee outside this group raises confidential information within the scope of Apollo’s research, the firm cannot follow up and will need to cease communication and route followup through OpenAI.

That condition is significant. It comes in the wake of OpenAI firing three researchers for allegedly sharing confidential information with safety evaluation firm METR — which performed the Hugging Face incident post-mortem alongside independent AI safety research group Redwood.

Buck Shlegeris, CEO of Redwood, said in a post on Monday that the agreements announced don’t meet the bar of granting third parties “employee-like access,” saying it’s “disappointing but not surprising.”

OpenAI said that they'd have embedded evaluations with employee-like access; the agreements announced today obviously don't meet that bar.

This is disappointing but not surprising.

Maybe future agreements will be better. https://t.co/kmDbg1Dqav

— Buck Shlegeris (@bshlgrs) November 16, 2026

The pilot will also be limited in how much researchers can disclose. While Apollo will be allowed to publish independent conclusions of its findings, OpenAI will be able to limit evidence available to substantiate them publicly, which could make it hard for outsiders to independently assess the findings. Apollo has pushed back against this limitation, saying it believes researchers should be allowed to disclose the findings of the evaluation “to an appropriate outside audience even if the developer was trying to prevent the disclosure of unfavorable findings.”

That said, while the public cannot see the full supporting evidence, OpenAI says its Safety and Security Committee, which oversees safety on behalf of the OpenAI Foundation’s board, will get to review unredacted assessment reports and can communicate privately with evaluators.

While OpenAI is not granting all of Apollo’s conditions for a truly independent evaluation, the firm says the pilot is “an ambitious first step towards deeper access and disclosure rights,” and that it is working towards running full embedded evaluations with multiple AI developers in the future.

This article has been updated to include pilots with METR, Goodfire, and Transluce, as well as with information from OpenAI and comment from Redwood.

When you purchase through links in our articles, we may earn a small commission. This doesn’t affect our editorial independence.

原文版权归 TechCrunch · AI 及原作者所有,本站转载仅供阅读。如需删除,请联系 support@aisa.one。

事件综述

OpenAI 启动第一阶段嵌入式评估,四家机构独立审查对齐与安全

2026年11月16日,OpenAI 宣布启动第一阶段嵌入式评估,与 Apollo Research、METR、Transluce 和 Goodfire 四家机构合作,让外部专家深入访问其内部模型与系统,独立评估模型对齐、安全防护措施和监控系统。同日后续消息显示,METR 与 OpenAI 达成协议,将对其最大规模的 RL run 开展一次有限范围的对齐评估试点,METR 表示将发布一份或多份报告,透明描述评估过程。TechCrunch 报道称,与 Apollo Research 的试点于当周开始,重点调查模型是否试图破坏自身的对齐或控制机制,此举与 OpenAI 在白宫《超级智能协议》下的独立安全评估承诺相关;但访问存在明显限制:Apollo 未获得真正员工级权限,OpenAI 可基于技术、安全或 IP 理由扣留信息,员工若在授权小组外披露保密信息 Apollo 不得跟进。Redwood CEO Buck Shlegeris 公开表示这些协议未达到此前承诺的类员工访问标准。11月18日,The Decoder 进一步报道称,OpenAI 向四家机构开放内部 AI 系统的现场审计权限,审计机构可在 OpenAI 办公室接触未发布模型、源代码、训练数据和监控系统,并保密约谈开发者;独立安全委员会监督全程并接收未删节报告,审计机构可独立发布结论,但 OpenAI 可删减涉密内容并推迟报告发布以先修复风险,双方还计划于2027年初进行为期八周的后续审查。关于访问权限,后续报道称可接触源代码与训练数据,早先报道称 Apollo 未获员工级权限且信息可能被扣留,两种表述并存。

其他来源的报道 3