随着前沿 AI 模型的能力不断增强,安全措施在如何负责任地使用这些模型方面发挥着越来越重要的作用。对抗性蒸馏为不良行为者提供了一条途径,使他们能够在没有适当 safeguards(防护措施)的情况下复制这些能力。攻击者可能会未经授权利用前沿模型的输出来训练另一个具有类似能力的模型,然后部署该模型而不具备相当程度的保护。例如,对手可能会蒸馏出模型的网络安全能力,并利用生成的模型来协助实施网络攻击,而原始模型本会拒绝协助此类攻击,或其防护措施会阻止此类攻击。
我们对应对对抗性蒸馏的方法结合了对抗性测试、模型训练、分类器以及 API 级别的保护措施,并针对协调一致的提取模型能力的尝试进行调查。这些更新后的安全措施与我们用于生物安全和网络安全的措施更加一致。
我们特别强调防止攻击者提取我们模型的受保护推理过程:即模型在得出答案过程中内部生成的中间步骤,也就是其思维链(chain-of-thought)。这些推理痕迹可能包含最终答案所遗漏的信息,因此对于训练另一个模型来说,它们是极具价值的训练数据。
蒸馏是一种常见的人工智能学习技术,其中一个模型从另一个模型产生的数据进行学习。例如,一个模型可以从另一个模型的答案、推理过程,或其对哪些答案更好及原因的判断中学习。除了知识产权问题外,未经授权的蒸馏还带来严重的安全风险。通过对抗性蒸馏,第二个模型可以学习到先进的前沿能力,却未继承约束原始模型使用这些能力的防护措施。
这种担忧在网络安全和生物等双重用途领域尤为明显。一个帮助防御者发现软件漏洞的模型也可能帮助攻击者利用这些漏洞;一个辅助生物研究的模型也可能帮助某人开发有害病原体。如果此类能力被转移到防护措施较弱的模型中,它可能会提供原始模型本会拒绝的帮助,或者原始防护措施本会阻止的帮助。
我们的模型帮助用户进行编码和防御性网络安全任务。在大量这些良性输出上训练的第二个模型,可能会学到同样适用于进攻性网络安全任务的技能。为了更好地理解这一点,我们研究蒸馏(特别是涉及推理痕迹的情况)如何转移危险的网络安全能力。
在这种设置下,产生训练数据的模型称为教师模型,从中学习的模型称为学生模型。在一项关于网络安全能力转移的内部研究中,我们将用户通常可见的内容——最终响应、推理摘要和工具交互——作为蒸馏基线,并测试当学生模型能够访问教师模型的精确推理痕迹时,会有多少额外的网络安全能力发生转移。
我们从同一个预训练的学生模型开始每次训练运行。我们创建了一个监督微调数据集,主要由来自 Astra 级教师模型在处理编码和防御性网络安全任务时的轨迹组成。每条轨迹包括模型在执行任务过程中采取的所有步骤,包括任何工具交互及其最终响应。我们在微调后以及在网络安全和编码环境中的轻量级强化学习轮次之后,分别评估了每个学生模型的表现。
经过监督微调后,基于精确推理痕迹训练的学生模型通常优于基于推理摘要训练的学生模型。经过强化学习后,在两个基准测试中最大的推理预算下,基于精确痕迹的学生模型得分仍然更高,而基于摘要训练的学生模型在较低的预算下得分更高。1
我们的网络安全蒸馏结果表明,即使底层任务是符合我们使用政策允许的,保护推理痕迹也可以限制网络安全能力的转移。在许多此类交互中,精确的推理痕迹可以教会学生模型超越教师最初执行的任务的泛化技能。我们将继续研究在不同学生模型和蒸馏训练配方之间转移了多少额外能力。
我们在安全栈的两个互补层面上抵御对抗性蒸馏。在请求级别,模型行为、分类器检查和 API 控制限制了推理痕迹的提取。在活动级别,我们寻找请求级别和账户级别信号中的模式,以揭示协调一致的提取尝试。反过来,调查中揭露的攻击成为训练和测试我们安全措施的示例。
对抗性训练教导模型识别推理提取尝试并拒绝披露受保护的推理,同时保留对合法解释和帮助请求的响应。对抗性训练示例来自通过内部自动化红队测试、调查以及外部研究人员或行业合作伙伴负责任地披露的潜在安全弱点所确定的成功攻击。
借助 GPT-6.2 Astra 和 GPT-6.2 Sol,我们将模型的受保护推理与其生成它的对话历史绑定在一起。随着新消息的添加,模型可以基于其早期的推理进行构建。如果较早的消息被重写,我们的 API 会丢弃随之而来的推理(在新窗口打开)。我们还防止具有较弱防护措施的旧模型家族读取 GPT-6.2 Astra 和 Sol 的受保护推理。
当我们的系统检测到未经授权的对抗性蒸馏活动时,这些活动通常包括协调一致的提取我们模型推理的尝试,我们会调查负责的账户。当我们确定违规行为时,我们会根据我们的政策和条款对账户网络执行处罚。通过对每个活动的调查,我们生成见解,使我们能够升级防御措施,以便更好地检测类似活动。我们还将妥协指标与其他模型提供商共享,以便他们可以在自己的服务上调查相关的滥用行为。
为了衡量我们的请求级别安全措施协同工作的效果,我们在一个紧密镜像我们模型和安全措施生产部署的设置中测试攻击者是否仍能提取受保护的推理。在静态评估中,我们重放一组先前成功的攻击,看看哪些仍然会暴露受保护的推理。在自适应评估中,自动化 AI 红队测试人员会根据系统的响应调整其方法,这可以发现静态测试遗漏的弱点。
我们还改变测试人员可以看到的内容。黑盒测试给他们提供与外部用户相同的视图,而白盒测试则为他们提供有关系统的额外信息,以帮助他们开发更强的攻击。在我们的自适应黑盒评估中,我们测试了攻击者是否能够恢复模型在回答早期请求或当前请求时生成的受保护推理。在具有更新防护措施的 GPT-6.2 Astra 上,每种攻击类型的平均攻击成功率均低于 1%。这两个比率都远低于具有原始防护措施的 GPT-6 Astra。我们现在也将选定的安全措施改进应用于 GPT-6 Astra。
为了衡量我们工作的有效性,我们还跟踪蒸馏滥用的普遍程度,以及检测与执行之间的时间间隔。我们的目标是限制蒸馏者能够获得的数据量和价值,增加蒸馏活动的成本,并缩短对新攻击做出反应所需的时间。我们通过对抗性测试和调查不断改进这些防御措施。
查看原文
As frontier AI models become more capable, safeguards play an increasingly important role in how they can be used responsibly. Adversarial distillation creates a path for bad actors to reproduce these capabilities without appropriate safeguards. An actor may use a frontier model’s outputs without authorization to train another model with similar capabilities, then deploy it without comparable protections. For instance, an adversary might distill a model’s cyber capabilities and use the resulting model to help carry out cyberattacks that the original model would have refused to assist with or that its safeguards would have blocked.
We’re sharing how OpenAI detects and disrupts adversarial distillation, how we’ve strengthened protections for our models’ protected reasoning, and what we’ve learned that can help strengthen defenses across the industry.
Our approach to adversarial distillation combines adversarial testing, model training, classifiers, and API-level protections with investigations into coordinated attempts to extract model capabilities. These updated safeguards align more closely with those we use for biology and cybersecurity.
We place particular emphasis on preventing attackers from extracting our models’ protected reasoning: the intermediate steps a model generates internally while working toward an answer, known as its chain-of-thought. These traces can contain information the final answer leaves out, making them especially valuable training data for another model.
We’ve strengthened our defenses against protected reasoning extraction in GPT‑6.2 Astra and GPT‑6.2 Sol. In an internal adaptive evaluation using strong automated red-teaming, we tested whether attackers could recover protected reasoning the model had generated while answering an earlier prompt. The average attack success rate fell from 54.0% against GPT‑6 Astra with its original safeguards to 0.2% against GPT‑6.2 Astra with its updated safeguards.
Why distillation is a safety issue
Distillation is a common machine-learning technique in which one model learns from data produced by another. For example, a model can learn from another model’s answers, its reasoning, or its judgments about which answers are better and why. Beyond intellectual property concerns, unauthorized distillation poses serious safety risks. Through adversarial distillation, a second model can learn advanced frontier capabilities without inheriting the safeguards that govern how the original model can use them.
The concern is especially clear in dual-use domains like cybersecurity and biology. A model that helps defenders find vulnerabilities in software could also help an attacker exploit them; a model that assists with biological research could also help someone develop a harmful pathogen. If such capabilities were transferred to a model with weaker safeguards, it may provide assistance that the original model would have refused, or that the original safeguards would have blocked.
The importance of protecting exact reasoning traces for cyber safety
Our models help users with coding and defensive cybersecurity tasks. A second model trained on many of those benign outputs may learn skills that also apply to offensive cybersecurity tasks. To better understand this, we study how distillation, especially involving reasoning traces, may transfer risky cyber capabilities.
In this setting, the model that produces the training data is called the teacher, and the model that learns from it is called the student. In an internal study on how cyber capabilities transfer, we used the content normally visible to users—final responses, reasoning summaries, and tool interactions—as our baseline for distillation, and tested how much additional cyber capability transfers when the student has access to the teacher’s exact reasoning traces.
We started each training run from the same pretrained student model. We created a supervised fine-tuning dataset consisting primarily of trajectories from Astra-level teacher models working on coding and defensive cybersecurity tasks. Each trajectory includes all of the steps a model takes as it works through a task, including any tool interactions and its final response. We evaluated each student after fine-tuning and again after a lightweight round of reinforcement learning in cybersecurity and coding environments.
After supervised fine-tuning, the student trained on exact reasoning traces generally outperformed the student trained on reasoning summaries. After reinforcement learning, the exact-trace student still scored higher at the largest inference budget tested on both benchmarks shown, while the summary-trained student scored higher at lower budgets.1
Our cyber distillation results suggest that protecting reasoning traces can limit cyber capability transfer, even when the underlying task is one allowed under our usage policies. Across many such interactions, exact reasoning traces can teach a student skills that generalize beyond the tasks the teacher originally performed. We will continue studying how much additional capability transfers across different student models and distillation training recipes.
Defenses that reinforce one another
We defend against adversarial distillation at two complementary levels of our safeguard stack. At the request level, model behavior, classifier checks, and API controls limit reasoning trace extraction. At the campaign level, we look for patterns in request-level and account-level signals that can reveal coordinated extraction attempts. In turn, attacks uncovered in investigations become examples for training and testing our safeguards.
Our defenses against adversarial distillation combine safeguards for individual requests with campaign detection and enforcement. Findings from investigations and automated red-teaming help strengthen those safeguards.
Protecting individual requests and responses
Defensive training for model behavior
Adversarial training teaches models to recognize reasoning extraction attempts and refuse to disclose protected reasoning, while preserving responses to legitimate requests for explanations and assistance. Adversarial training examples come from successful attacks identified through internal automated red-teaming, investigations, and responsible disclosures of potential security weaknesses from outside researchers or industry partners.
Classifier safeguards
We use input classifiers to detect attempts to extract protected reasoning, and output classifiers to detect leaks of protected reasoning in assistant messages and tool-call arguments. These safeguards apply to both streaming and non-streaming responses.
Tool-call coverage is important because a model can expose protected reasoning in a tool’s arguments without including it in its final answer. Output classifiers can block suspected leaks before that content reaches the caller.
API-level protections
API-level protections restrict how protected reasoning can be reused across interactions and models. Our September disclosure described attacks that moved encrypted reasoning from one conversation into another and asked the model to reveal the reasoning in user-readable form. We strengthened protections across user, workspace, organization, and model-family boundaries, including closing a pathway that could expose another user’s encrypted reasoning.
With GPT‑6.2 Astra and GPT‑6.2 Sol, we tie a model’s protected reasoning to the conversation history that produced it. As new messages are added, the model can build on its earlier reasoning. If an earlier message is rewritten, our API drops the reasoning that followed it(opens in a new window). We also prevent older model families with weaker safeguards from reading GPT‑6.2 Astra and Sol’s protected reasoning.
Investigating and disrupting coordinated campaigns
When our systems detect unauthorized adversarial distillation campaigns, which often include coordinated attempts to extract our models’ reasoning, we investigate the accounts responsible. When we identify violations, we enforce against the network of accounts, consistent with our policies and terms of service. Through investigations of each campaign, we generate insights that allow us to upgrade our defenses in order to better detect similar activity. We also share indicators of compromise with other model providers so they can investigate related abuse on their own services.
This work extends beyond our own platform. Our consistent-standards policy establishes shared standards for safeguards and coordinated enforcement across providers offering our models. We work closely with partners where appropriate to uphold these standards and strengthen protections across deployments.
Sharing relevant findings across industry and government where appropriate can help disrupt campaigns and enable model developers and providers to better identify and defend against attempts to distill frontier models.
Measuring progress
Resistance to reasoning extraction
To measure how well our request-level safeguards work together, we test whether attackers can still extract protected reasoning in a setup that closely mirrors production deployments of our models and safeguards. In static evaluations, we replay a fixed set of previously successful attacks to see which still expose protected reasoning. In adaptive evaluations, automated AI red-teamers adjust their approach based on how the system responds, which can uncover weaknesses the static tests miss.
We also vary what testers can see. Black-box testing gives them the same view an external user would have, while white-box testing gives them additional information about the system to help them develop stronger attacks. In our adaptive black-box evaluation, we tested whether attackers could recover protected reasoning the model had generated while answering an earlier request or the current one. On GPT‑6.2 Astra with its updated safeguards, the average attack success rate was below 1% for each attack type. Both rates were substantially lower than on GPT‑6 Astra with its original safeguards. We’re now applying selected safeguard improvements to GPT‑6 Astra as well.
Attack type
GPT‑6 Astra defender with original safeguards
Adaptive extraction ASR (%)
GPT‑6.2 Astra defender with updated safeguards
Adaptive extraction ASR (%)
Extracting protected reasoning from an earlier request
54.0
0.2
Extracting protected reasoning from the current request
22.9
0.9
Attack success rate (ASR) in adaptive black-box red-teaming against GPT‑6 Astra and GPT‑6.2 Astra, aggregated across six attacker models. Lower ASR is better.
Campaign detection and enforcement
We have seen multiple unauthorized adversarial distillation campaigns on our platform this year, and shared initial findings in our September report. Our weekly enforcement actions against these campaigns have ranged in scale from thousands to tens of thousands of accounts, with the rate increasing significantly this past summer.
In order to measure the effectiveness of our efforts, we also track the prevalence of distillation abuse, as well as the time between detection and enforcement. Our objective is to constrain the volume and value of the data distillers are able to obtain, to increase the cost of distillation campaigns, and to shorten the time needed to respond to new attacks. We continue to improve these defenses through adversarial testing and investigations.
The national security implications of adversarial distillation
Adversarial distillation also poses a national security risk. A foreign actor that reproduces advanced capabilities from a U.S. model can deploy them without the protections its developer put in place. In February, we briefed(opens in a new window) Congress on activity indicating that DeepSeek was attempting to distill U.S. frontier models. In September, we reported disrupting a campaign to extract our models’ protected reasoning and attributed a core part of that activity to individuals associated with Moonshot AI, the developer of Kimi. We plan to share more soon about recent campaigns we’ve disrupted.
We support legislation that would give the U.S. government more tools to identify and sanction foreign actors that illicitly extract capabilities from American AI models. We’ve also told Congress we support the Chip Security Act, the AI OVERWATCH Act, and the MATCH Act. Stronger export controls can limit adversaries’ access to the hardware needed to train frontier models. Distillation defenses make it harder to acquire some of the same capabilities from models already trained. Both efforts help maintain a U.S. lead in frontier AI, giving developers more room to slow a release when they need time to test safeguards.
How others can build on this work
Supporting the safe proliferation of frontier capabilities is an important goal of our anti-distillation work. Our experience so far suggests several useful steps that other model developers and platforms can take:
-
Protect private reasoning across the full interaction. Check assistant messages and tool-call arguments for leaks, and restrict how encrypted reasoning can be reused across conversations and models.
-
Keep successful extraction attack logs found through red-teaming, production monitoring, or responsible disclosure as regression tests. Where appropriate, and consistent with applicable privacy policies, use those examples to improve model training and classifiers. When evaluating updated defenses, replay the original attacks and test whether an adaptive red-teamer can still find a way through those defenses.
-
Combine request-level mitigations with campaign-level investigation and enforcement.
-
Coordinate safeguard requirements and enforcement with deployment partners, and share expertise with other AI model developers, to ensure a high floor of distillation safeguards across the industry.
We have previously also shared findings through the Frontier Model Forum(opens in a new window) and appropriate government information-sharing channels to help other developers strengthen their defenses while keeping the details attackers could exploit private. We’ll continue working with researchers, developers, and partners to share what we learn about emerging attacks and effective defenses.
As we make frontier AI broadly available, our defenses against adversarial distillation aim to reduce the risk that its capabilities are reproduced without adequate safeguards, supporting our mission to ensure that artificial general intelligence is deployed safely and benefits all of humanity.