这种判断通常交给另一个 LLM——我们称之为_新颖性评判者（novelty judge）_。新颖性评判者评估一个研究想法相对于先前工作是否新颖，可以独立评估，也可以与其他想法进行比较。目前并没有构建此类评判者的标准方法；研究团队会选择模型、提示词、评估格式、推理预算，有时还会选择文献检索系统。这些选择会对最终结论产生实质性影响。然而，这些评判者往往仅在人类撰写的论文上进行验证，而不是在其实际旨在评估的 AI 生成想法上进行验证。
我们的新基准结果表明，新颖性评估本身远不可靠。在六种 LLM 评判者——GPT-5.1、GPT-5.2、GPT-5.4、Claude Sonnet 4.5、Claude Opus 4.5 和 Claude Opus 4.6——中，对评估设置的看似微小的变化会导致对同一想法的新颖性判断出现巨大差异。在一项实验中，仅更改提示词中的几个字，就会使模型识别两个想法中哪一个更具新颖性的准确率下降超过 50 个百分点。检索和额外的推理并不能始终改善判断结果，而且两个专门构建的新颖性评估系统的表现甚至不如简单的 Claude Opus 4.6 提示基线。
因此，我们围绕 ICLR 2026 同行评审构建了一个自动化流程。我们没有将论文录用作为新颖性的代理指标，而是从评审意见中挖掘审稿人明确讨论贡献原创性的段落。这一区别至关重要；只要论文在其他方面足够强，即使不太新颖也可能被录用。同样，一篇被拒稿但具有原创贡献的论文可能在其他方面存在弱点。
对于这个池子，我们将相同的 154 个经过验证的人类想法与 154 个由简单 LLM 生成器产生的想法配对。我们假设生成的想法比其人类撰写的对应想法新颖性较低；这些是弱标签，并非对所有想法都经过验证的判断。生成器仅接收一个研究领域和一个提出新颖想法的指令，没有文献搜索、工具、迭代或反馈。这为我们提供了一个保守的比较，即在基本生成器产生的想法与审稿人明确认可其原创性的 ICLR 前 10% 论文的贡献之间进行对比。
作为额外的检查，一位专家独立审查了 30 对_人类+生成_组合，重点集中在我们最强评判者出错的情况上，并将人类和 AI 生成的想法重写为通用结构，以测试评判者是否仅仅是对写作风格的差异做出反应。专家在所有 30 对中更喜欢人类想法，并认为没有任何生成的想法具有新颖性。
利用我们收集的人类撰写和 AI 生成的研究想法，我们测试了六种 LLM 评判者——GPT-5.1、GPT-5.2、GPT-5.4、Claude Sonnet 4.5、Claude Opus 4.5 和 Claude Opus 4.6——通过逐点比较（即绝对新颖性评分）和成对比较（即比较两个想法以找出最具新颖性的那个）来评估新颖性的效果。我们一次只改变一个因素：提示词、推理预算、评判者是否能查看相关论文、想法的格式、合并到最终结论中的判断数量，以及对于 AI 生成的想法——由哪个模型生成它们。
例如，我们向 GPT-5.4 展示了相同的_人类+生成_组合——每个组合包含一个经过验证的人类撰写想法和一个 AI 生成的想法——使用两种提示词。一种提示词告诉模型，审稿人在每对想法中恰好发现了一个具有新颖性的想法，并要求它识别出是哪一个；另一种提示词则简单地询问哪个想法具有新颖性。当我们切换到较简单的提示词时，准确率从 93.5% 降至 40.9%——低于随机水平——并且模型改变了超过一半相同组合的结论。
相比之下，同样的提示词变化在_仅人类_数据集上的准确率仅改变了约七个百分点，该数据集将具有较强和较弱新颖性证据的人类撰写想法配对。这表明，新颖性评判者在人类撰写的研究上可能显得相对稳定，但在 AI 生成的想法上行为却大相径庭。
话虽如此，人类撰写的想法也并非免疫。我们在_仅人类_数据集上也看到了显著的变化；同样的变化使 Claude Opus 4.6 的准确率移动了 21 个百分点。但这些变化在 AI 生成的想法上更为严重。
也没有普遍更好的提示词。例如，在单独评判想法时，删除一条警告评判者不要将琐碎或增量贡献视为新颖的指令，提高了某些评判者的准确率，却降低了其他评判者的准确率。换句话说，对一个模型有效的提示词可能无法转移到另一个模型。
我们还测试了新颖性评判者的常见用途：比较来自两个 AI 生成器的想法。我们将 Claude Opus 4.5 的想法与 Claude Sonnet 4.5 配对，将 GPT-5.4 与 GPT-5.1 配对。在测试的提示词变化中，平均而言，六种评判者中有四分之一到三分之一的结论发生了变化。这种分歧可以在不假设哪个生成器产生更具新颖性想法的情况下进行衡量，展示了评判者提示词的选择如何影响创意系统之间的比较。
更多的搜索和推理并不一定产生更好的新颖性判断。我们给予 GPT-5.6 Sol 极高的推理预算和仅限于 arXiv 的实时迭代搜索，然后将其与每个看到五个预检索摘要的标准检索评判者进行比较。（在此比较中的所有评判者针对_人类+生成_中的每个想法运行一次。）智能体评判者的宏观 F1 得分为 0.78——落后于 Claude Opus 4.6 的 0.89（相差 11 个点），大致处于较简单的 Claude Opus 4.5 和 GPT-5.4 设置的范围内——而在当前 API 价格下，每次决策的成本约为 10 倍。（在此背景下，宏观 F1 衡量评判者识别新颖和非新颖想法的能力；越高越好。）
专门构建的评估器准确性也没有更高，尽管成本要高得多。在_人类+生成_集合上，一个简单的 Claude Opus 4.6 提示词使用低推理并在三次调用中进行多数投票，实现了 0.854 的宏观 F1，总评估成本为 2.59 美元。相比之下，Idea Novelty Checker（一种基于文献的检索和重排序管道）得分为 0.642，成本为 30.66 美元；而 Sakana AI 的 AI Scientist（其新颖性评估器迭代搜索 Semantic Scholar）得分为 0.772，成本为 82.05 美元。（两个专用评估器均使用 Claude Opus 4.6，每次决策调用一次。）
我们的结果表明，基于 LLM 的新颖性评估仍然是一个主要的开放性问题。在人类撰写的研究上看起来可靠的评判者，在 AI 生成的想法上可能表现不同；有助于一个模型的提示词可能会损害另一个模型；更多的搜索或计算资源并不必然产生更一致的判断。
未来的工作应在评判者实际将评估的想法类型上测试评判者，测量跨提示词和模型的敏感性，改进先验文献的整合方式，并构建能够随着科学文献的增长和 AI 系统的改进而演变的基准。我们的基准和可重用数据管道只是一个起点——研究人员可以将它们应用于新的同行评审、其他科学社区以及未来几代的新颖性评判者。
自动创意不仅仅依赖于生成合理的研究方向；我们还需要一种方法来辨别这些想法是否真正新颖。通过使当前新颖性评判者的弱点更容易衡量，我们希望此次发布能帮助研究人员建立更稳健的方法来评估 AI 生成的
查看原文
📄 Paper | 📊 Data | 💻 Code
Today we’re releasing Old Ideas, Novel Problems, a new study of how reliably large language models can evaluate the novelty of scientific research ideas. Alongside the paper, we’re publishing a benchmark, data, and code for testing novelty judges, plus an automated pipeline researchers can use to build new evaluation sets.
At Ai2, we’ve been developing AI systems that support different parts of scientific discovery. DataVoyager helps researchers explore and analyze structured data, AutoDiscovery generates and tests hypotheses, and Theorizer synthesizes evidence across large bodies of literature into candidate scientific theories. Together with Asta, these projects point toward systems that can help researchers move from understanding prior work to proposing, testing, and refining new ideas.
But generating research ideas is only part of the challenge of automating scientific discovery. Those ideas also need to contribute something genuinely new, which means these systems need a way to distinguish a novel direction from one that mostly repackages existing work.
That judgment is often handed to another LLM—what we call a novelty judge. Novelty judges assess whether a research idea is new relative to prior work, either on its own or in comparison with another idea. There’s no standard way to build one; research teams choose a model, prompt, evaluation format, reasoning budget, and sometimes a literature-retrieval system. Those choices can materially affect the verdict. Yet these judges are often validated only on human-written papers rather than the AI-generated ideas they are actually meant to evaluate.
Our new benchmark results suggest novelty evaluation itself is far from reliable. Across six LLM judges – GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5, and Claude Opus 4.6 – seemingly small changes to the evaluation setup produce dramatically different judgments of the same ideas. In one experiment, changing just a few words in a prompt shifts a model’s accuracy in identifying which of two ideas is more novel by more than 50 percentage points. Retrieval and additional reasoning do not consistently improve judgments, and two purpose-built novelty evaluation systems perform worse than a simple prompted Claude Opus 4.6 baseline.
If AI is going to play a larger role in scientific discovery, we need more dependable ways to tell whether it is actually proposing something new. Our work provides tools for studying that problem, while showing how much remains unresolved.
Building a benchmark for novelty judges
To test how reliably novelty judges can distinguish more novel research ideas from less novel ones, we first needed examples where there was strong evidence about the originality of the underlying scientific contribution.
So we built an automated pipeline around ICLR 2026 peer reviews. Rather than treating paper acceptance as a proxy for novelty, we mined the reviews themselves for passages where reviewers explicitly discussed a contribution’s originality. That distinction is key; a paper can be accepted without being particularly novel as long as it is strong enough in other aspects. Likewise, a rejected paper with an original contribution can have other weaknesses.
We created a high-novelty pool of 154 human-authored research ideas by selecting accepted papers in the top-rated 10% of their ICLR research area whose reviewers positively identified novelty and where no reviewer disputed it. We applied the reverse criteria to build a lower-novelty pool: 145 ideas from rejected papers in the bottom 10% whose reviewers raised originality concerns without any reviewer positively identifying novelty. In a separate check of the review-mining process, an annotator agreed with the extracted novelty-signal polarity in 47 of 50 cases; none of the disagreements would have changed whether a reviewer was counted as praising or disputing the paper’s novelty.
Together, these two pools form our first evaluation setting, Human-Only, which tests whether novelty judges can tell apart human-authored ideas with strong evidence of higher or lower novelty. But in practice, novelty judges are increasingly being used to evaluate AI-generated ideas. So we created a second setting, Human+Generated.
For this pool, we paired the same 154 validated human ideas with 154 ideas produced by a simple LLM generator. We assumed the generated ideas were less novel than their human-authored counterparts; these are weak labels, not verified judgments for every idea. The generator received only a research area and an instruction to propose a novel idea, with no literature search, tools, iteration, or feedback. This gave us a conservative comparison between ideas from a basic generator and contributions from top-10% ICLR papers whose reviewers had explicitly endorsed their originality.
As additional checks, an expert independently reviewed 30 Human+Generated pairs, heavily weighted toward cases our strongest judge got wrong, and we rewrote the human- and AI-generated ideas into a common structure to test whether judges were simply reacting to differences in writing style. The expert preferred the human idea in all 30 pairs and judged none of the generated ideas novel.
Importantly, many of our core findings don’t depend on the labels at all. If two reasonable evaluation setups give opposite answers for the same pair of ideas, that disagreement itself reveals instability.
Small changes can produce very different verdicts
Using our collections of human-authored and AI-generated research ideas, we tested how well six LLM judges – GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5, and Claude Opus 4.6 – could evaluate novelty through both pointwise (i.e., absolute novelty ratings) and pairwise (i.e., comparing two ideas to find the most novel) comparisons. We varied one factor at a time: the prompt, reasoning budget, whether the judge could see related papers, how the ideas were formatted, how many judgments were combined into a final verdict, and – for AI-generated ideas – which model generated them.
The largest shifts in performance came from how we prompted the model to judge novelty.
For example, we showed GPT-5.4 the same Human+Generated pairs – each containing one validated human-authored idea and one AI-generated idea – with two prompts. One prompt told the model that reviewers had found exactly one idea in each pair to be novel and asked it to identify which one; the other simply asked which idea was novel. When we switched to the simpler prompt, accuracy dropped from 93.5% to 40.9% – below chance – and the model changed its verdict on more than half of the same pairs.
By contrast, the same prompt change shifted accuracy by only about seven percentage points on the Human-Only dataset, which pairs human-authored ideas with stronger and weaker evidence of novelty. That suggests a novelty judge can appear relatively stable on human-authored research but behave very differently on AI-generated ideas.
That said, human-authored ideas weren’t immune. We saw substantial shifts on the Human-Only dataset as well; the same change moved Claude Opus 4.6 accuracy by 21 points. The shifts were just less severe than on the AI-generated ideas.
There was also no universally better prompt. For example, when judging ideas individually, dropping an instruction that warned judges not to treat trivial or incremental contributions as novel improved accuracy for some judges while reducing it for others. A prompt that works well for one model may not transfer to another, in other words.
We also tested a common use of novelty judges: comparing ideas from two AI generators. We paired ideas from Claude Opus 4.5 with Claude Sonnet 4.5, and GPT-5.4 with GPT-5.1. Across the tested prompt changes, roughly a quarter to a third of verdicts changed on average across six judges. This disagreement can be measured without assuming which generator produces more novel ideas, showing how the choice of judge prompt can affect comparisons between ideation systems.
More search and reasoning didn’t necessarily produce better novelty judgments. We gave GPT-5.6 Sol an extra-high reasoning budget and live, iterative search restricted to arXiv, then compared it with standard-retrieval judges that each saw five pre-retrieved abstracts. (All judges in this comparison ran once per idea on Human+Generated.) The agentic judge scored 0.78 macro-F1 – 11 points behind Claude Opus 4.6 at 0.89, and in roughly the same range as simpler Claude Opus 4.5 and GPT-5.4 setups – while costing roughly 10× more per decision at current API prices. (In this context, macro-F1 measures how well a judge identifies novel and non-novel ideas; higher is better.)
Purpose-built evaluators weren’t more accurate either, despite costing substantially more. On the Human+Generated set, a simple Claude Opus 4.6 prompt using low reasoning and majority voting across three calls achieved 0.854 macro-F1 at a total evaluation cost of $2.59. By comparison, Idea Novelty Checker, a literature-grounded retrieval and reranking pipeline, scored 0.642 at $30.66, while Sakana AI’s AI Scientist, whose novelty evaluator iteratively searches Semantic Scholar, scored 0.772 at $82.05. (Both dedicated evaluators used Claude Opus 4.6, with one call per decision.)
In other words, the simple prompted baseline was both more accurate and considerably cheaper than either purpose-built system.
Taken together, these results suggest that novelty evaluation is particularly brittle on the AI-generated ideas these systems are increasingly being asked to judge.
Toward more reliable evaluation of novel research
Our results suggest that LLM-based novelty evaluation remains a major open problem. Judges that appear reliable on human-authored research may behave differently on AI-generated ideas; prompts that help one model can hurt another; and more search or compute does not necessarily produce more consistent judgments.
Future work should test judges on the kinds of ideas they will actually evaluate, measure sensitivity across prompts and models, improve how prior literature is incorporated, and build benchmarks that can evolve as the scientific literature grows and AI systems improve. Our benchmark and reusable data pipeline are a starting point—researchers can apply them to new peer reviews, other scientific communities, and future generations of novelty judges.
Automated ideation depends on more than generating plausible research directions; we also need ways to tell when those ideas are genuinely new. By making the weaknesses of current novelty judges easier to measure, we hope this release helps researchers build more robust ways to evaluate AI-generated research ideas.
Studio