Research · 2026-10-27

Can We Trust AI to Judge Whether a Research Idea Is New?

Researchers at Ai2 tested how well language models such as GPT and Claude judge the novelty of scientific ideas and found that their verdicts depend heavily on small changes in wording.

✨ Why AI needs a "novelty judge"

Scientists increasingly use AI to generate new research ideas. But an idea has to be genuinely new, not a rewording of what is already known. To check this, another language model is often used as a "novelty judge." It compares an idea with prior work and decides whether it contains something new.

The problem is that there is no standard way to build such a judge. Research teams choose the model, the question wording, the evaluation format, and whether the model may search for similar papers. Each of these choices can significantly change the result.

📌 What the study found

The Ai2 team tested six judges: GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5, and Claude Opus 4.6. They compared ideas written by humans with ideas produced by a simple model.

The wording of the question had the biggest effect. For example, when GPT-5.4 was given pairs of ideas and asked simply which one was novel, its accuracy fell from 93.5% to 40.9%, which is worse than guessing. In one experiment, changing just a few words in a prompt shifted accuracy by more than 50 percentage points.

The same kinds of changes affected judgments of human-written ideas too, though less strongly. Searching for related papers and extra reasoning did not consistently improve results. Two purpose-built novelty evaluation systems performed worse than a simple prompt to Claude Opus 4.6.

🔹 Why it matters

If AI is going to play a larger role in science, we need reliable ways to check whether it is actually proposing something new. For now, even judging novelty is unreliable, so readers should be cautious when AI claims that an idea is new.

The Ai2 study Old Ideas, Novel Problems provides tools for further research on this problem and shows how many questions remain unresolved.

Sources: 🔗 Hugging Face Blog