Research-idea evaluation must separate distance from direction

A 12 October preprint by Liu and colleagues distinguishes how far a research hypothesis lies from existing work from whether it moves outside established directions. Across 524,308 generated hypotheses, eight models and 1,325 topics in six natural-science fields, it reports lower median orthogonal novelty scores than human-authored papers. Its comparison corpus contains 26.9 million papers.

The method embeds titles and abstracts with SPECTER2 and measures the share of representation outside the topic’s existing subspace. The authors report a 15–23% median score deficit, but explicitly warn that this is a proxy: a high score need not mean a novel paper, nor a low score an unoriginal one. This is unreviewed preprint evidence, not a settled boundary on scientific invention.

Different novelty tests can disagree without contradiction

A 2024 independent study by Si, Yang and Hashimoto recruited more than 100 NLP researchers and found that reviewers judged generated ideas more novel, while slightly weaker on feasibility. That study also identifies limited diversity and unreliable model self-evaluation. It examines human assessments in a different domain and setup, not the new paper’s geometric construct.

Neither result should cancel the other by headline. A reviewer can find an idea surprising while its representation remains aligned with established directions. Conversely, an unusual embedding can reflect unfamiliar wording rather than a valuable scientific departure. This is an interpretation of the measurement problem, not proof that either study has isolated the cause of its findings.

For a research team, first specify the selection decision. Are you looking for useful extensions, an unexplored mechanism, a feasible experiment or a change of research agenda? Those are different objectives. A tool that helps with the first may be valuable without satisfying the last. A single league table of “novelty” would conceal that distinction.

Validate the selection process, not only the idea generator

A bounded local evaluation could compare human-origin, AI-origin and collaboratively developed proposals under the same brief. Blind the source where feasible, preserve the literature cutoff and require each proposal to identify the closest existing work. Ask assessors to score distinctness, feasibility and expected information gain separately. Record disagreements rather than averaging them away without explanation.

Then follow a manageable subset into actual research. Did the proposed experiment discriminate between alternatives? Did the result survive replication? Did the team discover that a supposedly new idea was already known? These are proposed follow-up questions; neither cited paper establishes their answers for your organisation.

Avoid tuning a generator until it maximises an unfamiliar proxy and then using that same score as independent validation. Keep a held-out assessment set and document any metric-driven changes. If an embedding measure is used for triage, sample rejected ideas too; otherwise the evaluation will never reveal what the filter excluded.

The new preprint compares generated titles and abstracts with published papers. Publication selection, model versions, topic construction and representation choice all limit transfer to real research programmes. The earlier human study has its own domain and judgement limits. Neither measures every stage from hypothesis through experiment to durable discovery.

The practical skill is model evaluation: define the construct, challenge the proxy and connect it to a decision. Teach researchers to explain why an idea should be tested, not just why its score is high. Continue using AI where it produces useful candidates, while retaining human responsibility for the agenda and empirical follow-through.