Independent review should start before the AI summary is shown

The reviewer may already have seen the answer

Reporting on 15 October renews attention to a practical question: is a reviewer’s recollection still independent after reading an AI summary? The underlying Sim, Eiger and Kohno manuscript was posted on 23 September, not this week.

The controlled human-subjects experiment recruited 360 US adults through Prolific; 331 completed both sessions and 328 entered analysis. Participants watched an animated accident, then read a summary 24–48 hours later. Researchers controlled whether a traffic-sign detail was consistent or misleading and whether the text was labelled AI- or human-written. All versions used AI-generated wording with controlled edits. Correct sign recognition was 83.6% after consistent information and 44.8% after misleading information. The source-label comparison was not statistically significant within either information condition.

This was a controlled recognition experiment, not a workplace trial or an estimate of current models’ spontaneous error rates. It tested a deliberately selected summary and a specific detail. The opened arXiv v1 notes a shortened AIES conference version; that separate version was not inspected here. The paper does not establish permanent memory damage, universal reviewer failure or a proven intervention for preventing the effect.

Preserve an independent first pass

Our recommendation is to test the order of review, not merely add another approval box. Before displaying a generated summary, ask the reviewer to record a short source-based account and point to the relevant passages or moments. Preserve that initial account separately. Only then reveal the AI version and ask for explicit discrepancies, omissions and unresolved questions.

This source-first sequence is an editorial proposal, not a safeguard shown to work by this experiment. Its purpose is to make the evidence path inspectable. A reviewer should be able to distinguish what they observed in the source, what the system proposed and what they later changed. A final agreement between person and machine cannot reveal that sequence by itself.

For a low-risk pilot, compare the usual summary-first workflow with this alternative on synthetic, non-sensitive material. Predefine the consequential details, use a reference answer created independently of the generated summary, and vary the error type. Measure both false acceptance and correction, along with review time and omissions introduced by the reviewer. Do not choose only conspicuous mistakes that make the proposed process look effective.

Keep a route to the original material available throughout the task. Ask reviewers to verify disputed details against that material rather than settle them through confidence or recollection alone. Record unresolved disagreements and permit escalation without forcing consensus. Training should include how to identify missing evidence, not only how to edit fluent wording.

Treat workflow protection as a hypothesis

The experiment’s narrow setting is a reason to test transfer, not to ignore the finding or impose a universal ban. Professional experience, repeated access to sources, longer documents and real organisational consequences may change the result. Neither favourable nor unfavourable transfer should be assumed.

The existing interview-summary record controls remain relevant where employment decisions are involved. This research adds an earlier question: when did the human first form their account? Before a high-stakes rollout, require evidence that the proposed review sequence improves error detection in the intended setting. A human signature is an accountability record, not proof of an independent cognitive check.