Econometric agents remove visible errors while leaving a harder review task

What the benchmark actually tests

In an 18 October column, Sebastian Galiani, Federico A. López and Raul A. Sosa describe 40 econometric tasks with simulated datasets and precomputed reference answers. They compare chatbot, self-repair and constrained-agent conditions in Stata, R and Python: 14,400 runs overall, 4,800 per condition. Correctness means matching reference numbers within tolerance.

Pooled loud failures—scripts failing to run or complete their output—fall from 15.8 to zero per 100 runs between chatbot and agent. Silent failures fall from 4.9 to 2.9. But the latter change is uneven: 4.7 to 0.8 for Claude Sonnet 4.6 in Claude Code, against 5.1 to 4.9 for GPT-5.6 Luna in Codex. The authors identify an out-of-tolerance standard error in shift-share instrumental variables as the hardest remaining agent task.

These are author-reported model-plus-harness results, not independent replication. The underlying NBER working paper was inaccessible during this check. Simulated, specified problems do not establish research quality on messy observational data or employment effects.

Change what the reviewer has to demonstrate

Our interpretation is that the review assignment should distinguish two deliverables: evidence that the programme ran, and evidence that it calculated the requested quantity. The first can be checked against execution logs. The second needs a reference outside the generated script's own reassurance. A clean log cannot answer whether the analyst requested the right uncertainty estimator or whether the implemented formula matches that request.

For an empirical research team, a useful proposed exercise is a small dataset with a calculation the reviewer can derive independently. Ask the reviewer to specify the estimator, expected output and tolerance before seeing the generated implementation. Then require an explanation of any discrepancy. This is a training design suggested by the error distinction; the benchmark does not test whether this exercise improves professional judgement.

The reference should not simply reproduce the same implementation in different syntax. If both paths inherit the same mistaken specification, agreement is weak reassurance. A reviewer could instead compare a limiting case, a hand-solvable example or an independently justified formula. Which reference is appropriate remains a statistical decision, especially when the result depends on assumptions rather than arithmetic alone. Define the tolerance before observing the answer, and preserve the requested specification alongside the test. Otherwise a reviewer could inadvertently adjust the acceptance rule to accommodate a plausible but incorrect result.

Keep the model difference visible

The pooled figures are useful for describing this experiment, but a team choosing a workflow should retain the separate model configurations. A large reduction in visible failures does not imply the same reduction in silent errors for every configuration. Equally, residual errors do not erase the reported gains. Both observations belong in the decision.

This differs from the earlier research-bottleneck analysis, which concerned how coding activity changes research work. Here the narrower question is what a reviewer must be able to establish after a script finishes. Before expanding delegation, ask for a worked reference problem and a defensible uncertainty calculation. That makes verification skill observable without treating this benchmark as proof that a particular review process, model or staffing structure will succeed.