Agent evaluations need a model–harness pairing record

What the preprint measures

The new preprint argues that an agent result belongs to a model–harness interaction, not to either component alone. It evaluates four general-purpose harnesses with four core models, plus two fixed native systems, on six benchmarks. Including stress conditions, the authors report 111 settings and 52,168 trajectories. The work is under review, so its methods and findings should not be treated as settled production evidence.

The proposed measures separate valid completion from task correctness, then add model–harness fit, cross-model fit, pairing-specific interaction residuals and token load, including failed attempts. Harness rankings varied across models and benchmarks; similar task scores could accompany different execution profiles. An independent earlier bug study also found agent problems across frameworks and components, although it does not validate the new metrics. This draft pins version 1 because later revisions changed the reported corpus and publication status; the 1,187-post description is historical rather than a statement about the current arXiv version.

Why attribution changes the decision

A failed task can end in several ways. The model may choose a poor plan. The harness may truncate context, expose an unsuitable tool, mishandle a result or fail to preserve an artifact. A pairing can also be weak even when neither component is weak in every setting. Replacing the model after every failure can therefore leave the mechanism untouched and erase evidence needed to learn.

Completion deserves its own field. A run that stops normally but produces the wrong answer differs from one that crashes, loops, exceeds a budget or claims a handoff that never occurred. Cost also matters: failed trajectories consume tokens and tool calls that disappear from success-only averages. These distinctions are diagnostic, not proof that one harness is universally superior.

The pairing record

For every material evaluation, record the exact model and version, harness and version, system instructions, tool schemas, permissions, context policy, task revision, verifier, stop condition and resource budget. Preserve whether the run completed validly, whether the artifact was delivered, which criterion failed and the token and tool-call load of the failed attempt.

Use a matrix rather than one leaderboard. Compare at least two models in the target harness and one model across at least two harnesses on the same frozen tasks. Inspect interaction residuals where the pair performs better or worse than component-level summaries predict. In production incidents, reproduce the original pair before testing a replacement; otherwise the investigation changes two variables at once.

Treat the matrix as a sampling design, not a promise to test every combination. Freeze representative tasks, repeat runs where stochastic variation could change a decision, and retain dispersion or confidence intervals rather than only the best score. When a vendor-native system cannot expose the same harness boundary, record that comparability limit instead of imputing a missing component score. The aim is a better fault tree, not synthetic ranking precision the experiment cannot support.

Decision rule

Do not approve a model or harness change from an aggregate agent score alone. Require a versioned pairing record and separate gates for valid completion, task correctness, artifact handoff and resource load. If the team cannot reproduce the failing pair, classify the root cause as unresolved rather than blaming a component. This extends Skills Intelligence’s earlier work on harness observability into procurement and incident evidence.