One monitorability score can hide the behavior class that gets through

The average answers only one question

OpenAI's 20 October methods note adds pair-matched comparison and a post-control rejection-gate view to g-mean². It re-analyses historical system-card data for seven agent models with GPT-5 as monitor; GPT-6 Astra is excluded because it used a different monitor. This is sponsor-generated evaluation evidence, not independent replication or production evidence.

Pair matching reduces bias when models complete different task sets, but shared tasks can still contain different behavior classes. The note's honeypot example contrasts an overt technique the monitor caught with a short wrapper that passed. Detection may therefore reflect behavioral legibility as well as the agent-monitor pair.

Retries change the control, not just throughput

At four non-adaptive attempts, the simulation found more monitor false negatives than at one. It samples without replacement from an observed rollout pool and does not model adaptation after rejection. In two illustrated honeypot slices, per-request bad-output probability stayed below 3% and deployment probability below 7%, while 41–45% of deployed outputs carried the target evaluation label. That label is not automatically real-world harm, and positive-only process evaluations do not cover the system's full output mix.

The practical decision is to replace a single acceptance threshold with a small evidence panel. For each agent-monitor pair, record canonical-instance coverage, behavior-class coverage, true- and false-positive rates where identifiable, deployment probability, bad-output probability, the share among accepted outputs and the retry budget. Keep monitor version, prompt, scaffold and authority boundary fixed or versioned. If a behavior class has too few examples, show the empty cell rather than averaging it away.

Before setting a gate, run three decision checks. First, split results by behavior class and ask whether the proposed threshold changes when an under-sampled class is removed. Second, list which instances each monitor accepts and rejects, rather than comparing ranks alone. Third, stress the retry budget at the actual product limit and record both accepted-output yield and the worst credible miss. A gate that lowers the headline rate by suppressing nearly all outputs creates an availability tradeoff, not a free safety gain.

Assign one owner to each failure mode, not one owner to the aggregate score. Evaluation owns sample coverage and uncertainty; the monitor team owns detection changes; the product owner owns retry and availability tradeoffs; security owns the authority boundary and response to a miss. Reopen the decision when any of those inputs changes. This prevents a favourable average from silently carrying a new scaffold, broader tools or a more permissive retry policy into production.

Use the matrix to decide what must be expanded before deployment: more examples of ambiguous shortcuts, a different monitor, a narrower tool boundary or a lower retry count. Do not convert the historical ordering into a claim about production safety. The analysis itself says behavior severity and difficulty should ideally be recorded, and it leaves adaptive retries unstudied.

This extends the newsroom's earlier release-gate record: a stop decision needs not only a score but a trace of which behaviors were observable under which monitor. For evaluation teams, the immediate action is to require behavior-class coverage and deployment yield in every monitorability review. One number is useful for summary; it is not a sufficient deployment argument.