A monitoring fix needs an independent closure retest

A patch is not a closure record

METR's 22 October account says one METR employee and one Redwood Research employee spent three weeks, from 21 July to 11 August, stress-testing OpenAI's internal monitoring for agent misalignment. The evaluators had access to internal code, configuration and documentation. They report several vulnerabilities, including one route by which hundreds of recent actions could have escaped monitoring. Public details are incomplete because the work concerned security-sensitive systems and OpenAI retained redaction rights.

The exercise evaluated the system that existed on or before 21 July. METR says it did not evaluate changes introduced during the pilot or the later expanded monitoring coverage. The findings therefore cannot be carried forward to the current system without a fresh version-bound test.

Reuters independently reported the pilot and OpenAI's statement that production-impacting issues had been patched. METR said it had not evaluated those remediation claims. That distinction is the decision point: remediation activity belongs to the system owner; closure is a separate evidence judgement.

Coverage figures in the public accounts are estimates for a particular internal system, period and eligibility rule. They are not a universal measure of OpenAI traffic or proof that unmonitored actions were harmful. The red-team also found controls that worked. A useful review must retain both results instead of turning a bounded pilot into a verdict on an organisation.

Build a closure packet

For every finding, record the affected system version, agent class, tool path, traffic population, monitor version and detection rule. State whether the gap concerns eligibility, ingestion, analysis, alerting or response. Put attempted and eligible actions beside alerts so that a falling alert count cannot be mistaken for improvement when the denominator changed.

The owner should attach the patch, regression test, deployment evidence and residual limitations. An independent evaluator should design a closure test from the original failure mechanism, not only the owner's happy path. Include nearby variants: long sessions, context compaction, chained read-only tools, delayed actions, monitor errors and explicit attempts to disable observation. Preserve failures as findings rather than tuning the test until it passes.

Use two outcomes. Remediated means the owner changed the system and produced internal evidence. Closed means an authorised reviewer reproduced the relevant conditions on the deployed version and found that the acceptance threshold was met. A finding may remain mitigated but open when disclosure limits, access or model changes prevent a comparable retest.

The counterargument is that detailed retesting can expose sensitive architecture. Use a layered record: a restricted technical packet for authorised reviewers and a public statement of scope, evaluator independence, date, version, classes of finding and unresolved limits. Neither layer needs exploit code to make the closure claim auditable.

Set reopening triggers. A model upgrade, monitor rewrite, new tool class, changed traffic eligibility, altered retention or a material bypass report should reopen the finding. Sampling should include actions the system allowed, blocked and never evaluated. Otherwise success describes only the slice it was designed to see.

Track the operational response separately from detector quality. A monitor can identify a suspicious trace while alert routing, triage authority or evidence retention still fails. Exercise the handoff from detection to containment and preserve the latency, false escalation, missed class and human-review burden. A closure decision should say which layers were actually tested.

Avoid one permanent coverage percentage. Publish a small coverage matrix by model class, environment, tool type and session state, with the measurement date and exclusions. When an unmeasured service enters scope, label it unknown rather than inheriting the nearest percentage. This prevents a precise historical estimate from becoming a timeless assurance claim.

The immediate decision is to stop treating “patched” as a terminal status in agent-control registers. This article does not establish that OpenAI's current monitoring is ineffective. It shows why the organisation claiming remediation and the party confirming closure should produce different evidence. The record should remain reviewable.