Kimi’s cyber filter needs a residual-risk ledger

The filter is a design claim, not a release verdict

Moonshot AI's Kimi K3.1 Pro launch note says the company removed cyber-related data from both training stages before the planned open-weight release, while retaining general-purpose coding tasks. The company says it does not want the weights to extend the absolute frontier of cyber capability. That is a meaningful design choice, but it is still a participant claim about a model whose downloadable weights are promised by 4 November.

Reuters reports Moonshot's internal xdev-bench-hard pass@3 result of 81.0% for the filtered model and 90.5% for a similar unfiltered version. Moonshot says the evaluation randomly selected runs and gave the agent vulnerable source, a short hint and a binary, without hard time or token limits. Reuters notes that the result has not been independently verified. The comparison cannot by itself establish a safety boundary: its task construction, sampling, tool access, absence of hard budgets and residual offensive capability matter as much as the percentage-point difference.

Open weights change the operating problem. A hosted provider can alter access, monitoring and safeguards after release. Widely distributed weights cannot be recalled in the same way. A pre-release decision therefore needs evidence about the residual model, not only the training material that was removed.

Build a residual-risk ledger before download

Start with a versioned capability map. List the cyber tasks tested, the environment, tools, time budget, success definition and baseline models. Separate vulnerability discovery, exploit construction, persistence and general coding. Record which results come from the filtered model, which come from the unfiltered comparator and which have been reproduced outside Moonshot.

Then define release conditions. A security team should name the tasks that would block local deployment, the containment needed for allowed work, who can load the weights and which logs must survive. Because the release is prospective, buyers should wait for the published weights, licence, model card and runnable evaluation artifacts before treating the announcement as an inspectable open-weight package.

The counterargument is that removing cyber-specific training could weaken defensive use. That is plausible, but the launch materials do not establish the downstream trade-off. Run a local, permissioned evaluation over representative secure-code review and remediation tasks, and separately test disallowed offensive workflows. Do not infer defensive value from general coding scores or infer safety from a lower internal cyber score.

This extends the newsroom's earlier local cyber retest analysis. The decision is not whether the model is “safe” in the abstract. It is whether a named version, under a named tool and network boundary, stays inside the organisation's residual-risk budget. Until the weights and artifacts are available, record the launch as a monitored claim and prepare the acceptance test rather than pre-approving deployment.

Turn the claim into a release gate

The gate should begin with artifact identity. Record the exact weight hash, licence, inference stack, quantisation and system prompt. Re-run a stable set of defensive tasks and prohibited offensive tasks, retaining attempts that fail as well as those that succeed. Compare the filtered release with the organisation’s current approved baseline; do not treat the unfiltered internal variant as a deployable comparator.

A useful test separates capability from control. Capability asks whether the model can complete a bounded cyber task. Control asks whether policy, refusal, monitoring and tool permissions prevent unacceptable use in the planned environment. Open weights change that control surface because a downstream operator can alter prompts, safeguards and tools. The release decision therefore needs evidence for the configuration being approved, not a general score.

Predeclare stop conditions: unexpected progress on disallowed tasks, material regression in defensive help, inability to reproduce the vendor’s version, or missing licence and provenance information. Name who can approve an exception and preserve the test bundle for later model or wrapper changes.

The counterargument is that delaying evaluation until release wastes preparation time. It does not. Teams can define tasks, scoring rules, secure sandboxes and reviewers now. They should simply label the result “test readiness,” not “model approval,” until the actual artifacts exist.