A 100% checklist score tests a skill package, not production competence
NVIDIA reports that DOCA agent skills lifted a model from 19% to 100% on 65 vendor-designed prompts. The useful next step is local reproduction across hardware, smoke tests and rollback.
What happened
NVIDIA published a 65-prompt evaluation in which an AI coding agent scored 19% without its DOCA skills package and 100% with the package.
Why it matters
The result shows that structured domain context can improve performance on a bounded vendor checklist. It does not independently establish safe deployment, transfer to local hardware or production reliability.
NVIDIA reports that its DOCA agent skills improved an AI coding agent’s score from 19% to 100% on a 65-prompt evaluation. The prompts covered setup, API use, build and runtime tasks. NVIDIA also publishes a provider checklist intended to make the skill package portable.
This is a useful vendor experiment, not an independent benchmark. The prompt set, expected answers, environment and skill package were designed by the same organisation. Build correctness represented only part of the test, and a checklist cannot observe every hardware state, security boundary, performance regression or recovery path.
Reproduce the mechanism locally
Choose ten representative tasks from your own backlog: environment setup, API selection, compilation, deployment, hardware interaction, failure diagnosis and rollback. Run them blind with and without the skill package. Keep the base model, tool permissions and time budget constant. Score source selection, command validity, build success, smoke-test result, unsafe action attempts and recovery quality.
Require provenance for every retrieved instruction and pin the skill version. A package that raises completion while silently widening permissions or using stale guidance is not a net gain.
The counterargument is that a 65-prompt result is already large enough to justify adoption. It is large enough to justify a controlled reproduction. It is not enough to estimate your error rate because local hardware, code, permissions and operator practice differ.
The immediate decision is a paired local test with a rollback gate. Promote the skill only when gains persist on unseen tasks and failures remain observable and reversible in production.