A skill library needs four gates, not one end-task score
SRA-Bench separates retrieval, incorporation and application under heavy distractor load. MacAgentBench independently shows that a skill library can drive results, making provenance and stage-level observability release requirements.
What happened
ACL published SRA-Bench with 5,400 instances, 636 manually constructed gold skills and 25,626 web-collected distractors, evaluating retrieval, incorporation and application separately.
Why it matters
A useful skill can improve an agent, but a large library also creates selection, conflict and provenance risks that an aggregate success rate cannot locate.
A library is a pipeline
SRA-Bench evaluates whether an agent can use external procedural knowledge without treating “skill use” as one event. Its 5,400 instances draw on 636 manually constructed gold skills and 25,626 web-collected distractors. The benchmark asks three separate questions: did the system retrieve the right skill, did it incorporate the instructions, and did it apply them correctly to the task?
That separation matters operationally. A failed task may come from not recognizing that help is needed, ranking a plausible but wrong skill, loading conflicting guidance, or executing a correct procedure badly. Adding more content can improve coverage and simultaneously increase distractor risk. The paper reports sensitivity to irrelevant skills and limits in need-aware loading, but its corpus is still a benchmark rather than a production repository with local permissions and maintenance history.
Libraries can move the headline result
MacAgentBench independently evaluates 676 task instances across 25 macOS applications. Across three frameworks and 16 models, the best configuration reached 73.7% Pass@1. The authors attribute its advantage primarily to the skill library rather than the framework design.
That is evidence that procedural context can matter, not proof that any library is safe or portable. Pass@1 collapses the trajectory into a final result. Two systems with similar scores can fail different sub-goals, use different permissions or rely on stale procedures. The operating question is therefore not simply whether skills raise completion, but where they enter the decision path and how a reviewer can trace them.
Install four release gates
First, test need detection: include tasks where no external skill should be loaded. Second, test retrieval against confusing neighbours and stale versions. Third, test application with a checkpoint that records which instruction changed the plan. Fourth, test the outcome and any side effects under the actual tool permissions.
Give each skill an owner, source, version, allowed tools, expiry condition and conflict rule. Log the exact skill identifiers selected for a run. A retrieval hit should not itself authorize execution; high-impact steps still need a policy gate. When a skill changes, rerun affected tasks instead of accepting a library-wide average.
This extends the Apple agent edit review gate: a reviewable edit is stronger when the procedure that produced it is also identifiable and versioned.
Decision rule
Do not ship a larger skill library because end-task success rose. Require clean no-skill controls, retrieval precision under distractors, traceable incorporation and side-effect checks for the permissions the agent will actually hold.