Professional-agent training needs a task-environment audit
WorkFactory scales to 112,686 tasks across 612 occupations and repairs missing context through probe execution. The gain is promising, but coverage and benchmark scores do not replace human checks of occupational validity.
What happened
A new preprint introduces information-support graphs and probe-guided repair for synthetic professional tasks, then fine-tunes three Qwen models and reports gains on four work benchmarks.
Why it matters
The paper addresses a real bottleneck—broken task environments—while exposing a second one: whether a solvable synthetic task is representative of the occupation it names.
Solvability is a useful quality gate
The WorkFactory preprint starts from professional intents, builds an information-support graph and runs probe executions to find missing or inconsistent inputs. It reports 112,686 tasks across 612 occupations, teacher demonstrations and fine-tuned Qwen models at three scales. All three improved over their matched base models on four professional-work benchmarks; the paper also reports diminishing returns.
This is a concrete response to a common training-data failure: a task may sound plausible yet be impossible because a reference, dependency or workspace state is absent. Planning and repair can reduce those defects before an expensive demonstration is collected.
The remaining boundary is occupational validity. Solvable does not mean representative, correctly scoped or safe to generalise to a job. The preprint is not peer reviewed, and its environment construction, probes and much of its quality loop are model mediated. Headline coverage should not be read as evidence that 612 occupations can be automated.
WorkWorlds offers independent counterevidence about context curation. Across 192 matched evaluations, moving from task-curated context to a full role-visible workplace reduced evidence access and criterion pass, with much of the difference occurring before sufficient evidence was reached. Curated environments can inadvertently pre-solve information localisation.
Add a three-part audit
Sample tasks by occupation, risk and source intent. First, ask experienced practitioners whether the deliverable, constraints and ambiguity resemble real work. Second, run a blind environment audit: can an agent locate the necessary evidence without task-specific curation leaking where it sits? Third, test defect detection with missing, stale and conflicting inputs.
Track separate rates for executable, occupationally valid, evidence-localisable and safely delegable tasks. Report benchmark gains with environment overlap and judge composition. Link the result to the agent-harness evaluation analysis: model and harness are a measured pair.
Decision rule
Do not approve professional-agent training from task count and benchmark gain alone. Require independent occupational review and an environment audit that separates solvability from realism, evidence discovery and delegation safety.