EnterpriseBench turns agent evaluation from a quiz into a decision scenario
The benchmark spans static questions, consulting dialogue, supply-chain feedback and a project simulator. Its own results—and a separate desktop benchmark—argue for task-specific gates rather than one reliability score.
What happened
ACL published EnterpriseBench in Findings of EMNLP 2026, testing nine agent methods with two backbone models across static and interactive enterprise settings.
Why it matters
The benchmark makes missing information, delayed effects and long-term trade-offs visible, but it does not establish that an agent is reliable in an organisation’s actual workflow.
Four different kinds of enterprise work
EnterpriseBench widens evaluation beyond finding facts or calculating a number. Its foundational suite reorganizes existing enterprise and financial question-answering data. Three interactive settings then test different failure modes: Consulting requires multi-turn information seeking, the Beer Game introduces delayed supply-chain feedback, and an Enterprise Digital Twin asks an agent to plan workforce, risk and projects over time.
The authors tested nine agent methods with two backbone models and report that none achieved stable, comprehensive reliability across tasks. That conclusion is bounded to the benchmark and configurations. A simulated business case is not a live organisation, and a benchmark score does not measure local data access, incentives, approval paths or the cost of a wrong action.
Long horizons remain a separate problem
DeskCraft provides an independent comparison in professional creative and engineering software. It evaluates 18 agents on 538 tasks, including interactive work where an agent must seek clarification or respond to feedback. The strongest reported model reached 31.6% on standard tasks and 27.6% on interactive tasks. The paper highlights persistent long-horizon and proactive-clarification failures.
The benchmarks are not directly comparable: one focuses on strategic reasoning and simulations, the other on desktop execution. Together they show why an aggregate score can conceal incompatible capabilities. An agent may retrieve the right fact yet fail when effects arrive later, or operate an interface while missing a question that a professional would ask.
Build an authority ladder
Start with three to five scenarios drawn from the real decision process. Include one incomplete brief, one delayed consequence, one competing objective and one case that should trigger escalation. Record success at the level of sub-decisions: what evidence was requested, which assumption changed, whether the agent noticed uncertainty and whether it stopped before an irreversible action.
Then define an authority ladder. Observation and draft generation can require a lower threshold than resource allocation or an external commitment. Advance only the specific scenario that passes; do not transfer success from static QA to interactive planning. Re-run the suite when the model, tools, data permissions or workflow change.
This complements the agent harness pairing evaluation: the unit of evidence should be the agent-plus-environment configuration, now tested across distinct decision dynamics.
Decision rule
Do not buy or authorize a “strategic agent” on one leaderboard number. Require scenario-level evidence for information seeking, delayed feedback and trade-offs, with a named stop condition for each authority tier.