llm-d 1.0 makes workload shape part of the infrastructure decision

The result is specific to a workload shape

The open-source llm-d 1.0 release adds a stable configuration API, multi-tenant quality-of-service controls, high-availability routing, coordinated upgrades and session-aware agent serving. IBM's announcement frames the release as a Day 2 production milestone and cites demonstrations of three-times output throughput and five-to-ten-times lower per-token cost for context-heavy agentic workloads.

Those figures should travel with their test conditions. The supporting IBM Research account describes GLM-5.2, an approximately 753-billion-parameter mixture-of-experts model with about 39 billion active parameters, distributed across 544 NVIDIA H100 GPUs. Its agent traffic had very long inputs, high prefix reuse and bursts of concurrency. One study of 219 Claude Code sessions found a median request of roughly 195,000 input tokens and 317 output tokens; 96% of main-agent requests reused at least 90% of a previous input verbatim.

The account also separates optimisations that can interact. Prefix-aware routing tries to send a request to a worker holding its cached context. CPU and peer-to-peer tiers preserve or move key-value blocks under GPU pressure. Prefill/decode disaggregation scales context processing separately from token generation, while wide expert parallelism spreads the model across nodes. Because the effects compound, one headline number does not isolate the contribution of any single component. Removing or shrinking one layer in a local architecture may change both throughput and latency.

Under those conditions, caching and routing have a large target. At 2,500 concurrent AutomationBench agents, the deployment reported 7,612 requests per minute and 6.05 million output tokens per minute, with zero preemptions. This is a contributor benchmark observation, not an independent reproduction.

Carry a fingerprint, not a headline multiplier

Before a platform team transfers the result, we propose recording a workload fingerprint: model and quantisation; accelerator type and count; median and tail input/output tokens; prefix-reuse distribution; concurrency and burst pattern; latency service-level objectives; tenant priorities; cache capacity; failure budget; and all-in cost boundary. The cost boundary should say whether it includes idle capacity, orchestration, storage, networking, observability and engineering labour. A per-token comparison with an API is not the same as total cost of ownership.

Then replay a representative trace through a small staged test. Report p50 and p95 time to first token, inter-token latency, throughput, queueing, cache hit rate, errors and upgrade interruption. Test low-reuse traffic as well as the high-reuse case; a router optimised for cached prefixes has less to recover when prompts are mostly novel. Run a tenant-contention case and an upgrade rollback. These are proposed acceptance checks, not claims that llm-d fails them.

Keep model quality outside the infrastructure numerator but inside the decision. A cheaper token from a different open model is not equivalent to an API token unless the task-level acceptance test says so. Compare only configurations that clear the same bounded quality and safety threshold, then report capacity and cost. Otherwise a systems benchmark can make a model substitution look like a serving improvement.

The earlier Muse Spark article separated kernel-level gains from service-level performance. The same boundary applies here at a larger scale. llm-d 1.0 adds operational mechanisms worth testing, but the purchase or standardisation decision should depend on the organisation's workload fingerprint and service objectives, not on a portable-sounding multiplier.