Goodput

Goodput is older than generative AI. DistServe applied it to the competing latency demands of prompt processing and token generation in 2024. Mooncake also optimized effective throughput subject to latency constraints. Later work revisited the metric itself, showing that apparently improved scores can conceal poor delivery behavior. These papers establish a useful engineering objective without making every implementation of the metric interchangeable.

Sources: s1, s2, s3

Increasing batch size or admitting more work can raise aggregate tokens per second while users wait too long. Goodput makes a serving comparison accountable to a defined experience. It helps evaluate scheduling, cache reuse and separation of prefill from decode. Report it alongside offered load, latency distributions, failures, rejected requests and hardware cost; otherwise a faster-looking configuration may simply serve an easier subset.

Sources: s1, s2, s3

Consider an illustrative one-minute load test with the same requests and hardware for two configurations. One completes 600 requests, of which 420 meet both latency thresholds. Its request goodput is seven per second. Another completes 540, of which 480 meet both thresholds, giving eight per second. The second has lower completion throughput but higher goodput under that explicit counting rule. Keep prompt lengths, output lengths and arrival patterns comparable.

Sources: s1, s2

Hard thresholds discard how far a request misses its target. Dropping slow requests or changing when tokens are delivered can also distort reported performance. Distinguish completed-request goodput from admission rate and raw token throughput, preserve failure accounting, and include end-to-end timing. Goodput measures delivery performance, not whether the generated answer is correct.

Sources: s2