Metrics Definition
Metrics definition specifies how an outcome or system property is measured so teams can interpret and compare evidence consistently. In AI work, the competency is choosing an operationally meaningful quantity and documenting its formula, population, time window and exclusions, while guarding against incentives that improve the number without improving the outcome.
What it is
A metric is an operational definition, not just a label such as accuracy or engagement. It identifies what counts, at which observation unit and over which eligible population. A rate requires a denominator; latency requires start and end events; model quality requires a target and evaluation protocol. Product outcomes, model performance, reliability and risk indicators answer different questions. Proxy measures can be useful but should be connected to the intended goal. Definitions also need versioning when logging or business processes change, otherwise a trend can mix incompatible quantities.
What the work involves
The practitioner starts from a decision or objective, defines the measure and tests whether it can be computed from available data. They specify eligibility, aggregation, missing events and subgroup breakdowns, then reconcile example cases manually with the implementation. They pair a primary outcome with relevant guardrails and name an owner for changes. Useful work produces a metric contract and calculation that stakeholders can inspect, including interpretation limits and a rule for comparing periods when the population or definition changes.
Illustrative example
A team measures whether an AI drafting tool improves support work. Instead of counting generated drafts, the analyst defines completed cases that meet review criteria and records the full staff effort through submission. The contract specifies which cases are eligible and how abandoned sessions are treated. A guardrail tracks incorrect accepted advice. Manually reconstructed examples check the event pipeline before the measures enter product decisions.
Limits and common mistakes
A well-defined proxy can still be optimized at the expense of the underlying goal. Missing events, changed eligibility and aggregate averages can hide poor outcomes for important groups. Model scores and business value are not interchangeable. Check denominator stability, measurement coverage and incentive effects. Metrics should support judgment alongside qualitative evidence; adding precision to a formula does not make an inappropriate objective desirable or a causal claim established.
Prerequisites
Related skills
- → is subcategory of: AI Product Management
Sources and further reading
- GOV.UK: measuring service success
Supports choosing metrics from service questions and combining them with user and operational evidence.
- Rules of Machine Learning
Supports metric choices, objective alignment and the distinction between modeled and broader goals.
Last updated: 2026-10-10