ML Monitoring

Monitoring spans several layers. Service metrics track errors, latency and capacity; data checks track missing values, schema changes and distribution shifts; prediction and outcome metrics assess model behavior when labels or feedback become available. These signals answer different questions. Drift indicates changed data, whereas degradation requires evidence about task performance. For generative systems, timing or token measurements can complement output-quality sampling. Monitoring differs from offline evaluation because it observes real deployment conditions, but it still needs a reference, interpretable metrics and a policy for deciding which changes warrant action.

The practitioner defines service and quality objectives, instruments the inference pipeline and creates dashboards or reports segmented by relevant populations. Delayed labels require a planned join between predictions and outcomes. Alert thresholds are tested for useful sensitivity and manageable noise. Useful artifacts include metric definitions, alert owners and a response playbook. Investigation checks ingestion and infrastructure before retraining. Rollback, fallback and data repair should be operationally available so an alert can lead to a concrete response rather than only another notification.

Labels may arrive late or be biased toward users who provide feedback. Aggregated metrics can conceal subgroup harm, while many alerts can overwhelm operators. A monitoring dashboard does not determine causality or automatically justify retraining. Good practice defines the meaning and limitations of each signal and validates response procedures. Monitoring maintains evidence about deployed behavior, complementing rather than replacing pre-release evaluation and controlled model changes.