Anomaly Detection

Detection methods can score distance from typical observations, estimate low-density regions, identify readily isolated points or model expected behavior over time. Outlier detection learns from data that may already contain unusual observations; novelty detection usually learns a reference of normal behavior and assesses new observations. Context matters: a large purchase may be ordinary for one account and unusual for another. Supervised detection is possible when reliable anomaly labels exist, but many applications have sparse or delayed labels. The key competence is matching the definition of deviation and the training setup to the event the organization actually needs to investigate.

Establish the reference population and decide whether detection operates on individual events, windows or sequences. Build features using only information available at the decision time, compare simple rules with statistical or machine-learning scores and set a threshold against realistic investigation capacity. Evaluate labeled cases where available, but also inspect normal events and changing conditions. The deliverable includes scores, alert explanations and a feedback process, so reviewers can distinguish model error, legitimate novelty and operational incidents while improving subsequent calibration.

Rarity and harmfulness are different properties. A common attack can look normal, while a legitimate new workflow can generate many alerts. Drift can make yesterday's reference inappropriate, and contamination assumptions influence thresholding. Accuracy alone is misleading when alerts are rare. Check false-alert burden, detection delay and sensitivity to known cases. Outlier-removal preprocessing also deserves scrutiny: automatically deleting flagged observations can erase important minority patterns or the very failures the system was meant to explain.