Data Cleaning

Cleaning transforms data according to assumptions about valid values and relationships. It may standardize formats, reconcile units, remove duplicate representations or handle missing information. These operations have different consequences: dropping a record changes the represented population, while imputing a value adds an estimate rather than recovering the original observation. An outlier may be a valuable rare case, so extremity alone is not evidence of error. Cleaning differs from feature preprocessing, which prepares model inputs, and from curation, which decides broader inclusion and use. The relevant mechanism is a documented correction tied to the task and source evidence.

The practitioner profiles data, identifies failure patterns and creates rules that distinguish valid exceptions from defects. They preserve raw inputs or suitable provenance and record material changes. Useful artifacts include a cleaning specification and before/after checks for affected fields and population coverage. Missing-value treatment should reflect why values are absent, and learned imputation stays within training boundaries when used for modeling. The team samples corrected records and evaluates downstream effects, ensuring that a cleaner-looking dataset has not lost information or introduced systematic distortion.

Automatic repair can hide upstream problems or encode unsupported assumptions. Missingness can carry information, and deduplication can remove legitimate repeated events if identity rules are wrong. Rules learned from the full dataset can leak information into model evaluation. Cleaning quality should be assessed through traceable decisions and downstream validity, not the disappearance of every null or outlier. Some defects should remain flagged until source evidence supports a correction.