Data Quality Management
Data quality management defines and maintains the properties data needs for a particular use. It turns expectations about correctness, completeness, consistency and timeliness into checks and remediation processes, so problems are investigated at their source rather than repeatedly patched in downstream analysis or model training.
What it is
Quality is fitness for purpose rather than a universal property of a dataset. A missing value may be valid in one workflow and a serious error in another; a fresh table may still contain incorrect measurements. Quality management connects explicit expectations to measurement, ownership and action. Schema and range checks detect certain errors, relationship checks examine consistency and source comparisons can reveal discrepancies. Governance provides responsibility for definitions and remediation. This differs from observability, which supplies evidence about changing pipeline behavior, and from cleaning, which performs specific corrections on data that already violates the required conditions.
What the work involves
The practitioner identifies critical data elements and agrees on checks with producers and consumers. They prioritize issues by downstream impact, define thresholds and implement validation at meaningful boundaries. Useful artifacts include an expectation suite, incident ownership and a remediation record. A failed check should explain affected records and consequences, not only produce a red indicator. The team evaluates whether blocking, quarantining or alerting is appropriate and measures recurring failures so process improvements address their causes rather than normalizing permanent manual repair.
Illustrative example
A forecasting dataset includes store opening hours. A validation suite detects negative durations and inconsistent time zones, while a source comparison finds stores whose hours were never updated. The team routes malformed records for correction and marks stale stores separately. Forecast evaluation then distinguishes model error from unreliable input coverage. A generic non-null check would have passed many of the problematic records and hidden their business impact.
Limits and common mistakes
Checks only detect the problems they encode, and passing a suite cannot prove every record correct. Rules can become obsolete when the domain changes or can reject legitimate exceptions. Overly broad alerts waste attention, while aggressive repairs may erase useful signals. Quality management should retain issue provenance and assess downstream effects. The strongest evidence is a maintained process that resolves meaningful defects and updates expectations as use cases evolve.
Prerequisites
- mediumSQL
Data quality checks often run as SQL assertions against data warehouses
Related skills
- → is subcategory of: Data Engineering
- ← is subcategory of: Data Cleaning
Sources and further reading
- Great Expectations: GX Core introduction
Official introduction to explicit data expectations and validation workflows.
Last updated: 2026-10-10