AI Fairness

Fairness concerns the full decision process: problem framing, labels, features, model behavior and actions taken from predictions. Group metrics summarize different questions. Demographic parity compares selection rates; equalized odds compares error behavior conditional on outcomes; calibration concerns the meaning of scores. These criteria need not be simultaneously achievable, especially when data distributions differ. The appropriate comparison therefore depends on the harm and decision being assessed. Individual and procedural fairness add further questions that group averages cannot settle. Removing a sensitive attribute is insufficient when other features act as proxies or historical labels encode unequal treatment.

The practitioner identifies relevant groups and harms, checks label validity and builds disaggregated evaluation with uncertainty estimates. They compare interventions in data, thresholds, training and downstream procedures, documenting the tradeoffs rather than advertising a model as unbiased. Evaluation should include intersections where sample size permits and consider who is missing from the data entirely. The main artifact is a context-specific assessment connecting measured disparities to an action: redesigning a target, collecting better evidence, changing allocation rules or adding a route for human review.

Fairness metrics do not determine which differences are unjust, and noisy or biased labels can make an apparently favorable metric misleading. Small groups require careful uncertainty reporting. A mitigation can improve one criterion while worsening another or changing who receives an opportunity. Fairness should be reassessed when populations, policies or system uses change; passing a historical dataset does not certify equitable effects in a new deployment.