Feature Selection
Feature selection chooses a subset of available variables for a model, balancing predictive value, redundancy, acquisition cost and interpretability. The competency is evaluating that choice inside a valid training procedure so a smaller feature set reflects generalizable evidence rather than accidental relationships in the evaluation sample.
Also searchable as: feature-selection, Variable Selection
What it is
Selection retains original variables, unlike feature extraction methods that construct a new representation. Filter methods score variables using predefined statistics; wrapper methods compare subsets through model performance; embedded methods select through the estimator's fitting behavior, such as sparsity-inducing penalties. These approaches answer different questions and can disagree when predictors are correlated or interact. Selection is part of model development, so using the eventual test set to choose variables contaminates its assessment. A feature's individual association also does not reveal its value conditional on other inputs or its causal role.
What the work involves
The practitioner identifies the objective for reducing features, removes impossible or unavailable inputs first and chooses a selection method compatible with the estimator. They place selection within cross-validation, inspect stability across splits and compare with a sensible full-feature baseline. They consider collection cost and operational availability alongside predictive behavior. The deliverable is a documented subset and selection procedure that can be repeated, with reasons for important exclusions and evidence that reduced complexity does not hide a material failure on relevant cases.
Illustrative example
A team builds a maintenance classifier from many correlated sensor summaries. The engineer evaluates a regularized selector inside grouped cross-validation, then examines whether selected variables vary substantially across folds. Two sensors provide nearly interchangeable signals, but one is missing on an entire device family. The final subset is tested on that family and chosen with the device-coverage constraint explicitly recorded.
Limits and common mistakes
Repeated selection against the same evaluation set can overfit even when the final model is simple. Correlated variables make importance unstable, and univariate filters can miss interactions. Removing a costly feature may alter subgroup behavior or eliminate an early warning signal. Inspect validation nesting, stability and deployment availability. Feature selection is not an explanation of causality, and a sparse model is not automatically interpretable without understanding what its retained variables represent.
Prerequisites
Related skills
- → is subcategory of: Feature Engineering
Sources and further reading
- Scikit-learn feature selection
Documents filter, recursive, sequential and model-based selection methods.
- Scikit-learn common pitfalls
Explains feature-selection leakage and pipeline-based prevention.
Last updated: 2026-10-10