Transfer Learning
Transfer learning reuses knowledge learned on one task or dataset to support another. It can use a fixed representation or adapt some or all model parameters. The skill is deciding what to transfer, how much to update and whether the source representation improves the target task under a fair evaluation.
What it is
A pretrained model contains representations shaped by its source data and objective. A target task can use those representations as features, replace an output head or fine-tune selected layers. The approach can reduce the amount of target-task fitting required, but its usefulness depends on compatibility between source and target. Frozen-feature training and full fine-tuning make different assumptions about what should change. Transfer learning is broader than language-model fine-tuning and appears in vision, audio and other domains. Competence includes recognizing negative transfer, where inherited features or adaptation choices make performance worse than an appropriate target-specific baseline.
What the work involves
Inspect the pretrained model's input conventions, source task and documented limitations. Establish a frozen-feature baseline before choosing layers to update, set learning rates and regularization accordingly and check target-label quality. Evaluate on target data separated by meaningful sources or time, comparing with simpler or from-scratch alternatives where practical. Save the base revision, preprocessing and adaptation artifacts. The result should explain which parameters were reused or changed and provide evidence that the transfer helps the intended target population rather than only fitting a small development set.
Illustrative example
In an illustrative inspection task, an engineer adapts an image encoder to a new material. They first train a small classifier on frozen features, then compare partial fine-tuning. The target images have different lighting and texture from the source data, so validation uses a separate acquisition batch. If deeper adaptation improves training but not the new batch, the engineer returns to a more constrained update and investigates data coverage before increasing capacity.
Limits and common mistakes
Source-task success does not guarantee useful target features. Large domain differences, incompatible preprocessing or small noisy target datasets can cause negative transfer or overfitting. Fine-tuning can forget previous behavior and changes the artifact that must be evaluated. A pretrained checkpoint's license and documentation remain relevant. Transfer learning differs from simply reusing code and from training an entire model from scratch. Compare adaptation choices on the actual target task and preserve enough provenance to reproduce the inherited representation.
Prerequisites
- hardDeep Learning
Transfer learning is meaningless without understanding what pretrained representations are and how fine-tuning modifies learned features
Related skills
- → is subcategory of: Machine Learning
- ← is subcategory of: LoRA / QLoRA
Sources and further reading
- Dive into Deep Learning: Fine-Tuning
Frozen representations, adaptation and target-task transfer.
Last updated: 2026-10-10