LoRA / QLoRA
LoRA adapts a model through trainable low-rank weight updates; QLoRA combines that adaptation with a quantized frozen base. The competence is selecting adapter targets and capacity, controlling memory use and verifying the complete loaded model. Low-rank adaptation and reduced-precision storage solve different parts of the training problem.
What it is
LoRA represents an update to a weight matrix as a product of two smaller trainable matrices while the original weight remains frozen. The rank controls the update's representational capacity, and scaling influences its contribution to the layer. Target-module choices determine which computations can adapt. QLoRA retains this adapter-training idea while storing the frozen base in a low-bit format and computing through dequantized values. Its published recipe includes NormalFloat quantization and memory-management techniques. Quantized base weights are not simply trained like ordinary full-precision weights. The canonical label groups related techniques, but a reproducible run must state whether it uses ordinary LoRA or a particular quantized-base configuration.
What the work involves
Inspect the architecture and choose target modules based on the task rather than copying names from another model. Check trainable parameters, rank, scaling and saved task heads. For QLoRA, confirm the quantization backend, compute dtype and device support, and test a small backward pass. Keep evaluation sources outside adapter training and compare with the base on target and retained tasks. Measure training peak memory and deployment behavior separately. Save base revision, adapter and quantization settings, then reproduce a fresh load and test any merge or export path that deployment requires.
Illustrative example
An illustrative team adapts a model to produce consistent laboratory report fields. Limited accelerator memory leads them to test QLoRA. They verify that only the adapter parameters update and inspect samples where a required field is absent. Held-out reports determine whether increasing rank improves field accuracy or merely fits familiar templates. The final loader reconstructs the exact quantized base and adapter, because loading the adapter onto a different revision can change the measured behavior.
Limits and common mistakes
Low rank restricts the available update and may be insufficient for some adaptations. A low trainable-parameter count still leaves base weights, activations and optimizer state to account for. Quantized computation can add approximation and compatibility constraints. Adapter merging and re-quantization require separate validation; they need not reproduce the original loading path exactly. LoRA is not a quantization algorithm, and QLoRA is not a guarantee of quality equivalence to full tuning. Evaluate their chosen configuration on the actual task.
Prerequisites
- hardLLM Fine-Tuning
LoRA/QLoRA are parameter-efficient alternatives to full SFT — you must understand what full fine-tuning does before learning to approximate it cheaply
- mediumLinear Algebra
LoRA factorizes weight updates into low-rank matrices (W = BA where B,A are low-rank) — linear algebra explains why this works
Related skills
- → is subcategory of: LLM Fine-Tuning
- → is subcategory of: Transfer Learning
Sources and further reading
- LoRA: Low-Rank Adaptation of Large Language Models
Frozen base weights and trainable low-rank matrix updates.
- QLoRA: Efficient Finetuning of Quantized LLMs
Quantized-base adapter training and its published numerical and memory techniques.
Last updated: 2026-10-10