Model Quantization
Model quantization represents model quantities with fewer bits to reduce storage or computation requirements. The skill is choosing a numerical format and supported execution path, calibrating where necessary and measuring the resulting quality and resource tradeoff. A smaller weight file alone does not establish faster or reliable inference.
What it is
Quantization maps values from a higher-precision representation to a more limited set using scales, offsets or other encoding parameters. A scheme may quantize weights, activations or both, with parameters shared across a tensor, channel or group. Post-training quantization converts an existing model, sometimes using representative calibration data; quantization-aware training incorporates the effect during optimization. Algorithms differ in how they control approximation error and handle unusually large values. Low-bit storage also requires compatible kernels or dequantization during execution. Quantization changes numerical representation and is distinct from LoRA, which changes the set of trainable adaptation parameters.
What the work involves
Start with an evaluated higher-precision baseline and identify whether memory, throughput or deployment support is the constraint. Select a method supported by the target hardware and model architecture. For calibration-based schemes, use representative inputs without borrowing the final test set. Record bit width, grouping, calibration construction and excluded modules. Check task quality, long-input behavior and retained capabilities, then measure peak memory and latency at realistic batch sizes. Save the quantization configuration with the artifact. The useful result is a verified runtime configuration with an explicit accuracy-cost tradeoff, rather than a compression ratio presented as sufficient validation.
Illustrative example
An illustrative assistant needs to fit on one workstation accelerator. The engineer compares a lower-bit weight representation with the baseline using the same questions and decoding settings. A held-out suite includes code generation and long instructions. The compressed model fits memory, but one format slows short requests because conversion overhead dominates. Another compatible kernel performs better in the workload. The deployment decision uses actual request measurements and error inspection together.
Limits and common mistakes
Outliers and accumulated rounding error can damage particular layers or tasks, even when average loss changes little. Hardware support, batch size and memory bandwidth determine whether reduced precision improves speed. Calibration data can miss important input ranges. Quantizing weights does not automatically reduce activation or attention-cache memory. Adapter training over quantized weights is a combined procedure, not proof that quantization itself performs adaptation. Validate artifacts after conversion and loading, including configuration and kernel compatibility.
Prerequisites
Quantization compresses Transformer weights from FP16 to INT4/INT8 — understanding what the weights represent is essential for assessing quality trade-offs
- mediumLinear Algebra
Quantization is approximation of floating-point matrices with lower-precision representations — linear algebra explains the error propagation
Related skills
- → is part of: LLM Inference Serving
Sources and further reading
- Hugging Face Transformers: Quantization overview
Supported quantization families, calibration and hardware-dependent execution.
- GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
A specific post-training weight quantization method and error-control approach.
Last updated: 2026-10-10